Monday, September 13, 2010

A satisfying ending to a Domino 8.5.2 installation dilemma

Today I had one of those "walk-on-water" moments that occasionally make consulting satisfying. Over the weekend I assisted a client in upgrading nine Domino servers from version 7.0.2 to 8.5.2. The marathon upgrade went generally smoothly. The main problem with it was there were just not enough hours in the weekend to upgrade (via compact -c) all of the databases on all of the servers. In particular, the headquarters mail server has an eye-popping number of multi-gigabyte mail databases on it. Those top execs just can't bear to part with any of their mail, and of course you can't enforce quotas against them; they'd have you (a lowly tech) drawn-and-quartered if you tried. So the one mail server never did get all of its databases upgraded to ODS 51. But except for that and a couple other minor problems, everything was working great on Monday morning.
Then a field technician in England reported that he couldn't set up a Notes workstation for a user who had been registered the previous week. The Notes setup kept failing with the following error message:
The Policy and/or Settings document assigned to you has been edited by an unauthorized person. Please notify your Administrator that you cannot proceed with the client setup.
Some Googling revealed that this error message arises when a policy or settings document is (as the message plainly says) signed by one who is unauthorized to sign it.. How could this happen? As follows: An administrator might in the past have created Policy and/or Settings documents that worked just fine. But then the administrator left the company and his account was removed from the domain. As a result, the retired administrator's signatures on the Policy and/or Settings documents were no longer those of an authorized signer, which is defined as one who has Editor access to the Domino Directory and is assigned the Policy Creator and Policy Modifier roles. See Technotes 1110644 and 1205444. Both of these Technotes state further that, to re-sign a Policy or Settings document you can, among other techniques, edit and save it.
How did this apply to us? That was not clear. As we could plainly see in the Policies and Settings views, the signer of all of the the Policy and Settings documents in the domain was a generic user named "Administrator" who is still to this day being used to create new users and who has the requisite access rights and roles assigned. So the error message seemed to be bogus, telling us something which was demonstrably untrue. Nonetheless, we tried editing/saving the policy and settings documents to apply new signatures. And, according to the Policies and Settings views, this worked. The views showed that each newly saved document had a new signer. But even though this new signer also had sufficient rights and roles assigned, we discovered that we could not register a new user into the England OU. So our problem was not only that we couldn't set up a new workstation for users registered before the upgrade, but also that we couldn't register new users after the upgrade.
We were scratching our heads over this and hunkering down for a long siege against this problem, when one of us had the bright idea of looking at the console of the mail server in question. (In this particular domain, there are several regional OUs. Each region also has a dedicated mail server. Also, each OU has an organizational policy with several OU-specific settings documents assigned to it. So our problem was affecting the England OU, the England mail server, and the England-specific policy and settings documents.)
There on the England mail server's console, in bright purple letters, was a cascade of error messages, repeating every few seconds:
Policies and settings documents signed by XXXXX are no longer valid because this person does not have the required access level or roles to the Domino Directory.

More Googling revealed that this new error message had the same cause as the earlier message, that is, that a Policy or Settings document was signed by an insufficiently authorized entity. The server was trying to apply a mail policy to the mail users on the server, but it didn't like the signature on the policy or mail settings document. However, this message told us one other bit of information that the earlier message had left out, which was that the Policy or Settings document was signed by the mail server itself, not, as the Policies and Settings views claimed, by the Administrator who actually created or last edited each document. Either the error message or the views were lying to us.
But the server did not in fact have the Policy Creator or Policy Modifier roles assigned to it in the ACL of the Domino Directory. So that meant that the error messages were perhaps not bogus after all. Assigning those roles to the server in the ACL should have solved our problem. We'll never know, however, if it would have done so, because our second bout with Google revealed one other fact -- that there was another (newer?) way to re-sign the Policies and Settings documents:

Select the documents in the Policies or Settings views, then in the Actions menu choose Resign Policy.

We tried that and it worked like magic. The cascade of messages in the server's console abruptly stopped. Thereafter, we found we could once again register new users and set up Notes for them. Later we checked the other mail servers and discovered that the Mexico server was experiencing the same problem. So we re-signed all of the policies and settings documents. Just like that, the Mexico server's messages stopped too.

Conclusion: Something happened that shouldn't have when we upgraded two of the nine servers. Apparently that something is that, somehow, the upgraded servers' signatures were applied to the Policy and/or Settings documents. Also, the Policies and Settings views tell us that the last editor of a document is the signer of it, which makes sense. The Technotes cited above tell us that too. But in our case saving the Policy or Settings document did not apply a new signature to it. So the view is misleading in that regard (and so are the Technotes). I think I'll tell Lotus about this.

Sunday, August 29, 2010

Lotus Notes Domino 8.5.2 so far - painful

So far, we've attempted one Domino and four Notes 8.5.2 upgrades. The server installation went off cleanly, but took forever. Of the four Notes installs: One completed without errors (on a relatively clean, new Windows 7 machine). One completed but with problems (on a clean, old Vista machine), and I'll try a repair and see if that fixes it; otherwise, it's uninstall/reinstall. Two failed completely (on a Windows 7 and a Vista machine, both with lots of widgets). One was remedied by uninstall/reinstall. The other will go that route too when I find the time. The error, some of them, anyway, have to do with provisioning problems and MUI problems, and that's all I know at this time.
Update: Repair of the one machine (where 8.5.2 installed by "with problems") had no positive effect. The machine is useable. But the Home tab reads "%str.HomeTitle.Keywords". That's the only problem at the moment, one that is not a killer. But the fact that the title of the Home tab is broken in that way implies that I'll encounter other problems. I think I'll just uninstall/reinstall and see if that fixes it.
The one failure (Win7) was on Carlos's machine. He blamed the failure on a music widget. How he decided that was the cause of his problem, I don't know. But un/reinstall did fix it, so he says.
The other failure (Vista) was on my primary machine. It also has lots of third-party widgets on it. We'll see what happens when I un/reinstall there.

Sunday, August 15, 2010

Some thoughts about Lotus Domino high availability

Someone wondered out loud today if a new Domino domain we're about to configure should rely on Domino clustering or Windows clustering to provide high availability. It occurred to me awhile back that another way of providing high availability would be to virtualize our servers. I'd intended to write something about this. Maybe now I will.

The goal is for our servers to be available for something approaching 24x7x365. Here is my first pass at this:

Domino clustering
  • Up to six Domino servers can be clustered together.
  • The servers keep close tabs on each other.
  • They communicate frequently with each other.
  • They consume a lot of network bandwidth. Therefore, it's a good practice for the servers to be connected to a second network segment, dedicated to inter-server communications.
  • They consume a lot of processor time. Clustered servers a noticeably slower to  respond to a user request than are non-clustered servers.
  • Not every cluster member need be identical. They can have different processors, different amounts of RAM and disk capacity, different OSes, different physical platforms, and different subsets of databases on them.
  • You can take down a cluster member for maintenance whenever you want (though you'll want to gently shoo the users off the server first) and Notes users will automatically fail over to another member of the cluster.
  • You can set up the cluster to fail HTTP users over as well.
  • You can tune each cluster member to turn away new requests upon reaching a defined level of activity - load balancing.
  • Domino admins would manage the cluster.
  • Domino Enterprise or Utility license.
Windows clustering
  • Two Windows servers share access to a single drive array. One is the active server and runs a Domino server that resides on the shared drive array. The other is a hot backup and would take over control of the drive array and restart the Domino server from the array should the active server go down for any reason.
  • A short outage (less than a minute?) would exist until the Domino server came back up.
  • You could configure the two Windows servers to share two arrays. Each server could be active on one array, passive on the other. That way, you wouldn't be wasting a whole server for the sole purpose of insuring that the Domino server comes back up quickly should the active Windows box go down.
  • Windows admins would manage the cluster.
  • Special Windows licensing?
Virtual machines
  • A virtual machine on which a Domino server resides and runs would itself reside on a LUN on a Storage Area Network (or a similar "independent-disk" technology). Any VM host that can access the LUN could host the VM running the Domino server.
  • If the host on which the Domino-resident VM was running should go down for any reason, another host could take control of the LUN and restart the Domino VM within a short period of time. This is the high availability.
  • If the VM on which Domino runs or Domino itself had to go down for any reason, the Domino server is out of service.
  • One could piggyback Domino clustering on top of the virtualization, so that, when a VM running Domino (or Domino itself running on the VM) goes down, Domino failover would cause users to find another Domino server in the cluster. (Could one also run Windows clusters comprised of two VMs? Would there be any benefit to doing so?)
  • The primary benefit of virtualizing the machine on which Domino runs is to make more intensive use of available hardware resources. One could run multiple VMs on a single VM host, rather than have to run each hosted machine on its own, dedicated physical box.
  • Another benefit of virtualization is that any machine dependent on old technology could continue to run as a virtual machine where it might not be able to continue existence as a physical machine if the underlying hardware/software platform is no longer available. Don't know that this is relevant to Domino.

Tuesday, July 27, 2010

Lotus Notes installation pains

I'm trying to install a Notes 8.5.1 client onto a computer running Windows Vista Home Premium. I've done this before without problems, but this time it's a struggle. Don't know why. The 8.5.1 install went okay. Installed from the "All clients" kit. Chose to install every option.
Then I tried to install Fixpack 3. It succeeded, but broke my Notes installation. When I tried to run Notes afterward, it informed me that ndgts.dll couldn't be found. Uninstalled Fixpack 3. Repaired Notes. Installed Fixpack 1. Same result.
I've found several references to this problem through Google, but no solutions. People seem to have solved their own problems. But nobody has documented exactly how they did so, as far as I could see.
Apparently, though, I'm having the problem on this machine because I've run earlier versions of Notes on it. I was running 8.0.2. Uninstalled it. Renamed its directory. Installed 8.5.1 into the default directory. Everything worked. I install any of the 3 fixpacks, though, and Notes stops working. What could the fixpack installers be doing that's messing me up?
I've managed a workaround that I hope has fixed the problem. After installing the Fixpack (which broke the installation as expected) I ran the Notes 8.5.1 installer in Repair mode. It repaired Notes. Notes ran. About Notes shows that the fixpack is still installed. I tested this initially with Fixpack 1, again with Fixpack 3. I hope repair mode didn't overwrite anything important with older code.
Problem is: This installation is different from two other supposedly identical (but not upgraded) installations.
  • ndgts.dll is in the Application Extension folder, not the Notes folder as in the other installations.
  • ndgts.sym isn't installed anywhere that I can see. It's in the Notes folders in the other installations.
  • ndgts.dll is dated 9/29/09 in this installation, 5/24/10 in the others. These are the dates of the files in the 8.5.1 and 8.5.1 FP3 installation folders, respectively. On checking further I see that every file that is dated 5/24/10 in the other installations is dated 9/29/09 in this one. So I'm not confident that FP3 is truly, fully installed on this workstation.
Hmm. What to do next? Reinstall! Whoopee!
So I uninstalled Notes 8.5.1, then copied the whole IBM directory to an external drive. 6037 items, 3.32 GB worth of old stuff, all so I can hold on to a few old databases. I'll have to remember to delete this stuff when I get the old DBs back into a working Notes installation.
Part Two
So, I reinstalled Notes All Clients 8.5.1. Ran it. Everything works. Made a copy of ndgts.dll. Installed Fixpack 3. Ran Notes. Drum roll...
IT WORKS! Frabjous day! Calloo! Callay!
And the copy of ndgts.dll that's in there is dated 5/24/10, as God intended! But it's in the Application Extension folder. Oh well. Who cares? It works. Time to tip-toe away before it breaks again.
BTW, I've been running everything "as administrator" even though I'm logged in as an admin. Don't know if that's necessary. But I'm paranoid about everything at this point, so "belt and suspenders" feels comforting to me about now.
What a pain!

Domino Security From 30,000 Feet

To access any bit of data on a Domino server, a user has to get over these hurdles:
  • Network or "KVM" access to the server
    • KVM access means actual or virtual access to the server's keyboard, video, and mouse. 
    • Users with KVM access may be able to bypass some of the other security measures. 
  • Authenticate with the server 
  • Server ACLs (in the Server doc, under the Security tab) 
  • File System ACL if user has requested a file in the html directory. 
  • Directory Link ACL if the data is in a database that is in a linked folder 
  • Database ACL if the data is in a database
  • Inside the database:

    • View ACL (controls who can use a view)
    • Form ACL (controls who can create documents with a form)
    • Document access
      • Section ACL (controls who can edit the fields in the section)
      • $Readers field and fields of Readers or Authors data type
        • $Readers and Readers fields control who can see/read a document
        • Authors fields control who (having Author access in the database ACL) can edit the document
      • Encrypted fields (user must have a decryption key to decrypt the data in the field)