Monday, February 18, 2019

A Traveler user's iPhone stopped working over the weekend; interesting reason why

Monday morning I received notice that a Notes Traveler user's iPhone had stopped sending/receiving messages. I see this sort of thing occasionally and I generally respond by issuing a Tell Traveler User command to obtain the device ID of the user's mobile device, then issuing a Tell Traveler Reset command to resync the devices. That almost always resolves the user's problems.

But this time when I issued the Tell Traveler User command it came back with a raft of errors I had never seen before. The first one was that the user's name wasn't in the mail database's ACL.

So I opened the Domino Directory to the People view and saw that the user's Person document had two (count 'em, two) replication/save conflict documents. I thought, aha, maybe Traveler is getting misled by all the Person documents for this user.

I compared the content of the three documents and none of the name fields (or for that matter any fields in the first few tabs) were different among the three documents. But I did see that the Last Updated field under the Administration tab was different for all three. They were all updated the previous Friday, late in the day by IAM (the SSO service used by the organization). The "winner" Person document was the most recently edited, so I deleted the two conflict documents.

Then I opened her "winner" Person document and saw that she had been renamed at some point in the past (because Domino preserves a user's former names when it renames a user, say, with a new married name). I noticed also that her mail database's file name was formed from her first initial and former last name, not her new last name. That was normal.

Then I opened her mail database and saw three unexpected things:
  • The title of the database was still set to her former name; 
  • The ACL had only her former name, not her new name in it; and
  • The Owner field was still set to her former name, not her new name.
All should have been set to her new name when she was renamed. I wondered if someone had attempted to rename the user manually instead of correctly telling the Administration Process to rename her. Occasionally a Windows administrator, unfamiliar with Notes architecture, will assume they can do that and, in the process, will make a mess of everything - not that I expected anyone at this company to be so dumb.

It occurred to me to have a look at the Administration Requests database to see if there were any Rename-related documents in it. Sure enough, there was an Initiate Rename in Domino Directory document. It had been created late the previous Friday, and the request had been carried out. But, curiously, there were no follow-on Rename documents. By now there should have been a whole train of them.

The Administration Process, running on each Domino server, checks the Administration Requests database every minute or so throughout the day. When it discovers new requests it attempts to carry them out. If it succeeds, it typically generates the next request in a given series. Then, when it checks again a minute later (or maybe an hour, a day, or a week later, depending on the nature of the request), it carries out that one, and so on until the whole process of (in this case) renaming the user is complete.

I checked Administration Help and read about the Initiate Rename in Domino Directory step of the Rename process and it became clear to me what was going on. After the Administration Process carries out the steps required by the Initiate Rename in Domino Directory document (which are to make certain changes in the Person document, among them adding user's new name to the top of the list of names in the User Name field), it waits for the user to log into Notes. When the user does that, Notes will check with the user's mail server to see if it needs to respond to any changes made regarding the user on the server. When Notes does so, it discovers that the user has been renamed, and it makes a number of local changes as a result:
  • Notes pulls the user's new certificate down from the server and merges it into the User ID, which as a result includes the user's new name along with her former name;
  • Notes renames the user in the ACLs of all local databases and in configuration files such as notes.ini; and
  • After Notes has done all that, it creates the next Rename request in the Administration Requests database for the user: Rename Person in Domino Directory.
At this point the Administration Process can complete the renaming process. That is, it can carry out the steps defined by the Rename Person in Domino Directory document and all of the documents that will follow it. It will rename the person in a raft of places, including (but not limited to) group documents, ACLs of various databases throughout the domain (including, most importantly from Traveler's point of view, the user's mail database), and Names fields in any databases in the domain where it locates the user's former name.

So what must have happened, I concluded, is that the user was renamed in Notes so late on the previous Friday that her copy of Notes had not had the opportunity to update itself and create the Rename Person in Domino Directory document. So the user was renamed in the Person document, thanks to the Initiate Rename in Domino Directory document, but no place else. As a result, Traveler could not see that the newly renamed user had sufficient rights to the mail database and stopped updating the user's iPhone. The user could see over the weekend that her iPhone had stopped functioning; so she opened a support ticket, the one that was assigned to me.

Late Monday morning I telephoned the user. Because it was a holiday (President's Day), she still had not attempted to open and log into Notes on her laptop. I asked her to do so and, voila, all the dominoes described above started falling and, voici, eventually her iPhone started working again. Oh la la!