Configuring SAML authentication on multiple Domino servers.

Copyright 2022 by Rob Kirkland. All rights reserved.

It seems that HCL has neglected to publish the full set of instructions for configuring SAML authentication between Domino HTTP servers and Microsoft ADFS. The documentation as published results in one properly configured Domino server. But if you’re setting up multiple Domino servers to host a single Web site behind a load balancer / IP sprayer (i.e., you have configured multiple servers as hosts in a Web Site document), what you’ll end up with is one server (the one on which you performed the configuration work) that properly redirects authentication requests to ADFS and other servers that cannot do so and end up prompting users for credentials themselves.

The missing steps are:

1.     Export the Service Provider certificate, created during the first Domino server’s (“Server1”) configuration, from the first Domino server’s ID file.

2.     Import the SP certificate into the ID files of the other Domino server(s) hosting the Web site.
Important note: This includes any servers not currently existing or not currently hosting the Web site, but that may do so in the future.)

NOTE: Roberto DeLaRosa of HCL Support provided me with the instructions below. If you ever find yourself receiving support from Robi, consider yourself lucky; you are in the care of a competent and resourceful individual. 

TIP: Consider composing the set config commands below in a Notepad document, then proofreading, then pasting into the Domino Command field in the Server Console.

TIP: For a better understanding of what you’re going to do below, examine Server1’s ID file and look at its Internet Certificates. The steps below will export them from Server1 and import them into Servers 2 through n. Subtip: If there are multiple Internet certificates in the export file, and you don’t want to import them all into the target server(s), consider editing the export file to remove the unwanted certificate(s).

TIP: Make backup copies of the server ID files before performing the steps below. Name the backup copies such that it is clear how they are different from other copies of the ID files. Maybe even create a readme.txt file to clarify. And then put everything, including the export ("p12") file created below, in a zip file. Finally (and most importantly), stash the zip file in a secure place and delete any unsecured copies of it or its content.

Part 1: Exporting the Certificate from Server1’s ID file

1.     Run Domino Administrator and connect to Server1, the server that properly redirects users to the ADFS Identity Provider for authentication.

2.     Open the IdP Configuration document in the idpcat.nsf

a.     Click the Certificate Management Tab

b.     Take note of the Company name including the exact case.

c.      Take note of the Certificate public hash value.

3.     Navigate to the Server, Status tab and select the Domino console. Click the “Live” button. (Alternatively, you can perform the next steps in the server's local console.)

4.     Use “set config” to set the SAMLCompanyName variable in Server1's notes.ini file. The value is the “Company Name” observed in the step above. Append upper case “CN=” to the company name (this is case sensitive). Incorrect case will cause subsequent commands to fail.

a.     Syntax:

set config SAMLCompanyName=CN=<Company name>

 Where <Company name> is the content of the Company name field in the IdP Configuration document.

b.     Example:

set config SAMLCompanyName=CN=MyAmazingCompany

5.     Use "set config" to define the Certificate public hash value (as defined in the document).
TIP: A copy and paste from the IdP Configuration document helps insure accurate transcription.

a.     set config SAMLPublicKeyHash=<hash value>

 Where <hash value> is the content of the Certificate public hash value (base 64) field of the IdP Configuration document.

b.  Example:

set config SAMLPublicKeyHash=eWJ6S2cJd+6861u+XSpmDA==

6.     Export the certificate to a file, secured with a password.

a.     Syntax:

certmgmt export saml pkcs12 file.p12 password

Where file.p12 a file name to be created.

Where password will be the password of the p12 file to be created.

b.     IMPORTANT NOTE: This process will export the private key also, which is why the export file is password protected. Please follow proper security protocols for storage, transport, and password protection of this file.

c.  Example command:

certmgmt export saml pkcs12 mykeys.p12 s3cuR1ty@!F1rsT

d.     Note: The file will be saved into Server1’s data directory.

Part 2: Importing the certificate into the other Domino servers’ ID files.

In this part we will import the p12 file created in Part 1 into the ID file(s) of Server2.

NOTE: If more than two servers will host the Web site for which you created the subject IdP Configuration, you will repeat this procedure on servers 2 through n.

NOTE: If additional servers may be added or substituted as Web Site hosts in the future, and their ID files exist now, consider importing the certificate into their ID files as well. If new but not yet registered servers will host the Web Site, consider saving the export file (and its password) somewhere secure. (See the earlier tip about making and securing a zip file.)

1.     Copy the p12 file from Server1 to the data directory of each target server (where you wish to import).

NOTE for Linux users: If a target Domino server resides on a Linux host, make sure that the p12 file is owned by the owner of the other Domino processes.

2.     Launch the Domino Console from the administrator's client. (Alternatively, open the local console on the target server.)

3.     Enter the import command using the same details as before for filename and password. NOTE: Since the file is in the server's Domino\data directory, there's no need to enter a path.

a.     Syntax:

certmgmt import pkcs12 file.p12 password

b.     Example:

Certmgmt import pkcs12 mykeys.p12 s3cuR1ty@!F1rsT

c.      Confirm that your import was successful by issuing the following command:

certmgmt show all

 The command will display the details of the certificate(s) that was/were imported.

4.     Restart the HTTP task on server2.

5.     Test SAML authentication on server2.

NOTE: If the Web servers are behind a load balancer, you my need to disable the HTTP service on all but the one Domino server being tested.

TIP: An easy way to determine which server you are connecting to is to open the homepage.nsf database. First, edit the design of the Intro page of the homepage.nsf database, adding computed text with a value of @ServerName to one of the table cells.

6.     Optional cleanup: Delete the p12 files from data directories of servers 1 through n. (But, as previously suggested, keep a copy somewhere safe if you might need to perform more imports in the future. Otherwise, you will have to create a new p12 file by performing another export at that time.)

Opening attachments in HCL Verse and Chrome

I've often been frustrated when trying to open or download attachments when using IBM/HCL Verse in a Chrome browser window. I always end up looking at a raw version of the file. And I have to struggle to get back to the referring message. Then, after fumbling around with it a bit, I switch to Firefox or Edge to open the file. It occurred to me this morning (duh!) that I should look up the procedure for handling attachments in HCL's documentation. There I found a solution: "(Google Chrome only) Save attachments by using drag and drop."  

Oh, so that's how you do it! And, okay, it's a Chrome quirk that's causing the problem. 

So I tried drag and drop and it did solve my problem.

But drag and drop is sort of a cumbersome procedure, especially when using a laptop computer with a small, crowded screen and a touch pad. You have to set up your desktop for it before doing it, arranging windows side-by-side, that sort of thing. Then you have to make sure you don't let up off the mouse button or touch pad during that long, perilous journey across the desktop.

So, still not satisfied, it occurred to me to try right-clicking on the attachment in the message. In the menu that popped up I saw "Save link as...". Looked promising. Tried it. A Save As dialog opened. I navigated to the folder where I wanted to save the attachment. Clicked Save. Opened the saved file (in Excel in my test). Ta-da! It worked. There was my file.

Suggestion to @HCLDSSup: Add this right-click procedure to the product documentation as an alternative to drag and drop in Chrome. It's much less cumbersome. And, thanks, Reader, for reading this all the way to the end.

Updall options for Note/Domino v9.0.1 documentation

 I use the Notes/Domino maintenance tools pretty frequently to keep Domino servers and Notes workstations running at their best and to fix issues that may arise from time to time. When I run them from a command prompt I like to refer to each utility's Options pages in the product documentation to make sure that I use appropriate arguments on the command line, depending on what I need each tool to do for me. I've been doing this for years - no, decades - and you would thing that, by now, I would know all the arguments by heart. But I have never bothered to memorize them because, I don't know, I guess I'd rather put my organic storage device (a/k/a my brain) to other uses.

Anyway, I've noticed that, since HCL acquired Notes/Domino from IBM and took over the documentation of the produce, the Options pages for some of the utilities have disappeared from the documentation. The links to Updall Options in the online Domino 11 documentation, for example, no longer take one to the page that lists and explains the uses of all of the switches available to be used with the command. If I hunt long and hard enough, sometimes I may find what I'm looking for. But it feels like a real waste of time. So, for my own benefit and that of my other reader (and, okay, yours, too, if you want), below this paragraph I am quoting the content of the Domino 9.0.1 Updall Options page. I didn't try to fix any links in the quoted text and I don't know, offhand, if IBM or HCL may have made or be planning to make changes in later versions of the utility):

Updall options
The Updall task manages database full-text indexes.

Note: You can run the Updall task on a server, or you can use the dbmt tool that now includes the Updall task as well as other options instead of running Updall alone. See the related topics for more information.

You can use several methods of running the Updall task on a server.

• From Task -> Start tool in the Domino® Administrator -- Use this method if you don't want to use command-line options.
• Using the Load Updall console command -- Use this method if you are comfortable using command-line options or if you want to run Updall directly at the server console when there is no Domino Administrator running on the server machine.
• Program document that runs Updall -- Use this method to schedule Updall to run at particular times.
• Run Updall on a Win32 platform -- Use this method if you are unable to run Updall at the server console. This method requires that you use the "n" prefix -- for example, nupdall - R.

When you use these methods, you can include options that control what Updall updates. For example, you can update all views and not update any full-text search indexes.

The following tables describe the options you can use with Updall (Task -> Start ). The second column lists the equivalent command-line options that you use when you use a console command to run Updall and when you schedule Updall to run in a Program document.

Use this syntax when you use the Load updall console command:

Load updall databasepath options

For example:

Load updall SALES.NSF -F

You can specify multiple options -- for example:

Load updall -F -M

Table 1. Updall - Basic options

Option in Task - Start tool	Command-line option	Description
Index all databases Index only this database or folder	databasepath This option is used when running Updall as a console command.	Choose the option to index all databases if you want updall to process all databases on the server. Choose the option to specify a database or folder if you want updall to limit processing to the specified location. To update a database in the Domino data folder, enter the file name, for example, SALES.NSF. To update all databases contained in a subfolder of the data folder, specify the path relative to the data folder, for example, DOC\README.NSF.
Update this view only	database -T viewtitle	Updates a specific view in a database. Use, for example, with -R to solve corruption problems. Note: -T cannot be used with .IND (indirect) files.

Table 2. Updall - Basic options - more

Option in Task - Start tool	Command-line option	Description
Update: All built views	-V	Updates built views and does not update full-text indexes.
Update: Full text indexes	-F	Updates full-text indexes and does not update views.
Update: Full text indexes: Only those with frequency set to: Immediate or Hourly	-H	Updates full-text indexes assigned "Immediate" or "Hourly" as an update frequency.
Update: Full text indexes: Only those with frequency set to: Immediate, Hourly, or Scheduled	-M or -S	Updates full-text indexes assigned "Immediate," "Hourly," or "Scheduled" as an update frequency.
Update: Full text indexes: Those with frequency set to: Immediate, Hourly, Daily, or Scheduled	-L	Updates full-text indexes assigned "Immediate," "Hourly," "Daily" or "Scheduled" as an update frequency.

Table 3. Updall - Rebuild options

Option in Task - Start tool	Command-line option	Description
Rebuild: Full-text indexes only	-X	Rebuilds full-text indexes and does not rebuild views. Use to rebuild full-text indexes that are corrupted.
Rebuild: All used views	-R	Rebuilds all used views. Using this option is resource-intensive, so use it as a last resort to solve corruption problems with a specific database.
Rebuild: Full-text indexes and additionally: All unused views	database -C	Rebuilds unused views and a full-text index in a database. Requires you to specify a database.

Table 4. Updall - Search Site options

Option in Task - Start tool	Command-line option	Description
Update database configurations: Incremental	-A	Incrementally updates search-site database configurations for search site databases.
Update database configurations: Full	-B	Does a full update of search-site database configurations for search site databases.

Option for running Updall as part of dbmt

Updall performs the following tasks by default. These are also tasks that the database maintenance tools performs:

• purges deletion stubs
• expires soft deleted entries
• updates unread lists

Because the database maintenance tool is meant to replace (and improve upon) running updall nightly, you can use the following new option for updall to skip the tasks the preceding tasks, making updall faster when you run it for any one-time purpose.

-nodbmt

When you run updall as part of dbmt, Domino also ensures that the following views are built for databases with a template name of StdR7Mail, StdR8Mail, StdR85Mail and StdR9Mail:

• $Inbox
• $Drafts
• $All
• ($RepeatLookup)
• ($ToDo)
• ($Calendar)
• ($Haiku_TOC)
• ($Alarms)
• ($iNotes)
• ($Users)
• ($iNotes_Contacts)
• ($ThreadsEmbed)

After these views are built, they will not be discarded due to non-use.

Error message in Notes 11.0.1: "Insufficient memory - local heap is full"

I recently (like last week) upgraded Notes on my main workstation from 10.0.1 to 11.0.1FP1. The installation went uneventfully. But when I ran Notes and tried to open my mail database, Notes locked up and presented me with an error I hadn't seen before:

 Insufficient memory - local heap is full

I immediately Googled the error, but got nothing very useful in return. So then I decided to run the standard array of Notes maintenance tasks: Fixup. Compact -c. Updall. Not much help there either. But I noticed that Notes wasn't failing until I clicked the Mail or Calendar links in either the Task Bar or the Open List. So I tried opening mail manually, via Ctrl+O. That did seem to have a positive effect. My mail opened! And I was able to work with it for awhile. But eventually the error message popped up again and I had to kill and restart Notes to get back to work again.

So, as a last resort, I decided to give HCL Support a try. And pretty quickly I had a positive result. I searched my error message in the KnowledgeBase and got a direct hit - KB0081067.. (I wonder why it didn't turn up in my Google search.)

The fix in the KBase article directed me to carry out what amounted to a fresh reconfiguraton of Notes. I carried it out. It worked in that, afterwards, Notes could open my mail without the lockup. But it was a problem for me because I lost all my Desktop folders and tiles and all my bookmarks.

And it seemed like more of a workaround than a solution to me. Yeah, it might get Notes to run and open my mail DB. But it didn't provide any clue as to why the error was occurring and it didn't reassure me at all that the error wouldn't occur again some day. And, for me, losing all my tiles and bookmarks was a painful solution. I sort of live and die by my Notes configuration.

So I opened a support ticket. A nice, knowledgeable support tech named Nic responded and agreed that, yes, my tiles and bookmarks would be wiped out. And, no, the fix in the KBase article was not a permanent fix.

I asked if increasing the size of the local heap would be a sensible thing to do. Nic said, yes it would, but the new, bigger heap would consume about 2 GB of RAM. My workstation has 16 GB of RAM, so I asked how to proceed. Nic provided me with a link to this additional KBase article that described the procedure. I followed it. Notes is running. Mail is opening. So far, so good.

I suggested to Nic that the first KBase article needs to be amended to: 1) add a caveat about losing one's Notes configuration if one follows the instructions; 2) add the fact that following the reconfiguration procedure isn't necessarily a permanent fix; 3) state that one should alternatively consider increasing the heap size if one has sufficient RAM; and 4) provide the link to the second article describing the procedure for increasing the heap size. Nic agreed that the article should be amended with those items.

By the way, my experience with HCL Tech Support has been generally positive so far. And, thanks, Nic, for your helpful support.

Is it time to renew your Domino ID Vault certificates?

IBM issued a Technote today detailing the procedure for renewing ID Vault Trust Certificates and Password Reset Certificates. They expire after 10 years. ID Vaults were first introduced in Domino 8.5, which was released December 2018, 10+ years ago now. So early adapters of the ID Vault will increasingly be having to renew their certificates.
The Technote describes:

• The error message that signals that your certificates have expired ("Not a valid ID or the ID is corrupted");
• The procedure for determining the expiration dates of your certificates; and 
• The procedure for renewing them. Unfortunately you can't just recertify them. You have to remove the expired ones then issue new ones.

My favorite feature of Sametime Meetings

Call me a dinosaur, but I love Sametime Meetings. Here's why. It has the best chat functionality of any meeting software I've seen. What I like about it in particular are two things:

• You can categorize entries in the chat window according to five predefined categories.
• When the meeting ends, Sametime generates a meeting report that organizes the chat entries by category.

The five categories are:

• Group Chat
• Minutes
• Action Item
• Question
• Starred Item

If you just enter text in the chat window, your entry defaults to Group Chat. But you can select another category before you hit Enter. That entry and all following entries are under the category you selected until you select another one.

When you end the meeting you see the dialog below, where you can choose to generate a meeting report or not, and where to store it. I have a subscription to Sametime Meetings in IBM Connections Cloud, where I also have subscriptions to IBM Notes Mail and IBM Connections. So the dialog defaults to saving the report to "My Files", my cloud-based file storage area.

I don't go around testing the features of meeting products. So it may be that other meeting software has these features too; but I haven't seen them in WebEx, GoToMeeting, or Zoom.

And I really don't in my life have much call to conduct meetings. But were I, say, the Chair of a regularly scheduled meeting I might designate someone in the meeting to take meeting minutes by entering them in the meeting chat window. And I would put all action items and unresolved questions there too.

By entering that information right in the chat stream for all to see, the meeting attendees could act as proofreaders, flagging errors as they occur. Then, after the meeting ends I would use the meeting report to follow up on action items, unresolved questions, and important ("Starred") items, perhaps distributing parts of the report to the people assigned to carry out each item.

What's not to like about this? (Now, if only the process of installing the screen-sharing browser plug-in were a little easier and faster...)

A Traveler user's iPhone stopped working over the weekend; interesting reason why

Monday morning I received notice that a Notes Traveler user's iPhone had stopped sending/receiving messages. I see this sort of thing occasionally and I generally respond by issuing a Tell Traveler User command to obtain the device ID of the user's mobile device, then issuing a Tell Traveler Reset command to resync the devices. That almost always resolves the user's problems.

But this time when I issued the Tell Traveler User command it came back with a raft of errors I had never seen before. The first one was that the user's name wasn't in the mail database's ACL.

So I opened the Domino Directory to the People view and saw that the user's Person document had two (count 'em, two) replication/save conflict documents. I thought, aha, maybe Traveler is getting misled by all the Person documents for this user.

I compared the content of the three documents and none of the name fields (or for that matter any fields in the first few tabs) were different among the three documents. But I did see that the Last Updated field under the Administration tab was different for all three. They were all updated the previous Friday, late in the day by IAM (the SSO service used by the organization). The "winner" Person document was the most recently edited, so I deleted the two conflict documents.

Then I opened her "winner" Person document and saw that she had been renamed at some point in the past (because Domino preserves a user's former names when it renames a user, say, with a new married name). I noticed also that her mail database's file name was formed from her first initial and former last name, not her new last name. That was normal.

Then I opened her mail database and saw three unexpected things:

• The title of the database was still set to her former name; 
• The ACL had only her former name, not her new name in it; and
• The Owner field was still set to her former name, not her new name.

All should have been set to her new name when she was renamed. I wondered if someone had attempted to rename the user manually instead of correctly telling the Administration Process to rename her. Occasionally a Windows administrator, unfamiliar with Notes architecture, will assume they can do that and, in the process, will make a mess of everything - not that I expected anyone at this company to be so dumb.

It occurred to me to have a look at the Administration Requests database to see if there were any Rename-related documents in it. Sure enough, there was an Initiate Rename in Domino Directory document. It had been created late the previous Friday, and the request had been carried out. But, curiously, there were no follow-on Rename documents. By now there should have been a whole train of them.

The Administration Process, running on each Domino server, checks the Administration Requests database every minute or so throughout the day. When it discovers new requests it attempts to carry them out. If it succeeds, it typically generates the next request in a given series. Then, when it checks again a minute later (or maybe an hour, a day, or a week later, depending on the nature of the request), it carries out that one, and so on until the whole process of (in this case) renaming the user is complete.

I checked Administration Help and read about the Initiate Rename in Domino Directory step of the Rename process and it became clear to me what was going on. After the Administration Process carries out the steps required by the Initiate Rename in Domino Directory document (which are to make certain changes in the Person document, among them adding user's new name to the top of the list of names in the User Name field), it waits for the user to log into Notes. When the user does that, Notes will check with the user's mail server to see if it needs to respond to any changes made regarding the user on the server. When Notes does so, it discovers that the user has been renamed, and it makes a number of local changes as a result:

• Notes pulls the user's new certificate down from the server and merges it into the User ID, which as a result includes the user's new name along with her former name;
• Notes renames the user in the ACLs of all local databases and in configuration files such as notes.ini; and
• After Notes has done all that, it creates the next Rename request in the Administration Requests database for the user: Rename Person in Domino Directory.

At this point the Administration Process can complete the renaming process. That is, it can carry out the steps defined by the Rename Person in Domino Directory document and all of the documents that will follow it. It will rename the person in a raft of places, including (but not limited to) group documents, ACLs of various databases throughout the domain (including, most importantly from Traveler's point of view, the user's mail database), and Names fields in any databases in the domain where it locates the user's former name.

So what must have happened, I concluded, is that the user was renamed in Notes so late on the previous Friday that her copy of Notes had not had the opportunity to update itself and create the Rename Person in Domino Directory document. So the user was renamed in the Person document, thanks to the Initiate Rename in Domino Directory document, but no place else. As a result, Traveler could not see that the newly renamed user had sufficient rights to the mail database and stopped updating the user's iPhone. The user could see over the weekend that her iPhone had stopped functioning; so she opened a support ticket, the one that was assigned to me.

Late Monday morning I telephoned the user. Because it was a holiday (President's Day), she still had not attempted to open and log into Notes on her laptop. I asked her to do so and, voila, all the dominoes described above started falling and, voici, eventually her iPhone started working again. Oh la la!