Configuring SAML authentication on multiple Domino servers.

Copyright 2022 by Rob Kirkland. All rights reserved.

It seems that HCL has neglected to publish the full set of instructions for configuring SAML authentication between Domino HTTP servers and Microsoft ADFS. The documentation as published results in one properly configured Domino server. But if you’re setting up multiple Domino servers to host a single Web site behind a load balancer / IP sprayer (i.e., you have configured multiple servers as hosts in a Web Site document), what you’ll end up with is one server (the one on which you performed the configuration work) that properly redirects authentication requests to ADFS and other servers that cannot do so and end up prompting users for credentials themselves.

The missing steps are:

1.     Export the Service Provider certificate, created during the first Domino server’s (“Server1”) configuration, from the first Domino server’s ID file.

2.     Import the SP certificate into the ID files of the other Domino server(s) hosting the Web site.
Important note: This includes any servers not currently existing or not currently hosting the Web site, but that may do so in the future.)

NOTE: Roberto DeLaRosa of HCL Support provided me with the instructions below. If you ever find yourself receiving support from Robi, consider yourself lucky; you are in the care of a competent and resourceful individual. 

TIP: Consider composing the set config commands below in a Notepad document, then proofreading, then pasting into the Domino Command field in the Server Console.

TIP: For a better understanding of what you’re going to do below, examine Server1’s ID file and look at its Internet Certificates. The steps below will export them from Server1 and import them into Servers 2 through n. Subtip: If there are multiple Internet certificates in the export file, and you don’t want to import them all into the target server(s), consider editing the export file to remove the unwanted certificate(s).

TIP: Make backup copies of the server ID files before performing the steps below. Name the backup copies such that it is clear how they are different from other copies of the ID files. Maybe even create a readme.txt file to clarify. And then put everything, including the export ("p12") file created below, in a zip file. Finally (and most importantly), stash the zip file in a secure place and delete any unsecured copies of it or its content.

Part 1: Exporting the Certificate from Server1’s ID file

1.     Run Domino Administrator and connect to Server1, the server that properly redirects users to the ADFS Identity Provider for authentication.

2.     Open the IdP Configuration document in the idpcat.nsf

a.     Click the Certificate Management Tab

b.     Take note of the Company name including the exact case.

c.      Take note of the Certificate public hash value.

3.     Navigate to the Server, Status tab and select the Domino console. Click the “Live” button. (Alternatively, you can perform the next steps in the server's local console.)

4.     Use “set config” to set the SAMLCompanyName variable in Server1's notes.ini file. The value is the “Company Name” observed in the step above. Append upper case “CN=” to the company name (this is case sensitive). Incorrect case will cause subsequent commands to fail.

a.     Syntax:

set config SAMLCompanyName=CN=<Company name>

 Where <Company name> is the content of the Company name field in the IdP Configuration document.

b.     Example:

set config SAMLCompanyName=CN=MyAmazingCompany

5.     Use "set config" to define the Certificate public hash value (as defined in the document).
TIP: A copy and paste from the IdP Configuration document helps insure accurate transcription.

a.     set config SAMLPublicKeyHash=<hash value>

 Where <hash value> is the content of the Certificate public hash value (base 64) field of the IdP Configuration document.

b.  Example:

set config SAMLPublicKeyHash=eWJ6S2cJd+6861u+XSpmDA==

6.     Export the certificate to a file, secured with a password.

a.     Syntax:

certmgmt export saml pkcs12 file.p12 password

Where file.p12 a file name to be created.

Where password will be the password of the p12 file to be created.

b.     IMPORTANT NOTE: This process will export the private key also, which is why the export file is password protected. Please follow proper security protocols for storage, transport, and password protection of this file.

c.  Example command:

certmgmt export saml pkcs12 mykeys.p12 s3cuR1ty@!F1rsT

d.     Note: The file will be saved into Server1’s data directory.

Part 2: Importing the certificate into the other Domino servers’ ID files.

In this part we will import the p12 file created in Part 1 into the ID file(s) of Server2.

NOTE: If more than two servers will host the Web site for which you created the subject IdP Configuration, you will repeat this procedure on servers 2 through n.

NOTE: If additional servers may be added or substituted as Web Site hosts in the future, and their ID files exist now, consider importing the certificate into their ID files as well. If new but not yet registered servers will host the Web Site, consider saving the export file (and its password) somewhere secure. (See the earlier tip about making and securing a zip file.)

1.     Copy the p12 file from Server1 to the data directory of each target server (where you wish to import).

NOTE for Linux users: If a target Domino server resides on a Linux host, make sure that the p12 file is owned by the owner of the other Domino processes.

2.     Launch the Domino Console from the administrator's client. (Alternatively, open the local console on the target server.)

3.     Enter the import command using the same details as before for filename and password. NOTE: Since the file is in the server's Domino\data directory, there's no need to enter a path.

a.     Syntax:

certmgmt import pkcs12 file.p12 password

b.     Example:

Certmgmt import pkcs12 mykeys.p12 s3cuR1ty@!F1rsT

c.      Confirm that your import was successful by issuing the following command:

certmgmt show all

 The command will display the details of the certificate(s) that was/were imported.

4.     Restart the HTTP task on server2.

5.     Test SAML authentication on server2.

NOTE: If the Web servers are behind a load balancer, you my need to disable the HTTP service on all but the one Domino server being tested.

TIP: An easy way to determine which server you are connecting to is to open the homepage.nsf database. First, edit the design of the Intro page of the homepage.nsf database, adding computed text with a value of @ServerName to one of the table cells.

6.     Optional cleanup: Delete the p12 files from data directories of servers 1 through n. (But, as previously suggested, keep a copy somewhere safe if you might need to perform more imports in the future. Otherwise, you will have to create a new p12 file by performing another export at that time.)

Opening attachments in HCL Verse and Chrome

I've often been frustrated when trying to open or download attachments when using IBM/HCL Verse in a Chrome browser window. I always end up looking at a raw version of the file. And I have to struggle to get back to the referring message. Then, after fumbling around with it a bit, I switch to Firefox or Edge to open the file. It occurred to me this morning (duh!) that I should look up the procedure for handling attachments in HCL's documentation. There I found a solution: "(Google Chrome only) Save attachments by using drag and drop."  

Oh, so that's how you do it! And, okay, it's a Chrome quirk that's causing the problem. 

So I tried drag and drop and it did solve my problem.

But drag and drop is sort of a cumbersome procedure, especially when using a laptop computer with a small, crowded screen and a touch pad. You have to set up your desktop for it before doing it, arranging windows side-by-side, that sort of thing. Then you have to make sure you don't let up off the mouse button or touch pad during that long, perilous journey across the desktop.

So, still not satisfied, it occurred to me to try right-clicking on the attachment in the message. In the menu that popped up I saw "Save link as...". Looked promising. Tried it. A Save As dialog opened. I navigated to the folder where I wanted to save the attachment. Clicked Save. Opened the saved file (in Excel in my test). Ta-da! It worked. There was my file.

Suggestion to @HCLDSSup: Add this right-click procedure to the product documentation as an alternative to drag and drop in Chrome. It's much less cumbersome. And, thanks, Reader, for reading this all the way to the end.

Updall options for Note/Domino v9.0.1 documentation

 I use the Notes/Domino maintenance tools pretty frequently to keep Domino servers and Notes workstations running at their best and to fix issues that may arise from time to time. When I run them from a command prompt I like to refer to each utility's Options pages in the product documentation to make sure that I use appropriate arguments on the command line, depending on what I need each tool to do for me. I've been doing this for years - no, decades - and you would thing that, by now, I would know all the arguments by heart. But I have never bothered to memorize them because, I don't know, I guess I'd rather put my organic storage device (a/k/a my brain) to other uses.

Anyway, I've noticed that, since HCL acquired Notes/Domino from IBM and took over the documentation of the produce, the Options pages for some of the utilities have disappeared from the documentation. The links to Updall Options in the online Domino 11 documentation, for example, no longer take one to the page that lists and explains the uses of all of the switches available to be used with the command. If I hunt long and hard enough, sometimes I may find what I'm looking for. But it feels like a real waste of time. So, for my own benefit and that of my other reader (and, okay, yours, too, if you want), below this paragraph I am quoting the content of the Domino 9.0.1 Updall Options page. I didn't try to fix any links in the quoted text and I don't know, offhand, if IBM or HCL may have made or be planning to make changes in later versions of the utility):

Updall options
The Updall task manages database full-text indexes.

Note: You can run the Updall task on a server, or you can use the dbmt tool that now includes the Updall task as well as other options instead of running Updall alone. See the related topics for more information.

You can use several methods of running the Updall task on a server.

• From Task -> Start tool in the Domino® Administrator -- Use this method if you don't want to use command-line options.
• Using the Load Updall console command -- Use this method if you are comfortable using command-line options or if you want to run Updall directly at the server console when there is no Domino Administrator running on the server machine.
• Program document that runs Updall -- Use this method to schedule Updall to run at particular times.
• Run Updall on a Win32 platform -- Use this method if you are unable to run Updall at the server console. This method requires that you use the "n" prefix -- for example, nupdall - R.

When you use these methods, you can include options that control what Updall updates. For example, you can update all views and not update any full-text search indexes.

The following tables describe the options you can use with Updall (Task -> Start ). The second column lists the equivalent command-line options that you use when you use a console command to run Updall and when you schedule Updall to run in a Program document.

Use this syntax when you use the Load updall console command:

Load updall databasepath options

For example:

Load updall SALES.NSF -F

You can specify multiple options -- for example:

Load updall -F -M

Table 1. Updall - Basic options

Option in Task - Start tool	Command-line option	Description
Index all databases Index only this database or folder	databasepath This option is used when running Updall as a console command.	Choose the option to index all databases if you want updall to process all databases on the server. Choose the option to specify a database or folder if you want updall to limit processing to the specified location. To update a database in the Domino data folder, enter the file name, for example, SALES.NSF. To update all databases contained in a subfolder of the data folder, specify the path relative to the data folder, for example, DOC\README.NSF.
Update this view only	database -T viewtitle	Updates a specific view in a database. Use, for example, with -R to solve corruption problems. Note: -T cannot be used with .IND (indirect) files.

Table 2. Updall - Basic options - more

Option in Task - Start tool	Command-line option	Description
Update: All built views	-V	Updates built views and does not update full-text indexes.
Update: Full text indexes	-F	Updates full-text indexes and does not update views.
Update: Full text indexes: Only those with frequency set to: Immediate or Hourly	-H	Updates full-text indexes assigned "Immediate" or "Hourly" as an update frequency.
Update: Full text indexes: Only those with frequency set to: Immediate, Hourly, or Scheduled	-M or -S	Updates full-text indexes assigned "Immediate," "Hourly," or "Scheduled" as an update frequency.
Update: Full text indexes: Those with frequency set to: Immediate, Hourly, Daily, or Scheduled	-L	Updates full-text indexes assigned "Immediate," "Hourly," "Daily" or "Scheduled" as an update frequency.

Table 3. Updall - Rebuild options

Option in Task - Start tool	Command-line option	Description
Rebuild: Full-text indexes only	-X	Rebuilds full-text indexes and does not rebuild views. Use to rebuild full-text indexes that are corrupted.
Rebuild: All used views	-R	Rebuilds all used views. Using this option is resource-intensive, so use it as a last resort to solve corruption problems with a specific database.
Rebuild: Full-text indexes and additionally: All unused views	database -C	Rebuilds unused views and a full-text index in a database. Requires you to specify a database.

Table 4. Updall - Search Site options

Option in Task - Start tool	Command-line option	Description
Update database configurations: Incremental	-A	Incrementally updates search-site database configurations for search site databases.
Update database configurations: Full	-B	Does a full update of search-site database configurations for search site databases.

Option for running Updall as part of dbmt

Updall performs the following tasks by default. These are also tasks that the database maintenance tools performs:

• purges deletion stubs
• expires soft deleted entries
• updates unread lists

Because the database maintenance tool is meant to replace (and improve upon) running updall nightly, you can use the following new option for updall to skip the tasks the preceding tasks, making updall faster when you run it for any one-time purpose.

-nodbmt

When you run updall as part of dbmt, Domino also ensures that the following views are built for databases with a template name of StdR7Mail, StdR8Mail, StdR85Mail and StdR9Mail:

• $Inbox
• $Drafts
• $All
• ($RepeatLookup)
• ($ToDo)
• ($Calendar)
• ($Haiku_TOC)
• ($Alarms)
• ($iNotes)
• ($Users)
• ($iNotes_Contacts)
• ($ThreadsEmbed)

After these views are built, they will not be discarded due to non-use.

Error message in Notes 11.0.1: "Insufficient memory - local heap is full"

I recently (like last week) upgraded Notes on my main workstation from 10.0.1 to 11.0.1FP1. The installation went uneventfully. But when I ran Notes and tried to open my mail database, Notes locked up and presented me with an error I hadn't seen before:

 Insufficient memory - local heap is full

I immediately Googled the error, but got nothing very useful in return. So then I decided to run the standard array of Notes maintenance tasks: Fixup. Compact -c. Updall. Not much help there either. But I noticed that Notes wasn't failing until I clicked the Mail or Calendar links in either the Task Bar or the Open List. So I tried opening mail manually, via Ctrl+O. That did seem to have a positive effect. My mail opened! And I was able to work with it for awhile. But eventually the error message popped up again and I had to kill and restart Notes to get back to work again.

So, as a last resort, I decided to give HCL Support a try. And pretty quickly I had a positive result. I searched my error message in the KnowledgeBase and got a direct hit - KB0081067.. (I wonder why it didn't turn up in my Google search.)

The fix in the KBase article directed me to carry out what amounted to a fresh reconfiguraton of Notes. I carried it out. It worked in that, afterwards, Notes could open my mail without the lockup. But it was a problem for me because I lost all my Desktop folders and tiles and all my bookmarks.

And it seemed like more of a workaround than a solution to me. Yeah, it might get Notes to run and open my mail DB. But it didn't provide any clue as to why the error was occurring and it didn't reassure me at all that the error wouldn't occur again some day. And, for me, losing all my tiles and bookmarks was a painful solution. I sort of live and die by my Notes configuration.

So I opened a support ticket. A nice, knowledgeable support tech named Nic responded and agreed that, yes, my tiles and bookmarks would be wiped out. And, no, the fix in the KBase article was not a permanent fix.

I asked if increasing the size of the local heap would be a sensible thing to do. Nic said, yes it would, but the new, bigger heap would consume about 2 GB of RAM. My workstation has 16 GB of RAM, so I asked how to proceed. Nic provided me with a link to this additional KBase article that described the procedure. I followed it. Notes is running. Mail is opening. So far, so good.

I suggested to Nic that the first KBase article needs to be amended to: 1) add a caveat about losing one's Notes configuration if one follows the instructions; 2) add the fact that following the reconfiguration procedure isn't necessarily a permanent fix; 3) state that one should alternatively consider increasing the heap size if one has sufficient RAM; and 4) provide the link to the second article describing the procedure for increasing the heap size. Nic agreed that the article should be amended with those items.

By the way, my experience with HCL Tech Support has been generally positive so far. And, thanks, Nic, for your helpful support.

Is it time to renew your Domino ID Vault certificates?

IBM issued a Technote today detailing the procedure for renewing ID Vault Trust Certificates and Password Reset Certificates. They expire after 10 years. ID Vaults were first introduced in Domino 8.5, which was released December 2018, 10+ years ago now. So early adapters of the ID Vault will increasingly be having to renew their certificates.
The Technote describes:

• The error message that signals that your certificates have expired ("Not a valid ID or the ID is corrupted");
• The procedure for determining the expiration dates of your certificates; and 
• The procedure for renewing them. Unfortunately you can't just recertify them. You have to remove the expired ones then issue new ones.

My favorite feature of Sametime Meetings

Call me a dinosaur, but I love Sametime Meetings. Here's why. It has the best chat functionality of any meeting software I've seen. What I like about it in particular are two things:

• You can categorize entries in the chat window according to five predefined categories.
• When the meeting ends, Sametime generates a meeting report that organizes the chat entries by category.

The five categories are:

• Group Chat
• Minutes
• Action Item
• Question
• Starred Item

If you just enter text in the chat window, your entry defaults to Group Chat. But you can select another category before you hit Enter. That entry and all following entries are under the category you selected until you select another one.

When you end the meeting you see the dialog below, where you can choose to generate a meeting report or not, and where to store it. I have a subscription to Sametime Meetings in IBM Connections Cloud, where I also have subscriptions to IBM Notes Mail and IBM Connections. So the dialog defaults to saving the report to "My Files", my cloud-based file storage area.

I don't go around testing the features of meeting products. So it may be that other meeting software has these features too; but I haven't seen them in WebEx, GoToMeeting, or Zoom.

And I really don't in my life have much call to conduct meetings. But were I, say, the Chair of a regularly scheduled meeting I might designate someone in the meeting to take meeting minutes by entering them in the meeting chat window. And I would put all action items and unresolved questions there too.

By entering that information right in the chat stream for all to see, the meeting attendees could act as proofreaders, flagging errors as they occur. Then, after the meeting ends I would use the meeting report to follow up on action items, unresolved questions, and important ("Starred") items, perhaps distributing parts of the report to the people assigned to carry out each item.

What's not to like about this? (Now, if only the process of installing the screen-sharing browser plug-in were a little easier and faster...)

A Traveler user's iPhone stopped working over the weekend; interesting reason why

Monday morning I received notice that a Notes Traveler user's iPhone had stopped sending/receiving messages. I see this sort of thing occasionally and I generally respond by issuing a Tell Traveler User command to obtain the device ID of the user's mobile device, then issuing a Tell Traveler Reset command to resync the devices. That almost always resolves the user's problems.

But this time when I issued the Tell Traveler User command it came back with a raft of errors I had never seen before. The first one was that the user's name wasn't in the mail database's ACL.

So I opened the Domino Directory to the People view and saw that the user's Person document had two (count 'em, two) replication/save conflict documents. I thought, aha, maybe Traveler is getting misled by all the Person documents for this user.

I compared the content of the three documents and none of the name fields (or for that matter any fields in the first few tabs) were different among the three documents. But I did see that the Last Updated field under the Administration tab was different for all three. They were all updated the previous Friday, late in the day by IAM (the SSO service used by the organization). The "winner" Person document was the most recently edited, so I deleted the two conflict documents.

Then I opened her "winner" Person document and saw that she had been renamed at some point in the past (because Domino preserves a user's former names when it renames a user, say, with a new married name). I noticed also that her mail database's file name was formed from her first initial and former last name, not her new last name. That was normal.

Then I opened her mail database and saw three unexpected things:

• The title of the database was still set to her former name; 
• The ACL had only her former name, not her new name in it; and
• The Owner field was still set to her former name, not her new name.

All should have been set to her new name when she was renamed. I wondered if someone had attempted to rename the user manually instead of correctly telling the Administration Process to rename her. Occasionally a Windows administrator, unfamiliar with Notes architecture, will assume they can do that and, in the process, will make a mess of everything - not that I expected anyone at this company to be so dumb.

It occurred to me to have a look at the Administration Requests database to see if there were any Rename-related documents in it. Sure enough, there was an Initiate Rename in Domino Directory document. It had been created late the previous Friday, and the request had been carried out. But, curiously, there were no follow-on Rename documents. By now there should have been a whole train of them.

The Administration Process, running on each Domino server, checks the Administration Requests database every minute or so throughout the day. When it discovers new requests it attempts to carry them out. If it succeeds, it typically generates the next request in a given series. Then, when it checks again a minute later (or maybe an hour, a day, or a week later, depending on the nature of the request), it carries out that one, and so on until the whole process of (in this case) renaming the user is complete.

I checked Administration Help and read about the Initiate Rename in Domino Directory step of the Rename process and it became clear to me what was going on. After the Administration Process carries out the steps required by the Initiate Rename in Domino Directory document (which are to make certain changes in the Person document, among them adding user's new name to the top of the list of names in the User Name field), it waits for the user to log into Notes. When the user does that, Notes will check with the user's mail server to see if it needs to respond to any changes made regarding the user on the server. When Notes does so, it discovers that the user has been renamed, and it makes a number of local changes as a result:

• Notes pulls the user's new certificate down from the server and merges it into the User ID, which as a result includes the user's new name along with her former name;
• Notes renames the user in the ACLs of all local databases and in configuration files such as notes.ini; and
• After Notes has done all that, it creates the next Rename request in the Administration Requests database for the user: Rename Person in Domino Directory.

At this point the Administration Process can complete the renaming process. That is, it can carry out the steps defined by the Rename Person in Domino Directory document and all of the documents that will follow it. It will rename the person in a raft of places, including (but not limited to) group documents, ACLs of various databases throughout the domain (including, most importantly from Traveler's point of view, the user's mail database), and Names fields in any databases in the domain where it locates the user's former name.

So what must have happened, I concluded, is that the user was renamed in Notes so late on the previous Friday that her copy of Notes had not had the opportunity to update itself and create the Rename Person in Domino Directory document. So the user was renamed in the Person document, thanks to the Initiate Rename in Domino Directory document, but no place else. As a result, Traveler could not see that the newly renamed user had sufficient rights to the mail database and stopped updating the user's iPhone. The user could see over the weekend that her iPhone had stopped functioning; so she opened a support ticket, the one that was assigned to me.

Late Monday morning I telephoned the user. Because it was a holiday (President's Day), she still had not attempted to open and log into Notes on her laptop. I asked her to do so and, voila, all the dominoes described above started falling and, voici, eventually her iPhone started working again. Oh la la!

Notes/Domino security vulnerability patched by IBM. You should apply this fix soon.

IBM has discovered and (on Friday, December 14, 2018) released a patch for a security vulnerability in NSD (Notes System Diagnostics) for Windows. So now is a really good time to upgrade your Windows-based Domino servers to 9.0.1FP10IF5 and your Windows-based Notes clients to version 9.0.1FP10IF6. (Or you could upgrade them to version 10.) Here's the Technote with the details. 

Beware of stray cables

I just watched this scary video demonstration by Kevin Mitnick of KnowBe4 of a lightning cable that infects any computer you plug it into (well, the demo used a Windows 10 computer) with malware. In the demo Kevin suggests that we stop leaving cables plugged into our work computers, implying that the demo lightning cable could be swapped in when our back is turned. And don't use any old cable that you might find lying around? "You need to stop, look, and think", he says, "before you plug any device into your computer."

But Kevin leaves a lot of other questions unanswered:

• How can we determine if a cable is malicious?
• How can we tell if a cable we buy in a store is malicious or not? 
• Do we have to stop buying non-Apple branded lightning cables now?
• Are Apple branded cables save, even?
• Can we use anti-malware software to protect ourselves if such a cable is plugged into our machine?

Hey, reader, sleep well tonight!

IBM Notes 9.0.1, MacOS High Sierra, and Java 8. Part 2.

After I wrote my Jan 24 post about running Notes on MacOS in Basic Mode, IBM released Notes 9.0.1 for MacOS Interim Fix 13. IF13 provides a fix for the problem I described in that post, which was that upgrading Java on the Mac to a version higher (more recent) than Java 8 Update 151 caused Notes to fail to start. That surprised me because, on the Windows platform Notes provides its own JVM; you can install whatever Oracle JVMs you like under Windows without affecting Notes at all. But it turns out that Notes running on the MacOS platform does not come with its own JVM and does rely on the Oracle JVM that you install on the machine. And, of course, Java 8 Update 152 caused Notes to choke and die.

In any case, the workaround at that time was to run Notes in Basic Mode, which effectively reverts Notes to running the old Release 7 Notes client written in C++, naked of the Expeditor wrapper that provides the new features of  Notes that debuted in Release 8. In Basic Mode, Notes does not use any Java-based features.

Another odd thing that I discovered since writing my last post is that the Notes 9.0.1 installer for the MacOS platform is "broken" with respect to MacOS High Sierra. The first time you run it on a machine running MacOS, it fails in the Provisioning stage, with the following error message:

File /Applications/IBM Notes.app/Contents/MacOS/rcp/rcplauncher.properties not found. Provisioning process failed to launch or was terminated before status could be determined.

Then the installation fails.

The fix for this is, of all things, to rerun the installer. The second time around it succeeds all the way through. Go figure.

How to run IBM Notes in Basic Mode on MacOS

Late last week a RockTeam client notified me that a user upgraded Java on his Macintosh to Java 9.0.1, then discovered that IBM Notes would no longer start on the machine. The client is a software publisher and the user is a developer. He figured out how to run Notes in Basic Mode (i.e., without the Eclipse wrapper that provides additional, Java-based functionality to the Notes client, which is known as "Standard Mode") and concluded that his upgrade of Java must have caused the problem.

My client asked me to help figure out what the problem was and how to get Notes running again in Standard Mode. With IBM's help (I opened a PMR) I soon discovered two things:

• This IBM document states that 64-bit Notes running on MacOS does not support Basic Mode.
• This IBM document states that 64-bit Notes will not run on MacOS if you upgrade Java to version 8 Update 152 (or later).

The first document above turns out to be inaccurate; Notes will in fact run in Basic Mode on the Mac. The second document is accurate; Notes will not currently run in Standard Mode if you install Java 8 Update 152 or later on your Mac. So the user has two options: Downgrade or remove Java; or settle for running Notes in Basic Mode. Running in Basic Mode, one loses the Open button, the full-text search field in the upper right corner of the Notes window, and the right sidebar and all the Java-based apps that it contains.  The subject user has decided for now to live with Basic Mode. The company isn't a big user of the "Social Edition" features of Notes, so Basic Mode probably meets all of this user's current needs.

What interested me about this was that, from the way he described the problem, it was obvious that our user was not a Notes guru, knowledgeable about Notes's different running modes. He is a developer, though, and knowledgeable about Java and Eclipse. So he was able to just figure out how to get Notes to run without Eclipse. I thought that was pretty ingenious of him and asked how he managed to do it. So far the only answer I've received from him is "by brute force".

But I did some testing myself and learned that you can indeed start Notes in Basic Mode on the Mac. Here are the two ways I found:

• Set the variable UseBasicNotes=1 in Notes Preferences. Notes Preferences is the Mac equivalent of notes.ini, where one would set this variable on the other supported Notes platforms. When you set this variable, Notes always starts in Basic Mode, i.e., without trying to wrap itself in Eclipse. If you want to run Notes in Standard Mode, you have to remove this variable or reset its value to "0".
• Issue this command in Terminal:
  

  "/Applications/IBM Notes.app/Contents/MacOS/notes" -basic

  • NOTE: The quotes are necessary because of the space between "IBM" and "Notes.app". 
  • Notice also (my fellow geek) that I appended "-basic" to the command. Under Windows you could append either "-sa" or "-basic", but "-sa" did not work for me under MacOS.

I know there's a way to create a script to run the above command with a mouse click (or two). I'm not a UNIX guru, so I don't know how by heart. When I find some time, I'll figure it out and post that information here.

How to locate a document with a bad NLO reference (or What to do when fixup -d -j fails)

A user has a local mail database. Every time it replicates with the mail server, replication takes 5-8 minutes as it tries repeatedly to replicate the same 1500+ documents. Running fixup, compact, updall against the mail database on the server revealed this error message:

The database <pathname>.nsf attempted to access a missing file: H:\DAOS\0007\E5128368DF400D54DE01F369AFAAF560FCB85F350007FAC9.nlo: File does not exist

It seems that the replication process was getting hung up because of a document contained an invalid NLO reference.

The quick and dirty way to deal with a problem like this is to issue the following fixup command, which deletes documents that contain invalid NLO references:

load fixup -d -j <pathname>.nsf

This command did not, however, resolve the problem for us because (I discovered later) the problem document included two attachments, both pointing to NLO objects. But the second pointer was valid, so fixup decided not to delete the document.

The solution to this problem was to locate the document that had the bad NLO reference. Here is the way to do that:

1. Set the following notes.ini variable on the mail server (no server restart needed): Debug_DAOS_Diagnostics=1
2. Run the following command at a console prompt: Tell DAOSMgr listnlo MAP -V <pathname>.nsf
   • This causes the creation in the mail server's Data folder of a text file, listnlo.txt, that contains a list of all NLO references in <pathname>.nsf.
3. Make a local copy of listnlo.txt. It is a comma-delimited file. Open it in a spreadsheet program or a text editor.
4. Search for the NLO's hash key, which is the file name of the NLO file.
   • In my case, I searched for "E5128368DF400D54DE01F369AFAAF560FCB85F350007FAC9".
5. The Note ID (hex format) of the document appears in the second column of the row that contains the search term.
6. In Domino Administrator under the Files tab, select the database in question and use Find Note (Tools pane, Database section) to find the problem document by its Note ID. Alternatively you can use a third-party tool, such at Ytria's ScanEZ.
7. Open the subject database on the mail server and use the information provided by the Find Note dialog (document date, subject, etc.) to locate the problem document in the database.
8. Decide what to do about the problem document. Possible fixes include:
   • Delete the problem document.
   • Delete the problem attachment from the document.
   • Restore the problem attachment from backup.
9. Optional but recommended: Reset  the notes.ini variable, Debug_DAOS_Diagnostics, to 0 or NULL.
   • se co Debug_DAOS_Diagnostics=0; OR
   • se co Debug_DAOS_Diagnostics=

Thanks to the following bloggers for this fix: Cristian D'Aloisio, Ralf Petter, Ulrich Krause

Copyright 2017 by Rob Kirkland

Here are four good reasons to upgrade your IBM Notes mail clients to the latest Fix/Feature Pack

Reason #1: It fixes a Denial of Service vulnerability.

Reason #2: It fixes another Denial of Service vulnerability.

Reason #3: It fixes an Open Source zlib vulnerability.

Reason #4: It fixes an Open Source libpng vulnerability.

If your users are running Notes 9.x for Windows, you want to upgrade them to Notes 9.0.1 FP9.

If your users are running 64-bit Notes 9.x for Mac, you want to upgrade them to Notes 901 64-bit Mac IF11.

If your users are running Notes 8.5.3 for Windows, you want to upgrade them to Notes 8.5.3 FP6 IF15. Then start preparing to upgrade your Domino servers and Notes clients to 9.0.1, because Domino/Notes 8.5.3 will be going out of support soon. You can upgrade your users to Notes 9.0.1 before you upgrade your servers to Domino 9.0.1.

If your users are running earlier than Notes 8.5.3, you want to upgrade them because you are running an unsupported version of Notes/Domino. Plus, the four vulnerabilities listed above are just the four most recently fixed vulnerabilities. Your old, unsupported copies of Notes harbor others too.

If your mail databases are hosted in IBM Connections Cloud or if you have implemented IBM Verse on-premises, you could migrate your users from IBM Notes to IBM Verse. But if you are running IBM Verse on-premises, I recommend that you upgrade your Domino mail servers to Domino 9.0.1 FP9. (I'll tell you why in my next post.)

And remember that, if you are hosting your Notes mail on on-premises Domino servers, your Notes/Domino licensing probably includes IBM SmartCloud Notes licensing. That means you can migrate your users' mail databases to IBM Connections Cloud-based Domino mail servers at no additional cost beyond your current licensing costs. If you are not sure what your Notes/Domino rights and restrictions are, ask me; I can help you to sort that out.

Also, if you don't think you can migrate away from Notes to Verse because you are still running Notes-only Domino applications, that's okay, because Notes is a great product. But if you would like to upgrade your apps, ask me about your options for browser-enabling your Notes applications. There are lots of options available.

Finally, if you are thinking about migrating from Notes/Domino to Exchange/Outlook or Office 365 or Google Apps or SoHo or whatever, ask me why you should reconsider. IBM Connections Cloud is superior to those other platforms -- runs circles around them, in fact -- in any number of ways. And it is getting better at an accelerating pace.

Addendum: I can help you automate (or at least streamline) your users' Notes upgrade process, so you don't have to babysit each upgrade.

IBM Notes 9.0.1 Feature Pack 9 is available and here is my favorite enhancement

IBM Notes 9.0.1 Feature Pack 9 includes this:

High resolution support for the Notes® ClientThe Notes® client on Windows correctly scales text and icons when high resolution monitors or custom DPI settings are used. 

 This might seem like a minor fix, but for me it is great news. If you have shopped for Windows computers lately you may have noticed that many of the laptops now sport fantastically high screen resolutions. I recently bought (then regrettably returned) an HP Spectre laptop (aircraft carrier, really - it was BIG) with a 4000x3000 pixel display. Then I bought (and kept) a Microsoft Surface that has a 3000x2000 pixel screen. The problem I had with both of them was that IBM Notes, which is my bread-and-butter software, couldn't cope with such high resolution screens. Either the text would be so small that I needed a magnifying glass to read it or, if I tried to use Windows or Notes text resizing tools to enlarge the text, it would enlarge in unsatisfactory ways. For example, the text would be large enough to read, but the line height would not change, resulting in the tops of the letters being hidden and the whole exercise of reading the text very unsatisfactory.

I had found two workarounds for the situation. At first I simply reduced the screen resolution of my new laptop to an old-fashioned 2048x1024 (or less) so that I could read the content in my Notes windows. That worked fine. But it irritated me to have spent all that money and to be unable to use one of the nicest features of my new laptop. 

Later I noticed that, if I started my computer in high resolution mode with no external monitor attached, the text and icons in my Notes window would be properly sized and would look okay. But then if I attached an external monitor (my highest resolution one being 2048x1024) and moved the Notes window onto it, it would not resize properly. But then if I rebooted with the external monitor still attached, the Notes window, when reopened on that monitor, would display properly sized text and icons. But then if I moved the Notes window back onto the native monitor, the text in it would again be improperly sized. Or if I simply unplugged the external monitor, forcing the windows on it to move onto the native monitor, same result. But rebooting would again fix the text sizing issue.

In other words, my second workaround was to reboot the laptop (or, as I later discovered would also work, to simply log out of Windows, then log back in) whenever I wanted to connect to and use my external monitor or to disconnect from it. Doing so would fix the text in the Notes window. But it was a hassle to have to shut down and reopen the various programs I typically run every time I needed to connect my laptop to or disconnect it from my external monitor. All in all, a pretty unsatisfactory situation.

All that is fixed now with the happy release of Notes 9.0.1 Feature Pack 9. Now, if I unplug my external monitor, Notes repositions itself to the native monitor and resizes its text and icons exactly as it should. And if I plug the external monitor back in, Notes reverses the process exactly as it should - everything properly formatted and sized, all text easily readable. I am a very happy camper. 

Thwart spearphishing attacks by using digital signatures in IBM Notes Mail

In case you aren't inclined to read this post through and through, despite the beauty of its prose, here's the bottom line: Spearphishers cannot easily spoof a digitally signed message; so you should enable default digital signing of messages by all of your users to prevent spearphishing attacks.

Recently a friend told me about a security breach at his company. The breach was a textbook spearphishing attack. A member of the accounting department received an email message that purported to be from the company CEO. In the message, the “CEO” directed the recipient to wire umpteen thousands of dollars to a certain bank account. The email arrived late Friday afternoon and urged the recipient to wire the money “before close of business today”. The CEO was, of course, not around, so could not be reached to verify. The recipient bookkeeper did as directed and the company never saw the money again. 

Even more recently (because it's tax season, I suppose) I read that someone has been very successfully using spearphishing messages to trick companies into sending their employees' W-2 forms to the spearphisher. This spearphisher has succeeded so far in collecting some 120,000 W-2 forms from a number of organizations.

It occurred to me that if the victim companies used IBM Notes Mail they could easily have thwarted these spearphishing attacks.

Phishing attacks in general are a type of social engineering attack in which the attacker blasts a deceptive email out to as many recipients as possible, hoping to trick some of the recipients into responding in a way that will enable the attacker to rip off the respondent in some way.  Spearphishing attacks are phishing attacks aimed at a single recipient. The email (or other attack vector) is finely tuned to trick the recipient into trusting the sender and responding positively. The email typically purports to be from a trusted, authoritative executive within the recipient’s organization; the email may direct the recipient to send money to some bank account and do it immediately because time is of the essence, as in my first example. Or, as in my second example, the goal may be to get the recipient to give away confidential information such as, oh, I don’t know, how about: The W-2 forms of every one of the organization’s employees!

The feature of Notes that could have prevented these spearphishing attacks is digital signing of documents. IBM Notes for decades now has included a feature permitting senders to sign outgoing messages digitally. This is not a “written” signature at the bottom of outgoing messages, but rather an encrypted hash of the message that accompanies the message and enables the recipient to verify that 1) the message really came from the purported sender and 2) the content of the message was not altered en route to the recipient. When the recipient of the message opens it, Notes verifies the integrity of the hash and assures the recipient that the message is genuine. In the message header it displays “This message is digitally signed." In the Status Bar (bottom of the window) it briefly displays "Signed by <sender’s fully distinguished name> on <date> <time>, according to <certifier name>”. 

A spearphisher could not easily spoof such a message because to do so he would have to obtain the sender’s private key, which only exists in the sender’s Notes user ID file. The spearphisher would have to obtain a copy of the purported sender’s ID file and learn its password (or, even harder, obtain a certifier ID and learn its password). That is, he would have to compromise both parts of a Notes user’s two-factor security. Not impossible, but not an easy thing for a total and remote stranger to pull off. 

Digital signing of messages has been available in Notes since at least 1993. But it is voluntary by default. Notes users have to check a box to digitally sign any message before sending it. (They can check another box to digitally encrypt the message, too, if they want.) As you might guess, in most organizations hardly anyone ever checks the boxes or has any clue about why they might want to do so. 

What Notes organizations can do (and should do soon, because spearphishers are clearly getting really good at their craft) is enable digital signatures by default, so that messages are digitally signed unless the sender turns off the feature.

First, of course, Notes administrators should notify their users that they will enable this feature.

Before that, though, and most importantly of all, Notes admins should educate their users so that

• The users become appropriately paranoid about responding to mail that asks them to do potentially problematic things.
• The users know why digital signing is important.
• The users know that they should always look for the notice that the message is digitally signed before assuming a message from a purported Notes user is genuine. 

And as I write and think about how serious this spearphishing/ransomware plague is getting, it occurs to me that,  in parallel with educating the users, the Notes admins should get management on board and get HR to pitch in and revise personnel policies to make it clear to everyone in the organization that thou shalt [do certain things] and thou shalt not [do certain other things] with respect to email.

And all of that being said and done, and now that the users know what to expect, what to do, and why it's important, the admins should enable default digital signing of Notes mail. 

Enabling default digital signing of messages is very easy or moderately easy depending on a number of factors. Enabling it is very easy if all of an organization’s mail users use Notes to send and receive mail, and the only mail the organization is concerned about protecting with digital signatures is internal mail. An admin sets a particular field in a policy and applies the policy to the target users. Done.

Enabling default digital signing of messages is moderately easy if users also use Web browsers or non-Notes (POP, IMAP, IMSMO) mail programs or mobile devices to send and receive mail, because you can’t enable default digital signatures in non-Notes clients by policy. Rather, you have to convince users to enable it as a default in their user preferences. (That pretty much means that you really must educate your users in the importance of digital signatures, and not just pay lip service to it.) And you have to make sure that their ID files have either been merged into their mail databases or reside in an ID Vault. (But if your non-Notes mail users' mail resides on IBM cloud-based mail servers, their IDs must reside in the cloud-based ID Vault; merging their ID into their mail database won't be sufficient.)

Finally, if you also want to give users the option to sign messages addressed to recipients whose mail does not reside on Domino servers, the implementation process becomes not so easy at all. I'll be happy to discuss the complications involved in that process if either you or my other reader expresses interest in reading about it. But for now, I'll leave this discussion here:

• Educate management, HR, and your users about the dangers of spearphishing and how to cope with it.
• For your Notes mail users, use policies to enable default digital signing of messages to other Notes users.
• For your non-Notes, Domino-based mail users who have Notes IDs, encourage them strongly to enable the preference to sign outgoing mail by default. And get their Notes IDs merged into their mail databases or an ID Vault.
• For your non-Notes, Domino-based mail users who do not have Notes IDs, re-register them to generate Notes IDs for them.
• If you have non-Domino email users (Exchange, O365, Google, whatever), take two aspirin and call me in the morning.

Jumping over hurdles to upgrade Domino

Here’s what I had to do to the other day to apply Interim Fix 2 to a Domino server running version 9.0.1 Fixpack 4: Shut down Domino, the Domino Controller, the Java Console, NSD, and (wait for it, wait for it) Windows Management Instrumentation. Huh? Apparently WMI or one of its dependent apps grabs onto one or more Domino DLLs and won’t let go. Or some damn thing. One of the dependent apps was VMware Helper. I wonder if that was the culprit.  I remember running into a similar problem years ago upgrading Domino on a Xen VM. That might explain why one only sees problems like this occasionally (when least welcome, of course).

Anyway, until I stopped the WMI service, I kept getting this error when I tried to apply IF2:  “Notes/Domino related process is still running”. Thanks to Rainer Brandl for that tip.

This morning, I tried the same thing on another Domino server and it didn’t work. Same error message. More research. Thanks to Daniel Nashed for the following workaround:

Set Domino and NSD to Manual in Windows Services, rename the Domino directory, restart Windows, rename the Domino directory back to original, apply the patch, reset the services to automatic, restart NSD and Domino. 

Daniel didn’t actually say anything about resetting the Windows services. That was my own “belt-and-suspenders” add-on. Don’t know if it was really necessary. But I got past the error message. Finally. 

But IF2 still wouldn’t install because blah blah blah. Look at the UPGRADE.LOG. Wrong version of something in the Domino directory. I suspected I may have failed to notice if IF1 was installed on this server. So I ran the IF1 installer, which offered to uninstall IF1, and then did so. Then I succeeded, finally, in installing IF2. 

Had that not worked, I might have tried these other things: 

• Run the Fixpack 4 installer. See what it says. Depending on what it says, maybe uninstall/reinstall it, then try to install IF2 again.
• Run the 9.0.1 installer in Repair mode. Then reinstall FP4 and IF2.
• Uninstall 9.0.1, strip everything from the Program directory, reinstall all.

Luckily I didn’t have to get that radical. 

I love working on Sunday mornings.