Friday, December 15, 2017

How to locate a document with a bad NLO reference (or What to do when fixup -d -j fails)

A user has a local mail database. Every time it replicates with the mail server, replication takes 5-8 minutes as it tries repeatedly to replicate the same 1500+ documents. Running fixup, compact, updall against the mail database on the server revealed this error message:
The database <pathname>.nsf attempted to access a missing file: H:\DAOS\0007\E5128368DF400D54DE01F369AFAAF560FCB85F350007FAC9.nlo: File does not exist
It seems that the replication process was getting hung up because of a document contained an invalid NLO reference.
The quick and dirty way to deal with a problem like this is to issue the following fixup command, which deletes documents that contain invalid NLO references:
load fixup -d -j <pathname>.nsf
This command did not, however, resolve the problem for us because (I discovered later) the problem document included two attachments, both pointing to NLO objects. But the second pointer was valid, so fixup decided not to delete the document.
The solution to this problem was to locate the document that had the bad NLO reference. Here is the way to do that:
  1. Set the following notes.ini variable on the mail server (no server restart needed): Debug_DAOS_Diagnostics=1
  2. Run the following command at a console prompt: Tell DAOSMgr listnlo MAP -V <pathname>.nsf
    • This causes the creation in the mail server's Data folder of a text file, listnlo.txt, that contains a list of all NLO references in <pathname>.nsf.
  3. Make a local copy of listnlo.txt. It is a comma-delimited file. Open it in a spreadsheet program or a text editor.
  4. Search for the NLO's hash key, which is the file name of the NLO file.
    • In my case, I searched for "E5128368DF400D54DE01F369AFAAF560FCB85F350007FAC9".
  5. The Note ID (hex format) of the document appears in the second column of the row that contains the search term.
  6. In Domino Administrator under the Files tab, select the database in question and use Find Note (Tools pane, Database section) to find the problem document by its Note ID. Alternatively you can use a third-party tool, such at Ytria's ScanEZ.
  7. Open the subject database on the mail server and use the information provided by the Find Note dialog (document date, subject, etc.) to locate the problem document in the database.
  8. Decide what to do about the problem document. Possible fixes include:
    • Delete the problem document.
    • Delete the problem attachment from the document.
    • Restore the problem attachment from backup.
  9. Optional but recommended: Reset  the notes.ini variable, Debug_DAOS_Diagnostics, to 0 or NULL.
    • se co Debug_DAOS_Diagnostics=0; OR
    • se co Debug_DAOS_Diagnostics=
Thanks to the following bloggers for this fix: Cristian D'Aloisio, Ralf Petter, Ulrich Krause

Copyright 2017 by Rob Kirkland

Friday, September 1, 2017

Here are four good reasons to upgrade your IBM Notes mail clients to the latest Fix/Feature Pack

Reason #1: It fixes a Denial of Service vulnerability.

Reason #2: It fixes another Denial of Service vulnerability.

Reason #3: It fixes an Open Source zlib vulnerability.

Reason #4: It fixes an Open Source libpng vulnerability.

If your users are running Notes 9.x for Windows, you want to upgrade them to Notes 9.0.1 FP9.

If your users are running 64-bit Notes 9.x for Mac, you want to upgrade them to Notes 901 64-bit Mac IF11.

If your users are running Notes 8.5.3 for Windows, you want to upgrade them to Notes 8.5.3 FP6 IF15. Then start preparing to upgrade your Domino servers and Notes clients to 9.0.1, because Domino/Notes 8.5.3 will be going out of support soon. You can upgrade your users to Notes 9.0.1 before you upgrade your servers to Domino 9.0.1.

If your users are running earlier than Notes 8.5.3, you want to upgrade them because you are running an unsupported version of Notes/Domino. Plus, the four vulnerabilities listed above are just the four most recently fixed vulnerabilities. Your old, unsupported copies of Notes harbor others too.

If your mail databases are hosted in IBM Connections Cloud or if you have implemented IBM Verse on-premises, you could migrate your users from IBM Notes to IBM Verse. But if you are running IBM Verse on-premises, I recommend that you upgrade your Domino mail servers to Domino 9.0.1 FP9. (I'll tell you why in my next post.)

And remember that, if you are hosting your Notes mail on on-premises Domino servers, your Notes/Domino licensing probably includes IBM SmartCloud Notes licensing. That means you can migrate your users' mail databases to IBM Connections Cloud-based Domino mail servers at no additional cost beyond your current licensing costs. If you are not sure what your Notes/Domino rights and restrictions are, ask me; I can help you to sort that out.

Also, if you don't think you can migrate away from Notes to Verse because you are still running Notes-only Domino applications, that's okay, because Notes is a great product. But if you would like to upgrade your apps, ask me about your options for browser-enabling your Notes applications. There are lots of options available.

Finally, if you are thinking about migrating from Notes/Domino to Exchange/Outlook or Office 365 or Google Apps or SoHo or whatever, ask me why you should reconsider. IBM Connections Cloud is superior to those other platforms -- runs circles around them, in fact -- in any number of ways. And it is getting better at an accelerating pace.

Addendum: I can help you automate (or at least streamline) your users' Notes upgrade process, so you don't have to babysit each upgrade.

Friday, August 25, 2017

IBM Notes 9.0.1 Feature Pack 9 is available and here is my favorite enhancement

IBM Notes 9.0.1 Feature Pack 9 includes this:
High resolution support for the Notes® ClientThe Notes® client on Windows correctly scales text and icons when high resolution monitors or custom DPI settings are used. 
 This might seem like a minor fix, but for me it is great news. If you have shopped for Windows computers lately you may have noticed that many of the laptops now sport fantastically high screen resolutions. I recently bought (then regrettably returned) an HP Spectre laptop (aircraft carrier, really - it was BIG) with a 4000x3000 pixel display. Then I bought (and kept) a Microsoft Surface that has a 3000x2000 pixel screen. The problem I had with both of them was that IBM Notes, which is my bread-and-butter software, couldn't cope with such high resolution screens. Either the text would be so small that I needed a magnifying glass to read it or, if I tried to use Windows or Notes text resizing tools to enlarge the text, it would enlarge in unsatisfactory ways. For example, the text would be large enough to read, but the line height would not change, resulting in the tops of the letters being hidden and the whole exercise of reading the text very unsatisfactory.

I had found two workarounds for the situation. At first I simply reduced the screen resolution of my new laptop to an old-fashioned 2048x1024 (or less) so that I could read the content in my Notes windows. That worked fine. But it irritated me to have spent all that money and to be unable to use one of the nicest features of my new laptop. 

Later I noticed that, if I started my computer in high resolution mode with no external monitor attached, the text and icons in my Notes window would be properly sized and would look okay. But then if I attached an external monitor (my highest resolution one being 2048x1024) and moved the Notes window onto it, it would not resize properly. But then if I rebooted with the external monitor still attached, the Notes window, when reopened on that monitor, would display properly sized text and icons. But then if I moved the Notes window back onto the native monitor, the text in it would again be improperly sized. Or if I simply unplugged the external monitor, forcing the windows on it to move onto the native monitor, same result. But rebooting would again fix the text sizing issue.

In other words, my second workaround was to reboot the laptop (or, as I later discovered would also work, to simply log out of Windows, then log back in) whenever I wanted to connect to and use my external monitor or to disconnect from it. Doing so would fix the text in the Notes window. But it was a hassle to have to shut down and reopen the various programs I typically run every time I needed to connect my laptop to or disconnect it from my external monitor. All in all, a pretty unsatisfactory situation.

All that is fixed now with the happy release of Notes 9.0.1 Feature Pack 9. Now, if I unplug my external monitor, Notes repositions itself to the native monitor and resizes its text and icons exactly as it should. And if I plug the external monitor back in, Notes reverses the process exactly as it should - everything properly formatted and sized, all text easily readable. I am a very happy camper. 

Thursday, April 6, 2017

Thwart spearphishing attacks by using digital signatures in IBM Notes Mail

In case you aren't inclined to read this post through and through, despite the beauty of its prose, here's the bottom line: Spearphishers cannot easily spoof a digitally signed message; so you should enable default digital signing of messages by all of your users to prevent spearphishing attacks.

Recently a friend told me about a security breach at his company. The breach was a textbook spearphishing attack. A member of the accounting department received an email message that purported to be from the company CEO. In the message, the “CEO” directed the recipient to wire umpteen thousands of dollars to a certain bank account. The email arrived late Friday afternoon and urged the recipient to wire the money “before close of business today”. The CEO was, of course, not around, so could not be reached to verify. The recipient bookkeeper did as directed and the company never saw the money again. 

Even more recently (because it's tax season, I suppose) I read that someone has been very successfully using spearphishing messages to trick companies into sending their employees' W-2 forms to the spearphisher. This spearphisher has succeeded so far in collecting some 120,000 W-2 forms from a number of organizations.

It occurred to me that if the victim companies used IBM Notes Mail they could easily have thwarted these spearphishing attacks.

Phishing attacks in general are a type of social engineering attack in which the attacker blasts a deceptive email out to as many recipients as possible, hoping to trick some of the recipients into responding in a way that will enable the attacker to rip off the respondent in some way.  Spearphishing attacks are phishing attacks aimed at a single recipient. The email (or other attack vector) is finely tuned to trick the recipient into trusting the sender and responding positively. The email typically purports to be from a trusted, authoritative executive within the recipient’s organization; the email may direct the recipient to send money to some bank account and do it immediately because time is of the essence, as in my first example. Or, as in my second example, the goal may be to get the recipient to give away confidential information such as, oh, I don’t know, how about: The W-2 forms of every one of the organization’s employees!

The feature of Notes that could have prevented these spearphishing attacks is digital signing of documents. IBM Notes for decades now has included a feature permitting senders to sign outgoing messages digitally. This is not a “written” signature at the bottom of outgoing messages, but rather an encrypted hash of the message that accompanies the message and enables the recipient to verify that 1) the message really came from the purported sender and 2) the content of the message was not altered en route to the recipient. When the recipient of the message opens it, Notes verifies the integrity of the hash and assures the recipient that the message is genuine. In the message header it displays “This message is digitally signed." In the Status Bar (bottom of the window) it briefly displays "Signed by <sender’s fully distinguished name> on <date> <time>, according to <certifier name>”. 

A spearphisher could not easily spoof such a message because to do so he would have to obtain the sender’s private key, which only exists in the sender’s Notes user ID file. The spearphisher would have to obtain a copy of the purported sender’s ID file and learn its password (or, even harder, obtain a certifier ID and learn its password). That is, he would have to compromise both parts of a Notes user’s two-factor security. Not impossible, but not an easy thing for a total and remote stranger to pull off. 

Digital signing of messages has been available in Notes since at least 1993. But it is voluntary by default. Notes users have to check a box to digitally sign any message before sending it. (They can check another box to digitally encrypt the message, too, if they want.) As you might guess, in most organizations hardly anyone ever checks the boxes or has any clue about why they might want to do so. 

What Notes organizations can do (and should do soon, because spearphishers are clearly getting really good at their craft) is enable digital signatures by default, so that messages are digitally signed unless the sender turns off the feature.

First, of course, Notes administrators should notify their users that they will enable this feature.

Before that, though, and most importantly of all, Notes admins should educate their users so that
  • The users become appropriately paranoid about responding to mail that asks them to do potentially problematic things.
  • The users know why digital signing is important.
  • The users know that they should always look for the notice that the message is digitally signed before assuming a message from a purported Notes user is genuine. 
And as I write and think about how serious this spearphishing/ransomware plague is getting, it occurs to me that,  in parallel with educating the users, the Notes admins should get management on board and get HR to pitch in and revise personnel policies to make it clear to everyone in the organization that thou shalt [do certain things] and thou shalt not [do certain other things] with respect to email.
And all of that being said and done, and now that the users know what to expect, what to do, and why it's important, the admins should enable default digital signing of Notes mail. 

Enabling default digital signing of messages is very easy or moderately easy depending on a number of factors. Enabling it is very easy if all of an organization’s mail users use Notes to send and receive mail, and the only mail the organization is concerned about protecting with digital signatures is internal mail. An admin sets a particular field in a policy and applies the policy to the target users. Done.

Enabling default digital signing of messages is moderately easy if users also use Web browsers or non-Notes (POP, IMAP, IMSMO) mail programs or mobile devices to send and receive mail, because you can’t enable default digital signatures in non-Notes clients by policy. Rather, you have to convince users to enable it as a default in their user preferences. (That pretty much means that you really must educate your users in the importance of digital signatures, and not just pay lip service to it.) And you have to make sure that their ID files have either been merged into their mail databases or reside in an ID Vault. (But if your non-Notes mail users' mail resides on IBM cloud-based mail servers, their IDs must reside in the cloud-based ID Vault; merging their ID into their mail database won't be sufficient.)

Finally, if you also want to give users the option to sign messages addressed to recipients whose mail does not reside on Domino servers, the implementation process becomes not so easy at all. I'll be happy to discuss the complications involved in that process if either you or my other reader expresses interest in reading about it. But for now, I'll leave this discussion here:
  • Educate management, HR, and your users about the dangers of spearphishing and how to cope with it.
  • For your Notes mail users, use policies to enable default digital signing of messages to other Notes users.
  • For your non-Notes, Domino-based mail users who have Notes IDs, encourage them strongly to enable the preference to sign outgoing mail by default. And get their Notes IDs merged into their mail databases or an ID Vault.
  • For your non-Notes, Domino-based mail users who do not have Notes IDs, re-register them to generate Notes IDs for them.
  • If you have non-Domino email users (Exchange, O365, Google, whatever), take two aspirin and call me in the morning.