Saturday, June 5, 2010

Issues with Lotus Domino 8.5.1 64bit on Windows 2008 64bit

We experienced significant issues with windows 2008 64 and IBM Lotus Domino 8.5.1, despite the platform being officially certified:
    http://www-01.ibm.com/support/docview.wss?uid=swg21366754

The biggest problems were related to problems in closing the lotus domino server. The servers took ages to shutdown, and the processes were blocked in (apparently) a kernel race condition in the network interface card drivers (we had a significant percent of the cpu time being spent in the system kernel).

Installing FP1 or FP2 did not help.These malfunctions were completely unrelated to DAOS features.

We also observed some errors probably related to the java subsystem used by the domino server.
A typical symptom was that the domino server console was not closing after the "Server Shutdown Completed" message, and additional logs were still printed, without the server going down. The server restart time was completely unreliable despite the setting in the transaction log configuration to "favor restart time".

The problems completely disapperared after complete reinstallation on windows 2003 64, and currently we are satisfied of the performance and stability of our Rome Domino 64 bit Installations. To our knowledge there are currently no particular domino related benefits in using windows 2008 64 bit envronments instead of Windows 2003/64.

We did not perform any specific registry tweaking on the current domino production servers. We are currently running Domino Release 8.5.1FP2 HF55 on Windows/2003/64 5.2 on most customer installations.

Further analysis that we performed on another customer, later, pointed out a possible cause of this problem related to the windows 2008 drivers of the broadcom network cards.
    http://seasideit.blogspot.com/2009/06/server-2008-receive-side-scaling-and.html
    http://technet.microsoft.com/en-us/network/dd277646.aspx

On a small customer with a windows 2008 64 bit domino installation we performed the aforementioned changes in the broadcom nic parameters, and since then we had the domino server running flawlessly.
Both installations are using servers with on-board broadcom network interface cards.

We are not currently recommending lotus domino 64 bit installations on windows 2008 64.
Broadcom gigabit devices are currently used by many server producers (HP, DELL, SIEMENS just to name a few).
The reliability of these installations will probably change in the near future, with new fixes being delivered by both IBM and Microsoft.


We can not be 100% sure that the Broadcom issue was the exact (or only) cause of the troubles we had, but the evidences are quite strong.

Another potential issue to be aware of is the use of Windows NTFS compression which seem to generate console errors and attachment access problems from webmail if used in conjuction with DAOS. This is not a commonly adopted production solution, but we observed the problem during a migration transition.




Marco (@mgua on twitter)




.