Saturday, June 5, 2010

Issues with Lotus Domino 8.5.1 64bit on Windows 2008 64bit

We experienced significant issues with windows 2008 64 and IBM Lotus Domino 8.5.1, despite the platform being officially certified:
    http://www-01.ibm.com/support/docview.wss?uid=swg21366754

The biggest problems were related to problems in closing the lotus domino server. The servers took ages to shutdown, and the processes were blocked in (apparently) a kernel race condition in the network interface card drivers (we had a significant percent of the cpu time being spent in the system kernel).

Installing FP1 or FP2 did not help.These malfunctions were completely unrelated to DAOS features.

We also observed some errors probably related to the java subsystem used by the domino server.
A typical symptom was that the domino server console was not closing after the "Server Shutdown Completed" message, and additional logs were still printed, without the server going down. The server restart time was completely unreliable despite the setting in the transaction log configuration to "favor restart time".

The problems completely disapperared after complete reinstallation on windows 2003 64, and currently we are satisfied of the performance and stability of our Rome Domino 64 bit Installations. To our knowledge there are currently no particular domino related benefits in using windows 2008 64 bit envronments instead of Windows 2003/64.

We did not perform any specific registry tweaking on the current domino production servers. We are currently running Domino Release 8.5.1FP2 HF55 on Windows/2003/64 5.2 on most customer installations.

Further analysis that we performed on another customer, later, pointed out a possible cause of this problem related to the windows 2008 drivers of the broadcom network cards.
    http://seasideit.blogspot.com/2009/06/server-2008-receive-side-scaling-and.html
    http://technet.microsoft.com/en-us/network/dd277646.aspx

On a small customer with a windows 2008 64 bit domino installation we performed the aforementioned changes in the broadcom nic parameters, and since then we had the domino server running flawlessly.
Both installations are using servers with on-board broadcom network interface cards.

We are not currently recommending lotus domino 64 bit installations on windows 2008 64.
Broadcom gigabit devices are currently used by many server producers (HP, DELL, SIEMENS just to name a few).
The reliability of these installations will probably change in the near future, with new fixes being delivered by both IBM and Microsoft.


We can not be 100% sure that the Broadcom issue was the exact (or only) cause of the troubles we had, but the evidences are quite strong.

Another potential issue to be aware of is the use of Windows NTFS compression which seem to generate console errors and attachment access problems from webmail if used in conjuction with DAOS. This is not a commonly adopted production solution, but we observed the problem during a migration transition.




Marco (@mgua on twitter)




.

2 comments:

Matt Bidwell said...

Have you experienced any benefits from using 64 bit Domino in general? I've read the IBM information and it does not sound like there are any solid benefits. I would like to hear your opinion.

Marco Guardigli said...

Matt,

Definitely switching to 64 bits environment helped me a lot in most of my biggest domino installations.

The issue I mentioned in my post was impacting on that domino version with Windows 2008 64 bits. That customer is working fine with windows 2003 64 bits and with about 8Tb of main domino storage, in two two-node clusters.

I found that the primary benefits of 64 bits are in the memory management areas, and in the overall performances (mostly of router, httpd, ldap, indexer, amgr).

In the 32 bits environments, non linear memory addressing techniques are needed for the operating systems to "see" over the 2^32 bit boundary.

Switching to 64 bits architectures on current server hardware is definitely a needed infrastructural improvement for most enterprise software built for supporting 64 bit systems.

My suggestion is to be careful in mixing 64 and 32 bits software in any integration, so to stay away from untested environments. Switch to 64 bits if you can migrate all your modules to 64 bits.

Before production migration, carefully examine and test all your components, like backup agents, antivirus components, and the like.


At the moment, I did not perform testing with virtualized 64 bit domino systems nor I run 64bit-vm/64bit-host solutions in any enterprise customers.


regards,
marco