[LordZordec]

Hello all!

We recently implemented software that automatically replicates data from our local Exchange 2003 front-end and back-end servers to a pair of Exchange servers at a hosted facility about 50 miles away. In the event of a failure of our Exchange servers or a hole in the ground opens and swallows our building, the special failover software will automatically change DNS entries to point email clients to the new servers and then start Exchange services on them.

A few weeks ago, we did a failover test, and as far as the Exchange portion went, the test was smooth as silk. We shut down our local Exchange servers, and in a matter of ten minutes, the failover servers were up and running, and all was well...except the BES.

We could access email via the OWA portal and Microsoft Outlook. However, the BES REFUSED to talk to the failover servers and continue letting email flow.

The local Exchange servers and the failover servers have different names, but there was a DNS forwarding rule put in place that forwards all traffic sent to the original servers over to the failover servers.

So, email clients had no issues with this. The BES refused to work with this arrangement.

The following errors were in the event log:

The Kerberos client received a KRB_AP_ERR_MODIFIED error from the server host/XXXX-dc03.XXXX.org. The target name used was HOST/DBSI-MAIL. This indicates that the target server failed to decrypt the ticket provided by the client. This can occur when the target server principal name (SPN) is registered on an account other than the account the target service is using. Please ensure that the target SPN is registered on, and only registered on, the account used by the server. This error can also happen when the target service is using a different password for the target service account than what the Kerberos Key Distribution Center (KDC) has for the target service account. Please ensure that the service on the server and the KDC are both updated to use the current password. If the server name is not fully qualified, and the target domain (XXXX.ORG) is different from the client domain (XXXX.ORG), check if there are identically named server accounts in these two domains, or use the fully-qualified name to identify the server.

Also, for each and every Blackberry user, there was one of these:

User Doe, John not started

{jdoe@XXXX.org} MAPIMailbox::MAPIMailbox - OpenMsgStore (0x8004011d) failed, MailboxDN=/o=XXXX/ou=First Administrative Group/cn=Recipients/cn=ppicarie, ServerDN=/o=XXXX/ou=First Administrative Group/cn=Configuration/cn=Servers/cn=XXXX-MAIL/cn=Microsoft Private MDB

We have verified that this is not a permissions issue for the Blackberry administrator account. It does have full rights to the Exchange server and the user mailboxes.

The BES refused to talk to the failover Exchange servers for two hours, and then suddenly started working. However, when we attempted to failover the BES to its mirrored companion at the hosting facility, it wouldnt work at all. We suspect that, since we are getting the same errors on both, its the same issue causing both.

So, what could it be? Any help would be appriciated.

[MisterGriffiths]

Two hours! Isn't that the default Information Store Caching period in Exchange 2003?

I'm not sure I totally understand whether the actual Exchange server name is changing or not.

Also, what version of BES?

[mahoward]

Is this XOSoft? If so I am very interested in this issue as we are going to be using it for our failover solution as well. Please post any information you discover, thanks!

[Edit]: I just remembered that the XOSoft product also modifies the Service Principal Name (SPN) for the destination Exchange server to allow Kerberos to keep working. Perhaps the product you have is not doing that?

[LordZordec]

Thanks for the replies...

This is Exchange 2003, BES 5.0, and XOSOFT.

In the failover scenario, the name of the Exchange server DOES change.

Primary Exchange Server=XXXX-MAIL
Failover Exchange Server=DBSI-MAIL

[freakinvibe]

The only Exchange fail-over solution that worked for us with BES was Exchange clustering as it is transparent to the BES. All other solutions we tested didn't really work with BES.

[LordZordec]

Originally Posted by freakinvibe

The only Exchange fail-over solution that worked for us with BES was Exchange clustering as it is transparent to the BES. All other solutions we tested didn't really work with BES.

What kinds of issues did you have? Were they similar to the issues described above?

[freakinvibe]

We had connection issues after failover. The implementation of the MAPI/CDO access from BES to Exchange is not very good with sudden name and IP address changes. It is probably a DNS caching issue, but we didn't invest time to find the root cause.

[LordZordec]

Originally Posted by freakinvibe

We had connection issues after failover. The implementation of the MAPI/CDO access from BES to Exchange is not very good with sudden name and IP address changes. It is probably a DNS caching issue, but we didn't invest time to find the root cause.

Well, our plan is to fail over the email servers and then run the handheldcleanup.exe -u command to hopefully redirect the BES to the right server for mailboxes.

The key word there is "hopefully".

[LordZordec]

Here is the solution that we are going to attempt tonight:
1. Fail over the Exchange servers.
2. Force AD replication.
3. Run the handheldcleanup.exe -u tool.

We will let you know what happens.

[Harry Azol]

Just curious - did you look at the Exchange 2007 geographically disperse CCR cluster option instead of paying for a 3rd party product on 2003?

[LordZordec]

Originally Posted by Harry Azol

Just curious - did you look at the Exchange 2007 geographically disperse CCR cluster option instead of paying for a 3rd party product on 2003?

Yep.

The higher-ups wanted email high availability NOW, but they were not willing to shell out the cash for the Exchange 2007 upgrade until NEXT year. So, we are shelling out cash for XOSOFT now to give them what they want now even though we could give it to them a few months from now WITHOUT spending a begrillion dollars on a failover product that will not have any other use for one Exchange 2007 is in place.

Do they teach "common sense" anywhere in business school???

[LordZordec]

Originally Posted by LordZordec

Here is the solution that we are going to attempt tonight:
1. Fail over the Exchange servers.
2. Force AD replication.
3. Run the handheldcleanup.exe -u tool.

We will let you know what happens.

The outcome: The BES started to work after 1 hour instead of 2, and when we failed over to the other BES at our remote site, it worked like a charm (vs not working at all - big improvement).

The process isnt perfect yet, but the two key things that made it work were 1) forcing AD replication, and 2) running the handheldcleanup.exe tool.

[mahoward]

Thanks for the updated information, m'lord! We have the EXACT same higher up reasoning as you do except we are waiting to move to 2010 and skipping 2007. Glad to hear it is working for you with the new process.

[mahoward]

BTW, do you have a separate AD site where your failover target Exchange server exists? We are going with a single AD site stretched across the primary and DR physical locations which would reduce the AD replication latency delay.

[LordZordec]

Originally Posted by mahoward

BTW, do you have a separate AD site where your failover target Exchange server exists? We are going with a single AD site stretched across the primary and DR physical locations which would reduce the AD replication latency delay.

We actually have a separate AD site for our remote failover facility, which I am sure is a factor in this as well.

Hmmmmm...now the wheels in my head are spinnning though...