BES Express 5.0.4 - Messaging Agent failing each night (5305)
 

Since patching my E2010 boxes with Windows Updates this week, the last 4 evenings I've seen the BlackBerryAgent.exe miss heartbeats then fail to start with error code 5305. The problem starts after midnight, and once it starts failing, I can't get it to start properly / behave unless I restart all BES services.

This link discusses that error, but all of those solutions imply that it never s tarts properly. Mine just starts failing during the evenings. I've just been getting up early to restart BES servicse, at which point it's fine the rest of the day.

Has anyone encountered this?

MAGT log snippet:

[30480] (02/01 03:45:09.354):{0x1280} [CFG] BES failover mode is ACTIVE
[41120] (02/01 03:45:09.354):{0x1280} [CFG] Address Lookup is enabled
[41122] (02/01 03:45:09.354):{0x1280} [CFG] LDAP search is enabled, LDAP PIM search is disabled, LDAP ALP search is disabled
[35035] (02/01 03:45:09.573):{0x1280} LDAP: Create 0x00a40a44
[30460] (02/01 03:45:09.573):{0x1280} [CFG] LDAP Info: Host Name = DCServerName.domain.local
[41123] (02/01 03:45:09.573):{0x1280} [CFG] Using up to 10 external CDO Helpers in keep-alive mode
[41124] (02/01 03:45:09.573):{0x1280} [CFG] AddOwnerApptID is disabled
[41125] (02/01 03:45:09.573):{0x1280} [CFG] SetLocaleIDs is disabled
[30302] (02/01 03:45:09.573):{0x1280} [CFG] OTAFM hard-delete server support is disabled, rescans are 0x00
[40532] (02/01 03:45:09.573):{0x1280} [CFG] vCard conversion support has been enabled
[40892] (02/01 03:45:09.573):{0x1280} [CFG] GlobalRescan config: messaging:O, calendar:O, pim:O
[41126] (02/01 03:45:09.573):{0x1280} [CFG] EnableExceptionStackTrace is disabled
[40000] (02/01 03:45:09.573):{0x1280} [CFG] MAPI Encoding is 1
[10277] (02/01 03:45:09.573):{0x1280} BlackBerry Messaging Agent BESServerName Agent 1 failed to start. Error code 5305
[50106] (02/01 03:45:09.573):{0x1280} Stopping BlackBerry Mailbox Agent 1 for Server BESServerName
[40507] (02/01 03:45:09.589):{0x1A88} ExtUDPLogThread: Code on closing = 0
[30000] (02/01 03:45:09.589):{0x1A88} [DIAG] EVENT=Unregister_thread, THREADID=0x1A88, THREADNAME="ExtUDPLogThread"
[35035] (02/01 03:45:09.620):{0x1280} LDAP: Delete 0x00a40a44
[30065] (02/01 03:45:09.651):{0x1280} BlackBerry Mailbox Agent 1 for Server BESServerName shutdown complete
====== END OF LOG ======

Re: BES Express 5.0.4 - Messaging Agent failing each night (5305)
 

are you stopping BES during the updates?

Is BES installed on mail server or on dedicated box

proper boot sequence
DC, GC, SQL and Exchange, BES

Re: BES Express 5.0.4 - Messaging Agent failing each night (5305)
 

I am not. To be honest I used to, but the last couple of years BES recovered without issue so I stopped since it was a massive headache. Still, with multiple days of this, and the intermittent (yet scheduled) nature, I'm naturally suspect that a simple DC reboot would fix this. I'll give it a go either way, but I suspect something else is at play.

Re: BES Express 5.0.4 - Messaging Agent failing each night (5305)
 

I'm also finding that handheldcleanup.exe is not running. Despite typing in the exact BESName, I get:

Open() failed for BESName
OpenDB failed to open Database

I have run this before without issue, and wonder if it is related.

Re: BES Express 5.0.4 - Messaging Agent failing each night (5305)
 

could be related, check with SSMS to see if you can access it

Re: BES Express 5.0.4 - Messaging Agent failing each night (5305)
 

The credentials check out, and I can telnet to the SQL server from the BES on the SQL port. Plus profiler shows the BES doing its usual work.

I'm going to investigate NSPI tonight, since we are mixed between 2003 and 2008R2 DCs, though why that issue would crop up now is beyond me given our user count has been pretty static / low at ~100 users. Our throttling policies are all kosher.

Re: BES Express 5.0.4 - Messaging Agent failing each night (5305)
 

NSPI limit fix here

http://www.blackberry.com/btsc/KB17325

Re: BES Express 5.0.4 - Messaging Agent failing each night (5305)
 

No luck with the NSPI limit fix. Also re-set the throttling policies and re-created the MAPI profile.

Going to re-set permissions and possibly do a repair of BES files. Curioser and curioser what's causing this morning MAPI logon failures. The times are inconsistent outside of the 12AM-7AM consistency.

I wonder if upgrading from MAPI/CDO 1.2.1 (6.5.8211.0) to (6.5.8244.0) would be worthwhile. They're both 2011 releases.

Thanks for all your help.

Re: BES Express 5.0.4 - Messaging Agent failing each night (5305)
 

I would use latest version or CDO if possible

Re: BES Express 5.0.4 - Messaging Agent failing each night (5305)
 

I've made a little progress - I have not yet updated MAPI/CDO, but did do a network capture. At the time we got this error in the ALRT log:

[30000] (02/10 02:14:13.728):{0x20A4} [Alarm::ActivateAlarm] Queuing alarm: <N/A> | BlackBerry Messaging Agent BESServer Agent 1 (Application Event Log on BESServer.domain.local) | 02/10/2013 02:14:13 (EFFF2825) -> BlackBerry Messaging Agent BESServer Agent 1 failed to start. Error code 5305

The network capture recorded:

1064653 2:14:13 AM 2/10/2013 8963.3886465 BlackBerryAgent.exe CAS.Domain.Com BESServer MSRPC MSRPC:c/o Fault: Call=0xE Context=0x0 Status=0x1C010014 Cancels=0x0 {MSRPC:11522, TCP:11520, IPv4:7}

That status (0x1C010014) indicates the server (one of the CASes) is too busy to fulfill the request.

Combine this with the IOS 6.1 issues with Exchange we're seeing, and I wonder if they're not related, since that update was released a day before we started seeing issues.

Anyway, we'll see what I find out next. Just need to collect some perf logs from the CASes.

Re: BES Express 5.0.4 - Messaging Agent failing each night (5305)
 

IOS issues have killed my IIS a few times in past weeks from the nefarious syncs.

BYOD Nightmare: iOS Devices Hobble Exchange Servers When They Synch - CIO.com

iOS 6.1 banned from corporate servers due to Exchange snafu | ZDNet

Re: BES Express 5.0.4 - Messaging Agent failing each night (5305)
 

Regrettably, throttling our IOS 6.1 users did not resolve the nightly issues.

MAPI / CDO upgrade is on the table for the weekend, but I wanted to get your thoughts on CAS Arrays. Both our MAPI profile and EWSCASURL/EWSSCPURL point to an NLB'ed address. Would you recommend pointing at a single CAS for those, or is the method we're using supported? We'd had no issues up until this month with pointing directly to the load-balanced address, but finding solid documentation is somewhat tricky.

Oooh, another possibility: http://support.microsoft.com/kb/2810616.

Endless!

Re: BES Express 5.0.4 - Messaging Agent failing each night (5305)
 

After using our hosts file to point the BES at a single CAS instead of the array, we no longer see the nightly 5305s. It appears one of the CASes (SCOM hints at which) is the culprit! Why it gets busier at night is beyond me, but that's an issue for a different forum.