Message delays on BES 5.0.2
 

I have about 35 users on a VMWare server and everything has been going great, but now email is delayed for several minutes when compared to Outlook. It used to show up before it would show up in Outlook. I do not have any errors in my event logs but i do have a few weird warnings and also noticed a couple of things in the logs, but they don't reference a certain user id. My ping time to the exchange server is <1ms. I have assigned myself a static mailbox agent and now i am getting my emails instantly like before. Here is what I have found:


Event Type: Warning
Event Source: BlackBerry Messaging Agent stw-bb-01 Agent 1
Event Category: None
Event ID: 20000
Date: 8/4/2010
Time: 3:59:37 PM
User: N/A
Computer: STW-BB-01
Description:
RIM_HrResolveProxysFromDN - ResolveProxyAddressesMAPI failed from -2146762751

Event Type: Warning
Event Source: BlackBerry Policy Service
Event Category: None
Event ID: 20000
Date: 8/4/2010
Time: 3:59:27 PM
User: N/A
Computer: STW-BB-01
Description:
SCS::PerformITAdminQueueHealthCheck - HealthCheck found issues

Event Type: Warning
Event Source: BlackBerry Messaging Agent stw-bb-01 Agent 1
Event Category: None
Event ID: 20000
Date: 8/4/2010
Time: 3:26:18 PM
User: N/A
Computer: STW-BB-01
Description:
ExtUDPNtfThread::NotifyDatabase: COM Error 0x80040E57 - IDispatch error #3159 - Source: "Microsoft OLE DB Provider for SQL Server" - Description "Microsoft OLE DB Provider for SQL Server" - Command "ExtUDPNtfThread::NotifyDatabase"

I also noticed this in my MAGT logs
[45063] (08/04 14:27:15.172):{0x16F4} {ConnectionPool::GetConnection()}: Wait time of 328ms exceeded 100ms. 2 thread(s) are still waiting. Pool may need more connections to reduce wait time.
[45063] (08/04 14:27:15.172):{0x16D4} {ConnectionPool::GetConnection()}: Wait time of 328ms exceeded 100ms. 1 thread(s) are still waiting. Pool may need more connections to reduce wait time.

and these in my POLC logs
[40000] (08/04 09:41:03.929):{0x14F0} {}RequestHandler::DoWork - Processing ITADMINQUEUE_HEALTHCHECK request
[40000] (08/04 09:41:03.929):{0x1500} {}RequestHandler::DoWork - Processing QUEUE_KEY_REGEN_EVENTS request
[40000] (08/04 09:41:03.929):{0x14F0} {email@domain.com, PIN=xxxxxxxx, UserId=23}SCS::CheckForFailedAppDeliveryRequests - Command SEND_APC_APP, Status 6, StatusMessage "45162".
[20000] (08/04 09:41:03.944):{0x14F0} SCS::PerformITAdminQueueHealthCheck - HealthCheck found issues
[40000] (08/04 09:41:03.944):{0x1500} {email@domain.com, PIN=xxxxxxxx, UserId=89}SCS::CheckDBandQueueKeyRegenRequests - Queuing REQUEST_KEY_GENERATION request
[40000] (08/04 09:41:03.960):{0x14F8} {}RequestHandler::DoWork - Processing QUEUE_STALE_ITPOLICIES request
[40000] (08/04 09:41:03.960):{0x14F4} {}RequestHandler::DoWork - Processing CHECK_FOR_UNAPPLIED_ITPOLICIES request
[40000] (08/04 09:41:03.960):{0x14F4} {}RequestHandler::DoWork - Completed CHECK_FOR_UNAPPLIED_ITPOLICIES request
[20000] (08/04 09:41:04.132):{0x14F0} SCS::RequeueInProgressCommands - 3 commands were requeued
[40000] (08/04 09:41:04.132):{0x14F0} {}RequestHandler::DoWork - Completed ITADMINQUEUE_HEALTHCHECK request


Any help would be greatly appreciated.

I have the exact same issue
 

Anybody has an idea what is the solution?
i have been struggling with this for almost a week
there is no latency between the servers, i see a lot of "mail queued through rescan" in my MAGT log

Any resolution to this?
 

I have a few of these as well..

RIM_HrResolveProxysFromDN - ResolveProxyAddressesMAPI failed from

on new 5.0.2 system. Still on SQL 2000. Have not yet moved to SQL 2008. I wonder if that is any issue.

Alice

does it go away if you restart the dispatcher and controller? if so how long does it work normally for?

what version of mapi and cdo do you ahve installed? what version of exchange?

Re: Message delays on BES 5.0.2
 

We had a case on this recently (some users saw delayed messages, some were unaffected). We have Exch 2010 SP1, and ever since that patch was applied, things haven't worked correctly.

Well, it turns out, Blackberry is aware that there is an issue and they are "actively working with Microsoft on a fix", but no ETA.

They're clearly not publishing this as an active issue, but t.Support admitted that it was known (it only took a 5 hour call this time around to find this out).

The workaround was as indicated, move the affected users to a static message agent for now.

Re: Message delays on BES 5.0.2
 

Matt, we have the same issue and but it only took 3.5 hours on the phone for them to id this as a Microsoft fix in progress...
I just have to tell users to be patient.

Re: Message delays on BES 5.0.2
 

I just got off the phone with T Support. They said that assigning a user account to a static messaging agent would work but it's not a great solution since it would probably cause more issues if you had a bunch of people doing that.

It was recommended to use the newest version of the MAPI/CDO client on our BES. You do not need to upgrade the client on the Exchange 2010 box. Here is the link.

Download details: Microsoft Exchange Server MAPI Client and Collaboration Data Objects 1.2.1

This should get your MAPI/CDO client up to 6.5.8190.

This resolved the issue for me.

Re: Message delays on BES 5.0.2
 

Nice.. Thanks for posting your solution. I'll have to take a look. We seem to see some of the same errors.

Re: Message delays on BES 5.0.2
 

That's great that this worked for you, wish it would work for us. We're using that CDO version and still seeing the issue. The static agent method isn't flawless either certainly, thankfully we only have about 65 BES users at the moment (all of whom are probably ready to get ActiveSync devices at this point).

Re: Message delays on BES 5.0.2
 

I've been having this issue as well, but we haven't moved to SP1 on Exchange 2010 ye, still running build number 14.0.639.21, which the internet tells me is RTM.

Not seeing any errors jump out at me in the log file, it's just......delayed.

Re: Message delays on BES 5.0.2
 

For the record, updating the MAPI/CDO on the BES seems to have solved the problem, at least on my device. I'll report back if there are any further issues after talking with some of the users.

Re: Message delays on BES 5.0.2
 

scratch that, worked for a bit, now back to the way it was. Getting delays sending and receiving from the device.

I tried to send a message from the device to an outside email account, on the device the sent item has the checkmark confirming it went through. on the BES server the last line item relating to my address is:

The message was *finally* actually delievered to my gmail box 8 minutes after it was sent from the device, and the line item finally appears in the MAGT log file

7 1/2 minutes between any reference to my account in the log files for sending the message.

Any thoughts?

Re: Message delays on BES 5.0.2
 

How much RAM do you have on your MBX server?

Re: Message delays on BES 5.0.2
 

the exchange server has 4.5GB of ram, which is currently at 50% utilization, and the BES has 3GB, again 50% utilization.

I also just got a message on my Blackberry that arrived in my outlook inbox 26 minutes earlier.

Re: Message delays on BES 5.0.2
 

did you ever find a resolution to this issue? I believe we are having an identical issue.

Re: Message delays on BES 5.0.2
 

This problem does not go away!!!!
Does anyone has information about it?

Re: Message delays on BES 5.0.2
 

We've been having the exact issue. We recently transitioned from Exchange 2003 to Exchange 2010. We've been running in co-existence and had our BES server delivering messages for boxes on both servers. When we had 30-40 mailboxes on 2010 we did not experience any issues with Blackberry. However, once we moved the majority of our boxes we ended up having significant delays in sending messages, receiving messages, and performing lookups.

Our environment:
BES
ESX 4.0
1vcpu
4GB Memory
170 users
CDO 1.21 installed
5.02 SP2 MR2 (latest build)

Exchange
Single server
Dual 6-core 2.66Ghz
12GB memory
Windows 2008 R2 64 bit
Exchange 2010 SP1 w/ Update Rollup 2

Over the past three days my team has probably spent 10-15 hours on the phone with RIM support trying a myriad of things as well as checking all settings and intergration pieces and there has been no change.

Things RIM had us try/check:
1. Throttling Policy
2. BESAdmin roles/permissions
3. BES version
4. CDO/MAPI version on Exchange & BES
5. Recreate service account for BES, reset permissions, create new throttling policy, and set new service account
6. Change rescan interval from 15 to 5
7. Decrease the max number of mailboxes per agent. (default is 150, we have about 170 BES users.) We decreated it to 60, so that the users would be spread over 3 dynamic agents

The only work around that we've tried that we've found works (and works well) is to move users to static mailbox agent. RIM initially was hesitant to have us do this, but on the 3rd day of support calls fully endorsed the idea. In speaking with the technicians they've indicated there should be no difference in the static versus dynamic agent except for the number of users on each; that said, we've had no luck with dynamic agents, and the static agent users work flawlessly. They origionally said to limit it to 5 static agents, but have now said 10 static agents would be fine. We have 10 users per agent and those users on the static agents are running great. It's not solution nor a pleasant work around, but it removes some of the urgency from the matter.

RIM points the finger at Microsoft for the issue, focusing on an RPC access issue. I'm not convinced this is the issue as Outlook clients connect over RPC and we've not had any issues with them. Additionally, restarting the RPC service on Exchange does not temporarily alleviate the problem.

We're currently running performance monitors on Exchange and BES to give to RIM for analysis. We've also ordered 16GB more memory for our Exchange server, as we've seen some people indicate memory alleviates (but does not resolve) the issue.

Re: Message delays on BES 5.0.2
 

I'm having a similar issue, though I only have about 90 blackberries currently. Was RIM able to provide any further detail based on your performance monitoring?

Adding additional memory seems to have helped for now, but I don't know how long it will take before these delays start to creep into the environment again.

Also, I've been having a hard time finding info on decreasing the max number of mailboxes per agent in BES 5. I've found info for BES 4, but I don't know if it applies to BES 5, and I'm hesitant to try it. Can you let me know where the setting is or what registry value to add/change?

Thanks

Re: Message delays on BES 5.0.2
 

Hey all...
Us too, as well.

And our Exchange 2010 environment is net new...
Our BES 5.0.2 enviroment, also, net new
Our MAPI/CDO version are latest
We also jacked our processor and RAM for years of growth..
That said.. I have seen delays, anywhere from 1 to 8 minutes.

Re: Message delays on BES 5.0.2
 

I tried the Static mailbox agent on a few devices yesterday, and implemented it on all of our users (about 45) today, and mail delivery seems to be on time now. I'll have to see over the course of a few days if it holds up, but I know as soon as I put myself into a static agent mail delivery times went from 15-25 minutes(basically whenever the BES did a rescan), to instant. Every other user I've talked to who I made the change to reported similar success.

Not really a feasible solution if you have 400 users on your BES, but for smaller shops it may do the trick.