Message delays on BES 5.0.2
 

I have about 35 users on a VMWare server and everything has been going great, but now email is delayed for several minutes when compared to Outlook. It used to show up before it would show up in Outlook. I do not have any errors in my event logs but i do have a few weird warnings and also noticed a couple of things in the logs, but they don't reference a certain user id. My ping time to the exchange server is <1ms. I have assigned myself a static mailbox agent and now i am getting my emails instantly like before. Here is what I have found:


Event Type: Warning
Event Source: BlackBerry Messaging Agent stw-bb-01 Agent 1
Event Category: None
Event ID: 20000
Date: 8/4/2010
Time: 3:59:37 PM
User: N/A
Computer: STW-BB-01
Description:
RIM_HrResolveProxysFromDN - ResolveProxyAddressesMAPI failed from -2146762751

Event Type: Warning
Event Source: BlackBerry Policy Service
Event Category: None
Event ID: 20000
Date: 8/4/2010
Time: 3:59:27 PM
User: N/A
Computer: STW-BB-01
Description:
SCS::PerformITAdminQueueHealthCheck - HealthCheck found issues

Event Type: Warning
Event Source: BlackBerry Messaging Agent stw-bb-01 Agent 1
Event Category: None
Event ID: 20000
Date: 8/4/2010
Time: 3:26:18 PM
User: N/A
Computer: STW-BB-01
Description:
ExtUDPNtfThread::NotifyDatabase: COM Error 0x80040E57 - IDispatch error #3159 - Source: "Microsoft OLE DB Provider for SQL Server" - Description "Microsoft OLE DB Provider for SQL Server" - Command "ExtUDPNtfThread::NotifyDatabase"

I also noticed this in my MAGT logs
[45063] (08/04 14:27:15.172):{0x16F4} {ConnectionPool::GetConnection()}: Wait time of 328ms exceeded 100ms. 2 thread(s) are still waiting. Pool may need more connections to reduce wait time.
[45063] (08/04 14:27:15.172):{0x16D4} {ConnectionPool::GetConnection()}: Wait time of 328ms exceeded 100ms. 1 thread(s) are still waiting. Pool may need more connections to reduce wait time.

and these in my POLC logs
[40000] (08/04 09:41:03.929):{0x14F0} {}RequestHandler::DoWork - Processing ITADMINQUEUE_HEALTHCHECK request
[40000] (08/04 09:41:03.929):{0x1500} {}RequestHandler::DoWork - Processing QUEUE_KEY_REGEN_EVENTS request
[40000] (08/04 09:41:03.929):{0x14F0} {email@domain.com, PIN=xxxxxxxx, UserId=23}SCS::CheckForFailedAppDeliveryRequests - Command SEND_APC_APP, Status 6, StatusMessage "45162".
[20000] (08/04 09:41:03.944):{0x14F0} SCS::PerformITAdminQueueHealthCheck - HealthCheck found issues
[40000] (08/04 09:41:03.944):{0x1500} {email@domain.com, PIN=xxxxxxxx, UserId=89}SCS::CheckDBandQueueKeyRegenRequests - Queuing REQUEST_KEY_GENERATION request
[40000] (08/04 09:41:03.960):{0x14F8} {}RequestHandler::DoWork - Processing QUEUE_STALE_ITPOLICIES request
[40000] (08/04 09:41:03.960):{0x14F4} {}RequestHandler::DoWork - Processing CHECK_FOR_UNAPPLIED_ITPOLICIES request
[40000] (08/04 09:41:03.960):{0x14F4} {}RequestHandler::DoWork - Completed CHECK_FOR_UNAPPLIED_ITPOLICIES request
[20000] (08/04 09:41:04.132):{0x14F0} SCS::RequeueInProgressCommands - 3 commands were requeued
[40000] (08/04 09:41:04.132):{0x14F0} {}RequestHandler::DoWork - Completed ITADMINQUEUE_HEALTHCHECK request


Any help would be greatly appreciated.

I have the exact same issue
 

Anybody has an idea what is the solution?
i have been struggling with this for almost a week
there is no latency between the servers, i see a lot of "mail queued through rescan" in my MAGT log

Any resolution to this?
 

I have a few of these as well..

RIM_HrResolveProxysFromDN - ResolveProxyAddressesMAPI failed from

on new 5.0.2 system. Still on SQL 2000. Have not yet moved to SQL 2008. I wonder if that is any issue.

Alice

does it go away if you restart the dispatcher and controller? if so how long does it work normally for?

what version of mapi and cdo do you ahve installed? what version of exchange?

Re: Message delays on BES 5.0.2
 

We had a case on this recently (some users saw delayed messages, some were unaffected). We have Exch 2010 SP1, and ever since that patch was applied, things haven't worked correctly.

Well, it turns out, Blackberry is aware that there is an issue and they are "actively working with Microsoft on a fix", but no ETA.

They're clearly not publishing this as an active issue, but t.Support admitted that it was known (it only took a 5 hour call this time around to find this out).

The workaround was as indicated, move the affected users to a static message agent for now.

Re: Message delays on BES 5.0.2
 

Matt, we have the same issue and but it only took 3.5 hours on the phone for them to id this as a Microsoft fix in progress...
I just have to tell users to be patient.

Re: Message delays on BES 5.0.2
 

I just got off the phone with T Support. They said that assigning a user account to a static messaging agent would work but it's not a great solution since it would probably cause more issues if you had a bunch of people doing that.

It was recommended to use the newest version of the MAPI/CDO client on our BES. You do not need to upgrade the client on the Exchange 2010 box. Here is the link.

Download details: Microsoft Exchange Server MAPI Client and Collaboration Data Objects 1.2.1

This should get your MAPI/CDO client up to 6.5.8190.

This resolved the issue for me.

Re: Message delays on BES 5.0.2
 

Nice.. Thanks for posting your solution. I'll have to take a look. We seem to see some of the same errors.

Re: Message delays on BES 5.0.2
 

That's great that this worked for you, wish it would work for us. We're using that CDO version and still seeing the issue. The static agent method isn't flawless either certainly, thankfully we only have about 65 BES users at the moment (all of whom are probably ready to get ActiveSync devices at this point).

Re: Message delays on BES 5.0.2
 

I've been having this issue as well, but we haven't moved to SP1 on Exchange 2010 ye, still running build number 14.0.639.21, which the internet tells me is RTM.

Not seeing any errors jump out at me in the log file, it's just......delayed.

Re: Message delays on BES 5.0.2
 

For the record, updating the MAPI/CDO on the BES seems to have solved the problem, at least on my device. I'll report back if there are any further issues after talking with some of the users.

Re: Message delays on BES 5.0.2
 

scratch that, worked for a bit, now back to the way it was. Getting delays sending and receiving from the device.

I tried to send a message from the device to an outside email account, on the device the sent item has the checkmark confirming it went through. on the BES server the last line item relating to my address is:

The message was *finally* actually delievered to my gmail box 8 minutes after it was sent from the device, and the line item finally appears in the MAGT log file

7 1/2 minutes between any reference to my account in the log files for sending the message.

Any thoughts?

Re: Message delays on BES 5.0.2
 

How much RAM do you have on your MBX server?

Re: Message delays on BES 5.0.2
 

the exchange server has 4.5GB of ram, which is currently at 50% utilization, and the BES has 3GB, again 50% utilization.

I also just got a message on my Blackberry that arrived in my outlook inbox 26 minutes earlier.

Re: Message delays on BES 5.0.2
 

did you ever find a resolution to this issue? I believe we are having an identical issue.

Re: Message delays on BES 5.0.2
 

This problem does not go away!!!!
Does anyone has information about it?

Re: Message delays on BES 5.0.2
 

We've been having the exact issue. We recently transitioned from Exchange 2003 to Exchange 2010. We've been running in co-existence and had our BES server delivering messages for boxes on both servers. When we had 30-40 mailboxes on 2010 we did not experience any issues with Blackberry. However, once we moved the majority of our boxes we ended up having significant delays in sending messages, receiving messages, and performing lookups.

Our environment:
BES
ESX 4.0
1vcpu
4GB Memory
170 users
CDO 1.21 installed
5.02 SP2 MR2 (latest build)

Exchange
Single server
Dual 6-core 2.66Ghz
12GB memory
Windows 2008 R2 64 bit
Exchange 2010 SP1 w/ Update Rollup 2

Over the past three days my team has probably spent 10-15 hours on the phone with RIM support trying a myriad of things as well as checking all settings and intergration pieces and there has been no change.

Things RIM had us try/check:
1. Throttling Policy
2. BESAdmin roles/permissions
3. BES version
4. CDO/MAPI version on Exchange & BES
5. Recreate service account for BES, reset permissions, create new throttling policy, and set new service account
6. Change rescan interval from 15 to 5
7. Decrease the max number of mailboxes per agent. (default is 150, we have about 170 BES users.) We decreated it to 60, so that the users would be spread over 3 dynamic agents

The only work around that we've tried that we've found works (and works well) is to move users to static mailbox agent. RIM initially was hesitant to have us do this, but on the 3rd day of support calls fully endorsed the idea. In speaking with the technicians they've indicated there should be no difference in the static versus dynamic agent except for the number of users on each; that said, we've had no luck with dynamic agents, and the static agent users work flawlessly. They origionally said to limit it to 5 static agents, but have now said 10 static agents would be fine. We have 10 users per agent and those users on the static agents are running great. It's not solution nor a pleasant work around, but it removes some of the urgency from the matter.

RIM points the finger at Microsoft for the issue, focusing on an RPC access issue. I'm not convinced this is the issue as Outlook clients connect over RPC and we've not had any issues with them. Additionally, restarting the RPC service on Exchange does not temporarily alleviate the problem.

We're currently running performance monitors on Exchange and BES to give to RIM for analysis. We've also ordered 16GB more memory for our Exchange server, as we've seen some people indicate memory alleviates (but does not resolve) the issue.

Re: Message delays on BES 5.0.2
 

I'm having a similar issue, though I only have about 90 blackberries currently. Was RIM able to provide any further detail based on your performance monitoring?

Adding additional memory seems to have helped for now, but I don't know how long it will take before these delays start to creep into the environment again.

Also, I've been having a hard time finding info on decreasing the max number of mailboxes per agent in BES 5. I've found info for BES 4, but I don't know if it applies to BES 5, and I'm hesitant to try it. Can you let me know where the setting is or what registry value to add/change?

Thanks

Re: Message delays on BES 5.0.2
 

Hey all...
Us too, as well.

And our Exchange 2010 environment is net new...
Our BES 5.0.2 enviroment, also, net new
Our MAPI/CDO version are latest
We also jacked our processor and RAM for years of growth..
That said.. I have seen delays, anywhere from 1 to 8 minutes.

Re: Message delays on BES 5.0.2
 

I tried the Static mailbox agent on a few devices yesterday, and implemented it on all of our users (about 45) today, and mail delivery seems to be on time now. I'll have to see over the course of a few days if it holds up, but I know as soon as I put myself into a static agent mail delivery times went from 15-25 minutes(basically whenever the BES did a rescan), to instant. Every other user I've talked to who I made the change to reported similar success.

Not really a feasible solution if you have 400 users on your BES, but for smaller shops it may do the trick.

Re: Message delays on BES 5.0.2
 

Did you try re-creating the BES mapi profile? I had similar issues lately and a MAPI profile recreation fixed it. I can find the documentation if needed but it's a knowledge base article on RIM's site...

Re: Message delays on BES 5.0.2
 

KB article: 13374

"The BlackBerry® smartphone might receive email messages up to 15 minutes after the messages arrive on the source mailbox. This is by design. The BlackBerry® Infrastructure checks for new email messages every 15 minutes."

Re: Message delays on BES 5.0.2
 

That is for BIS. ;-)

Re: Message delays on BES 5.0.2
 

Ah. So it is...
That revealed however, is there anyone out there, with BES 5.0.x, and Exchange 2010.. who has more than 100 users.. and never experiences these delays which so many of us are seeing?

Re: Message delays on BES 5.0.2
 

What service pack on Exchange 2010?

Re: Message delays on BES 5.0.2
 

Are there any errors in router log?

Re: Message delays on BES 5.0.2
 

OK - So I called RIM.
I got to second level support.
Got myself a sharp cat on the line, who knew his such and such.
Told him the whole story... and I sent him the link to this thread, which he read.

So, then he said: ZIP UP your BES logs and your application log from your server, send them along, and we'll have a look.

Well, wouldn't you know.. after he, and a senior development analyst had a look at all of the logs, it was determind that there were no problems with my BES 5.0.2 / Exchange environment.


I do want to point out the following...
Have any of you had at the ActiveSync policies in Exchange 2010?
I have... It allows for the similar granularity of BES policies.
iPhones and Windows 7 phones are super fast with emails, GAL look ups, ect.
Microsoft has seemingly taken some time to develop this, nicely.

Can't help but wonder if MS is making it 'quirky' for BES on purpose ;)
( did anyone manage a Lotus shop, when Office 2000 came out?!?! )

Re: Message delays on BES 5.0.2
 

I'm not sure if this will help, but I have been fighting this issue for 3 weeks and believe that I have finally resolved it.

For the record, I tried all the work arounds. Static agents worked for about 15 minutes then started doing the same thing. Up to 30 minute delays in new emails to blackberry devices.

Little background about our environment. We run BES 5.0.2 on a windows 2003 server virtualized in VSphere. We are currently running a mixed mode Exchange environment with Exchange 2003 on Server 2003 physical boxes, and Exchange 2010 on a Server 2008 VSphere box. SQL is a 2005 Enterprise Virtual system. THe SQL server was P2V'ed, and the others were built from scratch. We have 80 users that have blackberry's. No other devices are allowed so we dont have any Iphones or Droid devices.

Troubleshooting and description. Our issue began 3 weeks ago when we moved a large chunk of our MAC users to the Exchange 2010 environment. Users started complaining of delayed new messages. I went through the typical steps of verifying cell service, sending test message from BES (Which were never delayed) and PIN to PIN messages.

Looking through the logs I got "Send Pipe Full" errors in the Router log. The Controller log showed a few errors that seemed to be related. I decided to open a ticket with RIM. I wont go into the saga of trying to get help with them, just believe that they are proud of their support staff and make you pay for it. 3 hours on the first call and I got nowhere. The Tech said he couldn't see anything wrong based on our log files. After arguing, we agreed that I would get a specific users incident, time and message number and call back. When I called back, it was about 30 minutes and the tech determined that UDP packets between the BES and the Exchange server were being dropped. He referenced this Blackberry KB article KB14139 and explained that I had network issues.

I wont bore you with the next steps, but just tell you that we discovered our VM's running an older version of the Virtual NIC (E1000). We updated our BES and our Exchange 2010 CAS server to use the VMware enhanced NIC and changed TX and RX size to 2048, disabled the Interrupt moderation.

THis seems to have resolved our problem. I dont know why the old NIC's were used to begin with, perhaps it was defaulted to when the system was created.

Hope this helps someone else. OR atleast points you int he right direction.

Re: Message delays on BES 5.0.2
 

???

Re: Message delays on BES 5.0.2
 

Rtm, Ru4

Re: Message delays on BES 5.0.2
 

We are seriously considering rolling back Exchange 2010 SP1 back to RTM. This is simply getting unacceptable. I've tried throwing more RAM at the server, using static agents, moving people around in those static agents, and nothing works for more than a couple days.

Re: Message delays on BES 5.0.2
 

I am on Exchange 2010, RTM RU4 and am too, suffering.

Re: Message delays on BES 5.0.2
 

Can anyone else confirm that they are having this issue on Exchange 2010 RTM? I don't want to have us roll back to RTM only for the problem to continue. My manager would not be happy. =/

Re: Message delays on BES 5.0.2
 

Have you (anyone) tried assigning a static agents to any users?

Re: Message delays on BES 5.0.2
 

Yes people have tried that.

Re: Message delays on BES 5.0.2
 

Hi
did anybody tried the solution that Rueburbon published
Blackberry support are leading me on a wild goose chase with perfmon etc for weeks now without admitting they have a problem in the software
so if anyone tried the solution please update.

Re: Message delays on BES 5.0.2
 

NIranec, is part/whole of your BES environment on VMWARE?

Re: Message delays on BES 5.0.2
 

Yes Mikey-AGBoston, my whole environment is on a VMWARE

Re: Message delays on BES 5.0.2
 

Did you investigate your VNIC settings, as Rueburbon discovered and altered?
Our CAS is on VMware.. so I am begining to wonder.
The most perplexing issue is that during non-business, off peak hours, emails arrived within the minute of being sent...

Re: Message delays on BES 5.0.2
 

The reason for that is with less emails being sent during off peak times, the queue has time to process. When the higher volume of emails come though the high latency means it can't keep up. I'm encountering the same issue with our environment.

our enviornment is virtualized, but running Hyper-V.

Re: Message delays on BES 5.0.2
 

W'bar,
Have you tried adding a couple of users to a static agent, and if so what were your findings?

Re: Message delays on BES 5.0.2
 

the static agent worked fine for a couple of days, but the issues returned after that. I then added users that were in one static agent to another, and they worked fine for a couple of days, and the issues returned again.

My favourite is that whenever I want to activate a device, I have to move a user to a new static agent first before the device will activate. It's so much fun.

Re: Message delays on BES 5.0.2
 

KB17325-Issues with the BlackBerry Enterprise Server in a Windows Server 2008 based Microsoft Active Directory environment

If you've got any 2008 domain controllers, check out this KB article.

Easiest way to identify if you're affected by this issue is to change the registry settings as instructed in the KB on your DC's. Watch the logs for a few minutes, and it'll confirm you've got the issue.

We were affected by the issue and it was very difficult to troubleshoot because our symptoms didn't match what the KB describes. However, increasing the number of NSPI connections immediately resolved the issue.

Re: Message delays on BES 5.0.2
 

RU3 announced - possible related BES fix

How it applies:
As I understand 'it'

1.Message received in user mailbox
2.A UDP notification message is sent to the Blackberry Agent process handling the user
3.Message gets received on device

If anything keeps that UDP packet from getting to the agent (high network traffic or a million other reasons) the BES scans the users mailbox every 20 minutes for missed messages and then delivers them to the device which is why RIM is notorious for saying 10-20 minute delays are normal. RIM wonxxx8217;t guarantee the immediate delivery due to the limitations of the UDP protocol

This is why when the BES xxx8211; Exchange connection is optimal we see near immediate delivery and when periods of high use (ie. Business hours) we are more likely to see the delays of up to 20 minutes.
That said, Microsoft announced:
UDP Notification Support Re-added to Exchange 2010 for Roll-Up 3 (SP1 RU3)
(( I would post a link to the article, but I am prohibited from doing so, due to few than 10 posts, here ))

Re: Message delays on BES 5.0.2
 

That's all well and good, but with an expected release date in March, that is waaaay too long for a possible fix.

The issue is so maddening too. My blackberry has been fine today, but others haven't.

Re: Message delays on BES 5.0.2
 

Let me rephrase. I can work around it until March, but this is really frustrating.

The only thing that seems to be working for me is moving users to different static agents when mail delivery becomes a problem for them. I can literally move a user from one static agent into another and it begins to work immediately, and will work well for anywhere from 2-5 days, and then it starts to slow down again.

I only have 45 users, so I can live with having people call/email me when their email starts to get bad again and I can move them to a different agent. But it's just going to require a lot of babysitting until it can be fixed.

Re: Message delays on BES 5.0.2
 

Can you all confirm whether you're seeing similar issues with Outlook 2003?

In Outlook 2003, e-mail messages take a long time to send and receive when you use an Exchange 2010 mailbox

Also - Here's a link that I believe Mikey intended to post:

http://msexchangeteam.com/archive/20...28/457845.aspx

Re: Message delays on BES 5.0.2
 

I agree with kjarrodc, we had the same issue plus user's getting a red x when trying to send external email.

Bouncing the Dispatcher was a temporary fix but increasing the NSPI limit from the new default for Win 2k8 Svr of "50" to 2,000 has resolved these issues.

With the default NSPI limit, the first 50 users served are just fine, user 51 on though, can't get an exit.

--BesMstr

Re: Message delays on BES 5.0.2
 

This wins, if your Domain Controllers are on 2008...

Re: Message delays on BES 5.0.2
 

we have a domain controller on 2008 R2. I tried the fix, it's made the problem a little better, but not resolved.

We've also been seeing the Red X problem, as well as having difficulties getting new devices activated. March can't come fast enough.....

Re: Message delays on BES 5.0.2
 

please sign up for the webcast

Mitigating potential messaging delays:
BlackBerry and the Microsoft Exchange Server 2010


Lobby

Re: Message delays on BES 5.0.2
 

We have the same issues, RIm support is aware of this issue. It's a software design issue between BES 5.02 for Exchange and Exchange 2010. I have had this issue for about a month now. Some devices even stop receiving messages altogether, untill the Blackberry controller is restarted. I opened a ticket with MS as per RIMS request, I just need to know when this is going to be fixed.

so far I applied these bandaids, and still no love:
enabled 10 agents ber BES server
KB22812 Intermittent mail delivery issues via Load balancerenabled LDAP lookup
KB03193 enable ldap
KB20157 ews Calander helper


John Quest Software.

For more info, go to the Blackberry communities forum, and you'll see many other companies have the same issue.

Re: Message delays on BES 5.0.2
 

Ok, so it's too early in the game to get too excited about the recent development we've had, but being that delay seems to have disappeared, we're pretty hopeful.

A few things have happened in the last 10 days or so:
--February 28, 2011 - BES has released MR4 for BES 5.0.2 (you still need to do the supplimental fix in addition to MR4 which fixes 3 dll files)

--February 25, 2011 - Microsoft has released CDO 6.5.8211.0, which needs to go on your blackberry servers (and we put it on our client access server for Exchange as well).

--February 28, 2011 - Blackberry has released a technical note called: "Managing The Blackberry Enterprise Server Performance in a Microsoft Exchange 2010 Environment" (you can get here by going to the blackberry documentation area, under documentation for administrators, it's the latest note - I can't post links on this forum, otherwise I would link it).

Inside this last document, it talks about setting up direct LDAP queries to active directory to prevent these queries being made through MAPI (amongst other things like the CDO upgrade). Well, we implemented this last night and so far, all the delays seem to have vanished (though we're going to get a regular day test today, so we're not out of the woods yet). You can get more information on this through Blackberry KB article 03193 (we didn't do the SSL connection one as indicated and it still works ok).

I'll be sure to update everyone as we get more feedback, but again, so far so good.

Our topology (all running on vSphere 4.1):
Exchange 2010 SP1 (no update rollup) Mailbox Servers
Exchange 2010 SP1 (no update rollup) CAS array (there is a hardware load balancer on this array, but BES has a host file entry to always just hit the 1st CAS server in the site)
BES 5.0.2 MR4 + Interm Security Fix deployed in an HA configuration

Re: Message delays on BES 5.0.2
 

I applied the updates to our environment last night as well, it'll be a few days before I know whether it really worked. Aside from the fact that it was a pain in the ass to do the MAPI, MR4, and the interm security release at one time it went smoothly.

Re: Message delays on BES 5.0.2
 

I have updated with MR4, and the new MAPI/CDO (..yes, and the PDF security patch).
So far, deliveries to the handhelds are running close to the minute of actual reception, and have been for the past three hours.
(we have 380+ users)

I am optimistic - Stay tuned

Re: Message delays on BES 5.0.2
 

Marked Improvement
My deliveries have gone from batches of emails, delivered in 15 minutes intervals... to delays of zero to three minutes of the originating time stamp.

Re: Message delays on BES 5.0.2
 

Anyone experienced this yet or know under what conditions it can occur?

Exchange 2010 SP1 Rollup 3 and BlackBerrys sending duplicate messages - Exchange Team Blog - Site Home - TechNet Blogs

Re: Message delays on BES 5.0.2
 

I still have a handful of users complaining about delays, while some of us have seen the delays nearly disappear. Have people out there also jacked up the CAS/Mailbox servers memory and then seen much improvement?

Re: Message delays on BES 5.0.2
 

Well, about 10 days after applying the MR4 upgrade, and the new MAPI/CDO, everything seems to be working pretty much perfectly for us. I'm back to hearing my blackberry buzz before I even see the message in outlook.

We haven't even applied SP1 RU3 yet.

Re: Message delays on BES 5.0.2
 

OK - we encountered our first issue.
Activations.
Has any tried to conduct an activation since the upgrade to MR4 (w/new MAPI+CDO)?

Re: Message delays on BES 5.0.2
 

I have one user who is now reporting email issues. excuse me while I go and pull the BES server out of the rack and throw it out a window.

At least it's only one....I guess.....

Re: Message delays on BES 5.0.2
 

I've been fighting message delay issues on BES for about three years now. We have approximately 6,300 BES users and about 17,000 exchange mailboxes and on a typical day in the past would find that 10-15 minute message delays were not abnormal.

I hope that my experiences and pain can be of help to some of you.

For us, the problem turned out to be over provisioned disks. Increased IOPS on BES users is not the same issue in 2010 as it was in 2003, but it is still a very valid performance metric that needs to be considered. In Exchange 2010 “I/O Database Reads Average Latency” must be below 20ms, and RPC Average Latency must be no higher than 35ms average, and no higher than 50ms spikes. For our environment, as long as our disks are performing within these numbers messages flow to handheld devices in less than 60 seconds, confirmed using BoxTone.

Never under estimate the importance of disk IO.

Re: Message delays on BES 5.0.2
 

Hi Guys,
We had the same problem and it was the default throttoling policy - I know a couple of posts have talked about this but that was the fix for us.

Re: Message delays on BES 5.0.2
 

My env't is 5.0.2 MR5, mixed Exchange (2003, SP2 and 2010, Sp1, RU3). 2200 BB users across 4 HA BESs running on Win2K8 VMs each having 2 x dual-core CPUs and 8GB RAM.

I think I have been pointed towards the solution to this. Whilst the throttling policy changes are important they don't fully resolve - at least not for me. In my env't the problem was all down to the way BES handles Messaging Agents.

RIM have configured it such that a single Messaging Agent handles all BES to Exch2010 traffic for the first 500 users. This always struck me as a very large number of users for a single Messaging Agent and it turns out to be the case because each Messaging Agent can only handle a maximum of 100 threads. So once you go above 100 Exch2010 users, or once you have sufficient hung threads you're going to see latency and possible red Xs when users try to send.

Giveaway entry in the MAGT logs was multiple instances of "Failed to reach user's mailbox"


Fix is a registry hack and it has worked 100% in my env't. This changes Agent allocation such that after making this change users will be divided evenly amongst the number of agents specified in the values below:


Take HA BES out of Automatic failover mode.

Make change on the live Primary BES first then implement on the standby afterwards. The standby will actually spawn the same number of agents as the primary as it gets the AgentID from the SQL database. The reason you must make the change on the registry is in the event that both servers are shutdown and the Standby comes up first, triggers a failover and everyone is back on 1 agent. This will only work on a server with more than 10 users.

Stop Controller, Dispatcher & Router BlackBerry services in this sequence.

At HKEY_LOCAL_MACHINE\SOFTWARE\Research In Motion\BlackBerry Enterprise Server\Dispatcher create a DWORD registry value called AgentOptimizeAlgorithm (capitalisation is critical) and give it a value of 1.

At HKEY_LOCAL_MACHINE\SOFTWARE\Research In Motion\BlackBerry Enterprise Server\Agents create a DWORD registry value called NumAgents (capitalisation is critical). 5 is the default number of agents, recommendation is to increase to a value of 10.

At HKEY_LOCAL_MACHINE\SOFTWARE\Research In Motion\BlackBerry Enterprise Server\Agents create a DWORD registry value called NumAgentsFullServer (capitalisation is critical). 5 is the default number of agents, recommendation is to increase to a value of 10.

Start Router, Dispatcher & Controller BlackBerry services in this sequence.

Repeat on Standby BESs.

Put back into Automatic HA failover state.

Re: Message delays on BES 5.0.2
 

We have a mixed Exchange 2007/2010 environment; if you have a CAS array, make sure to put a HOSTS file entry on each BES pointing to just ONE of the CAS boxes in your array and also make sure that the mailbox for the BES account defined in your BES MAPI profile has been migrated to 2010 because you will have issues if it is on a 2007 Exchange server. In addition, make sure you have the latest MAPI/CDO installed (currently 6.5.8211.0) and you have Exchange 2010 patched to the latest Rollup (4) as well as the latest MR for BES 5.0.2 or 5.0.3. Microsoft also recommends this reg change on your BES boxes:

"we have managed to reduce the mail delays by implementing the below (we were seeing delays of up to 30 mins, they are now seeing delays of <5 mins);


try the following reg key to limit the number of users per session. By default it is 50 and 25 should be a good number to test with. You will neeed to restart the agents and this can be done by restarting the BlackBerry Dispatcher service. I will recommend to stop the Controller, restart the dispatcher, and then start the controller service. This way we are in the clear and this key should get picked up at that point.

HKLM\SOFTWARE\Research In Motion\BlackBerry Enterprise Server\Agents\

DWORD Type
MaxMailboxesPerSession

*Enter 25 as a decimal value, not hex*

Note: BES has a limit of 2000 users per server, so if the key is set to 25 users per session you can have a maximum of 40 sessions."

Hope that helps; it did for us.