[drlchua]

We are currently running BES 5.0.2 and recently upgraded to Microsoft Exchange 2010 SP1 from Exchange 2003. For testing, we moved 5-10 users and were moved with no problem, users were able to send/receive from their bb devices fine.

When we moved all our users over the weekend, we started getting 5-30mins delays before users can receive emails on their blackberries. if we move a user to a static box then mail flow seems normal.

Fixes we've tried:

1) checked Send As permissions for BESAdmin account
2) added MaxSessionsPerUser = 100000
3) Set all RCA Parameters to $null

is there anything we missed to fully migrate to Exchange 2010 and not have these email delay issues?

thanks in advance for your help.

[Gnyff]

Well, this really isn't going to help you, but there seems to exist a memory leak in the rpc client access service in Exchange 2010 Sp1, which can cause a delay in the delivery on the mails to your BES.
I've only seen it sofar on one device in our setup, but it apparently come down to the amount of memory you have in your Exchange server. The less amount you have the sooner the issue appears.
In my case I experienced a delay up to 15-20 min on various mails while others were right on depending on time of day. (I'm in Europe on a US server so not really affected before US wakes up)
A friend of mine had the same issue in their environment and MS eventually provided them a hotfix for the memory leak for their problems.
The hotfix were issued prior to the Exchange 2010 Sp1 Rollup 2 update, so unless the RU2 has been applied that might be an option.

[knottyrope]

check out this thread
BES 5.0.2 message to user deley after migrate - BlackBerry Support Community Forums

[ManuNarayan]

Just to add to the issue as described:

We've been having the exact issue. We recently transitioned from Exchange 2003 to Exchange 2010. We've been running in co-existence and had our BES server delivering messages for boxes on both servers. When we had 30-40 mailboxes on 2010 we did not experience any issues with Blackberry. However, once we moved the majority of our boxes we ended up having significant delays in sending messages, receiving messages, and performing lookups.

Our environment:
BES
ESX 4.0
1vcpu
4GB Memory
170 users
CDO 1.21 installed
5.02 SP2 MR2 (latest build)

Exchange
Single server
Dual 6-core 2.66Ghz
12GB memory
Windows 2008 R2 64 bit
Exchange 2010 SP1 w/ Update Rollup 2

Over the past three days my team has probably spent 10-15 hours on the phone with RIM support trying a myriad of things as well as checking all settings and intergration pieces and there has been no change.

Things RIM had us try/check:
1. Throttling Policy
2. BESAdmin roles/permissions
3. BES version
4. CDO/MAPI version on Exchange & BES
5. Recreate service account for BES, reset permissions, create new throttling policy, and set new service account
6. Change rescan interval from 15 to 5
7. Decrease the max number of mailboxes per agent. (default is 150, we have about 170 BES users.) We decreated it to 60, so that the users would be spread over 3 dynamic agents

The only work around that we've tried that we've found works (and works well) is to move users to static mailbox agent. RIM initially was hesitant to have us do this, but on the 3rd day of support calls fully endorsed the idea. In speaking with the technicians they've indicated there should be no difference in the static versus dynamic agent except for the number of users on each; that said, we've had no luck with dynamic agents, and the static agent users work flawlessly. They origionally said to limit it to 5 static agents, but have now said 10 static agents would be fine. We have 10 users per agent and those users on the static agents are running great. It's not solution nor a pleasant work around, but it removes some of the urgency from the matter.

RIM points the finger at Microsoft for the issue, focusing on an RPC access issue. I'm not convinced this is the issue as Outlook clients connect over RPC and we've not had any issues with them. Additionally, restarting the RPC service on Exchange does not temporarily alleviate the problem.

We're currently running performance monitors on Exchange and BES to give to RIM for analysis. We've also ordered 16GB more memory for our Exchange server, as we've seen some people indicate memory alleviates (but does not resolve) the issue.

[ManuNarayan]

We are nearing an end with RIM support. After 24 hours of extensive exchange logs, they have deemed it is an issue with high rpc latency and is an 'exchange performance' issue that we must take up with Microsoft. Indeed, we do seem to have high RPC latency, in the 150-200ms range, but only when BES services are running. When we turn off BES services latency drops to 2ms and is steady. When we turn services up, latency sits around 50ms for 5-10 minutes, drops to 5ms for a few minutes and then immediately skyrockets to over 100ms. Our server is sized well for our number of mailboxes and is probably sized for significant growth.

Has anyone had any contact with microsoft or put in a support call with them? We are in a tough spot as we do not have a support contract with Microsoft, and every other aspect of Exchange is working without any error. MAPI connections from Outlook, OWA, ActiveSync, imap, smtp, no other problems exist. My options, as RIM has left them, is to either enter into a paid support call with Microsoft (and I can already hear them blaming blackberry) or to begin to migrate users from BES to ActiveSync devices.

Anyone have any other ideas?

[ianamac]

Well the issues that is outlined is EXACTLY what we have since Exchange was upgraded the SP1 and we have to upgrade BES to 5.0.2. We are running everything on a Virtual Evironment with 2 Exchange Servers in different localities and i BES server in first locality.

After 5 days of pain we set up 6 static agents for one of the localities to see how many users is best per agent. This is still going on but for those users one of these agents have stopped complaining.

The issue all centres around hung threads that BES doesnt / cant (answer depends on if you are talking to MS or RIM!) get the answer from Exchange Datastore.

So Static Agents help but isnt the best solution!

[ManuNarayan]

ianamac, that is exactly what we've done. We have about 170 devices. We put 10 people into 10 static agents and those 100 users now have blackberry performance on par with Pre 2010 SP1. (near instant delivery)

What's funny is that is RIMs work around, yet they indicate a static versus dynamic agent is 'the same' where the only difference is static vs. dynamic allocation to the agent. We now have only about 20 users running on each of the dynamic agents and they appear to only get messages on mailbox rescan.

[knottyrope]

I have seen a few posts that show tossing more RAM at BES and teh MBX server can help with this issue.

Look in task manager and see what BAS and EWS is using for RAM and get the free RAM up to a gig or more if possible.

For the MBX server I have seen people double the recommended (microsoft reccomended) RAM and the issue is gone. Why? because the mailboxes are being put into RAM instead of being accessed via disk.

[ManuNarayan]

Thanks for the advice. We did read that as well actually and are adding 16GB of memory to the server tonight. I am not sure if it will resolve our issue as we have 300GB in mailbox databases, so certainly not enough RAM for all mailbox data to be loaded in memory.

That said, the moderator over at blackberryforums.com.au has setup a 2010 SP1 test environment with BES. 64GB of RAM, dual quads, 5mailboxes, and the issue is present. So I'm not convinced memory will really solve the issue.

[ManuNarayan]

We did our RAM upgrade from 12GB to 28GB and so far the issue seems to have gone away. I'm running performance logs to check RPC latency as well as messaging tests. The RAM installation came with a full box restart, so we do need to see how performance handles after load + a few days of usage.

Messages have gone from being delivered only after rescan (15 minutes) to near instant.

[mungoblack]

Setting this reg key on the BES server has also helped. It is not a fix but a temporary workaround.

HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Research In Motion\BlackBerry Enterprise Server\Agents

New DWORD
RescanPollInterval
set to 10

setting less than 10 didnt help us as we got into a cycle of rescans before the previous rescan had finished. 10mins approx still isnt ideal but its better than the 30mins+ delays i had been seing.

[kjarrodc]

Are you all also having problems with Outlook 2003?


Another similar thread:
http://www.blackberryforums.com/bes-...5-0-2-a-3.html

For what it's worth, I'm betting this has to do with the fact that in Exchange 2010 SP1, udp notifications have been removed.
Discontinued Features: Exchange 2010 SP1 Help


Udp notifications are being re-added:
You Had Me At EHLO... : UDP Notification Support Re-added to Exchange 2010

[AL72]

We are also seeing issues with Outlook 2003 and another 3rd party app that talks to Exchange 2010.

We have been working with RIM and they have acknowledged there is a known issue with RPC latency (more and more of their customers are reporting this problem) and a specialist team is now working with Microsoft to develop a fix. This may take some time...month or more has been indicated. Here's hoping it's sooner than later!!

[wunderbar]

Originally Posted by AL72

We are also seeing issues with Outlook 2003 and another 3rd party app that talks to Exchange 2010.

We have been working with RIM and they have acknowledged there is a known issue with RPC latency (more and more of their customers are reporting this problem) and a specialist team is now working with Microsoft to develop a fix. This may take some time...month or more has been indicated. Here's hoping it's sooner than later!!

Well, Exchange 2010 SP1 RU3 is bringing back the UDP system what is what the BES and outlook 2003 use. that, in theory, should fix it, and RU3 is due in March. I'm basically holding out until then. Not much else I can do.....

[garycutri]

To resolve this issue you need to install BES 5.0.2 MR4 and update to the new MAPI MAPI\CDO client (6.5.8211.0): http://www.blackberryforums.com.au/f...cy-issues.html

[wunderbar]

I will be doing this tonight and will report back on how it goes. I just want to get this bloody thing fixed.

[wunderbar]

I did the install last night. it was a massive PITA to do it all in one go, but it did go smoothly. the performance on my device was better than before, but I wasn't experiencing any real problems myself.

It's going to be a few days before I'll know the real results, but I am hopeful.

[scottj3]

Does anyone know if MS pulled the MAPI CDO version 1.2.3 files? When I went to the link that was supposed to contain that version on the MS Download site the version still says 1.2.1. Interesting though that the release date says Feb 25th 2011. The "real" version 1.2.1 files have been out nearly a year. Not sure what MS is saying here.

[freakinvibe]

The MAPI CDO version has been on version 1.2.1 for a long time. It is the sub-version that changes all the time, for example

6.5.8153.0

6.5.8211.0

So the 25th February update was just an increase in the sub-version number.

[wunderbar]

just as an update to this, so far so good. I have about 1/3 of my users back into the dynamic agent, the rest are still in static agents, but so far no complaints, and no issues that I can see. I'm going to be moving users off of the static agents over the course of the next week and we'll see if it keeps it up.