[JPenrose]

Our BES environment:

Exchange 2003 (no SP)
BES 4.0 SP5 HF5
BB 7250 handhelds running 4.0 code

After updating BES from 4.0 (no SP) to SP5a HF1-5 (incremental) on Sunday we're experiencing anywhere between 30 sec and 3 min mail redirection delays. Prior to the SP/HF updates (which completed error free) our handhelds were receiving email almost instantly. Now users are reporting unusually long delays.

The exchange queues are green and clean and mail to desktops seems to be running normally (sub-second delivery). A test email while writing this had a 1:55sec delay to the handheld where I used to see instant redirection. Doesn't seem to matter if its internally delivered mail or inbound from the Internet.

Our CDO.dll and MAPI32.dll versions are 6.5.7226.0 on both the Exchange server and the BES server. The BES server is installed on an SMTP relay box on the internal domain which is how we've been doing things for years.

Can anyone offer some advice/answers/guidance from experience?

Thanks in advance,

JPenrose MCSE/A

[dev]

Have you checked your BES logs (MAGT, SYNC) to see what kind of activities are being recorded? You may have several rescans going on, hung threads being generated. A reboot of the BES may clear the hung threads if there are any....but this may not resolve the issue. Ensure there are DNS resolution issues, or permission issues between the BES service account and the Exchange mailboxes. The latter shouldn't exist, since it worked fine before the upgrade...but have a look at the logs to get a better picture of what's happening

[JPenrose]

Thanks for the reply dev,

I had a look at the BES logs specifically the two you pointed out. I'm not going to pretend I know exactly what I'm looking for as far as identifying trouble goes but I can say that what I see I understand just not in as far as I could point out trouble.

I see a ton of entries in MAGT that go like this:

[40690] (10/23 02:22:03):{0x158C} {user1@domain.com} Starting message status rescan
[40691] (10/23 02:22:03):{0x158C} {user1@domain.com} Message status rescan completed
[40267] (10/23 02:22:09):{0x17A8} {user2@domain.com} Starting calendar rescan
[30019] (10/23 02:22:09):{0x17A8} {user2@domain.com} MAPIMailbox::DoICS ICS Synch Successful.
[40701] (10/23 02:22:09):{0x17A8} {user2@domain.com} Calendar rescan completed

on and on these entries go but not any that look like rescans over and over for the same user for the same reason. I don't really know what trouble looks like in here to honest.

SYNC log looks a lot like this:

[40000] (10/23 04:25:38):{0xFD8} [SRP] Send data, Tag=363, Submit=1, Size=42
[40000] (10/23 04:25:38):{0xFD8} [SRP] EVENT=Send_SEND, VERSION=1, TAG=363, SIZE=42
[46019] (10/23 04:25:38):{0xDB8} [SYNC-DSession] DSession is finished. [user1:53, SID=1950757935, CLID=57, ECLID=4]
[40000] (10/23 04:25:39):{0xFD4} [SRP] EVENT=Receive_STATUS, VERSION=1, TAG=363, STATUS=1
[40000] (10/23 04:25:39):{0xFD4} [SRP] Received status DELIVERED, Tag=363, AckRequired=0
[36009] (10/23 04:26:07):{0xE14} [SYNC-Gate] Performing user config change check.
[46077] (10/23 04:26:07):{0xE14} [SYNC-Gate] Finished user config change check. No change detected.
[40000] (10/23 04:26:39):{0xFD8} [SRP] EVENT=Send_PING, VERSION=1, TAG=842
[40000] (10/23 04:26:39):{0xFD8} [SRP] Ping 842 sent
[40000] (10/23 04:26:39):{0xFD4} [SRP] EVENT=Receive_PING_RESPONSE, VERSION=1, TAG=842
[40000] (10/23 04:26:39):{0xFD4} [SRP] PingResponse 842 received
[40000] (10/23 04:27:04):{0xFD4} [SRP] EVENT=Receive_RECEIVE, VERSION=1, TAG=12223859, SIZE=102
[40000] (10/23 04:27:04):{0xFD4} [SRP] Received datagram, Tag=12223859, Size=102
[46042] (10/23 04:27:04):{0xDC0} [SYNC-DSession] Received 68 bytes from device. [user2:30, SID=1950865647, SEQ=1/1]
[46052] (10/23 04:27:04):{0xDC0} [SYNC-DSession] Contains 1 record(s) and 1 command(s) from DevMgmt.Handheld Agent [user2:30]

again, on and on that goes.

Our DNS infrastructure is pretty solid and reliable. I see no obvious issues either before or after the updates.

How would I identify a hung thread from log data?

During the update I did about 6 reboots of the BES but I can certainly schedule another.

The account I used for the updates is our master BESAdmin account which is the one used to originally install the BES server and also has full mailbox access so no issue there and no change.

Could it be a handheld code issue?

Thanks again for the help.

JPenrose MCSE/A

[dev]

It's not a handheld code issue, the code is fine for the BES version. If there are older devices with older codes (3.6-3.8) all they will experience is slow performance with respect opening attachements and viewing mail...not mail redirection to the devices.
Also have a look at your event viewer (as well as the Task Manager)for any errors related to any of the BES services. The Task Manager would indicate which service or task is occupying most of the CPU cycles.
If there were hung threads...the MAGT logs would indicate it. That's provided that the logging levels are set sufficiently high (at least level 4) to get some verbose output. To monitor/identify the issue try raising the logging level for the following logs - MAGT, SYNC, ROUT and DISP.

Also have a look at the Exchange server as well - Task Manager, Event Viewer to see if there are any issues on that side as well.

[JPenrose]

Clearly, your Kung-Foo is very strong.

If you don't mind looking over some results I will post some stuff that presents itself as a [red] flag when I'm looking over the logs etc?

I did observe that the user of our only 7100R running code 4.1 on a different carrier than the rest of us gets his redirected email instantly. This was always the case for him and the rest of us before the update using the main carrier and a 7250. I'm not sure if that means anything but it was worth the mention. The fact that this user is GPRS on carrier B and most units are CDMA on carrier A may be something.

Thanks again, dev.

JPenrose MCSE/A

[dev]

Ahh ...those Bell devices.....hopefully those affected users are in an area of good coverage (not in a basement office somewhere). We use both Rogers and Bell and the EDGE/GPRS devices (Rogers) seem to function better all the time. The CDMA can be slow sometimes.

[JPenrose]

The CDMA are TELUS Mobility with a few newer ones running on EVDO and the GPRS are indeed Rogers [we must both be Canadian]. We have about 6 users on GPRS b/c of TELUS coverage issues. The Rogers units rock while the TELUS units are always a half step behind. Here, if you're using a CDMA it's b/c you can actually get a signal otherwise you're on GPRS.

TELUS gave us a re contracting deal that was just too sweet to pass up on. Bell and Rogers couldn't touch it (too bad).

-=J=-

[dev]

Oh..yeah I forgot Telus (they have the fastest network). I just don't like their devices.
Yes I'm am Canadian....to mimic that old AD that ran a couple of years ago I think..
What's the Pending count like for those affected users? If you need help you can PM me.

[JPenrose]

The bizzare thing is that the pending count stands at 0 for all but 3 users [we have 53] one of which is away on vacation somewhere in Asia. The other two have 10 and 1 respectively but that's not all that unusual. Messages are taking their sweet time to land but are not pilling up inexplicably in the pending queue.

Any point in doing an enterprise activation as a test?

-=J=-

[dev]

What you could do for one of the affected users (or have them try it) as a test is..a simulated EA.
Simply have the user go to the Enterprise Activation option. Then with the option opened, hold down the ALT key and type the following letters "CNFG" without the quotes. For those 7100 series devies they'll have to type hit the
keys twice to get the right letter.
A new view will open up. Scroll down to the "Wireless Sync" option and change it from "Yes" to "No" and save it. Then go back in and change the option to yes. This will simulate an EA and will pull down all the recent service books again from the BES server. Then test mail redirection for that user to how fast the delivery may be.
Another way to test delivery times is to address mail to the affected users with the following in the subject field <$confirm>. The angled brackets should be included. The result is similar to ping testing a device to see if its alive and how the response is.

[JPenrose]

Good stuff dev.

I'll try those and post the results here tomorrow.

-=J=-

[JPenrose]

Well, after some deliberation and thought to dev's suggestions I elected to schedule a restart of the BES server before cranking up the logging and getting down to business.

The last HF I installed was HF5 for SP5a which installs then wants a reboot to configure itself (just like the previous 4 did). I obliged the installer with the reboot, let it configure and basically just walked away after testing mail to a GPRS unit. The mail arrived on time (~ 2 seconds after sending) so we figured we were good. Nope. The CDMA devices were experiencing as much as a 3+ minute delay in redirection while the GPRS devices were on time. After the reboot I did on Tuesday the CDMA devices immediately got with the program. Lesson for me: do a restart after a HF that wants to configure itself after requesting a restart.

The good news is that no one really noticed or if they did they didn't say anything. The jury is still out as to whether the SP and HFs fixed our calendaring issues. We still have the embedded JPG issue (where you have to retrieve the image twice b/c the first request fails) but RIM says this is fixed in BES 4.1 and is under consideration for HF support.

Thanks to fellow CAN dev for helping out.

Keep up the great work here; it's a BES admins #1 stop for help.

JPenrose MCSE/A

[dev]

Good to hear...still try the "ping test" to give you an idea of the roundtrip delivery times for the users in general.