We have two BES in our HQ location and about 700 users. Most of the users' mailboxes are in the HQ site (multiple mailbox servers) and while most of them are on one BES, a handful are on the second BES. The second BES is mostly for the remote site users and those user mailboxes are homed all over the US. All of the remote users have Nextel 7520 devices with the latest version (4.1.0 I think?) on the handhelds and the BES is v4.0 with 2 service packs. We run Exchange 2000 SP3.
Our HQ users are happy as clams no matter what BES they are on. Our remote users have a lot of problems, with the most common being that the device stops sending and recieving and the only way we can fix it is to remove service books and rerun enterprise activation. Sometimes the enterprise activation takes many hours. We are averaging 4 or more problems a week from this set of users even though there's only about 40 of them. This compares to one or two similar problems a month from our 650 HQ users.
I have no idea how to begin troubleshooting this issue--- can anyone give me suggestions on what to do to improve stability of service for users whose mailboxes are homed outside of the area but BES is in HQ?
The description you provided of the network environment implies that your second BES server has to make MAPI connections to remote mailbox servers. The minimum latency as recommended by RIM is <35ms between the BES and Exchange servers. It is also recommended that the BES and Exchange servers reside on the same LAN to facilitate that minimum requirement. But we all know that isn't the case in most organizations.
What you can try to do to lessen the issue is possibly reduce the polling interval between the second BES and the remote Exchange servers that it "speaks" to.
Also have a look at the MAGT, SYNC, ROUT and DISP logs for the second BES server and look for "packets" or "response" not received from devices that's assigned to those 40+ users.
You could also try sending a "ping" test (using "confirm" in the subject line) to those devices to determine the round-trip time.