[Jadey]

OK, there is a really odd problem with my BES, not even sure how to describe it.

Sys info:
* Lotus Domino Server (Release 7.0.2 for Windows/32) - BES is NOT the Domino mail server, it actually connects to four mail servers (2 on same LAN in UK, 2 via VPNs in USA/CA)
* BlackBerry Enterprise Server, Version 4.1.3.22
* Windows 2000 Server

Problem seems to be:
* In a nutshell, BES slowly gives up processing anything for users.
- I do not ever receive a BES alert email
- When I check the server, Domino is running
- When I check "show tasks" on Domino, BES is running
- When I check services, all BlackBerry services are running

How does problem present?
- One by one, BES seems to just start "ignoring" the fact it has certain users.
- Watching the server console, it never seems to scan these users' inboxes for mail. Nor does it attempt to contact the device. Users find that they just "don't receive" any mail, and can have problems sending mail and completing lookups.
- Server does not show errors for those users, it just *ignores* the fact they exist
- Affected users are on different mail servers, so this is not (obviously) linked to one of the 4 domino servers running mail
- Only "fix" seems to be restarting Domino & BES
- Once this issue has appeared (it does not happen for some time after a restart), Domino & BES will not shut-down cleanly, I normally have to end task on Domino
- Have to put all services to manual before server reboot, and start them when server is up - when they are on auto, Domino & BES do not start reliably
- Domino does not seem able to start BES services any more, even when BES services are set to automatic. I bring up Domino, which says it has loaded BES but nothing processes, then I have to go to each BB service and manually start them.
- Different users are affected on different days
- Problem starts with one user, then another, then another, till everyone affected, and this makes it hard to know when server is problematic. For example, today my BB fine, 2 colleagues on same mail server had no mail since early hours of morning.


Any error messages?
* Only in logs, never on BES or Domino console
* These messages will start appearing in Application Logs on Windows for affected users:
---------------------------------
Event Type: Warning
Event Source: BlackBerry Messaging Agent XXXXXXXX
Event Category: None
Event ID: 20148
Date: 01/05/2008
Time: 08:40:00
User: N/A
Computer: XXXXXXXX
Description:
The description for Event ID ( 20148 ) in Source ( BlackBerry Messaging Agent BLACKBERRY ) cannot be found. The local computer may not have the necessary registry information or message DLL files to display messages from a remote computer. You may be able to use the /AUXSOURCE= flag to retrieve this description; see Help and Support for details. The following information is part of the event: Thread: *** No Response *** Thread Id=0x175C, Handle=0x8F0, WaitCount=121, Last Activity: New Message for user Mark Smith/OU/ORG.
---------------------------------

AND

---------------------------------
Event Type: Warning
Event Source: BlackBerry Messaging Agent XXXXXXXX
Event Category: None
Event ID: 20149
Date: 01/05/2008
Time: 08:40:00
User: N/A
Computer: XXXXXXXX
Description:
Thread 175C, utilization=0.0000%, failed health check 121 times
---------------------------------

Has anyone seen anything like this before?
Any ideas where to start?!
Any requests for further info/Win logs/BES logs please let me know and I will post.

The result of this is that I have a defective and very difficult to manage BES service, has been going on for a week or so now, and I am running out of ideas!

PLEASE HELP IF YOU CAN!