[Arkanian]

Server: BES 5 SP2 MR2, Windows Server 2003, Domino 8.5.1

Lately we have had an issue where the pending internal events are growing. Example yesterday it was at 200 and now it is at 236. It is taking around 3 days for new users to get the application policy to resolve so I think it is processing but it is processing slower than the new ones that are being added. We do have a BES Pool and it is configured TCP. We even stopped the other BAS to see if that would make a difference and it didn't.

I have a query from RIM that is supposed to identify a stuck job but it comes back empty.

Right now I am at a loss why this is happening. Has anyone seen this issue before?

[Arkanian]

Well I found that I had 3 jobs stuck in a running status so I stopped execution and changed them to start on Monday but now my pending internal events just shot up 20 more!

[Arkanian]

Well...the server is finally playing nice and around 240 events just got processed. Looks like it was a combination of one of the BAS services getting out of sync and the 3 hung jobs.

[Arkanian]

Update: we upgraded to 5.0.3 MR1 and Domino 8.5.2. Our internal events are at 655 now! I can see that sometimes it does process an event but it is doing it slowly. I also see that some events that are in the running status the tasks are complete. Has anyone dealt with this before? Trying to figure out how to make it go down. We also have another server as the HA server in the pool.

[DarthBBerry]

It's an issue with BAS processing the tasks. You probably have multiple BAS instances all running in a DNS pool. Each BAS instance is fighting to be the singleton (or primary) BAS instance.

Stop all instances of BAS.
Reboot the BAS server you want to be the singleton right now. Be sure all other instances are stopped.
Let it run for about 20-30 minutes. The jobs should process and clear up. *The KB says 5 minutes, but after being on the phone with T-Support... 20-30 is better.
Now (if you want) restart the other BAS instances in the pool.

This is a known singleton error with BAS.
KB23803-Jobs and Tasks are not being reconciled by the BlackBerry Administration Service.

Lastly, since you have already upgraded to SP3, try this article.
KB26461-BlackBerry Administration Service reconciliation issues after upgrading to BlackBerry Enterprise Server 5.0 SP3

[RadHaz75]

also, if you have bas in multiple sites, this can occur if the latency is above 20ms. jboss is gets really cranky with any kind of latency. i had this exact issue and basically have to keep one of my bas servers off cause any time i start it tasks start building cause it cant keep up with the latency between the 2

[Arkanian]

Yeah I got only 1 of the BAS turned on right now but it didn't do any good. My internal events finally went down from 655 to 652 but nothing is happening right now. I have a ticket opened with RIM on this issue.

[Arkanian]

Support told me that my issue is a known issue and should be addressed in SP3 MR2. They did tell me to turn off all of my BES servers and only bring up 1. So I am doing that right now. Before I was just shutting down the BAS service.

[Arkanian]

After the restart the tasks are still 652 but I see a TON of these in the log:
processFinishedTaskLocal jobTaskEntity not found for taskId = 56734
processFinishedTaskLocal Current task (57246) state is SUCCESS, but the dependency task is in a completed state, ignore the notification!

[Arkanian]

It is almost like there are tasks in the DB that are not tied to a job and thus the BAS never clears them out.

I tried to search for these tasks but the BAS says that it cannot find them.

[Arkanian]

Well it finally went down to 643 so something is happening...but it is going very slowly.

[Arkanian]

well it has been stuck at 643 for some time now. Support is still looking into it but so far nothing has worked.

[Arkanian]

Going to be installing MR2 to see if that fixes the issue as MR2 was supposed to address this specific problem.

[noname]

Arkanian, no MR is gonna resolved your issue. RIM has issued a Critical Advisory for DT 1216175 which DarthBBerry has given you in KB26461.

You should not have upgraded your BES to 5.0 SP3 if you still have stuck pending internal events.

Source: http://docs.blackberry.com/en/admin/...1-5.0.3-US.pdf

Internal events are stucked due to environmental issues causing communications between the BAS instances to failed. The many BAS instances you have in your setup, the higher the chances of getting stuck jobs. I don't under why one needs to install so many BAS instances in a setup. For BAS pooling, just setup 2 in a pool and preferably all located in the same LAN and avoid using TCPPing communication mode. A BAS instance is like a web-server, and with the web admin console URL, you can access it for administration from 100 workstation's IE browsers if you like. So for redudancy, just install 2 instances, one to standby in case the other is not working.

One can still function with 1 BAS instance for a domain of 10 BES servers with many people access from multiple workstation to the web admin console address using IE browsers to admin these servers remotely.

[noname]

And btw, for those stuck jobs, you will require RIM's help to determine if they are in bad states and they will work out scripts to help you clear them. Also running the script will not immediately drop those jobs and most likley you will need to restart the BAS-AS service after applying the script and allow it to process all gracefully within 2-3 hours, depending on how powerful your server is.