BlackBerry Forums Support Community               

Closed Thread
 
LinkBack Thread Tools
Old 01-14-2011, 02:45 PM   #1 (permalink)
Thumbs Must Hurt
 
Join Date: Jun 2008
Model: 9810
PIN: N/A
Carrier: AT&T
Posts: 130
Post Thanks: 0
Thanked 0 Times in 0 Posts
Default Stuck "Pending internal event processes"

Please Login to Remove!

Server: BES 5 SP2 MR2, Windows Server 2003, Domino 8.5.1

Lately we have had an issue where the pending internal events are growing. Example yesterday it was at 200 and now it is at 236. It is taking around 3 days for new users to get the application policy to resolve so I think it is processing but it is processing slower than the new ones that are being added. We do have a BES Pool and it is configured TCP. We even stopped the other BAS to see if that would make a difference and it didn't.

I have a query from RIM that is supposed to identify a stuck job but it comes back empty.

Right now I am at a loss why this is happening. Has anyone seen this issue before?
Offline  
Old 01-14-2011, 03:09 PM   #2 (permalink)
Thumbs Must Hurt
 
Join Date: Jun 2008
Model: 9810
PIN: N/A
Carrier: AT&T
Posts: 130
Post Thanks: 0
Thanked 0 Times in 0 Posts
Default Re: Stuck "Pending internal event processes"

Well I found that I had 3 jobs stuck in a running status so I stopped execution and changed them to start on Monday but now my pending internal events just shot up 20 more!
Offline  
Old 01-14-2011, 04:39 PM   #3 (permalink)
Thumbs Must Hurt
 
Join Date: Jun 2008
Model: 9810
PIN: N/A
Carrier: AT&T
Posts: 130
Post Thanks: 0
Thanked 0 Times in 0 Posts
Default Re: Stuck "Pending internal event processes"

Well...the server is finally playing nice and around 240 events just got processed. Looks like it was a combination of one of the BAS services getting out of sync and the 3 hung jobs.
Offline  
Old 05-16-2011, 04:13 PM   #4 (permalink)
Thumbs Must Hurt
 
Join Date: Jun 2008
Model: 9810
PIN: N/A
Carrier: AT&T
Posts: 130
Post Thanks: 0
Thanked 0 Times in 0 Posts
Default Re: Stuck "Pending internal event processes"

Update: we upgraded to 5.0.3 MR1 and Domino 8.5.2. Our internal events are at 655 now! I can see that sometimes it does process an event but it is doing it slowly. I also see that some events that are in the running status the tasks are complete. Has anyone dealt with this before? Trying to figure out how to make it go down. We also have another server as the HA server in the pool.
Offline  
Old 05-17-2011, 07:10 AM   #5 (permalink)
Wireless Sith Lord
 
DarthBBerry's Avatar
 
Join Date: Jan 2007
Location: Online
Model: iOS 6
Carrier: Verizon x2
Posts: 1,458
Post Thanks: 2
Thanked 27 Times in 22 Posts
Default Re: Stuck "Pending internal event processes"

It's an issue with BAS processing the tasks. You probably have multiple BAS instances all running in a DNS pool. Each BAS instance is fighting to be the singleton (or primary) BAS instance.

Stop all instances of BAS.
Reboot the BAS server you want to be the singleton right now. Be sure all other instances are stopped.
Let it run for about 20-30 minutes. The jobs should process and clear up. *The KB says 5 minutes, but after being on the phone with T-Support... 20-30 is better.
Now (if you want) restart the other BAS instances in the pool.

This is a known singleton error with BAS.
KB23803-Jobs and Tasks are not being reconciled by the BlackBerry Administration Service.

Lastly, since you have already upgraded to SP3, try this article.
KB26461-BlackBerry Administration Service reconciliation issues after upgrading to BlackBerry Enterprise Server 5.0 SP3
__________________
DarthBBerry
6-Time BlackBerry World Champion (2007-2012)
BlackBerry® Certified Support Specialist v5.0
BlackBerry® Certified System Administrator v5.0
Offline  
Old 05-17-2011, 09:10 AM   #6 (permalink)
Talking BlackBerry Encyclopedia
 
RadHaz75's Avatar
 
Join Date: Feb 2006
Location: Philadelphia
Model: 9650
OS: 6.0.0.524
PIN: BALL
Carrier: Verizon Wireless
Posts: 456
Post Thanks: 2
Thanked 4 Times in 4 Posts
Default Re: Stuck "Pending internal event processes"

also, if you have bas in multiple sites, this can occur if the latency is above 20ms. jboss is gets really cranky with any kind of latency. i had this exact issue and basically have to keep one of my bas servers off cause any time i start it tasks start building cause it cant keep up with the latency between the 2
__________________
Two months ago, I saw a provocative movie on cable TV. It was called The Net, with that girl from the bus.
Offline  
Old 05-17-2011, 11:33 AM   #7 (permalink)
Thumbs Must Hurt
 
Join Date: Jun 2008
Model: 9810
PIN: N/A
Carrier: AT&T
Posts: 130
Post Thanks: 0
Thanked 0 Times in 0 Posts
Default Re: Stuck "Pending internal event processes"

Yeah I got only 1 of the BAS turned on right now but it didn't do any good. My internal events finally went down from 655 to 652 but nothing is happening right now. I have a ticket opened with RIM on this issue.
Offline  
Old 05-17-2011, 01:32 PM   #8 (permalink)
Thumbs Must Hurt
 
Join Date: Jun 2008
Model: 9810
PIN: N/A
Carrier: AT&T
Posts: 130
Post Thanks: 0
Thanked 0 Times in 0 Posts
Default Re: Stuck "Pending internal event processes"

Support told me that my issue is a known issue and should be addressed in SP3 MR2. They did tell me to turn off all of my BES servers and only bring up 1. So I am doing that right now. Before I was just shutting down the BAS service.
Offline  
Old 05-17-2011, 01:47 PM   #9 (permalink)
Thumbs Must Hurt
 
Join Date: Jun 2008
Model: 9810
PIN: N/A
Carrier: AT&T
Posts: 130
Post Thanks: 0
Thanked 0 Times in 0 Posts
Default Re: Stuck "Pending internal event processes"

After the restart the tasks are still 652 but I see a TON of these in the log:
processFinishedTaskLocal jobTaskEntity not found for taskId = 56734
processFinishedTaskLocal Current task (57246) state is SUCCESS, but the dependency task is in a completed state, ignore the notification!
Offline  
Old 05-17-2011, 02:19 PM   #10 (permalink)
Thumbs Must Hurt
 
Join Date: Jun 2008
Model: 9810
PIN: N/A
Carrier: AT&T
Posts: 130
Post Thanks: 0
Thanked 0 Times in 0 Posts
Default Re: Stuck "Pending internal event processes"

It is almost like there are tasks in the DB that are not tied to a job and thus the BAS never clears them out.

I tried to search for these tasks but the BAS says that it cannot find them.

Last edited by Arkanian : 05-17-2011 at 02:25 PM.
Offline  
Old 05-17-2011, 02:41 PM   #11 (permalink)
Thumbs Must Hurt
 
Join Date: Jun 2008
Model: 9810
PIN: N/A
Carrier: AT&T
Posts: 130
Post Thanks: 0
Thanked 0 Times in 0 Posts
Default Re: Stuck "Pending internal event processes"

Well it finally went down to 643 so something is happening...but it is going very slowly.
Offline  
Old 05-17-2011, 04:43 PM   #12 (permalink)
Thumbs Must Hurt
 
Join Date: Jun 2008
Model: 9810
PIN: N/A
Carrier: AT&T
Posts: 130
Post Thanks: 0
Thanked 0 Times in 0 Posts
Default Re: Stuck "Pending internal event processes"

well it has been stuck at 643 for some time now. Support is still looking into it but so far nothing has worked.
Offline  
Old 05-18-2011, 04:18 PM   #13 (permalink)
Thumbs Must Hurt
 
Join Date: Jun 2008
Model: 9810
PIN: N/A
Carrier: AT&T
Posts: 130
Post Thanks: 0
Thanked 0 Times in 0 Posts
Default Re: Stuck "Pending internal event processes"

Going to be installing MR2 to see if that fixes the issue as MR2 was supposed to address this specific problem.
Offline  
Old 05-21-2011, 03:15 AM   #14 (permalink)
BlackBerry Extraordinaire
 
noname's Avatar
 
Join Date: Sep 2005
Location: Congested Islet of "Foreign Talents" (> 45% of workforce) - Singapore.
Model: Z10
OS: 10.0.0
PIN: NUKE(PAP)
Carrier: Singtel
Posts: 1,504
Post Thanks: 6
Thanked 9 Times in 9 Posts
Default Re: Stuck "Pending internal event processes"

Arkanian, no MR is gonna resolved your issue. RIM has issued a Critical Advisory for DT 1216175 which DarthBBerry has given you in KB26461.

You should not have upgraded your BES to 5.0 SP3 if you still have stuck pending internal events.

Source: http://docs.blackberry.com/en/admin/...1-5.0.3-US.pdf

Internal events are stucked due to environmental issues causing communications between the BAS instances to failed. The many BAS instances you have in your setup, the higher the chances of getting stuck jobs. I don't under why one needs to install so many BAS instances in a setup. For BAS pooling, just setup 2 in a pool and preferably all located in the same LAN and avoid using TCPPing communication mode. A BAS instance is like a web-server, and with the web admin console URL, you can access it for administration from 100 workstation's IE browsers if you like. So for redudancy, just install 2 instances, one to standby in case the other is not working.

One can still function with 1 BAS instance for a domain of 10 BES servers with many people access from multiple workstation to the web admin console address using IE browsers to admin these servers remotely.
__________________
Native but 4th class citizen of a nation governed by idiots who import congestions & contention.

Last edited by noname : 05-21-2011 at 03:17 AM.
Offline  
Old 05-21-2011, 03:20 AM   #15 (permalink)
BlackBerry Extraordinaire
 
noname's Avatar
 
Join Date: Sep 2005
Location: Congested Islet of "Foreign Talents" (> 45% of workforce) - Singapore.
Model: Z10
OS: 10.0.0
PIN: NUKE(PAP)
Carrier: Singtel
Posts: 1,504
Post Thanks: 6
Thanked 9 Times in 9 Posts
Default Re: Stuck "Pending internal event processes"

And btw, for those stuck jobs, you will require RIM's help to determine if they are in bad states and they will work out scripts to help you clear them. Also running the script will not immediately drop those jobs and most likley you will need to restart the BAS-AS service after applying the script and allow it to process all gracefully within 2-3 hours, depending on how powerful your server is.
__________________
Native but 4th class citizen of a nation governed by idiots who import congestions & contention.
Offline  
Closed Thread


Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On

Similar Threads for: Stuck "Pending internal event processes"
Thread Thread Starter Forum Replies Last Post
Messaging Agent Event ID 20000 devinewind22 BES Admin Corner 3 03-11-2009 10:47 AM
event log warnings kburd CH BES Admin Corner 1 02-22-2006 05:25 PM
Process [application name] queue overflow; oldest event dropped KaTTdaddY Developer Forum 2 02-21-2006 12:26 PM
MsiInstaller error in event log jeidson BES Admin Corner 12 08-19-2005 02:10 PM
BES not working cybercity BES Admin Corner 2 08-04-2005 10:18 AM





Copyright © 2004-2014 BlackBerryForums.com.
The names RIM © and BlackBerry © are registered Trademarks of BlackBerry Inc.