[Arkanian]

BES 5 SP1 MR1 Domino

I am having an issue right now where any new user will not be assigned the IT Policy. If I click on the Policies tab I get the following:
Reconciliation is still in progress. Information is not yet available. Please wait, and try again later.

Has anyone seen this before?

[Arkanian]

ok, we restared the SYNC and POL services to no luck, reboot the server to no luck, and failed over to our 2nd BES server with no luck. Not sure what is going on.

[Arkanian]

Well I found out that this article talkes about it:
View Document

Problem is that KB20668 has this fix for the issue but I can't find that KB Article. Does anyone know where this KB is?

[Arkanian]

So support told me that 1 job is hung and causing the backup. Supposed to be creating a script for me to run to clear them out.

[Arkanian]

Does anyone know where these jobs live? Database? If so what table?

[noname]

Software Configurations and IT policy are managed by a new feature in BES 5.0 called "Jobs" and "Job Tasks". This is something which I feel is badly designed and a good example is your case - stuck jobs. There should be a mean to find out what jobs are stuck and give us the option to purge and release them - just like a print queue job in a computer.

Anyway, for all benefits, please refer to snip below and the link to the online version of the Admin Guide that shows it.

[Quote]
When you create a software configuration and assign it to user accounts, change a software configuration that you assigned to user accounts, or assign or change an IT policy, the BlackBerry® Administration Service creates jobs to deliver the resulting objects or settings to BlackBerry devices. You can change the default settings that control how the BlackBerry Administration Service creates jobs and delivers job tasks to BlackBerry devices. You can also change the default settings that the BlackBerry Administration Service uses to deliver IT policies, BlackBerry Java® Applications, BlackBerry® Device Software, and standard application settings to BlackBerry devices.
[Unquote]

Link: Managing the default distribution settings for jobs

[Arkanian]

Thanks. Support is supposed to give me a script here in a few hours that is going to clean my queue. Looks like the only thing that we can do per your document is just to change the default timeouts. So what is your settings set to on this?

[noname]

I just turn on throttling for all IT policy tasks in jobs by selecting "Enabled to reduce load on system" and tweaked the related settings as I go along.

What did RIM says about your ITAdminQueue table? I guess it's the asynchronous background processing nature of the BAS that could have caused duplicate/stale records amd hence causing the jobs to stuck? It will be interesting to see how they explain to you. Do you mind sharing?

[Arkanian]

So far RIM support is not being very helpful. Supposed to call this morning.

[Arkanian]

ok, after looking at my standby server I see the following every hour:
(05/04 03:50:12:761):{WorkManager(2)-46} [com.rim.bes.bas.reconciliationmanager.Reconciliati onManagerBean] [INFO] [BBAS-1009] {u=SystemUser, t=88537} Reconciliation is running
(05/04 03:50:12:807):{WorkManager(2)-46} [com.rim.bes.bas.reconciliationmanager.Reconciliati onManagerBean] [WARN] [BBAS-2029] {u=SystemUser, t=88537} _processPendingReconciliation cannot find entity for key 9908
(05/04 03:50:12:870):{WorkManager(2)-46} [com.rim.bes.bas.reconciliationmanager.Reconciliati onManagerBean] [WARN] [BBAS-2029] {u=SystemUser, t=88537} _doProcessPendingReconciliation internal reconciliation exception while executing task 12479, com.rim.bes.bas.util.BASCouldNotCompleteRequestRol lbackException: _processPendingReconciliation cannot find entity for key 9908


Looks like this task could be the one that is stuck. Going to see what support thinks about this.

[Arkanian]

ok...RIM blew me off when I sent this to them. They are now wanting me to replace the Device and Vendor XML files. Not sure what this has to do with my job being stuck. My support guy said he won't be in the office until tomorrow now.

[Arkanian]

So I replaced my XMLs like they said and sent the log files to them. Made it clear that I needed to do this before they went any further. Is it so hard for them to just give me a script to clear the pending events? They keep growing. I don't care the root cause I just need it fixed and then we can go back and look at the root cause.

[Arkanian]

So support finally gave me a query to run to see if there is a stuck job and low and behold it shows an issue. So now I wait for there response.

[wistowg]

Had the same issue a while back and got the following scripts from RIM

/**Identify stuck job**/
SELECT jobs.*
FROM BASJobs jobs, BASJobTasks tasks
WHERE jobs.JobId = tasks.jobId
AND tasks.state = 0
AND tasks.objectTypeId = 7
AND tasks.LastStateChangeDate < dateadd (day, -2, getDate ())
AND (1 = (SELECT count ( * )
FROM basJobTasks tasks2
WHERE tasks2.JobId = jobs.JobId))

Once you have a job ID of the stuck job you can run the below script, replace XXXX with the job ID number obtained from the previous script.


DECLARE @JobId int
/*----------------------------------------------------------------------------*/
/* Edit this section only */
/*----------------------------------------------------------------------------*/
SET @JobId = XXXX -- the Job ID we wish to update
/*----------------------------------------------------------------------------*/
/* End of editable section */
/*----------------------------------------------------------------------------*/
UPDATE BASJobs SET State = 4 WHERE JobId = @JobId
UPDATE BASJobTasks SET State = 6, LastStateChangeDate = getDate() WHERE JobId = @JobId
GO

[Arkanian]

yep that is the query that they sent me but I don't have any tasks with state 0. So I am not sure what to do. RIM is looking at it.

[Arkanian]

So RIM is now telling me that they think it is a configuration issue with the BAS. Not sure why it would just stop processing jobs. We have a primary and standby server so maybe the BAS on the other server is hanging up the other one not sure. Very strange. Going to shutdown the standby server and restart the BAS service on the primary server to see if that makes a difference.

[Arkanian]

well all is fixed. We had the BAS Pool configured incorrectly. So we are good now.

[TargetIT]

Thanks for the updates. Anything in general you can tell us about how it was incorrectly configured?

[Arkanian]

Basically we had the BAS Pool configured but the main BAS service on the primary server was pointed to the actual server name and not the DNS Round Robin name for the BAS Pool. So we had it point to the BAS Pool DNS name and then restarted the BAS service on both the primary and standby servers. Once we did that the BAS processed all of the pending jobs and life was good.

Just surprised it took support this long to have me run the query to see if there was a hung job to determine that the BAS was configured incorrectly.