[cschrage]

I totally agree with Spydertech...wait to all the bugs are ironed out.
BTW - this is my first issue with a BES SP...going all the way back to V3.6

Originally Posted by Spydertech

If I had another server to upgrade, I wouldn't do it. Not until I figure out what went wrong.

I don't know if it was something I did, something I have configured incorrectly, or just plain dumb luck (bad) at this point.

I know I have performed many other updates and upgrades, and never had a problem until now....

Then again, I never had a car accident, until my first screw up...;)

Spydertech

[merlin]

Does anyone getting these errors use BackupExec (or similar) and a remote agent to backup their BES directories -the folders on the BES server, not the databases on a seperate SQL box?

I am still getting these errors everyday, BUT ONLY if I schedule my Backup server to backup the BES server(s) using BackupExec...

If I get backupexec to skip the folders on the BES server then I don't get the errors - so at the moment I am only backing up the SQL server and nothing on the BES'es...

Merlin

[BESAdminOSI]

I am using Backup Exec 12d to perform backups of the file system on my BES server. The database is backed up via NetApp SAN. I too did not have this problem until I applied SP6 last week. It's fine except for the fact that I get over 20K emails a night. I have removed all email settings now so I don't get alerted until there is a fix for this. Was not able to find anything on RIM's support site yet...

[jibi]

...has anyone contacted RIM yet? I've been running SP6 in test for months now and have not noticed such an error. Still waiting to push into production, though. Are there any surrounding log entries? Going by the event logs and/or alert messages isn't a good way to locate the root of a lot of issues with BES, to be honest.

[bort13]

Called RIMM regarding this issue: RIMM said to a) shut off the BB Alerter service and b) wait for the next maintenance release.

Edit: it might be only BackupExec release 11 and later. Also, they instructed us to reboot if the high CPU condition happens.

Also \/ \/ \/ I open a ticket online and answer all the questions then wait thirty minutes and call 'em.

[cschrage]

My ticket is still open...no response yet. Love being a TX1 customer!

[knottyrope]

Originally Posted by merlin

Does anyone getting these errors use BackupExec (or similar) and a remote agent to backup their BES directories -the folders on the BES server, not the databases on a seperate SQL box?

I am still getting these errors everyday, BUT ONLY if I schedule my Backup server to backup the BES server(s) using BackupExec...

If I get backupexec to skip the folders on the BES server then I don't get the errors - so at the moment I am only backing up the SQL server and nothing on the BES'es...

Merlin

I use backupexec 10d with no issues every week night. I do have the SQL connector license for it.

[Spydertech]

OK - here is some more, probably useless, info...

We do use Backup Exec Remote Agent with the SQL connector.
We do a full backup every night. We exclude the MSSQL directory for file backup - we use the SQL Option to back up the databases. I also have a scheduled database backup to a flat file occurring before Backup Exec kicks off.

But, none of that has changed, yet the errors have stopped for the last couple of days.

The only change I made was to have the Blackberry Alert Service start using the Local System Account (with Interact with Desktop enabled).

The service has been running since the change Monday, and I have no errors anymore.

I don't know if that fixed it or not. I know you are supposed to use the BES Admin user account for the services, but everything seems to be fine. The logs are populating just like they did before I installed SP6...

And, no, I never contacted RIM - I find better answers here.

Spydertech

[jimglore]

I am working with RIM on this issue as I am experiencing the same thing after upgrading to SP6 from 4.1.4 on Friday. The 100% CPU is being by the BBAgent.exe, BBController.exe and services.exe (this service should always be at 0% unless processing a stop/start of a service. I have seen that once I stop/start the BBController service the CPU for the entire server returns to normal.

I am have uploaded my MS app log and today's BB log to RIM. I am still on hold with them and will provide further details once I have them.

[cschrage]

RIM support got back to me this morning.

The issue you are experiencing is on that we are well aware of and one that has been escalated to our development team. At this time there is no resolution for the issue and the only workaround would be to disable the BlackBerry Alert Service. The issue should be resolved in MR1 of 4.1.6 but we do not have an estimated date for this release.

[Takkl]

I also have a case with RIM support...but have not been helpful in my situation. Everytime backup kicks in the events flood the apps logs and spike up the cpu, despite lowering the diag value. Even though BES is still functional, I do not feel comfortable with the high cpu (around 90%) and constant logged events.

Here's my temporary fix:
So far I've isolated the issue to the LOGS. I've excluded the LOGS path from backup (Backup Exec 10d) and issue has not come back. I've tested a few times with backup.

[jimglore]

I got a reply back this morning from RIM, but first what I have found since speaking with them yesterday. My issue is related specifically to the Controller service and the CTRL log. On a normal day our CTRL log is about 30k, it verfies OS, uptime, processes running etc, then runs health checks periodically throughout the day. Well on the two days I've had the CPU peg at 100% a health check runs and the following entry is logged:

[20000] (07/29 07:43:43.113):{0xAA4} AnalyzeEntry: Unable to retrieve event id

My CTRL log from Tuesday was 1.3GB. The above entry started at 7:43am and stopped when we rebooted at 11:25am. In the almost 4 hours the CPU was pegged this entry was logged over 17.5 million times. Same thing happened yesterday and that is what prompted my call to RIM. During our call we stopped the services according to their best practices. As soon as we stopped the Controller service CPU dropped to 0%, unfortunately we are not running the Alerting service so part of RIM's recommendation does not apply to us, however we lowered the logging level on the Controller Agent.

My email from RIM this AM shows that these issues are being looked at by the Development team under an SDR (Windows for us) and seperate SDR's exist for Domino and GW. They recommend the following if this happens:

"Our recommendation until we have a resolution is to turn the BlackBerry Controller and Alert log file size to 10Mb using the BlackBerry Server Configuration tool on the Logging tab.

We also recommend to turn off any BESAlert settings, this is so you do not receive multitude of error/warning messages.

While this issue is currently under investigation, your BlackBerry users should not notice any message flow issues."

So it looks like from what I have heard this will be part of the next MR, but unfortunately no one can tell me how soon or late that will be released.

[SAlbright]

I have turn off BESAlert and am not logging Controller and BESAlert - which has solved the massive logs problems BUT I have just discovered that Lookup of PIN numbers is not working. I'm assuming that the cause is BESAlert not running - can anyone verify this for me ? RIM Support online site is not available at this time (getting wierd error messages). Thanks !

REVISED 08/08/08 - I need to reset the "PIM SYNC Global Field Mapping" and everything is working OK on PINs

[wistowg]

Message from RIM this morning is that rather than issue a MR to fix they're going to wait till 4.1.7 - no timetable though. Recommended fix in the interim is to limit the size of the Agent and Controller logs to 10MB or so (Start > Programs > BES > BlackBerry Server Configuration > Logging).

[dbonline]

Here is the RIM response to my case for our Domino environment:

You seem to be experiencing an issue that some customers who have upgraded to BlackBerry Enterprise Server Version 4.1.6 have experienced where the Controller and Alert logs are growing exponentially. This is a previously reported issue and SDR200822 has been created to address this issue. It is expected to be resolved in Maintenance Release 1 for Service Pack 6.

Until Maintenance Release 1 is released we are asking customers who are experiencing this issue to perform the following:

1. Either turn off or turn down the Controller logging level.
2. Disable the BlackBerry Alert Service.

[cbnsingram]

Has anyone heard anything new since August 20th?

[Spydertech]

Nothing I've heard. And the Maintenance Release is still not available.
So for now, I have no controller log, and BESAlert Service is shut down.

[Arkanian]

**Bump**

We are about to upgrade our servers from 4.1.4 to 4.1.6 and was wanting to know if MR1 fixed this problem.

[Keyscan]

As per the release notes of BES 4.1.6 MR1, they fixed the logging problem that was discussed in this thread.

[dougb]

Sorry I landed up posting this in the wrong thread, but I think it should go here!

We have a similar configuration (2 BES in the same domain with external SQL Server) and have just upgraded both servers to 4.1.6 MR3. The morning after the upgrade, one of the BES servers is reporting exactly the saem problem which goes away after a reboot, only to re-appear again the next morning.

The only slight difference with us is that our controller log files are not unusually big so that clearly isn't the issue here.

So the real problem clearly hasn't been fixed by MR1.

Has anyone else got any news, or ideas on this one? We are not keen on the idea of having to turn off BESAlert.

Thanks,

DougB