BlackBerry Forums Support Community               

Closed Thread
 
LinkBack Thread Tools
Old 02-11-2008, 11:06 PM   #1 (permalink)
BlackBerry God
 
jibi's Avatar
 
Join Date: Oct 2004
Location: Jibi's Secret Place
Model: 8900
OS: 4.6.1.174
Carrier: AT&T
Posts: 11,310
Post Thanks: 0
Thanked 1 Time in 1 Post
Default BoxTone, Notifications, and Rapid Response

Please Login to Remove!

I was out of the office today when the outage happened, but I'm happy to say that very few people in our IT organization were confused as to how to troubleshoot the issue and how to utilize the tools made available to them. It's refreshing to see the implementation of tools and processes, even if at a bare minimum at this point in time, make an impact when something like today's outage comes up.

At 3:22:20 PM ET, we received our first BoxTone email notification letting us know one of our servers was in an unavailable status and had lost it's SRP connection. Within a minute, we were alerted for the rest of the BES servers we have in the United States (we do not have the international sites configured for notifications at this time). The outage was reported by RIM to have started at 3:20 PM ET. Not too shabby of a turnaround time.

Ten minutes after we first received notifications from our BoxTone connector, we received our first RIM email notification for the outage. This outage notification took over 5 minutes to deliver from RIM to our infrastructure. Over an hour after our initial BoxTone alert, we received our first notification from AT&T (ATTOM), which took nearly 20 minutes to deliver to our system after it was sent from AT&T.

Granted, a monitoring solution can do nothing to fix an outage, but it can certainly reduce the amount of time spent troubleshooting end-user issues when outages happen. This determination period is vital when dealing with thousands and thousands of users. Bulletins can be posted, internal notifications can be sent, Help Desk personnel can start notifying rather than troubleshooting ...all within minutes of an outage developing and quite often much more rapid than official vendor acknowledgement and notification. During these important minutes ticking away, vendors are typically in the process of drafting a response, gaining approvals to send the message to a select few hundred thousands customers, and straining their own mail queues; meanwhile the monitoring system is doing its thing - gathering real-time statistics, aggregating the data, sending alerts to internal technology groups, and helping deduce the outage's scope of impact in your own environment.

Here's what our environment looked like following the reconnection of SRP when messages were still increasing in the pending queues. Quite astonishing.

BES: North America
SABES: South America
PACBES: Asia-Pacific
EUBES: Europe

__________________
In the beginning the Universe was created. This has made a lot of people very angry and is widely regarded as a bad move.

Last edited by jibi : 02-11-2008 at 11:33 PM.
Offline  
Old 02-12-2008, 12:52 PM   #2 (permalink)
Knows Where the Search Button Is
 
Join Date: Feb 2006
Model: 8100t
Carrier: tmobile
Posts: 45
Post Thanks: 0
Thanked 0 Times in 0 Posts
Default

Great example.
Offline  
Old 02-13-2008, 04:03 PM   #3 (permalink)
New Member
 
Join Date: Feb 2008
Model: 8300
PIN: N/A
Carrier: rogers
Posts: 2
Post Thanks: 0
Thanked 0 Times in 0 Posts
Thumbs up

We are using Zenprise to monitor our BlackBerry infrastructure and I have to say that the software is amazing. I was alerted immediately when the outage started. When I called Rogers, they did not even know that the RIM network was down (I guess RIM hadn't notified them yet). I agree with JIBI that using monitoring software to identify outages means that we can proactively reach out to our user community before they end up flooding us with calls.

I've included a screenshot of the Zenprise console and one alert message. We were able to see pending messages growing for critical users, as well as immediately identify the root cause to be connectivity problems with the SRP network.


Attached Images
File Type: jpg ZenpriseDashboard.JPG (63.3 KB, 5 views)

Last edited by mingjing : 02-19-2008 at 04:19 PM.
Offline  
Old 02-13-2008, 10:27 PM   #4 (permalink)
BlackBerry God
 
jibi's Avatar
 
Join Date: Oct 2004
Location: Jibi's Secret Place
Model: 8900
OS: 4.6.1.174
Carrier: AT&T
Posts: 11,310
Post Thanks: 0
Thanked 1 Time in 1 Post
Default

Do you happen to have a screenshot of Zenprise in your own environment rather than the Zenprise test lab screenshot that they mass-mailed this morning? Just curious.
__________________
In the beginning the Universe was created. This has made a lot of people very angry and is widely regarded as a bad move.
Offline  
Old 02-14-2008, 08:02 AM   #5 (permalink)
New Member
 
Join Date: Feb 2008
Model: 8300
PIN: N/A
Carrier: rogers
Posts: 2
Post Thanks: 0
Thanked 0 Times in 0 Posts
Default

I have one alert message screenshot and one zenprise console issue warning screenshot.
Attached Images
File Type: jpg zenpriseConsole.JPG (66.4 KB, 52 views)
Offline  
Old 03-17-2008, 06:45 AM   #6 (permalink)
Knows Where the Search Button Is
 
mgaffney's Avatar
 
Join Date: Apr 2006
Location: Baltimore, Maryland
Model: 9000
Carrier: AT&T
Posts: 16
Post Thanks: 0
Thanked 0 Times in 0 Posts
Default

I posted an article about the BoxTone Dashboard during the BlackBerry outage on my blog. In the article, I use jibi's screen shot and compare it with mingjing's screen shot of the Zenprise User Dashboard. Give it a read if you get a chance:

The BoxTone Dashboard and the BlackBerry Outage

And just so you know, I work for BoxTone.
__________________
Michael Gaffney
Principal Software Architect
BoxTone
michael.gaffney@boxtone.com
Offline  
Old 03-20-2008, 01:49 AM   #7 (permalink)
Knows Where the Search Button Is
 
Join Date: Feb 2007
Location: Columbus
Model: 7130e
Carrier: Verizon
Posts: 25
Post Thanks: 0
Thanked 0 Times in 0 Posts
Default

mingjing, Zenprise 3.3 gets even better... Tons more visibility and alerting on user level issues. You will want to tune the alert filters somewhat though. Since 3.3 was installed, it even caught some hiccups with some of the international providers that was out of trend.

Not a Zenprise rep, but a happy user of it.
Offline  
Old 03-24-2008, 09:11 AM   #8 (permalink)
New Member
 
Join Date: Mar 2008
Model: 9630
PIN: N/A
Carrier: VZ
Posts: 12
Post Thanks: 0
Thanked 0 Times in 0 Posts
Default

highfall, screenshots or didn't happen
Offline  
Closed Thread


Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On





Copyright 2004-2014 BlackBerryForums.com.
The names RIM and BlackBerry are registered Trademarks of BlackBerry Inc.