BlackBerry Forums Support Community               

Closed Thread
 
LinkBack Thread Tools
Old 04-20-2007, 05:28 AM   #1 (permalink)
Grumpy Moderator
 
NJBlackBerry's Avatar
 
Join Date: Aug 2004
Location: Somewhere in the swamps of Jersey
Model: iPh6
Carrier: AT&T
Posts: 27,813
Post Thanks: 33
Thanked 442 Times in 382 Posts
Default RIM's Statement on BlackBerry Outage

Please Login to Remove!

RIM's in-depth diagnostic analysis of the service interruption that occurred in North America on Tuesday night is progressing well and RIM will continue to provide further information as it's available. RIM's first priority during any service interruption is always to restore service and then establish, monitor and maintain stability. Proper analysis can take several days or longer and RIM's commitment is to provide the most accurate and complete information possible in such situations.

RIM is pleased to report that normal conditions returned on Wednesday and the BlackBerry service continues to operate normally today.

RIM has been able to definitively rule out security and capacity issues as a root cause. Further, RIM has confirmed that the incident was not caused by any hardware failure or core software infrastructure.

RIM has determined that the incident was triggered by the introduction of a new, non-critical system routine that was designed to provide better optimization of the system's cache. The system routine was expected to be non-impacting with respect to the real-time operation of the BlackBerry infrastructure, but the pre-testing of the system routine proved to be insufficient.

The new system routine produced an unexpected impact and triggered a compounding series of interaction errors between the system's operational database and cache. After isolating the resulting database problem and unsuccessfully attempting to correct it, RIM began its failover process to a backup system.

Although the backup system and failover process had been repeatedly and successfully tested previously, the failover process did not fully perform to RIM's expectations in this situation and therefore caused further delay in restoring service and processing the resulting message queue.

RIM apologizes to customers for inconvenience resulting from the service interruption. RIM's root cause analysis and system enhancement process with respect to this incident is ongoing and RIM has already identified certain aspects of its testing, monitoring and recovery processes that will be enhanced as a result of the incident and in order to prevent recurrence.
Offline  
Old 04-20-2007, 06:43 AM   #2 (permalink)
BlackBerry Master
 
brucejr's Avatar
 
Join Date: Nov 2006
Model: 666
Carrier: ?
Posts: 3,645
Post Thanks: 0
Thanked 0 Times in 0 Posts
Default

Sometimes programs work like their supposed to, sometimes not. Thanks for the info NJBB!
__________________
The Purrfect 4
Offline  
Old 04-20-2007, 06:59 AM   #3 (permalink)
Grumpy Moderator
 
NJBlackBerry's Avatar
 
Join Date: Aug 2004
Location: Somewhere in the swamps of Jersey
Model: iPh6
Carrier: AT&T
Posts: 27,813
Post Thanks: 33
Thanked 442 Times in 382 Posts
Default

Absolutely - sh*t happens. It's how you respond that defines the incident. RIM clammed up, and I'm sure their official answer to that is "Proper analysis can take several days or longer and RIM's commitment is to provide the most accurate and complete information possible in such situations."

In this case, they behaved like engineers and not business people. While technology failing is an everyday occurance, not owning up to the problem quickly and giving interim updates was not the best way to handle this. IMHO.
Offline  
Old 04-20-2007, 07:10 AM   #4 (permalink)
EPIC MOD
 
ezrunner's Avatar
 
Join Date: Mar 2006
Location: Virginia Beach
Model: ZED10
OS: DOS 3.1
PIN: INK STICK
Carrier: Tmobile
Posts: 12,214
Post Thanks: 9
Thanked 6 Times in 6 Posts
Default

Thanks for the info NJ!!!
__________________
ZED 10
Offline  
Old 04-20-2007, 07:46 AM   #5 (permalink)
BlackBerry Extraordinaire
 
whsbuss's Avatar
 
Join Date: Feb 2005
Location: SE PA.
Model: 9800
OS: 4.6.0.304
Carrier: AT&T
Posts: 2,791
Post Thanks: 0
Thanked 0 Times in 0 Posts
Default

Any bets it was the BIS 2.3 upgrade that caused it?
Offline  
Old 04-20-2007, 09:46 AM   #6 (permalink)
Talking BlackBerry Encyclopedia
 
tprime's Avatar
 
Join Date: Apr 2006
Location: Jamaica,Queens,NYC
Model: 9700
OS: 6.0.0.448
Carrier: Tmobile
Posts: 253
Post Thanks: 0
Thanked 0 Times in 0 Posts
Default

Hopefully they'll learn from this mistake as well as the resulting feedback, backlash and negative publicity.
__________________
Tprime
7100t,8700g,8100,8120 now<9700 Bold and playbook
Lets Go Mets!
Offline  
Old 04-20-2007, 09:53 AM   #7 (permalink)
Talking BlackBerry Encyclopedia
 
ZeroKool's Avatar
 
Join Date: Aug 2006
Location: stl, MO
Model: 8330
PIN: nope
Carrier: verizon
Posts: 314
Post Thanks: 2
Thanked 0 Times in 0 Posts
Default

Did RIM send this in email to you or this info off the site?
__________________
Me likey BlackBerry
Offline  
Old 04-20-2007, 09:59 AM   #8 (permalink)
Talking BlackBerry Encyclopedia
 
Michbb's Avatar
 
Join Date: Nov 2005
Location: MN
Model: 8100
PIN: 23FAA477
Carrier: Tmo
Posts: 318
Post Thanks: 0
Thanked 0 Times in 0 Posts
Default

it was posted in the NY Times

Last edited by Michbb : 04-20-2007 at 10:01 AM.
Offline  
Old 04-20-2007, 09:59 AM   #9 (permalink)
Grumpy Moderator
 
NJBlackBerry's Avatar
 
Join Date: Aug 2004
Location: Somewhere in the swamps of Jersey
Model: iPh6
Carrier: AT&T
Posts: 27,813
Post Thanks: 33
Thanked 442 Times in 382 Posts
Default

I posted this from an article in The Wall St. Journal. I haven't heard squat from RIM directly.
Offline  
Old 04-20-2007, 10:07 AM   #10 (permalink)
BlackBerry Extraordinaire
 
stonent's Avatar
 
Join Date: Oct 2004
Location: Fort Worth, Texas
Model: 8220
PIN: FFFFFFFF
Carrier: T-Mobile
Posts: 1,734
Post Thanks: 0
Thanked 0 Times in 0 Posts
Default

I used to work with an IBM retiree who was a service technician for IBM for nearly 30 years. And in his 30 years he came up with the best explanation for computer problems.

"Fixing computers is an art, not a science."

As I assume many people here are in some sort of environment where they support things I'm sure there are many times where you can fix something but not really know what happened. I know it happens to me all the time.

Customer: "What caused it?"
Me: "I don't really know."
Customer: "Can you fix it?"
Me: "Yes."
__________________
Stonent

Prior: BES: 7510, 7520, 7290, 7230, 7130, 8220 (No Data Plan), Droid Eris
Post your question on the forum, don't PM me.

Last edited by stonent : 04-20-2007 at 10:09 AM.
Offline  
Old 04-20-2007, 10:30 AM   #11 (permalink)
Talking BlackBerry Encyclopedia
 
ZeroKool's Avatar
 
Join Date: Aug 2006
Location: stl, MO
Model: 8330
PIN: nope
Carrier: verizon
Posts: 314
Post Thanks: 2
Thanked 0 Times in 0 Posts
Default

I got an email from RIM on Wednesday...But nothing since then.

On April 17, 2007, at approximately 20:15 EDT, the BlackBerry Network Operations Center (NOC) started investigating monitoring alerts in regards to issues with BlackBerry service in the Americas. BlackBerry subscribers in the Americas were unable to receive messages and received an "x" when sending messages. Subscribers were unable to use BlackBerry Email Roaming to or from the Americas.
Initial investigations revealed an issue with the BlackBerry® Infrastructure. Subsequent troubleshooting efforts were not immediately successful in restoring service. At approximately 02:00 on April 18, in an effort to restore service, the components and services for the BlackBerry Infrastructure for the Americas were restarted. Since approximately 05:45, the service has been operating near expected traffic levels. Some subscribers may continue to see delays until message queues are cleared. The BlackBerry Support team is continuing to investigate reports of any remaining issues.
The root cause of this issue is under investigation.
Please note that the ability for subscribers to place telephone calls from voice-enabled devices was not affected during this event.
Thank you for your continued support, and we apologize for any inconvenience.
__________________
Me likey BlackBerry
Offline  
Old 04-20-2007, 01:26 PM   #12 (permalink)
Knows Where the Search Button Is
 
FlyerNut's Avatar
 
Join Date: Feb 2007
Location: Philadelphia
Model: 8300
Carrier: Cingular
Posts: 26
Post Thanks: 0
Thanked 0 Times in 0 Posts
Default

They did a really bad job of keeping the public aware, and hopefully someone will get a slap on not one, but both wrists.

That being said, I didn't expect much more from them, with my limited experience working with their support teams, it seems they prefer a heavy handed approach anyway. Oh, phone's not synching? Wipe it, delete and recreate the user is typically the answer I get for just about everything.
Offline  
Old 04-20-2007, 02:34 PM   #13 (permalink)
Retired BlackBerryForums.com Moderator
 
d_fisher's Avatar
 
Join Date: Oct 2005
Location: Columbus, OH
Model: 9700
OS: SID 6.7
Carrier: AT&T
Posts: 4,455
Post Thanks: 0
Thanked 2 Times in 1 Post
Default

Quote:
Originally Posted by FlyerNut View Post
They did a really bad job of keeping the public aware, and hopefully someone will get a slap on not one, but both wrists.
Not sure I agree. The outage took place after business hours (in most places) and the issue was resolved the next morning. How much notification does the public really need at 2:00 or 3:00 AM?
__________________
Doug

Remember, please try searching first!

Need a screenshot? ... Like JavaLoader?
Try using BBscreen .....Use JL_Cmder!
or BBScreenShooter!

[SIGPIC][/SIGPIC]
Offline  
Old 04-20-2007, 02:39 PM   #14 (permalink)
Grumpy Moderator
 
NJBlackBerry's Avatar
 
Join Date: Aug 2004
Location: Somewhere in the swamps of Jersey
Model: iPh6
Carrier: AT&T
Posts: 27,813
Post Thanks: 33
Thanked 442 Times in 382 Posts
Default

Personally I think they could have done much better. RIM has a market cap of $25 billion dollars, with 8 million customers. They HAVE to be ready to handle an emergency. Not changing their status phone line for 12 hours is not the right way to handle it. They had to be a little more forthcoming... Their customers and owners should expect that. They have one line of business - they are not a conglomerate. They had to own up quicker.

Again, just my opinion
Offline  
Old 04-20-2007, 02:47 PM   #15 (permalink)
BlackBerry God
 
penguin3107's Avatar
 
Join Date: Jan 2005
Model: iOS 5
Carrier: VZW
Posts: 11,701
Post Thanks: 1
Thanked 237 Times in 219 Posts
Default

Quote:
Originally Posted by d_fisher View Post
The outage took place after business hours (in most places)
This is 2007. There is no such thing as 'business hours' anymore, and hasn't for a while... especially for remote users who rely on the Blackberry and other mobile tools for their respective professions.

Thousands of users have 24/7 jobs nowadays. 2:00am - 3:00am is "normal business hours" for many people.
Doctors, IT admins, Lawyers, Government Workers, Security Personnel, project managers, etc, etc, etc... I'm sure there's a large group of people who would have loved to hear from RIM as to why the hell they were holding a brick in their hands for 12 hours.

And as I said in another thread... for a company that revolves around communication technology, they sure did a crap job in communicating to their own customer base.
__________________
BCSA
BES 5.0.3 MR4 :-: Exchange 2007 SP3 RU3
http://port3101.org

Last edited by penguin3107 : 04-20-2007 at 02:48 PM.
Offline  
Old 04-20-2007, 03:08 PM   #16 (permalink)
Knows Where the Search Button Is
 
Join Date: Apr 2006
Model: 9700
Carrier: Rogers
Posts: 30
Post Thanks: 0
Thanked 0 Times in 0 Posts
Default

This reminds me of scene from Terminator when Dyson's computer becomes fully aware and takes over. You guys need to critically examine how you use technology. The world lived before email and will continue to live without. Turn off your machines and see have it affects you guys. RIM wants to take over your lives and has.
Offline  
Old 04-20-2007, 03:36 PM   #17 (permalink)
BlackBerry God
 
penguin3107's Avatar
 
Join Date: Jan 2005
Model: iOS 5
Carrier: VZW
Posts: 11,701
Post Thanks: 1
Thanked 237 Times in 219 Posts
Default

Quote:
Originally Posted by eastend View Post
This reminds me of scene from Terminator when Dyson's computer becomes fully aware and takes over. You guys need to critically examine how you use technology. The world lived before email and will continue to live without. Turn off your machines and see have it affects you guys. RIM wants to take over your lives and has.
I realize you're just trying to make light of the situation, but let's be real... technology has vastly changed the world we live in. Yes, there are professionals who RELY on technology. This is not a bad, or uncommon thing. Just as you take for granted that the sun will rise, and your car will start, and birds will chirp... modern day professionals take for granted that their wireless carrier will provide uninterrupted service and their e-mail provider will continue routing mail as usual.

Yes, the world went on before these technologies ever existed... but you're incredibly naive if you think its possible to go back to those days without serious consequence for many people. See what happens when the network and/or phone system fails in your local hospital, or your local airport, or law enforcement agency. It's a BIG DEAL.

Your comment about RIM wanting to take over people's lives is just plain silly. Maybe that was your intention, I don't know.

No matter how you look at it, any major technical problem or service outage is a far bigger deal that you'd like to accept.
__________________
BCSA
BES 5.0.3 MR4 :-: Exchange 2007 SP3 RU3
http://port3101.org
Offline  
Old 04-20-2007, 03:56 PM   #18 (permalink)
BlackBerry Extraordinaire
 
snowskier79's Avatar
 
Join Date: Apr 2006
Location: Houston, TX
Model: Curve
PIN: No thanks, yo tienen una pluma.
Carrier: AT&T
Posts: 1,759
Post Thanks: 0
Thanked 0 Times in 0 Posts
Default

Quote:
Originally Posted by NJBlackBerry View Post
Absolutely - sh*t happens. It's how you respond that defines the incident. RIM clammed up, and I'm sure their official answer to that is "Proper analysis can take several days or longer and RIM's commitment is to provide the most accurate and complete information possible in such situations."

In this case, they behaved like engineers and not business people. While technology failing is an everyday occurance, not owning up to the problem quickly and giving interim updates was not the best way to handle this. IMHO.
Completely agree NJB!
Offline  
Old 04-20-2007, 05:00 PM   #19 (permalink)
Appleinator
 
Dubdub's Avatar
 
Join Date: Nov 2005
Location: New Hampshire
Model: App5
OS: AJBR549
PIN: Ask
Carrier: ATT & Verizon
Posts: 20,035
Post Thanks: 54
Thanked 782 Times in 742 Posts
Default

Quote:
Originally Posted by d_fisher View Post
Not sure I agree. The outage took place after business hours (in most places) and the issue was resolved the next morning. How much notification does the public really need at 2:00 or 3:00 AM?
I happen to agree. There was info out there Wednesday morning and most of us here also knew about it not long after it happened. I assume some government users would have been upset since they need to be connected 24/7, but even there they have alternate means - i.e. you pick up the phone and drop a dime (well 35 cents now I guess).
Offline  
Old 04-20-2007, 05:02 PM   #20 (permalink)
Appleinator
 
Dubdub's Avatar
 
Join Date: Nov 2005
Location: New Hampshire
Model: App5
OS: AJBR549
PIN: Ask
Carrier: ATT & Verizon
Posts: 20,035
Post Thanks: 54
Thanked 782 Times in 742 Posts
Default

Quote:
Originally Posted by eastend View Post
This reminds me of scene from Terminator when Dyson's computer becomes fully aware and takes over. You guys need to critically examine how you use technology. The world lived before email and will continue to live without. Turn off your machines and see have it affects you guys. RIM wants to take over your lives and has.
Cell phones, computers and blackberries are like spouses and significant others. You can't live with them and you can't live without them.
Offline  
Closed Thread


Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On





Copyright © 2004-2014 BlackBerryForums.com.
The names RIM © and BlackBerry © are registered Trademarks of BlackBerry Inc.