Looks like I'm back on online...it's been 30 min with no hiccups...

a company MANY times the size of RIM (microsoft) has had their Microsoft.com and MSN.com pages go down for hours a time. I am sure RIM has redundancy in place, several layers i would guess, but that doesnt mean it will solve every problem or prevent it from happening. We all hope it never happens again, but who knows?

Backup in Birmingham, AL

The staff at Dataoutates.com would like to thank everyone for the support that we've received over the last couple outages. The list size doubled today and I believe by the end of the night, it will have almost tripled. The traffic to the site is great and we're always looking for new ways to add more features to make people more aware of any outages.

<plug>A Digg submission wouldn't hurt once in awhile<end plug> ;)

Is anyone still down?

I'm still getting [BIPP] Connection failed from both ExchangeConnector and Blackberry Policy Service sources in my event logs.

I still am in Nashville (BIS)

Hmm, guess no one else is having issues with their BES still? I did a reboot and I'm still down and still getting the same errors in event logs.

I did a reboot when the outage first happened thinking it was a server issue. Hope I didn't screw something up....

Up in Philly

Up in NY.

Still some connection problems although email is flowing. I suspect this is due to email servers catching up.

I must have rebooted too early, I did another reboot and the packets are flowin.

Now I can go to bed, thanks RIM!

No connectivity issues for the past 20 minutes. Looks solid.....

Systems are online in NY, no issues and all bottle neck has been cleared on our side.

Any official word from Rim on what the problem was?

our company took a big hit from this! i myself got about 100 emails in 2 hours which killed my inbox abd bb battery.

I believe they are still investigating the RCA.

Not sure what the deal with this outage was, but I'm on VZW in Minneapolis, MN and a friend of mine is on TMO in Lawrence, KS, and neither of us experienced any issues today.

So I guess the BIS upgrade did not go over so welll......

Not just North America, I am in Brazil and was affected for at least 2 hours yesterday.

Let's look at the bright side: whatever happened yesterday is not likely to happen again. Something else will happen the next time. Am I Right, RIM?

This has been posted over at -= Dataoutages.com =-

-----Original Message-----
From: BlackBerry Support
To: xxxxxxxxxxxxxxxxxx
Sent: Mon Feb 11 21:51:08 2008
Subject: Re: RQST000046

Hello,

As requested please find the information regarding the technical difficulties experienced on the BlackBerry® Infrastructure below.

On February 11, 2008, at approximately 15:30, the BlackBerry® Network Operations Center began investigating an issue on the BlackBerry® Infrastructure. Some BlackBerry subscribers experienced delays receiving messages and received an "x" when sending messages until the issue was resolved. Subscribers were able to resend unsent messages after the incident cleared. No messages were lost.

The BlackBerry Support teams were engaged and implemented a series of corrective procedures to reduce the impact of the issue and restore service. This was completed at approximately 18:30; however, some subscribers may have continued to experience delays until the backlog of messages was processed.

The root cause of this issue is under investigation.

Thank you for your continued support, and we apologize for any inconvenience.

Sincerely,
BlackBerry Customer Support

Maybe RIM just needs to admit that somebody was performing maintenance on a production server and clicked [SHUT DOWN] instead of [LOG OFF] when they were finished. LOL

Heck, I've done that once. Once.

Either that or the cleaning lady uplugged the server to plug in her vacuum.

We had that happen once.

That's a lot of fun if you're remoting in. I've done this during an overnight maintenance window and had to drive in.

So did we....

Anyone have a problem with old messages showing back up? I have a user that says he received messages from last month that he deleted.

Hey Takeshi, if you're using HP servers, ILO is a blessed thing in that situation.

ugh i am in Northern Virginia, / Washington DC with AT&T. My berry is up and running. It had a lot of problems though re-syncing my inbox.

LOL

hey, one time i unplugged a pair of T-1s that connected a couple of buildings together. Everyone at the other building couldn't do anything! It was genius!

that's better than the photo with the oldie

If using IBM, they have Remote Supervisory Adapters (RSA) that do the same thing.

Both great technologies.

Of course if you virtualize your BES you can always just power it back on by clicking the play button.

That's not just any ordinary "oldie."

Does anyone think that RIM will announce what the actual problem was?

Took a while to catch up after the reboot, but back up in NYC...

that's the genius' grandma?

they will, they should; dataoutages would prolly get the first info and you should get that email if you sign up

For those with t-support, our Rep said something will be released 2-3 regarding root cause.

I doubt they will be more specific than "a component of the Blackberry infrastructure" and "steps are being taken to ensure it doesn't happen again."

Balsillie quotes in this article:

System upgrade may have caused BlackBerry outage, says RIM boss

Neither msn.com nor microsoft.com provide a consumable service. You don't pay MS to receive a service from either. Your analogy doesn't make sense. This would be more inline with an online stock brokerage not being available.

i didn't have any problems, then again maybe i didn't have an emails. either way its a win win situation for me

I 1st noticed around 3:30 EST yesterday and was back up by the time I had dinner with my fiance.