I'm on T-mo in Arizona and can send just fine, but I haven't received them back from my friend when testing... hmmm... how can I send but not receive? smtp=good / pop=broke? idk

gziz i think your right I am able to validate the SRP information and authenticate with RIMs servers not problem, however looking through the logs earlier there were all kinds of SRP authentication errors, but your right all the messages now are sitting in a pending state and the bes cant talk to the handhelds

as far as I know, and its not much, the RIM hub that is in contact with all of the handhelds has to be in contact with the carriers at all times to priperly route the messages via the PIN id it holds on all of thier devices, so perhaps that is where the problem is now

Im just happy there finally is a problem with our BES and I can point the finger at someone else! yipee!!

i hope its over with by the time my users wake up tomorrow.

ya i guess its diffrent for other people i know my bes can talk to rim because the pending count is going up so bes is doing its job of sending the messages to rims network, at that point its up to the device to contact rims network to pull that message.

for sure, well at least its good to see that this community here is on the ball and posting letting everyone know, as soon as I couldnt find a problem at our site this is the place I went to check before calling in with my tsupport number

great job everyone

I was about to turn in and responded to an email on my handheld from earlier. Lo and behold it doesn't send, check the BES lots of errors, but the SRP is back now... so I'm like wth? Get on hold with RIM, and meantime I decide to check in here... once again this place proves a great source of info.

Thanks fellas.

I'm VZW. And noticed the same issue. I shut down my BES for about 10 minutes. restarted and I'm not receiving a SRP Ping received. I assume the problem still exists?

assuming BB resolves the issue BES should take off on it's own correct?

Regards,

Jason

Agreed. My initial search for an explanation took me to this wonderful page: https://www.blackberry.com/status/EnterPIN.asp

Would be nice for RIM to post a status update of some sort.

In the meantime, this thread is a life saver. Thanks to all

Almost midnight DST, no change. Good night all.

well 7 of 2600 devices are showing current connect times now - can't get a <confirm> response yet though. i'm going to bed....

thank you all for this info - best source out there and we'll continue to monitor through the night - if anyone hears anything positive PLEASE POST as will we.

for the record.. RIMs hold music STINKS.

This site is keeping a running tab on the news.

Ain't that the truth... 22 minutes on hold and counting...

I manage multiple BES for multiple companies. All of them are down and BES Alert for all of them have been pounding my e-mail box (desktop of course). RIM is working on it because my SRP pings and SRP connections are going up and down like crazy (35 changes on each BES over the last 3.5 hours)according to BES Alert. SRP Status in BlackBerry Manager has been "connected" regardless of what BES Alert says. BlackBerry Server Configuration hasn't failed a connection test yet. This tells me that it is in the RIM Relay. The fact that PIN to PIN and BIS traffic have also stopped indicate to me that the RIM Relay is our culprit.

32 minutes on hold and counting...

It is 11:12pm central and my e-mail just kicked in. Over 100 BES Alerts just began to hit my Pearl. All of my BES appear to be up and running. Time to really test them.

Still dead in the water here WG....

I am getting dribs and drabs on only one of my three servers, hopefully this & Wireless Geek's comment above is a sign of good things to come...

BIS and PIN to PIN are either backed up (3.5 hours of work to catch up on), or still down.

This is the best site by far when trying to find out about these issues, we should work together to create a "Real-Time Outages" page that could be updated when one of us has issues and find out from RIM it is outage related. Then we would have one place to look instead of posting/searching in these forums. Just my $.02 - I could be wrong though...

Still NADA here, NJ Verizon

No e-mail in NYC still.
No browsing either.

hahah 42 minutes in and my cordless phone battery starts beeping(indicating imminent death...)... figures.

All four of my servers are dead in the water.... no traffic and tons of connection errors.

Houston, TX
Verizon and Cingular

Keep in mind with 4 hours of being down. There are tons of messages queued up. it will take a few hours to get caught up.

For those who are not up yet, check your SRP Address. You need to look at the BlackBerry Router tab in the BlackBerry Server Configuration app. There are at least 2 SRPs (RIM Relays) serving the US. Mine are on srp.us.blackberry.net. The other address is (I'm pretty sure) srp.na.blackberry.net.

Lets see if that explains the difference in experiences.

WG mine are:

Attempting to connect to :
srp.na.blackberry.net (206.51.26.33), port 3101
Successful
Attempting to connect to :
srp.na.blackberry.net (204.187.87.33), port 3101
Successful


But I've still got no traffic flowing on my BES.

They are looking at rolling back a couple of changes that were made over the weekend even though they were running clean for a couple of days. 30 mins to do this rollback.

both those hosts go to the same place, .us and .na are the same thing.

I'm sure glad tomorrow is my work from home day, now I'll just sleep an hour or so later than normal to make up being up dealing with this, this evening.

One of my BES Server started delivering..its still intermittent..not a 100% or even close.

We were on hold for 3.5 hrs (T2 service level) and just got through – RIM tech joined our DR bridge and provided following info (nothing new here):

RIM down for 4-5 hours – they w/could not provide information on cause
They expect to update their phone IVR (press 2/5) and the website for updates to ongoing outages
No ETA for resolution
If your SRP is connected, mail should begin flowing once service is back. If SRP goes down, it should reconnect by itself – in either case mail should begin flowing once reconnected.
This event started at 715p ET

keep the faith..

gziz, I won't put money on this, but I am pretty sure that they are different "routers" in different cities.

So far 1 "na" down and 1 "us" up. Anyone else?

Most of my t-mobile bb's are getting mail with up to 45 min delays, but none of my nextels are able to talk with the server. I still have no other data services on any devices.

Upgrade Status from Fail to Somewhat Intermittent. One of my bes servers showed some life for a few minutes in routing emails.

There really should be a status page on RIM's main support page. It's absolutely retarded that they don't have a network status page.

My work server is "na" and up and my home server is "us" and up. So it just depends on whether you are luck or not. Chances are it depend on how much traffic you see. I only have 22 users on my work server and 1 at home so they aren't real busy trying to play catch up.

just an update from my site in our Calgary AB data center, I am unable once again to validate our SRP Key with RIM and the SRP network tests are failing as well

haha 1hour 46min on hold!!!

The first news item I've seen on the outage:

NewsChannel4 Exclusive: Massive System Failure Affecting Blackberry Users - News Story - WNBC | New York

Completely agree - although we were told they'd post updates on site, i haven't seen anything.

Di, good point on # of users. I have just over 25 users between all of the BES I support, and all are running. It may be good to have a small shop tonight.