BlackBerry Forums Support Community               

Closed Thread
 
LinkBack Thread Tools
Old 10-20-2006, 01:58 PM   #1 (permalink)
Thumbs Must Hurt
 
Join Date: Oct 2005
Model: 8700
Carrier: Rogers
Posts: 56
Post Thanks: 0
Thanked 0 Times in 0 Posts
Default What do you think? - BES Failover/DR

Please Login to Remove!

Here's a scenario I want to run by anyone who's familiar with BES DR planning...

There are 2 BES' running concurrently (BES for Domino version 4.0, HF4, Domino version 6.5.4, Windows 2003). Each BES has its own SRP. BES1 is the Production BES, BES2 is the failover BES. All the users (50) are on BES1, 0 users on BES2. BES2 is pointing to the configuration database on BES1 (MSDE) using SQL Authentication. The MSDE database is being backed-up on schedule. The State databases on BES1 are replicating to BES2 (15 minute replication interval). This environment is all fine, and up and running. User moves between the two servers works fine.

Should BES1 go down, we simply restore the MSDE database on BES2, run the BES Server Configuration and point BES3 to the now LOCAL database (using Windows Trusted authentication). Then, through the BlackBerry Manager we can move users from BES1 (down) to BES2. There would be a very small window of outtage for the users, and redirection would continue without user interaction (new service books sent to the users when they are moved). Because the State databases are already on BES2, users would have the ability to forward and reply-with-history to messages already on their device.

Is this an effective Failover/DR architecture for BES?

Thanks!
Offline  
Old 10-20-2006, 03:14 PM   #2 (permalink)
x14
BlackBerry Extraordinaire
 
Join Date: Jul 2005
Location: NYC
Model: 9800
OS: 6.0.0.546
Carrier: AT&T
Posts: 2,344
Post Thanks: 0
Thanked 17 Times in 16 Posts
Default

It sounds good but it depends on your company's acceptable downtime. Our max downtime is 30 minutes.

Instead of using Domino to replicate the state databases we use a data replication product. Data replication at the file level keeps the state databases synchronized to the second.

So our setup is. BES and Domino is installed on Server1 and Server2. BES and Domino is up and running on Server1 while down on Server2. Data is replicated from Server1 to Server2 at the file system level. When Server1 goes down all we have to do is start BES and Domino on Server2 and everything continue when it last left off.
Offline  
Old 10-20-2006, 03:16 PM   #3 (permalink)
Talking BlackBerry Encyclopedia
 
Join Date: Oct 2005
Location: Toronto
Model: 8700R
Carrier: Rogers
Posts: 206
Post Thanks: 0
Thanked 0 Times in 0 Posts
Default

we have a similar setup here. our production BES is ghosted and mirrored on a backup server. as soon as the 1st one goes down, the 2nd one will connect and continue as the failover.
__________________
Ghetto stats: bberry 8130 :: desktop manager 4.2.1.8 :: handheld 4.2.2.163 :: Lotus Notes 7.02 :: BES 4.1?
Offline  
Old 10-20-2006, 04:04 PM   #4 (permalink)
Thumbs Must Hurt
 
Join Date: Oct 2005
Model: 8700
Carrier: Rogers
Posts: 56
Post Thanks: 0
Thanked 0 Times in 0 Posts
Default

Thanks for the replies. I'm starting to like the idea of one SRP, and replicating Profile AND state databases between the 2 servers....

- 2 Domino servers running concurrently (Different Domino Server names, obviously)
- BES sw (identical versions) installed on BOTH of them with the same SRP (Obviously the BES Task only running on one server at a time - ServerA)
- Profiles and State databases replicating between the two Domino servers
- ALL BES Services running on both servers concurrently
- MSDE database on ServerA
- Both Domino servers point to the same MSDE database on ServerA
- ALL users on BESA (BES Task running), No users on BESB (BES Task not running)

If BESA goes down:

- restore MSDE database from backup to ServerB
- Point ServerB to local MSDE database
- MOVE users from BESA to BESB using the BlackBerry Manager on BESB
- Start the BES task on BESB

Voila. All users are back up and running, no new service books to send to the devices (same SRP).
Offline  
Old 10-20-2006, 04:24 PM   #5 (permalink)
x14
BlackBerry Extraordinaire
 
Join Date: Jul 2005
Location: NYC
Model: 9800
OS: 6.0.0.546
Carrier: AT&T
Posts: 2,344
Post Thanks: 0
Thanked 17 Times in 16 Posts
Default

Quote:
Originally Posted by Hatchy
Thanks for the replies. I'm starting to like the idea of one SRP, and replicating Profile AND state databases between the 2 servers....

- 2 Domino servers running concurrently (Different Domino Server names, obviously)
- BES sw (identical versions) installed on BOTH of them with the same SRP (Obviously the BES Task only running on one server at a time - ServerA)
- Profiles and State databases replicating between the two Domino servers
- ALL BES Services running on both servers concurrently
- MSDE database on ServerA
- Both Domino servers point to the same MSDE database on ServerA
- ALL users on BESA (BES Task running), No users on BESB (BES Task not running)

If BESA goes down:

- restore MSDE database from backup to ServerB
- Point ServerB to local MSDE database
- MOVE users from BESA to BESB using the BlackBerry Manager on BESB
- Start the BES task on BESB

Voila. All users are back up and running, no new service books to send to the devices (same SRP).
There's 1 issue with this scenario. The problem is have 2 Domino server.

If you move all the users to BESB and the user run's the BlackBerry Desktop Manager it's going to look for BESA. The user would have to change it to BESB and back when BESA is up again. Of course you can tell the user not to use the desktop software when BESA is down.
Offline  
Old 10-20-2006, 04:47 PM   #6 (permalink)
BlackBerry God
 
jibi's Avatar
 
Join Date: Oct 2004
Location: Jibi's Secret Place
Model: 8900
OS: 4.6.1.174
Carrier: AT&T
Posts: 11,310
Post Thanks: 0
Thanked 1 Time in 1 Post
Default

Just to note, BES 5.0 will include built-in HA/DR/FO capabilities. Not really sure on all of the details, but I've heard it was quite nifty on the implementation. Just food for thought.
__________________
In the beginning the Universe was created. This has made a lot of people very angry and is widely regarded as a bad move.
Offline  
Old 10-20-2006, 06:51 PM   #7 (permalink)
Thumbs Must Hurt
 
Join Date: Sep 2005
Model: 8703e
Carrier: Verizon
Posts: 59
Post Thanks: 0
Thanked 0 Times in 0 Posts
Default

Quote:
Originally Posted by jibi
built-in HA/DR/FO capabilities.
Not familiar with these abreviations. Please explain..
Offline  
Old 10-20-2006, 06:53 PM   #8 (permalink)
Thumbs Must Hurt
 
Join Date: Sep 2005
Model: 8703e
Carrier: Verizon
Posts: 59
Post Thanks: 0
Thanked 0 Times in 0 Posts
Default

We run BES on Microsoft Virtual Server. We have VS running on two separate servers. BES goes down, restore BES VHD and VMC files to other VS and were backup running.
Offline  
Old 10-24-2006, 04:58 AM   #9 (permalink)
dev
CrackBerry Addict
 
dev's Avatar
 
Join Date: Apr 2005
Location: Toronto
Model: 8800
Carrier: Rogers
Posts: 571
Post Thanks: 0
Thanked 2 Times in 2 Posts
Default

[quote=Hatchy]Thanks for the replies. I'm starting to like the idea of one SRP, and replicating Profile AND state databases between the 2 servers....

Ensure that you setup one-way replication for the Profiles db. That means from the active do a "send only" to the standby Profiles db. The active Profiles db gets easily corrupted when there's another agent (replication agent) actively writing to it simultaneously as the BES message agent. Once corrupted it generates errors in the notes log which directly affects mail redirection. In short all mail redirection stops and Pending mail increases for all users. I experienced it when clustering the Profiles db in a DR scenario as well as when setting up full replication.
Offline  
Old 10-25-2006, 03:07 PM   #10 (permalink)
Thumbs Must Hurt
 
Join Date: Oct 2005
Model: 8700
Carrier: Rogers
Posts: 56
Post Thanks: 0
Thanked 0 Times in 0 Posts
Default

Quote:
Originally Posted by x14
There's 1 issue with this scenario. The problem is have 2 Domino server.

If you move all the users to BESB and the user run's the BlackBerry Desktop Manager it's going to look for BESA. The user would have to change it to BESB and back when BESA is up again. Of course you can tell the user not to use the desktop software when BESA is down.
Agreed, but it won't be a problem, noone is using the Desktop Manager.

Some of the Carrier support people seem to think that if a BES is pointing to a remote MSDE database, the configuration won't be supported. When I speak with RIM, they say it absolutely is. Very puzzling?
Offline  
Old 10-25-2006, 03:13 PM   #11 (permalink)
BlackBerry God
 
jibi's Avatar
 
Join Date: Oct 2004
Location: Jibi's Secret Place
Model: 8900
OS: 4.6.1.174
Carrier: AT&T
Posts: 11,310
Post Thanks: 0
Thanked 1 Time in 1 Post
Default

Quote:
Originally Posted by jmanford
Not familiar with these abreviations. Please explain..
high availability, disaster recovery, failover.
__________________
In the beginning the Universe was created. This has made a lot of people very angry and is widely regarded as a bad move.
Offline  
Old 10-26-2006, 09:53 AM   #12 (permalink)
Thumbs Must Hurt
 
siemp's Avatar
 
Join Date: May 2005
Location: Amsterdam
Model: 9000
Carrier: Vodafone
Posts: 65
Post Thanks: 0
Thanked 0 Times in 0 Posts
Question high availability for blackberry

Im trying to figure out a high availability for blackberry as well.
Is the next solution a workable situation?

BESA with x users SRP 1

BESB with y users SRP 2

all state dbs on both domino servers.

remote sql db

If one server fails I move all users from that server to the working server.
Any comments?
Is it the same as the first post?
Is the move of users an easy step?

Last edited by siemp : 10-26-2006 at 09:55 AM.
Offline  
Old 10-26-2006, 01:04 PM   #13 (permalink)
Thumbs Must Hurt
 
Join Date: Oct 2005
Model: 8700
Carrier: Rogers
Posts: 56
Post Thanks: 0
Thanked 0 Times in 0 Posts
Default

Siem, I think that that is the ideal way to do it (2 BES' running concurrently with their own SRP, sharing the same remote SQL database).

Similar to the first post, except I was using a local MSDE database on one of the servers, not a good as your architecture.

Moving users is a very simple process - you can drag and drop them. New service books will get sent to the devices (assuming coverage, if not, they are queued) and they'll become active. The State database which already resides on the server will be used (this is a replica of their State db from the other server). A new profile document will get created in Profiles.nsf.

You could easily test a user move (yourself). You'll see **Activation entries in the Notes log which confirm a successful activation on the new server. Because you have replicas of the State dbs, they'll be used. This allows you to reply-with-history and forward existing items on your device. If you didn't have a replica already, you could do the move, but you would lose this functionality (no reference to existing messages).

Regards,
Offline  
Old 12-04-2006, 05:48 PM   #14 (permalink)
CrackBerry Addict
 
amukhey's Avatar
 
Join Date: Sep 2004
Location: Los Angeles
Model: 9700
Carrier: T-Mobile
Posts: 750
Post Thanks: 0
Thanked 0 Times in 0 Posts
Default

Also, you are missing a few things:

--When migrating from Production to DR server, turn OFF blackberry services on Production server / turn BB windows svcs to manual and reboot the server/ shut down the server. If you dont, your SRP will be locked out based on the 5-1 rule.

--If using a sql server, ensure your server is in the AD group for accessing the sql database, besmgmt. If you dont, you will see authentication failure's on the Domino console.

--Registry key (Check Admin guide and BB KB) to enter user name and password for SQL Authentication.

--Do a DNS IP Change for Primary to DR BlackBerry server. -This means you have to shut down Primary server to avoid a conflict.

--Do a DNS IP Change for SQL Server.-This means you have to shut down Primary server to avoid a conflict.

There is more detail to this but this fail over scenario also depends on your Cluster Architecture and replication.

We recently did a cut over of all applications and servers to our DR site since our NY building was being shut down for maintenance (every 3 years).
Offline  
Old 12-05-2006, 02:19 PM   #15 (permalink)
Knows Where the Search Button Is
 
Join Date: Feb 2006
Model: 8100t
Carrier: tmobile
Posts: 45
Post Thanks: 0
Thanked 0 Times in 0 Posts
Default

You can also do this:

Just make a bat file with this, you can also automate this so it checks using monitoring. This is a setup for SQL environment, I did this along time ago and figured I would share it again. As for the State DB's just do data copies over. You can also have the 2 domino servers replicate, and just have an agent create new reps as they get created on the primary. If the data is out of date (restore of stat db for example) the user may receive some messages again.



This handles failure besides your SQL db. So for example you can do maint without effecting the userbase. As for the SQL db the only real way is to restore to a different server and point the BES boxes to the new host, which can be batched..

Enjoy.
(This is 4.0 environment there are some tweaks you need to do for a 4.1 environment)

c:
cd\temp
REM ************************************** Stop Blueberry Services
psservice \\<servername>-u <username> -p <password> stop "Blackberry Alert"
psservice \\<servername>-u <username> -p <password>stop "Blackberry Attachment Service"
psservice \\<servername>-u <username> -p <password> stop "Blackberry Controller"
psservice \\<servername>-u <username> -p <password> stop"Blackberry Mobile Data Service"
psservice \\<servername>-u <username> -p <password> stop "Blackberry Policy Service"
psservice \\<servername>-u <username> -p <password> stop "Blackberry Dispatcher"
psservice \\<servername>-u <username> -p <password> stop "Blackberry Router"
psservice \\<servername>-u <username> -p <password> stop "Blackberry Synchronization Service"

Sleep 5

OSQL -E -i c:\temp\Raspberry.sql

sleep 3


c:
cd\temp
REM ************************************** Start Raspberry Services
psservice psservice \\<servername>-u <username> -p <password> start "Blackberry Alert"
psservice \\<servername>-u <username> -p <password> start "Blackberry Attachment Service"
psservice \\<servername>-u <username> -p <password> start "Blackberry Controller"
psservice \\<servername>-u <username> -p <password> start "Blackberry Mobile Data Service"
psservice \\<servername>-u <username> -p <password> start "Blackberry Policy Service"
psservice \\<servername>-u <username> -p <password> start "Blackberry Dispatcher"
psservice \\<servername>-u <username> -p <password> start "Blackberry Router"
psservice \\<servername>-u <username> -p <password> start "Blackberry Synchronization Service"


The SQL script is as follows.

use BESMgmt
go
update ServerConfig set ServiceName = 'CN=Raspberry/OU=SRV/O=LAB where ID=1
update ServerConfig set MachineName = '<machinename.something.yada>' where ID=1
update MDSConfig set MDSHost = '<machinename.something.yada>' where ID=1
go

Last edited by Sagz : 12-05-2006 at 02:27 PM.
Offline  
Old 12-05-2006, 07:41 PM   #16 (permalink)
Talking BlackBerry Encyclopedia
 
dropscience's Avatar
 
Join Date: Jul 2006
Model: 8800
Carrier: Rogers
Posts: 216
Post Thanks: 0
Thanked 0 Times in 0 Posts
Default

Reading all this information makes me feel a bit out of the league amongst you experts. I don't have a DB background, nor do I really understand the logistics so I hope you guys can bare with me. We are going to add approximately 120 more users onto the BES but before I proceed I wanted to run some ideas with you guys.

Our scenario is this:

BES A (v4.0 w/SRP 1): 20 users
BES B (v4.1 w/SRP 2): ZERO users

The configuration database sits locally on BES A. I want to move it off onto a DBA managed "SQL farm" (is that the correct terminology?). This allows me to point both BES servers to the same remote config database.

I've been given direction to place both BES servers onto a VMWare environment. These are the following steps I think are correct:

1) Copy existing Configuration Database onto remote location.
2) Point BES A to the new remote location.
3) Install BES B (v4.1 w/SRP2) onto VMWare environment and also point it to the new remote location.

At this point, both BES servers are now pointing to the same configuration database which should allow me to move users freely back and forth between the two servers. Correct?

4) Move users from BES A to BES B.
5) Add remaining 120 users onto BES B.
6) Copy BES A onto VMWare environment.
7) Upgrade BES A to BES 4.1 (it's currently 4.0)
8) Use newly upgraded BES A (now 4.1) as my test BES and DR server.

Is this sound methodology? Also, forgive my ignorance but is "state database" the same as "configuration database"?

Thanks in advance for any suggestions and help!
__________________
Novell GroupWise 6.5 SP6
Blackberry Enterprise Server v4.1.2 Hotfix 3
Windows Server 2003 R2 SP1, SQL Server 2005

Last edited by dropscience : 12-05-2006 at 07:49 PM.
Offline  
Old 12-06-2006, 04:03 AM   #17 (permalink)
BlackBerry Extraordinaire
 
noname's Avatar
 
Join Date: Sep 2005
Location: Congested Islet of "Foreign Talents" (> 45% of workforce) - Singapore.
Model: Z10
OS: 10.0.0
PIN: NUKE(PAP)
Carrier: Singtel
Posts: 1,504
Post Thanks: 6
Thanked 9 Times in 9 Posts
Default

Hey dropscience,

You may like to take note that VMWare ESX Server version 2.5.2 or later is supported with BlackBerry Enterprise Server software version 4.1 Service Pack 2 or later only. (Source: KB04405 "Virtual Machine software is not supported for BlackBerry Enterprise Server installation")

For the cutover of the BESMgmt database local to BESA to a dedicated remote SQL server, you may like to refer to RIM's cutover article. (Source: KB-04261 "How To - Perform a cutover to another BlackBerry Enterprise Server ")

When you have point BESB to the BESMgmt database on this remote SQL server during installation, it will upgrade the database schema to that of version 4.1. All installs of BlackBerry Manager used to administer these 2 servers MUST BE version 4.1. (Source: KB-04505 "What Is - BlackBerry Manager does not function in mixed version domain")

One thing to remember is that if you have move users from BESA (4.0) to BESB (4.1), their SQL records will be stored in version 4.1 data structure. You must take care not to move a 4.1 user back to a 4.0 server as there are some known issues that email or PIM sync will stop after this kind of move.

Lastly, state databases are Lotus Notes database found under "\Domino\data\BES\state" folder. It is different from the SQL config database (BESMgmt). State database serves as a storage for the links that describes the state of data between BES and the user's mail database. (Source: KB-01477 "What Is - BlackBerry state database")

Once BESB is completely setup with all the users moved, you may like to setup a one-way replication for the "BES" folder (BESB -> BESA) to keep these data up to date.
__________________
Native but 4th class citizen of a nation governed by idiots who import congestions & contention.

Last edited by noname : 12-06-2006 at 04:05 AM.
Offline  
Old 12-06-2006, 07:24 AM   #18 (permalink)
Thumbs Must Hurt
 
Join Date: Jul 2006
Model: 8100
Carrier: 02
Posts: 149
Post Thanks: 0
Thanked 0 Times in 0 Posts
Default

Interesting that BES is not supported on virtual server, As I know that RIM run there bes servers on virtual servers with no issues.

As for DR if you build a second bes server with the exact same SRP key have them point to the remote database. When one server fails. shut it down. physically pull the network cable out of the back of the server plug it back into your exact replica power back up.

****This way is the quickest and you have not got to move any users, but as someone has already said if you have both SRP keys up at the same time. The SRP key would be diabled hence no mail. ***** A

*****Correct me if I am wrong but Just to let you know for those that move users, when you move users from one bes server to another if they have there blackberry turned off. It will only start delivering mails when it gets associated with the new bes server. So if you move a user. The user turns blackberry his off for a week. he turns his device on. It gets associated to the new bes server. New mail is delivered to that device all mail that was delivered during the week the user is off will never be delivered.
Offline  
Old 12-06-2006, 08:11 AM   #19 (permalink)
Talking BlackBerry Encyclopedia
 
dropscience's Avatar
 
Join Date: Jul 2006
Model: 8800
Carrier: Rogers
Posts: 216
Post Thanks: 0
Thanked 0 Times in 0 Posts
Default

This is why I visit this forum so often... for help and posts like the one by noname. THANK YOU! I read a few of the KB articles lastnight after my post and it's very reassuring to have some of my thoughts backed up with your knowledge!

On another note, the amount of BES users I have on BES A equates to no more than 20. I'm thinking it would be probably be easier to just do a knife-edge cutover to BES B when it's up. They're all aware that the BES connectivity is on a test basis and the amount of users might not justify the backup of the database.

What do you guys think?
__________________
Novell GroupWise 6.5 SP6
Blackberry Enterprise Server v4.1.2 Hotfix 3
Windows Server 2003 R2 SP1, SQL Server 2005
Offline  
Closed Thread


Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On





Copyright 2004-2014 BlackBerryForums.com.
The names RIM and BlackBerry are registered Trademarks of BlackBerry Inc.