first of all, get your domino server connected through the domino. from the existing mail server, try trace command
the NewBlackberryServer is the server name (that you named it once you created the server.id using your Administration Server) or it could be the ip address. both can be used.
once you run the command, you should be able to see something like
Connected to server Mail1/ABC (if you successfully connected both server)
Unable to find path to server Mail/ABC (if it fail)
if using the trace command and the server get connected, then you just install the BES on top of it.
otherwise, pls check your Connection Document in order for you to connect both server.
fyi, domino does not care what type of OSes in your Domino Name Network. it can be mix of windows, red hat, aix, and solaris. but pls verify that your domino is connected using the trace command.