So I have has a mare getting BES 4.1.4 running on an SBS 2003 box that also had Backup exec 11d installed. This then puts SQL 2005 Express in and this hijacks most of the SQL instances. I uninstalled BE 11d and then did:

1. Locate the MSDE files on the SBS 2003 CD 3 or download from Microsoft.
2. Open a command prompt (Start -> Run -> cmd -> clickOK).
3. On the Command Prompt, change to the directory where the MSDE files are located e.g. cd D:\SBS\MONITOR\MSDE
4. Enter the following command to create the new instance:

setup INSTANCENAME=”BESMgmt” SAPWD=”AStrongSAPwd” /L*v C:\MSDELog.log

5. Start the service: Start -> Run -> Services.msc -> Click OK. Scroll down to the instance you just created MSSQL$BESMGMT, select and click start service.
6. Check the log file to make sure everything installed correctly at C:\MSDELog.log

Then:

1. In Control Panel, open the Services tool, and then stop MSSQLSERVER and all other related services (such as SQLSERVERAgent).
2. On the Start menu, click Run, type regedt32, and then click OK to start Registry Editor.
3. Locate either of the following subkeys (depending on whether you installed MSDE as the default MSDE instance or as a named instance):
• HKEY_LOCAL_MACHINE\Software\Microsoft\MSSqlserver\ MSSqlServer\

-or-
• HKEY_LOCAL_MACHINE\Software\Microsoft\Microsoft SQL Server\Instance Name\MSSQLServer\
4. In the right pane, double-click the LoginMode subkey.
5. In the DWORD Editor dialog box, set the value of this subkey to 2, verify that the Hex option is selected, and then click OK.
6. Restart the MSSQLSERVER service and the SQLSERVERAgent service for this change to take effect.

This then installed the BES database ok. But syntax was key to it all:

If MSSQL$BESMGMT is your MSDE instance then the server syntax in the install has to be <machinename\BESMGMT> for the "server name". The rest the installer should pick up. I was thrown by the port number as it was set to dynamic in the registry and the installer choose 1433 (default SQL port). Anyway after a good 24 hour head scratch its running now! Phew...

Oh yes - yet to reinstall BE 11d....