First of all, RIM does not support the BES on a Virtual Machine..so keep that in mind if/when you talk to their support. Secondly if the configuration database is on the same box/VM as the BES, you may want to consider migrating it to a separate SQL Server. See article KB-03112 on the Blackberry Support website.
4.1.2 is supported under VMWare, so you may need to upgrade to the latest version to get support. I only have 45 users on my BES and have the same memory issue with nbes.exe. The problem started for me on 4.0.x and hasn't gone away even after the last few upgrades. I'm up to 4.1.2 with Hotfix#2 running on Domino 7.0.2, and it's still doing it.
Have you tried turning off Hyperthreading in BIOS?
Hyper-threading is Intel's name for a feature where a single processor can simultaneously run two threads. It's been built into the Pentium 4 since the start, but until recently only enabled for some Xeon chips. With all Pentium 4s from 3GHz speeds upwards, hyper-threading is enabled -- but you can turn it off in your computer BIOS. Not all software works well with hyper-threading.