There is no third-party app to force a restart when memory gets critically low. This must be done manually every time.

To rule out a faulty BlackBerry you should wipe it with JL_Commander and restore the operating system, using the latest version available for your model. Try running it without any third-party apps for a week or two and keep an eye on the free memory. If it holds, add apps back one by one. Wait a couple days between adding apps and again watch the memory. That's the only way to root out something like this.