I am looking for a program that can limit the amount of RAM an application is allowed to use. If possible, free and compatible with Windows 7 x64.
Process Lasso provides some advanced process monitoring options:
You can configure rules inside a process watchdog that triggers an action (e.g. terminating the process) when the memory usuage becomes greater than defined.
Here is a screenshot:
You can also disallow specified processes from running, log all processes run, and even set various other process rules. These rules, amongst many others, include automatically restarting or terminating a process after it reaches some CPU or memory threshold