(Developer of BEPUphysics here)
DigitalRune or Matali may do what you want. I don't have direct experience with them, but they're both pretty full featured as far as I know.
That said, BEPU and Jitter should work too. If you see the same sort of failure in both, it is likely that the problem is in the supplied mesh geometry or simulation configuration since it is unlikely that both engines have the same bug.
From a BEPUphysics perspective, some likely culprits based on the description are:
1) Inconsistent winding in triangles combined with a one-sided collision mode, causing some triangles to pull and others to push. Fix: make sure all triangles have a consistent winding, and then use a one-sided collision mode or Solid.
2) Double sided triangles combined with sufficiently high movement vs. time step to allow triangles to penetrate the other surface and get stuck. Fix: enable CCD (slower and can't guarantee separation in all cases), or switch to Solid or consistent one-sided collision (preferred).
3) Graphics offset from physics representation, causing apparent (but not actual) penetration between objects. Fix: use a debug drawer associated with the engine to check if the simulation runs correctly. For example, a representative test could be created in the BEPUphysicsDemos to make use of its drawer.
4) Simulation scaling issues. If the objects are extremely huge or tiny, the built-in thresholds and tuning factors won't be ideal. Fix: Use something like the ConfigurationHelper.ApplyScale (in the BEPUphysicsDemos) to inform the simulation of the desired scale.