I'm looking for a simulator that will enable me to listen to the output, and ideally run on Linux and / or OSX. A front end for visually laying out components would be excellent too, but is not strictly necessary.
I don't think something like that exists. You can use wav files as input (for sources) in LTspice and save the output in that format too, but I have not tried myself that function. It's apparently very slow.
And if just want to do audio programming (no real electronic components), there are plenty of options: SuperCollider, Max/MSP etc. The latter (which also has a free equivalent as Pure Data) actually uses blocks that implement DSP functions and which you connect as a circuit, but going from that to an actual implementation in electronics is a long way.