After testing SoundSource by Rogue Amoeba for about a month - I believe it to be the solution (or the best I have found yet).
https://rogueamoeba.com/soundsource/
It is not a free application, but for my use case, it was well worth the cost.
I can:
- mix and balance specific audio streams on an app by app basis
- modify specific input and output device sound
The latter is helpful for switching between I/O devices such as earbud out and in, vs earbud out and boom-mic in, etc. This is exactly the kind of solution I was looking for.
Here is what the expanded controls look like in a typical case:

Performance: Obviously, I can't give specifics, but 2019 Macbook Pro, running OS 12.4 -> I have not noticed any serious resource drains. If I have run into any problems, it has mainly revolved around a specific app or device connecting incorrectly, or audio being accidentally muted.