I would recommend using paramiko
anyways. I had done pretty much exactly what you described using paramiko
at a previous job. In fact, in addition to auth against a database and providing an sftp server, we also provided a virtual filesystem over sftp which served a database oriented view of the underlying filesystem along with custom ACL enforcement.
Your concern about complexity: I gave a talk about paramiko, although it is highly cringe worthy and I can't stand to watch the whole thing myself, hopefully might help you. The slides for the talk includes, at the bottom, references to other gentle introductory tutorials.
Your concern about performance: You really needn't worry because although there obviously is a bit of a performance difference using paramiko, you'll find that more often than not, the bottle neck would be with what happens in the application code after the ssh session is established. Paramiko by itself is pretty lightweight and acts more like a thin layer doing the connection handling. That said, if you feel like you can still limit Paramiko to just the sftp application side of things and let openssh handle all of the rest. To do this learn more about the openssh Subsystem
config option.
Custom Auth: Check the openssh config option AuthorizedKeysCommand
Hope that's useful for you.