I have come across another post on askubuntu that's also looking for a calibre alternative.
The accepted answer on that post basically suggests using calibres internal command line tool:
The Calibre install provides the command ebook-convert that will handle what you want, and there's no need to run Calibre.
ebook-convert file.pdf file.epub
The original answer goes into more detail in case that doesnt work as well.
If this does produce a meaningful speed difference compared to the GUI, then there's a chance this may be the result of a setting, or possibly a bug that could be causing the extra slowness.
My initial suspicion is that maybe the GUi could be defaulting to the "smart convert" that was mentioned in the askubuntu post. If you are interested in getting to the cause of this and possibly fixing the GUI, maybe try the command-line smart convert to see if its the GUI, or the "smartness" that's the cause of the slowness.
I suspect if the smart convert is at fault, there may be a setting to disable this or make it not the default when using the GUI. If its only slow in the GUI, then it may be a bug and might be worth checking the invormation on how to find an existing report for this or file a new report on the calibre bugs page in case it helps other users or the developers to find a fix.