I am designing a new app which is used to convert PDF, DOC, DOCX and XLS format into the EPUB format.
Is there any library I could use for this?
You can check out the LEADTOOLS Documents Converter SDK which has support for converting to and from the following Document Formats:
Disclaimer: I am an employee of this library
While I can't vouch for it meeting all of your needs, Google Drive/Apps has recently added support for epub files.