I am looking for a web-based, open source book publishing application.
I basically want a web-based application that will allow users to design and export books in PDF or EPUB format.
I would recommend GitBook.
GitBook is a command line tool (and Node.js library) for building beautiful books using GitHub/Git and Markdown (or AsciiDoc).
You can publish your book via GitBook (web-based) or desktop editor (provided by GitBook). There are some features that supported by GitBook:
GitBook is an open source software that released under Apache License version 2.0. You can check the source code here.
Edit: after reread your comment, you need a web-based software that user can design the book theme drag and drop. GitBook is not right choice for this since this software requires basic web programming.
While it might represent something of a learning curve, one possibility would be to use an online LaTeX editor/compiler.
N.b. These are not "self-hosted" options. See also this TeX.SE Q&A on using LaTeX for epub production.
This is a widely used, well known service. It has a rich set of templates, including one with directions for epub export.
One advantage to this service is that it allows users to write in Markdown, which might mitigate any perceived drawback to learning LaTeX. (N.b., this is for "simple documents", but again -- it could suffice).