0

I want to find software that will take one webpage within a website and given some limiting parameters combine it with all linked to information on all the pages into one document but each search "collate webpages"- gives create webpage or collate paper.

My specific problem is https://docs.godotengine.org/en/stable/classes/index.html

  • I want to reduce the complicated hierarchical structure of one high level node methods description to a single file and remove the white space and menu- the menu takes a third of my Kindle's Chrome page. It is doubtful anyone will do it manually.
Improve this question
8
  • 1
    Do you mean web-scraping and more general screen-scraping? Another term is "spidering" or "web-spider" (following all links and creating a graph, avoiding cycles), and more general "mirroring" (following links and making copies of all content found on a web site).
    – knb
    Commented Nov 16, 2021 at 12:19
  • 1
    Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking.
    – Community Bot
    Commented Nov 16, 2021 at 13:19
  • We will need much more information to give good recommendations here. Please take a look at What is required for a question to contain "enough information"? Then please edit your question and see if you can incorporate some of these improvements. Especially missing: what OS it should run on, how much it may cost, and what you want to achieve. It's not clear if you want to mirror the pages, or just collect links – and what output format you want then. Please edit your question accordingly and we can re-open it. Thanks!
    – Izzy
    Commented Nov 16, 2021 at 18:09
  • Just to clarify - you want to flatten the godot docs? and you are looking for a software that will do the rendering for you?
    – Lockszmith
    Commented Nov 19, 2021 at 18:07
  • Yes- basically.
    – user2617804
    Commented Nov 19, 2021 at 21:06

4 Answers 4

Reset to default
1
+50

Does this work?

It's not a single application, but a set of instructions on how to convert a website to an ebook.

https://mirekdlugosz.com/blog/2015/how-to-create-ebook-from-website-spanning-multiple-pages/

Improve this answer
1
  • Anyone care to test this a little to give a result
    – user2617804
    Commented Nov 24, 2021 at 23:29
0

Another option, if you are using Chrome (or Edge) is this, but it's a manual process of collecting each page and saving it as a chapter:

https://chrome.google.com/webstore/detail/save-as-ebook/haaplkpoiimngbppjihnegfmpejdnffj

Improve this answer
1
  • scratch this, I found a good one, posting an answer
    – Lockszmith
    Commented Nov 23, 2021 at 15:07
0

Check out WebToEpub Browser Extension

A browser extension (Chrome or Firefox) that:

  • Analyzes the web page you're on
  • Provides the list of URLs as chapters
  • Allows you to select which to save (multi select/unselect with ctrl-shift+click works).
  • Has a bunch of additional settings, including:
    • multi threaded page download
    • skipping images
    • Stylesheet
  • And then packs everything to an EPUB**.

** If it doesn't crash - which for OPs page is currently what's happening to me.

Improve this answer
4
  • Just tried your URL, it was about to process 1024 pages, but about 3/4 of the way in, it failed with an out of memory error, so there are limits of what's possible in a browser extension
    – Lockszmith
    Commented Nov 23, 2021 at 14:33
  • Another run. I'm impressed by this one. It got to ~90%, maybe ~95% of the way before crashing, memory footprint was way high - more than 4GB, but I assume it's because it's trying to hold the entity of the pages in memory. I'll try to split it, making the class reference a separate epub.
    – Lockszmith
    Commented Nov 23, 2021 at 14:43
  • No go, while this WAS designed for exactly what OP is looking for, it just fails more than not. I tried to create a ~370 pages ebook (out of the ~1030 links on the page), it failed with an out of memory error. So definitely has bugs
    – Lockszmith
    Commented Nov 23, 2021 at 14:48
  • This works to a fashion, but not with the GODOT page. The Firefox extension seems to be more stable than the chrome one. I have found another option, testing how it works now.
    – Lockszmith
    Commented Nov 23, 2021 at 16:33
0

The following Python script seems to be attempting to resolve the same issue.

https://github.com/izanaginookami10/web_to_epub

It's similar to the browser extensions I posted about earlier, but a command line/console tool.

Memory footprint is much more manageable, as it does not store everything in memory, but instead writes the HTML files to the local folder before parsing them.

It has a few bugs, so for OPs particular URL, 9 pages had to be removed otherwise the script would fail, I've opened an issue, but I don't know how responsive the dev is. You might try yourself if you feel comfortable with python.

As for OP's URL:

After a complete run on all but the 9 problem pages, which took 1145 seconds (~19 minutes). An epub was produced - see this link for the result (it's hosted on my home nextcloud server)

Improve this answer
2
  • Results don't produce anything such as Multimesh.
    – user2617804
    Commented Nov 24, 2021 at 8:36
  • Maybe this is still your answer - as you might be able to tweak the python code to do what you want. The code already does the heavy lifting of grabbing and rendering. it creates a xhtml files that it then consumes. you can either put a methodical pause and manipulate the xhtml files to your likings before allowing the script to continue with the rendering.
    – Lockszmith
    Commented Nov 24, 2021 at 13:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.