Put simply, I am looking for either a web browser or a browser plug‑in or extension that can save a web page so that, when loaded back in, it actually looks the same as it did when it was saved.
In principle, all browsers can do this, simply by saving “HTML Complete”, but in practice, Iʼve never seen one that does, and Iʼve used Firefox, Opera, and Vivaldi extensively, and have previously used Chrome and IE.
Problem Details
For sites that are coded in an extremely straightforward manner, most browsers do save them so as to load back correctly, but all too often the complexity of modern sites somehow messes up web browsers.
Examples of problems I see in the saved pages:
Distorted layout compared to how the original page looked at the time of saving. This is typically minor, but with some browser and web site combinations, layout errors can be so dramatic as to make the page practically unreadable, although all the wanted data is still there in the files. Drop‑down menus are a common source of catastrophic layout errors, though not the only one. (Itʼs ok if menus wonʼt actually pop up in the saved page, but itʼs not ok when they instead destroy the pageʼs quiescent layout.)
Saved pages take an extremely long time to load, in particular when there is no Internet connection, often because they are waiting to make useless accesses to servers for analytics.
Failure to render Twitter posts embedded in other sites. Firefox saves the tweets in the sense that you can find their text and images within the saved files, but when the saved page is loaded in a web browser, it doesnʼt render them. Chromium-based browsers do not even save the embedded data at all.
Similar failure to save or render embedded Instagram posts.
Saved Facebook pages typically omit “sticker” images in post comments (those silly large emoticons and cartoon figures). These use a background image within a
<div>or<span>, and Opera, for instance, does not save these images at all — you canʼt even find them manually in the saved resources, even though other images such as user avatars and posted photos are there. The saved page in thebackground-imageCSS property in the tagʼsstyleproperty continues to point to the live URL, which means that it typically renders correctly if youʼre connected to the Internet, but not if youʼre not, and not if that HTTP resource is ever taken down.Saved Facebook pages blur out the preview images for posts of videos, though they are not blurred when youʼre viewing the live site. The images are actually there in the saved files, and can even be found fairly easily via a DOM browser when the saved page is open — itʼs just that script or something is telling the browser to blur them.
Upvote and downvote counts not rendered on pages saved from Urbandictionary.com, even though the counts are actually present in the saved HTML file.
Problems with content that can be expanded or hidden on user request. It may not be in the same expansion state it was in when the page was saved, and/or the page loaded back in may not respond to requests to show or hide content, even though browsers generally do save JavaScript along with the page. The content is usually still in the saved HTML file even though it canʼt be expanded in the browser.
Some saved pages, when loaded back in, will for a brief moment show the page contents, and then immediately turn blank. This happens even without an Internet connection, and it can be seen in the saved files that the data is there — itʼs just getting hidden after page load, presumably by script (not sure if inadvertently or intentionally).
I have at times tried to print the page to a PDF file instead of save it as an HTML document, but in most cases the results of doing that are far worse! (HTML/CSS provide for a separate layout mode for printing a page, but most sites are not developed or tested to behave properly in this mode, and it also generally cannot produce a continuous layout, due to being constrained to actual paper sizes.)
Some browsers can save a capture of the rendered page as an image (pixel array); this does make a faithful save, but this is far from ideal for a number of reasons including it being impossible to search on or copy text.
Solution Details
What I want is for the browser to create HTML and CSS files, and save all external resources, in such a manner that upon loading in the saved page, the DOM will be in exactly the same state it was in when the page was saved, but with any and all images, icons, fonts, etc., it may have been pointing to now pointing to locally saved files. (No need to save streaming files, for which there are already good capture tools).
In thinking of how such a program would actually operate, I realize that client-side script in the general case becomes a huge problem. I would be fine with the program simply having an option to write the page in its current state without any of its script, for sites where the script is found to cause problems.
It would be acceptable for the page to instead be saved in some non-standard format that preserves the renderable objects and layout commands in a lower-level fashion, but this would be less ideal, and in this case ideally the format should be documented or easily reverse engineered.
I donʼt mind a solution such as a code library that requires some minimal programming to get the full solution, but any programming required canʼt comprise a large project; user-level software is preferred.
Also note that for some sites, user interaction with the site is required to pull in desired content from the server after page load, so a non‑interactive solution is not feasible.
The solution should hopefully be inexpensive, ideally free. It canʼt be a cloud‑based solution (thus precluding PageVault, for instance) or one where the purchased license is not perpetual. It should work on Windows, but in the very worst case *nix would be tolerable (but very annoying as Iʼd have to browse through a VM console).
Test Cases
A case being tested in a particular browser does not imply that the problem is unique to that browser. For any test case, I clear the cache and disconnect from the Internet prior to loading back the page; otherwise itʼs harder to spot things that didnʼt get saved as they could be getting fetched live.
Test Case #1
https://i-d.vice.com/en_us/article/59v8aq/adut-akech-instagram-depression-anxiety-mental-health
This page, saved in and loaded back in Vivaldi, demonstrates multiple issues. Behaviour is approximately the same with or without AdBlock.
If loaded without an Internet connection, it takes some 20 sec. even for the text to appear, and even after that it attempts connections for several minutes.
Ultimately, the Instagram post is never loaded, even though you can find its text and the first two pictures of the postʼs slideshow among the files. (The remaining two images in that slideshow are not saved at all, but thatʼs somewhat expected.)
The static close‑up picture of Adutʼs face at the beginning of the article is not loaded although it was saved.
The “i‑D” logo at the top of the page has a broken image symbol overlayed on it, and the vertical version below the article is missing. The article text is not in the original font, and there are a few other minor layout errors.
Test Case #2
Saved and loaded back in Opera, both with and without ad blocking. Again, multiple issues.
Although the bulk of the page loads quickly, without Internet connection, various external connections are attempted indefinitely.
The static picture of Evelyn Udell at the top of the article is not displayed although it is present in the files. All the article cover photos under “Most Popular Today” also donʼt show although they are also in the files. The preview picture for the news video appears black, and that image is not even saved. Ironically, the least wanted images — the ads — display fine, without ad blocking.
The Twitter post from LuliOrtizTV is totally omitted without a trace. The image in the tweet is not present in the saved files, and an exhaustive text search indicates not even the text was saved.
The layout is littered with missing or broken icons, such as on the Facebook, Twitter, G+ and Pinterest buttons. In some cases icons are replaced with boxes, such as the heart to the left of Recommend in the Disqus panel. Several of the icons in that panel are replaced with simpler versions, such as the Facebook “f”. The magenta FB and Twitter icons at the very top of the page are replaced with Asian characters.
Test Case #3
https://www.urbandictionary.com/define.php?term=Blunt
Saved and loaded back in Firefox. Again, multiple issues.
Without an Internet connection, itʼs about a minute before anything at all appears, and it attempts external connections for several more minutes.
The definitions are readable, but certain layout elements and especially the stuff at the top of the page is badly damaged, with the positioning within that area being massively incorrect. The Categories drop‑down has been barfed into the static layout instead of being hidden. The fonts of the labels such as “Vote” and “Store” have been defaulted. The graphical “urban dictionary” logo is missing entirely and looks to not even have been saved. The search bar is no longer even inside the blue area. Thereʼs even a strange “Loading...” animated passifier, awkwardly positioned at the far right of the page that never goes away.
While no big loss, the ads are not displayed, although present in the files.
The font in the definitions is somewhat different, with thicker underlines than in the original.
Worst of all, the vote counts beside the thumbs up and down are not displayed at all! They actually are in the saved HTML, in <span> elements, but are completely omitted from the layout.
The cookie notice doesnʼt respond to clicks to dismiss it, although that could be something server‑side.
If I run into other classes of failure, Iʼll try to add test cases for them.
