5
I think icanprove.com might fit your needs.
The screenshots are not presented online. They are stored in signed pdfs so you have to upload them themselves but on the other hand this gives you more control.
It offers a virtual remote browser (an old Firefox version) inside your browser which allows you to authenticate into websites. You have of course to ...
4
jsoup – Java Library for Fetching & Parsing
For fetching and parsing HTML, I use the Java-based library, jsoup. It manages amazingly well with mangled mal-formed HTML code.
Has a variety of ways to query, to answer questions like "Give me the third table in the document".
From the library’s web page:
jsoup is a Java library for working with real-...
3
HTTrack is a free (GPL, libre/free software) and easy-to-use offline browser utility.
For mirroring only images you need to set scan rules properly.
Limit external depth download and ensure that you have appropriate image filters (Options > Scan rules). The following will accept any images found even if they are linked from other servers:
+*.gif +*.jpg +*...
3
I don't know if it is what you are looking for but here is a great software you should look at: HTTrack Website Copier. I don't use it a lot but from what I saw up to date it is working great, download the website. Also, I checked it with Norton and it is virus safe.
3
I would suggest python possibly plus the scrapy library.
Free & Open Source
Python comes with a CSV library so you can quickly read and parse your csv file
Python has several url and web access libraries that can do what you need
Scapy simplifies getting and parsing the web pages.
3
Boilerpipe is written in Java and does exactly that.
You can try their demo. The demo does sometimes go over quota and becomes unavailable; if that happens just try again later.
I found it very useful and easy to implement.
3
First of all, there is no reason to use C++, stick with scripting languages (Python, Ruby, PHP or Javascript will do the trick just fine). Because you will be iterating frequently and also because the bottleneck will never be the parsing but rather the network latency. Also note that you should be working from the CLI.
The concept is the following:
...
3
Have a look at this python command line tool:
https://github.com/ckreibich/scholar.py
Example usage if you want a bibtex file:
$ scholar.py -c 1 --author "albert einstein" --phrase "quantum theory" --citation bt
@article{einstein1917quantum,
title={On the quantum theory of radiation},
author={Einstein, Albert},
journal={Phys. Z},
volume={18},
...
3
After grepping the archive.org FAQ just found that it does have a "Save page now" box under https://archive.org/web/ ...
I found this after noting that if a page had not been crawled when you search for it, it suggests: "Do you want to take a snapshot now?"
This makes it the best option so far, as it also has one of the optional operations: PDF save, and ...
answered Apr 12 '15 at 13:13
Ciro Santilli 冠状病毒审查六四事件法轮功
7991 gold badge6 silver badges15 bronze badges
2
As mentioned by @Parcier, archive.is does what I want.
It stores both screenshot and HTML, and says the date.
E.g. snapshot I just took from GitHub: https://archive.today/eN836
Does not seem to have any of the non-mandatory features.
The creator said in march 2015 that they don't do authentication: http://blog.archive.today/post/114635965191/you-should-...
2
Reader
It is a free Firefox add-on that converts the webpage into an improved reading format. That includes:
Stripping of Navigation panes, references, headers, and footers
Customizable fonts, colors, and margins
Before and after screenshots:
2
You can do exactly this sort of thing with python and scrapy.
Regarding your specific needs:
Free - Yes
Windows 7 - Cross platform so Yes
Open Source - Yes
GUI - Not inherently but there are a lot of python tools to allow you to roll your own GUI if you need them these include tk (comes with python), wxPython (my prefered) and lots of others.
There is ...
2
PyQt is one that might help outworking complex JS stuffed sites.
Selenium is ok, see an example.
DryScrape is ok, though I've not used it that much.
2
You haven't specified which country you are interested in.
In the UK, you can apply for access to the information from the DVLA under the Open Gov scheme.
2
jsoup is a really nice library to parse and scrape HTML pages.
It has a simple and easy to use API
It is actively developed and open source
It has no dependencies to other libraries
I used it in a project and it worked great.
2
Python can either use the exchangelib library to communicate directly with your Exchange 2013 Server, (Supports Microsoft Exchange 2007-2016 Server or Office365) via Exchange Web Services (EWS) unless your administator has disabled this feature, or you use the Win32Com library as demonstrated in this SO Question in conjunction with your local Outlook ...
2
Finally I went for OCRmyPDF (https://github.com/jbarlow83/OCRmyPDF), which uses tesseract for the actual OCR part (https://github.com/tesseract-ocr/tesseract) - as I understand tesseract is a OCR tool that has been open-sourced by Google.
OCRmyPDF has great documentation, also works from the command line and has many language packs:
ocrmypdf -l eng ...
1
The python Scrapy library can do exactly what you are looking for:
No GUI You can run interactively within a python or ipython shell or you can write a script.
Save all of the data from the walked pages
Depth limitation
Rate limitation so as not to get kicked off or banned
Crawl a single or linked sites
Filter sites that you don't wish to crawl
Scrapy &...
1
If the page is straight html, (not flash or query based), you should be able to do this very quickly and simply by using the python's Requests and Beautiful Soup libraries.
The best bit is that you can rapidly test it out using python or iPython interactivly.
1
I think that you need to take a look at Mendeley which has a citations manager plug in for various tool chains but to be able to add the papers to your library and then to be able to download the cited papers you will need to be subscribed to the journal(s) or libraries in question. Once you are you can add the cited papers to your library and download, ...
1
Slightly more work answer: Python + Scrapy
With python & scrapy you can fetch the web pages that you need, separate out the content from the other information, possibly also fetch other information linked from within the story, and collate it as you wish, there are lots of examples on the website & on stackoverflow.
Python can even create the ebook ...
1
Simplest answer: Pandoc
pandoc -s -r html chapter_url chapter_url chapter_url -o outputfilename.epub
where each chapter_url is a link to a webpage containing a chapter will take the whole of the webpages that you reference with the chapter_url as a chapter in the ebook.
Price: Free
Output formats: Lots
Platform: Windows/OS-X/Linux
Add ToC Option: -toc ...
1
A PhantomJS script can do the "saving the DOM as it is AFTER all the scripts made changes to the page" thing, but this example doesn't address the other parts of the problem, such as saving images locally.
var page = require('webpage').create();
var fs = require('fs');
var url = 'https://www.google.com/';
page.open(url, function(status) { // Load the web-...
1
The Mediawiki software includes such a "visual taxonomy tree" feature, it is accessible here:
https://en.wikipedia.org/wiki/Special:CategoryTree
You have to specify the root, and it shows the tree folded by default, very similar to the image in the question:
1
It is very easy when you use dryscrape python. On a Ubuntu PC, installation is as follows:
# apt-get install qt5-default libqt5webkit5-dev build-essential \
python-lxml python-pip xvfb
Usage as follows:
Import dryscrape as d
Import time #for refreshing or waiting for page
d.start_xvfb() #for using this in linux without Xserver
br = d.Session() ...
1
Do you need tools for automatically crawling websites, check for certain links, and extract data? You may contact with Essayhub Its a trusted service provider where you will get service.
1
If you need to crawl certain website for links, you may try sitechecker web crawler or screaming frog. They have a convenient interface and many useful functions:
1) website monitoring
2) detecting certain URLs
3) explore internal links and their anchors
1
You could do this with python and mechanize to fill in & submit the form, possibly plus possibly scrapy or BeautifulSoup to parse the resulting 2nd page. You can then use the supplied stmplib to create and send the required email on a change of the important items from what you stored in the previous run.
Free, most Linux machines will already have ...
1
Try website email extractor that is capable to extract email from a websites, webpages, list of websites etc. you can extract emails from university's website. It can extract emails from LinkedIn also.
See the picture.
1
Not sure if that fits your usecase, but Readability is an online service that can convert pages to more pleasant reading experience by extracting the main content. You can use it in Firefox as a bookmarklet or a dedicated addon. Readability was AFAIK one of the first (if not the first) websites to provide such a service.
Only top voted, non community-wiki answers of a minimum length are eligible
