6
votes
Accepted
Web service that serves as public proof that a given url contains something at a given time
I think icanprove.com might fit your needs.
The screenshots are not presented online. They are stored in signed pdfs so you have to upload them themselves but on the other hand this gives you more ...
4
votes
Web service that serves as public proof that a given url contains something at a given time
After grepping the archive.org FAQ just found that it does have a "Save page now" box under https://archive.org/web/ ...
I found this after noting that if a page had not been crawled when you search ...
4
votes
Accepted
Python & PDF parsing: any modern, powerful, well-maintained open-source libraries out there?
Finally I went for OCRmyPDF (https://github.com/jbarlow83/OCRmyPDF), which uses tesseract for the actual OCR part (https://github.com/tesseract-ocr/tesseract) - as I understand tesseract is a OCR tool ...
- 181
3
votes
Accepted
What are good tools for automatically crawling websites, check for certain links, and extract data?
I would suggest python possibly plus the scrapy library.
Free & Open Source
Python comes with a CSV library so you can quickly read and parse your csv file
Python has several url and web access ...
- 31k
3
votes
Whole website downloader
I don't know if it is what you are looking for but here is a great software you should look at: HTTrack Website Copier. I don't use it a lot but from what I saw up to date it is working great, ...
- 163
3
votes
Extracting the core content from a webpage
Boilerpipe is written in Java and does exactly that.
You can try their demo. The demo does sometimes go over quota and becomes unavailable; if that happens just try again later.
I found it very ...
- 71
3
votes
Extracting the core content from a webpage
First of all, there is no reason to use C++, stick with scripting languages (Python, Ruby, PHP or JavaScript will do the trick just fine). Because you will be iterating frequently and also because the ...
3
votes
Retrieve Google Scholar results
Have a look at this python command line tool:
https://github.com/ckreibich/scholar.py
Example usage if you want a bibtex file:
$ scholar.py -c 1 --author "albert einstein" --phrase "quantum theory" ...
- 31
3
votes
Download all words from Wiktionary for offline use
There's a Python script called dumpgenerator.py that will export a dump in XML format of any MediaWiki website (such as Wiktionary).
You can then import the XML file to your local install of ...
- 651
3
votes
API that gives VIN from a (US) License Plate Number
You haven't specified which country you are interested in.
In the UK, you can apply for access to the information from the DVLA under the Open Gov scheme.
- 1,738
3
votes
Software to download all images from an entire website
HTTrack is a free (GPL, libre/free software) and easy-to-use offline browser utility.
For mirroring only images you need to set scan rules properly.
Limit external depth download and ensure that you ...
- 680
2
votes
Web service that serves as public proof that a given url contains something at a given time
As mentioned by @Parcier, archive.is does what I want.
It stores both screenshot and HTML, and says the date.
E.g. snapshot I just took from GitHub: https://archive.today/eN836
Does not seem to ...
2
votes
Text or summary extractor from webpage
Reader
It is a free Firefox add-on that converts the webpage into an improved reading format. That includes:
Stripping of Navigation panes, references, headers, and footers
Customizable fonts, colors,...
- 5,451
2
votes
GUI scraper allowing the extract a specified element of a webpage for a list of URLs having a similar structure
You can do exactly this sort of thing with python and scrapy.
Regarding your specific needs:
Free - Yes
Windows 7 - Cross platform so Yes
Open Source - Yes
GUI - Not inherently but there are a lot ...
- 31k
2
votes
Python library for complex web scraping?
PyQt is one that might help outworking complex JS stuffed sites.
Selenium is ok, see an example.
DryScrape is ok, though I've not used it that much.
- 119
2
votes
Accepted
API that gives VIN from a (US) License Plate Number
There is one more option on RapidAPI that you can give it a try to check VINs for US license plates:
https://rapidapi.com/vincheckinfo/api/us-license-plate-to-vin/details
They charge $0.06, $0.08 or $...
- 36
2
votes
Accepted
Java library for scraping websites
jsoup is a really nice library to parse and scrape HTML pages.
It has a simple and easy to use API
It is actively developed and open source
It has no dependencies to other libraries
I used it in a ...
- 211
2
votes
Accepted
Scraping information from email to workflow
Python can either use the exchangelib library to communicate directly with your Exchange 2013 Server, (Supports Microsoft Exchange 2007-2016 Server or Office365) via Exchange Web Services (EWS) unless ...
- 31k
1
vote
Application to compile a single book from fanfiction.net chapters
Slightly more work answer: Python + Scrapy
With python & scrapy you can fetch the web pages that you need, separate out the content from the other information, possibly also fetch other ...
- 31k
1
vote
Application to compile a single book from fanfiction.net chapters
Simplest answer: Pandoc
pandoc -s -r html chapter_url chapter_url chapter_url -o outputfilename.epub
where each chapter_url is a link to a webpage containing a chapter will take the whole of the ...
- 31k
1
vote
Whole website downloader
A PhantomJS script can do the "saving the DOM as it is AFTER all the scripts made changes to the page" thing, but this example doesn't address the other parts of the problem, such as saving images ...
- 382
1
vote
Scrape / Extract Wikipedia Category Taxonomies into UI
The Mediawiki software includes such a "visual taxonomy tree" feature, it is accessible here:
https://en.wikipedia.org/wiki/Special:CategoryTree
You have to specify the root, and it shows the tree ...
- 18.8k
1
vote
Python library for complex web scraping?
It is very easy when you use dryscrape python. On a Ubuntu PC, installation is as follows:
# apt-get install qt5-default libqt5webkit5-dev build-essential \
python-lxml python-pip xvfb
Usage as ...
- 11
1
vote
API that gives VIN from a (US) License Plate Number
There are plenty of APIs available to do this lookup, but they are unfortunately not free. They can charge $0.20 cents per request Here is an Example http://vehicleregistrationapi.com/
- 111
1
vote
What are good tools for automatically crawling websites, check for certain links, and extract data?
Do you need tools for automatically crawling websites, check for certain links, and extract data? You may contact with Essayhub Its a trusted service provider where you will get service.
- 11
1
vote
What are good tools for automatically crawling websites, check for certain links, and extract data?
If you need to crawl certain website for links, you may try sitechecker web crawler or screaming frog. They have a convenient interface and many useful functions:
1) website monitoring
2) detecting ...
- 11
1
vote
Check a web page and send an email when a change is detected
You could do this with python and mechanize to fill in & submit the form, possibly plus possibly scrapy or BeautifulSoup to parse the resulting 2nd page. You can then use the supplied stmplib to ...
- 31k
1
vote
Download all words from Wiktionary for offline use
Try My Word List - Make Word Lists - Offline Dictionary app for Android.
It contains over 800,000 English words from Wiktionary.
You can track them, learn as you go, and mark them as mastered.
...
- 151
1
vote
Tool for Creating List of Businesses Located in Specific Suburb
I created a GoogleMap crawler years ago called Fantail.
Give it a shot and let me know if you need help.
Only top scored, non community-wiki answers of a minimum length are eligible