2022 Developer Survey is open! Take survey.
6 votes
Accepted

Web service that serves as public proof that a given url contains something at a given time

I think icanprove.com might fit your needs. The screenshots are not presented online. They are stored in signed pdfs so you have to upload them themselves but on the other hand this gives you more ...
user avatar
4 votes

Web service that serves as public proof that a given url contains something at a given time

After grepping the archive.org FAQ just found that it does have a "Save page now" box under https://archive.org/web/ ... I found this after noting that if a page had not been crawled when you search ...
user avatar
4 votes
Accepted

Python & PDF parsing: any modern, powerful, well-maintained open-source libraries out there?

Finally I went for OCRmyPDF (https://github.com/jbarlow83/OCRmyPDF), which uses tesseract for the actual OCR part (https://github.com/tesseract-ocr/tesseract) - as I understand tesseract is a OCR tool ...
user avatar
3 votes
Accepted

What are good tools for automatically crawling websites, check for certain links, and extract data?

I would suggest python possibly plus the scrapy library. Free & Open Source Python comes with a CSV library so you can quickly read and parse your csv file Python has several url and web access ...
user avatar
3 votes

Whole website downloader

I don't know if it is what you are looking for but here is a great software you should look at: HTTrack Website Copier. I don't use it a lot but from what I saw up to date it is working great, ...
user avatar
  • 163
3 votes

Extracting the core content from a webpage

Boilerpipe is written in Java and does exactly that. You can try their demo. The demo does sometimes go over quota and becomes unavailable; if that happens just try again later. I found it very ...
user avatar
3 votes

Extracting the core content from a webpage

First of all, there is no reason to use C++, stick with scripting languages (Python, Ruby, PHP or JavaScript will do the trick just fine). Because you will be iterating frequently and also because the ...
user avatar
3 votes

Retrieve Google Scholar results

Have a look at this python command line tool: https://github.com/ckreibich/scholar.py Example usage if you want a bibtex file: $ scholar.py -c 1 --author "albert einstein" --phrase "quantum theory" ...
user avatar
  • 31
3 votes

Download all words from Wiktionary for offline use

There's a Python script called dumpgenerator.py that will export a dump in XML format of any MediaWiki website (such as Wiktionary). You can then import the XML file to your local install of ...
user avatar
  • 651
3 votes

API that gives VIN from a (US) License Plate Number

You haven't specified which country you are interested in. In the UK, you can apply for access to the information from the DVLA under the Open Gov scheme.
user avatar
  • 1,738
3 votes

Software to download all images from an entire website

HTTrack is a free (GPL, libre/free software) and easy-to-use offline browser utility. For mirroring only images you need to set scan rules properly. Limit external depth download and ensure that you ...
user avatar
  • 680
2 votes

Web service that serves as public proof that a given url contains something at a given time

As mentioned by @Parcier, archive.is does what I want. It stores both screenshot and HTML, and says the date. E.g. snapshot I just took from GitHub: https://archive.today/eN836 Does not seem to ...
2 votes

Text or summary extractor from webpage

Reader It is a free Firefox add-on that converts the webpage into an improved reading format. That includes: Stripping of Navigation panes, references, headers, and footers Customizable fonts, colors,...
user avatar
  • 5,451
2 votes

GUI scraper allowing the extract a specified element of a webpage for a list of URLs having a similar structure

You can do exactly this sort of thing with python and scrapy. Regarding your specific needs: Free - Yes Windows 7 - Cross platform so Yes Open Source - Yes GUI - Not inherently but there are a lot ...
user avatar
2 votes

Python library for complex web scraping?

PyQt is one that might help outworking complex JS stuffed sites. Selenium is ok, see an example. DryScrape is ok, though I've not used it that much.
user avatar
2 votes
Accepted

API that gives VIN from a (US) License Plate Number

There is one more option on RapidAPI that you can give it a try to check VINs for US license plates: https://rapidapi.com/vincheckinfo/api/us-license-plate-to-vin/details They charge $0.06, $0.08 or $...
user avatar
  • 36
2 votes
Accepted

Java library for scraping websites

jsoup is a really nice library to parse and scrape HTML pages. It has a simple and easy to use API It is actively developed and open source It has no dependencies to other libraries I used it in a ...
user avatar
  • 211
2 votes
Accepted

Scraping information from email to workflow

Python can either use the exchangelib library to communicate directly with your Exchange 2013 Server, (Supports Microsoft Exchange 2007-2016 Server or Office365) via Exchange Web Services (EWS) unless ...
user avatar
1 vote

Application to compile a single book from fanfiction.net chapters

Slightly more work answer: Python + Scrapy With python & scrapy you can fetch the web pages that you need, separate out the content from the other information, possibly also fetch other ...
user avatar
1 vote

Application to compile a single book from fanfiction.net chapters

Simplest answer: Pandoc pandoc -s -r html chapter_url chapter_url chapter_url -o outputfilename.epub where each chapter_url is a link to a webpage containing a chapter will take the whole of the ...
user avatar
1 vote

Whole website downloader

A PhantomJS script can do the "saving the DOM as it is AFTER all the scripts made changes to the page" thing, but this example doesn't address the other parts of the problem, such as saving images ...
user avatar
  • 382
1 vote

Scrape / Extract Wikipedia Category Taxonomies into UI

The Mediawiki software includes such a "visual taxonomy tree" feature, it is accessible here: https://en.wikipedia.org/wiki/Special:CategoryTree You have to specify the root, and it shows the tree ...
user avatar
  • 18.8k
1 vote

Python library for complex web scraping?

It is very easy when you use dryscrape python. On a Ubuntu PC, installation is as follows: # apt-get install qt5-default libqt5webkit5-dev build-essential \ python-lxml python-pip xvfb Usage as ...
user avatar
1 vote

API that gives VIN from a (US) License Plate Number

There are plenty of APIs available to do this lookup, but they are unfortunately not free. They can charge $0.20 cents per request Here is an Example http://vehicleregistrationapi.com/
user avatar
1 vote

What are good tools for automatically crawling websites, check for certain links, and extract data?

Do you need tools for automatically crawling websites, check for certain links, and extract data? You may contact with Essayhub Its a trusted service provider where you will get service.
user avatar
1 vote

What are good tools for automatically crawling websites, check for certain links, and extract data?

If you need to crawl certain website for links, you may try sitechecker web crawler or screaming frog. They have a convenient interface and many useful functions: 1) website monitoring 2) detecting ...
user avatar
1 vote

Check a web page and send an email when a change is detected

You could do this with python and mechanize to fill in & submit the form, possibly plus possibly scrapy or BeautifulSoup to parse the resulting 2nd page. You can then use the supplied stmplib to ...
user avatar
1 vote

Download all words from Wiktionary for offline use

Try My Word List - Make Word Lists - Offline Dictionary app for Android. It contains over 800,000 English words from Wiktionary. You can track them, learn as you go, and mark them as mastered. ...
user avatar
1 vote

Tool for Creating List of Businesses Located in Specific Suburb

I created a GoogleMap crawler years ago called Fantail. Give it a shot and let me know if you need help.
user avatar

Only top scored, non community-wiki answers of a minimum length are eligible