4 votes
Accepted

Python & PDF parsing: any modern, powerful, well-maintained open-source libraries out there?

Finally I went for OCRmyPDF (https://github.com/jbarlow83/OCRmyPDF), which uses tesseract for the actual OCR part (https://github.com/tesseract-ocr/tesseract) - as I understand tesseract is a OCR tool ...
Jean Monet's user avatar
3 votes

Retrieve Google Scholar results

Have a look at this python command line tool: https://github.com/ckreibich/scholar.py Example usage if you want a bibtex file: $ scholar.py -c 1 --author "albert einstein" --phrase "quantum theory" ...
max's user avatar
  • 31
3 votes

Software to download all images from an entire website

HTTrack is a free (GPL, libre/free software) and easy-to-use offline browser utility. For mirroring only images you need to set scan rules properly. Limit external depth download and ensure that you ...
Davidenko's user avatar
  • 680
3 votes

Download all words from Wiktionary for offline use

There's a Python script called dumpgenerator.py that will export a dump in XML format of any MediaWiki website (such as Wiktionary). You can then import the XML file to your local install of ...
dr_'s user avatar
  • 681
2 votes

Python library for complex web scraping?

PyQt is one that might help outworking complex JS stuffed sites. Selenium is ok, see an example. DryScrape is ok, though I've not used it that much.
Igor Savinkin's user avatar
2 votes

API that gives VIN from a (US) License Plate Number

I was looking into this and found https://platetovin.com, it's not free but it worked well for me and is only $0.05 per lookup, seems to be the cheapest I can find.
Tomas Crofty's user avatar
2 votes
Accepted

API that gives VIN from a (US) License Plate Number

There is one more option on RapidAPI that you can give it a try to check VINs for US license plates: https://rapidapi.com/vincheckinfo/api/us-license-plate-to-vin/details They charge $0.06, $0.08 or $...
Zeyu's user avatar
  • 36
2 votes
Accepted

Scraping information from email to workflow

Python can either use the exchangelib library to communicate directly with your Exchange 2013 Server, (Supports Microsoft Exchange 2007-2016 Server or Office365) via Exchange Web Services (EWS) unless ...
Steve Barnes's user avatar
  • 31.5k
1 vote

Scrape / Extract Wikipedia Category Taxonomies into UI

The Mediawiki software includes such a "visual taxonomy tree" feature, it is accessible here: https://en.wikipedia.org/wiki/Special:CategoryTree You have to specify the root, and it shows the tree ...
Nicolas Raoul's user avatar
  • 19.4k
1 vote

Python library for complex web scraping?

It is very easy when you use dryscrape python. On a Ubuntu PC, installation is as follows: # apt-get install qt5-default libqt5webkit5-dev build-essential \ python-lxml python-pip xvfb Usage as ...
Harry1992's user avatar
1 vote

Saving several thousand urls as html

It's not obvious exactly what you want, but would this work (Linux bash shell): $ URL="https:…" $ lynx -dump -source "$URL" >tempfile If that's good enough, then: $ URL="...
Ray Butterworth's user avatar
1 vote

API for Commodities Stock markets

I have used marketstack.com for some stock price lookups, not sure whether they also support commodity prices. There is a related Guthub page, see github.com/apilayer/marketstack . Recently they sent ...
AdrianHHH's user avatar
  • 206
1 vote

How to bulk borrow and download borrowable-only books programmatically from the Internet Archive?

Yes the Internet Archive has an API details are here. There is a command line tool and a Python library. You will however be limited to only borrowing a total of 10 books at any time, for 14 days, ...
Steve Barnes's user avatar
  • 31.5k
1 vote

Python & PDF parsing: any modern, powerful, well-maintained open-source libraries out there?

I'm now the maintainer of pypdf and PyPDF2. I merged everything back into pypdf. pypdf is the way to go. PyPDF2 will be deprecated. Original answer: PyPDF2 is mainained again since April 2022. We ...
Martin Thoma's user avatar
  • 2,513
1 vote

internet data scraping tools

The answer depends on your skills. I would tell the problem to my friend ruby. She probably will use mechanize to do the job. If you need some guidance, consult the search engine of your choice for ...
mvw's user avatar
  • 136
1 vote

Web scraping tools and software available for free

For Python, my favorite is hands down Scrapy. If you want a GUI for it, Portia. ^^ If you want a GUI to install and run that Docker container: https://kitematic.com/
Brady coye's user avatar
1 vote
Accepted

How to scrape a dynamic webpage

I use Selenium to control a headless Chrome browser. This provides access to full browser functionality, such as javascript and dynamic content. You can programmatically control the browser in exactly ...
Daniel's user avatar
  • 609
1 vote

Web crawler that lets you specify depth of linked domains

The python Scrapy library can do exactly what you are looking for: No GUI You can run interactively within a python or ipython shell or you can write a script. Save all of the data from the walked ...
Steve Barnes's user avatar
  • 31.5k
1 vote

Web scraping tool to scrape a dynamic page based on current day

If the page is straight html, (not flash or query based), you should be able to do this very quickly and simply by using the python's Requests and Beautiful Soup libraries. The best bit is that you ...
Steve Barnes's user avatar
  • 31.5k
1 vote

Downloading all PDFs corresponding to a list of references

I think that you need to take a look at Mendeley which has a citations manager plug in for various tool chains but to be able to add the papers to your library and then to be able to download the ...
Steve Barnes's user avatar
  • 31.5k
1 vote

Software to download all images from an entire website

WFDownloader App can download images while crawling a website. It also meets your other criteria like being free and being able to run on Windows. You use its crawler mode to achieve this. Open the ...
Renato's user avatar
  • 11
1 vote

Tool for Creating List of Businesses Located in Specific Suburb

I created a GoogleMap crawler years ago called Fantail. Give it a shot and let me know if you need help.
Francesco Mantovani's user avatar
1 vote

Email Scraper that also scrapes Meta Data

Try website email extractor that is capable to extract email from a websites, webpages, list of websites etc. you can extract emails from university's website. It can extract emails from LinkedIn also....
Sarah Dylan's user avatar
1 vote

API that gives VIN from a (US) License Plate Number

There are plenty of APIs available to do this lookup, but they are unfortunately not free. They can charge $0.20 cents per request Here is an Example http://vehicleregistrationapi.com/
Fiach Reid's user avatar
1 vote

What are good tools for automatically crawling websites, check for certain links, and extract data?

Do you need tools for automatically crawling websites, check for certain links, and extract data? You may contact with Essayhub Its a trusted service provider where you will get service.
sopnodrosta's user avatar
1 vote

What are good tools for automatically crawling websites, check for certain links, and extract data?

If you need to crawl certain website for links, you may try sitechecker web crawler or screaming frog. They have a convenient interface and many useful functions: 1) website monitoring 2) detecting ...
NatalieF's user avatar
1 vote

Download all words from Wiktionary for offline use

You can also consider using the Websters Unabridged Dictionary The online version is from August 22, 2009 and has a Copyright Status Public domain in the USA. http://www.gutenberg.org/ebooks/...
Asher's user avatar
  • 111
1 vote

Download all words from Wiktionary for offline use

Try My Word List - Make Word Lists - Offline Dictionary app for Android. It contains over 800,000 English words from Wiktionary. You can track them, learn as you go, and mark them as mastered. ...
Smart Apps's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible