4
votes
Accepted
Python & PDF parsing: any modern, powerful, well-maintained open-source libraries out there?
Finally I went for OCRmyPDF (https://github.com/jbarlow83/OCRmyPDF), which uses tesseract for the actual OCR part (https://github.com/tesseract-ocr/tesseract) - as I understand tesseract is a OCR tool ...
- 181
3
votes
Retrieve Google Scholar results
Have a look at this python command line tool:
https://github.com/ckreibich/scholar.py
Example usage if you want a bibtex file:
$ scholar.py -c 1 --author "albert einstein" --phrase "quantum theory" ...
- 31
3
votes
Software to download all images from an entire website
HTTrack is a free (GPL, libre/free software) and easy-to-use offline browser utility.
For mirroring only images you need to set scan rules properly.
Limit external depth download and ensure that you ...
- 680
3
votes
Download all words from Wiktionary for offline use
There's a Python script called dumpgenerator.py that will export a dump in XML format of any MediaWiki website (such as Wiktionary).
You can then import the XML file to your local install of ...
- 681
2
votes
Python library for complex web scraping?
PyQt is one that might help outworking complex JS stuffed sites.
Selenium is ok, see an example.
DryScrape is ok, though I've not used it that much.
- 119
2
votes
API that gives VIN from a (US) License Plate Number
I was looking into this and found https://platetovin.com, it's not free but it worked well for me and is only $0.05 per lookup, seems to be the cheapest I can find.
- 46
2
votes
Accepted
API that gives VIN from a (US) License Plate Number
There is one more option on RapidAPI that you can give it a try to check VINs for US license plates:
https://rapidapi.com/vincheckinfo/api/us-license-plate-to-vin/details
They charge $0.06, $0.08 or $...
- 36
2
votes
Accepted
Scraping information from email to workflow
Python can either use the exchangelib library to communicate directly with your Exchange 2013 Server, (Supports Microsoft Exchange 2007-2016 Server or Office365) via Exchange Web Services (EWS) unless ...
- 31.5k
1
vote
Scrape / Extract Wikipedia Category Taxonomies into UI
The Mediawiki software includes such a "visual taxonomy tree" feature, it is accessible here:
https://en.wikipedia.org/wiki/Special:CategoryTree
You have to specify the root, and it shows the tree ...
- 19.4k
1
vote
Python library for complex web scraping?
It is very easy when you use dryscrape python. On a Ubuntu PC, installation is as follows:
# apt-get install qt5-default libqt5webkit5-dev build-essential \
python-lxml python-pip xvfb
Usage as ...
- 11
1
vote
Saving several thousand urls as html
It's not obvious exactly what you want, but would this work (Linux bash shell):
$ URL="https:…"
$ lynx -dump -source "$URL" >tempfile
If that's good enough, then:
$ URL="...
- 226
1
vote
API for Commodities Stock markets
I have used marketstack.com for some stock price lookups, not sure whether they also support commodity prices. There is a related Guthub page, see github.com/apilayer/marketstack .
Recently they sent ...
- 206
1
vote
How to bulk borrow and download borrowable-only books programmatically from the Internet Archive?
Yes the Internet Archive has an API details are here.
There is a command line tool and a Python library. You will however be limited to only borrowing a total of 10 books at any time, for 14 days, ...
- 31.5k
1
vote
Python & PDF parsing: any modern, powerful, well-maintained open-source libraries out there?
I'm now the maintainer of pypdf and PyPDF2. I merged everything back into pypdf. pypdf is the way to go. PyPDF2 will be deprecated.
Original answer:
PyPDF2 is mainained again since April 2022. We ...
- 2,513
1
vote
internet data scraping tools
The answer depends on your skills.
I would tell the problem to my friend ruby.
She probably will use mechanize to do the job.
If you need some guidance, consult the search engine of your choice for ...
- 136
1
vote
Web scraping tools and software available for free
For Python, my favorite is hands down Scrapy.
If you want a GUI for it, Portia.
^^ If you want a GUI to install and run that Docker container:
https://kitematic.com/
- 21
1
vote
Accepted
How to scrape a dynamic webpage
I use Selenium to control a headless Chrome browser. This provides access to full browser functionality, such as javascript and dynamic content. You can programmatically control the browser in exactly ...
- 609
1
vote
Web crawler that lets you specify depth of linked domains
The python Scrapy library can do exactly what you are looking for:
No GUI You can run interactively within a python or ipython shell or you can write a script.
Save all of the data from the walked ...
- 31.5k
1
vote
Web scraping tool to scrape a dynamic page based on current day
If the page is straight html, (not flash or query based), you should be able to do this very quickly and simply by using the python's Requests and Beautiful Soup libraries.
The best bit is that you ...
- 31.5k
1
vote
Downloading all PDFs corresponding to a list of references
I think that you need to take a look at Mendeley which has a citations manager plug in for various tool chains but to be able to add the papers to your library and then to be able to download the ...
- 31.5k
1
vote
Software to download all images from an entire website
WFDownloader App can download images while crawling a website. It also meets your other criteria like being free and being able to run on Windows. You use its crawler mode to achieve this.
Open the ...
- 11
1
vote
Tool for Creating List of Businesses Located in Specific Suburb
I created a GoogleMap crawler years ago called Fantail.
Give it a shot and let me know if you need help.
1
vote
Email Scraper that also scrapes Meta Data
Try website email extractor that is capable to extract email from a websites, webpages, list of websites etc. you can extract emails from university's website. It can extract emails from LinkedIn also....
- 11
1
vote
API that gives VIN from a (US) License Plate Number
There are plenty of APIs available to do this lookup, but they are unfortunately not free. They can charge $0.20 cents per request Here is an Example http://vehicleregistrationapi.com/
- 111
1
vote
What are good tools for automatically crawling websites, check for certain links, and extract data?
Do you need tools for automatically crawling websites, check for certain links, and extract data? You may contact with Essayhub Its a trusted service provider where you will get service.
- 11
1
vote
What are good tools for automatically crawling websites, check for certain links, and extract data?
If you need to crawl certain website for links, you may try sitechecker web crawler or screaming frog. They have a convenient interface and many useful functions:
1) website monitoring
2) detecting ...
- 11
1
vote
Download all words from Wiktionary for offline use
You can also consider using the Websters Unabridged Dictionary
The online version is from August 22, 2009
and has a Copyright Status Public domain in the USA.
http://www.gutenberg.org/ebooks/...
- 111
1
vote
Download all words from Wiktionary for offline use
Try My Word List - Make Word Lists - Offline Dictionary app for Android.
It contains over 800,000 English words from Wiktionary.
You can track them, learn as you go, and mark them as mastered.
...
- 161
Only top scored, non community-wiki answers of a minimum length are eligible