Questions tagged [scraping]
The scraping tag has no usage guidance.
80
questions
0
votes
1
answer
22
views
Framework/library to build a web scraper/crawler
Want to build a automatic web scrapper/crawler. At first I thought to build it with ‘CasperJS and PhantomJS’ then I was trying to study a bit before starting but it seams ‘CasperJS’ is a bit old and ...
0
votes
0
answers
18
views
Web Crawler alternatives for Sparkler
There was a Web Crawler called sparkler but today is defunct.
I'm looking for an alternative that could:
crawl the web and list top URLs with number of iterations
have a dashboard that could show the ...
0
votes
0
answers
30
views
Software that can actually screen scrape bot OCR
I wanted some recommendations on software for both Linux and Windows that can actually screen scrape and not OCR.
Screen Scrape -> OCR are generally not accurate even with trained models. I wanted ...
- 1
0
votes
0
answers
20
views
Is there a script or software to overwrite locally saved html files, so that they get saved with the same name and path?
I have a website i scraped with HTTracker, and it is scraped 1 level deep. HTTracker keeps the website structure, so if i click a link in the mainpage it sends me to another html file also saved in my ...
0
votes
2
answers
22
views
API for fetching HTML from web pages rendered after a certain period of time
Good day to all!
I am developing a node serverless framework application, and I'd like to know if there exists an API to fetch HTML pages from the web after a certain period of time has passed and it ...
0
votes
0
answers
15
views
Ways to automate data in excel from various websites
I work in IT for bank and my team work on obsolete software tech. So we have to daily make trackers to update our bank software with current ones in market. Every time We have to go on specific sites ...
- 1
0
votes
0
answers
76
views
Is there a more reliable pure Java based library than HtmlUnit or ui4j for returning and manipulating JavaScript rendered web pages?
I need a headless browser, or similar, pure Java library like HtmlUnit that supports JavaScript to reliably render all web page content just as a manual browser would do.
Although HtmlUnit claims to ...
- 1
1
vote
1
answer
29
views
Saving several thousand urls as html
I'm trying to download/scrape about 3000 urls and save them as HTML locally (I actually only need the contents of a certain div from each URL but the whole HTML works to). No images, js, css etc. are ...
- 11
0
votes
1
answer
41
views
API for Commodities Stock markets
I would like to have a clear picture about lithium investments all over the World.
I like this website where I can see all companies related to lithium on the Australian Market.
The website also ...
1
vote
0
answers
31
views
Android library for background web interaction
I would like to find a way for an Android app to interact with a web site, completely in the background without rendering the page in any view, without use of the main thread. The idea is for the app ...
- 111
0
votes
0
answers
100
views
Amazon product advertising API alternative
Because my website is not online yet i could not use amazon products API in it, so I'm looking for an alternative that would let me do that.
so can you recommend any good free alternative to Amazon ...
- 101
2
votes
1
answer
800
views
How to bulk borrow and download borrowable-only books programmatically from the Internet Archive?
How to bulk borrow and download those borrowable-only books programmatically from the internet archive?
And how can this be done?
Is there an archive.org API for borrowing books?
- 21
2
votes
0
answers
221
views
Software to archive/scrape an entire website built with modern techniques (lazy loading, AJAX, infinite scrolling etc.)?
There's plenty of recommendations for website crawler software which can archive entire websites. However in recent years the nature of web development has advanced to the point where old tools simply ...
- 171
4
votes
2
answers
2k
views
Python & PDF parsing: any modern, powerful, well-maintained open-source libraries out there?
I'm looking for well-maintained and well-documented powerful PDF parsing libraries for Python (mainly to extract and parse data from various types of PDFs with different/unpredictable structures, ...
- 181
-1
votes
1
answer
31
views
Web Scraping in Python
Do I need to learn advanced python programming skills in order for me to do web scraping using python? Or I should watch tutorials about web scraping and at the same time learning with python in it. I ...
- 99
1
vote
0
answers
7
views
Great scraping tools/ data mining tools for bulk of websites, extracting few emails from each?
I have been looking for good open source tools / softwares that can read certain patterns and scrape a bulk of provided websites...
My use case is like this:
I have a list of ~900 company websites ...
- 11
1
vote
0
answers
22
views
Software package for scraping/crawling that resists well to website changes
I am writing scrapers for around 100 websites using alltheplaces, built on Scrapy. My main problem is the sites keep changing. I am extracting data either from JSON files directly or by parsing them ...
- 11
-1
votes
1
answer
14
views
What Database type should I use for a scraped data comes from many websites?
I'm building a REST API using NodeJS storing a scraped data that comes from many websites ( the API I'm building is about selling - buying used cars )
There's a potential to scale in the future ...
- 101
2
votes
1
answer
109
views
PDF website crawler
I need an open source system that can crawl websites to gather PDF documents and then index them.
- 21
1
vote
0
answers
190
views
Is there a lighter alternative to puppeteer for simulated clicking?
I'm writing an application that navigates to a website, clicks a button on the website (which updates it) and then extracts information from the website. The application requires a headless browser, ...
0
votes
1
answer
69
views
Competitors price tracking tool to automate price tracking for 100000 products?
I need to build a competitors prices tracking tool for 100,000 products from around 15 competitors? What would be the best way to do it? Please recommend?
0
votes
1
answer
261
views
Recommendations for web scraping tools that require minimal installation
My work uses a custom database with addresses and other personal information. In my role I only have access to an html version.
I frequently need to make letters based on this information, but ...
- 11
1
vote
0
answers
18
views
Automation tool for getting information from various sites
I am searching for a software that I can use to get information from websites.
I've seen some 'scraping' and 'RPA' tools that you can make rules based on CSS or XPaths.
The thing is that there are ...
- 11
2
votes
0
answers
86
views
wget alternative with headless browser (webscraper with rendered JS)?
Is there a recursive web crawler like wget except that it executes javascript (in a headless browser)?
I just need to scrape some online documentation for a holiday, I don't want to write one myself (...
- 121
0
votes
1
answer
18
views
please suggest me an API or suggestion to list facilities around an piece of land
I am building an android app for property listings for farmlands and vacant lands, it should include a feature to extract list of nearby facilities like, roads, highways, water-bodies, factories, ...
- 3
2
votes
1
answer
58
views
internet data scraping tools
I'm a little distressed, for my job I need to get some info from a website, specifically http://www.museumsusa.org/museums/ , I need for every museum in America to get the name, address, phone number ...
- 21
1
vote
0
answers
136
views
Scrape Facebook public event pages
I need to web scrape facebook for some data.
I only need to visit 50 facebook pages every like 1 hour. I will be visting pages of events.
I need to scrape the event name, the event time, and the ...
- 11
1
vote
1
answer
511
views
Web scraping tools and software available for free
What are some free ready-to-use web scraping applications?
Usage: Monitoring online product prices, vendors, top 20 online market places in Asia and comparing price developments.
I am a beginner using ...
1
vote
1
answer
51
views
How to scrape a dynamic webpage
I want to scrape webpages with dynamically loaded content. As a user browsing this webpage, I have to click on an HTML element before the content of interest is dynamically loaded into the page.
I ...
- 113
0
votes
0
answers
23
views
getting the sci-hub url using the file title/URL
I have a .txt file that contains a list of 854 articles urls and titles (from the different journals and onferences sites). I want to have the list of sci-hub urls of thesse files. is there possible?
...
- 1
1
vote
1
answer
23
views
SalesForce and Excel - Chrome Extension the Best Solution?
I work with a sales team who frequently enter OPPs into SalesForce.
Part of this process requires the sales rep to manually:
Google customer City+State to find Zip Codes
Use an in-house spreadsheet ...
- 11
1
vote
0
answers
25
views
Parse SEC Edgar on Pyhthon
I would like to scrape tables from the SEC such as this one.
However, there is some JavaScript involved, and I don't know how to access specific tables, such as CONSOLIDATED BALANCE SHEETS.
Is there ...
- 111
1
vote
0
answers
24
views
Web Scraping for Coding Schools
Is there a fast and efficient way to create some web crawler or scraper to search for coding schools, their contact info, and location and put put them into an excel document or Google Sheet?
0
votes
0
answers
21
views
Price comparison/history webscraping, processing, and then more scraping and processing
What I'm basically trying to do is create an automated way to compare a CSV/JSON file with keywords to sold listings on eBay.
The overall flow would be something like this:
Scrape data from a ...
- 1
1
vote
1
answer
77
views
Web crawler that lets you specify depth of linked domains
I want to download every file (via HTTP, HTTPS and FTP, HTML, PHP as delivered, JS, CSS, linked PDF, images) from a domain, its subdomains, every domain linked from the previously mentioned pages and ...
- 121
3
votes
0
answers
136
views
Foolproof way to scrape articles
I'm looking for a very, very simple way to scrape all articles released on several websites based on these criteria:
I can't code, so it really has to be very straightforward (i.e. no Python or so) ...
- 31
1
vote
1
answer
110
views
Automate the download of the MP3 version of a show from a subscription-based RSS
I subscribe to a news site that frequently publish a page with a show in an enbeded video (the show) and a link to the MP3 of the show. The page is available to anyone but you need a valid ...
Sylario
0
votes
1
answer
480
views
Scraping information from email to workflow
I have an email delivered to a specific inbox which contains a number of fields I need to scrape.
The fields will be subsequently fed into a workflow system.
For example:
From: [email protected]
To:...
- 123
3
votes
0
answers
86
views
Download Image from Keywords
I have around 50000 sentences, I want relevant images closely or partially related to those sentences. Before google image search APIv1 was best suited for my need as now it is deprecated need to find ...
- 131
2
votes
0
answers
434
views
Mirroring a website like I see it after running JavaScript
My employer has created this wix website he wants me to replicate it from A to Z, I have done what I could, but some parts are still missing. He created the website and wants to move it to his server.
...
- 2,140
1
vote
0
answers
417
views
Fast tool to mirror a website
I'm interested in downloading the entire contents of a website (mirroring).
It took wget 8 hours to mirror a 3 GB website. I have also tried HTTrack, but the performance was the same. I've also ...
- 11
1
vote
2
answers
105
views
Web scraping tool to scrape a dynamic page based on current day
I need to scrape a theatre webpage to know whether the bookings for next week has been opened. I am planning to write a scraper if the page has the required HTML element for the next Friday based on ...
- 11
0
votes
0
answers
41
views
HTML link harvester
Can anyone suggest a program that can go through a list of URLs and scrape the URLs for a specific link?
I have roughly about 1000 links that I want to find if it has a specified page which ideally ...
- 183
1
vote
0
answers
125
views
A web scraper that recursively grabs all links on a page
I need to obtain all the links present in a group of web pages.
Most of these pages are part of four websites, but some of these sites are not domain.com but rather domain.com/~site. Some of these ...
2
votes
1
answer
82
views
Downloading all PDFs corresponding to a list of references
I am looking for a program, browser extension or website is that can download all PDFs corresponding to a list of references.
Example:
Given the input:
Paoloni JA, Appleyard RC, Nelson J, ...
4
votes
4
answers
7k
views
Software to download all images from an entire website
I'm looking for software that will recursively crawl a website to download all of the images from the entire website. (Not just from a single webpage.)
To illustrate:
If the site is www.example.com ...
user22923
0
votes
1
answer
2k
views
Tool to extract text from multiple URLs and place in a file
I am in a situation where I have to analyze text data. I have URLs of numerous websites from where I need to rip text and place it in a file for further analysis.
I tried few crawlers but they are ...
- 3
1
vote
3
answers
37
views
Application or suite to compare site after updates
I am looking for a way to automatically scan our site for changes after we launch an update to our site.
Of course, we have a service that scans for 404s, 500s, etc. But there are two problems we are ...
- 459
1
vote
0
answers
154
views
Gratis data scraping tool to replace Import.io
I need to be able to scrape data from multiple websites regularly. Some coding is OK, but due to heavily constrained time the fewer things I have to learn like XPath, the better.
I used to use Import....
- 599
2
votes
0
answers
151
views
Check if HTML pages contain structured data, and output which syntax/vocabulary they use
I have a list of HTTP URLs and I want to know if the corresponding HTML pages contain structured data markup, and if yes, which syntaxes and which vocabularies they use.
The tool can be anything: a ...
- 7,830