Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [scraping]

The tag has no usage guidance.

0
votes
0answers
8 views

Import.io, R or other for scraping subcategories and pages from Wikipedia and/or Government websites? [closed]

I am doing a research project and I want to create a data set which lists all the government entities in the world by country and by executive, judicial and legislative. I am scraping from this ...
1
vote
0answers
10 views

Automation tool for getting information from various sites

I am searching for a software that I can use to get information from websites. I've seen some 'scraping' and 'RPA' tools that you can make rules based on CSS or XPaths. The thing is that there are ...
1
vote
0answers
11 views

wget alternative with headless browser (webscraper with rendered JS)?

Is there a recursive web crawler like wget except that it executes javascript (in a headless browser)? I just need to scrape some online documentation for a holiday, I don't want to write one myself (...
0
votes
1answer
16 views

please suggest me an API or suggestion to list facilities around an piece of land

I am building an android app for property listings for farmlands and vacant lands, it should include a feature to extract list of nearby facilities like, roads, highways, water-bodies, factories, ...
0
votes
0answers
8 views

Web Scraper that Crawls Links Within PDFs

I have an interesting crawler requirement. Essentially I have a site that hosts a TON of PDF files and many of these PDF files have links to other PDF files. Said links are not present within html ...
2
votes
1answer
28 views

internet data scraping tools

I'm a little distressed, for my job I need to get some info from a website, specifically http://www.museumsusa.org/museums/ , I need for every museum in America to get the name, address, phone number ...
0
votes
0answers
6 views

Web Scraping Facebook advice [duplicate]

I need to web-scrape facebook. I want to scrape about 50 pages every hour. I want to scrape event pages, and get the event name, event location and event time. Would this be legal? Would it be noticed ...
1
vote
0answers
13 views

Scrape Facebook public event pages

I need to web scrape facebook for some data. I only need to visit 50 facebook pages every like 1 hour. I will be visting pages of events. I need to scrape the event name, the event time, and the ...
1
vote
1answer
36 views

Web scraping tool/software availabe for free?

I am looking for a free ready-to-use web scraping application. Any recommendations? Usage: Monitoring online product prices, vendors, top 20 online market places in Asia and comparing price ...
1
vote
1answer
28 views

How to scrape a dynamic webpage

I want to scrape webpages with dynamically loaded content. As a user browsing this webpage, I have to click on an HTML element before the content of interest is dynamically loaded into the page. I ...
0
votes
0answers
11 views

How to host a clone website on my local machine

I wanted to clone my website on localhost for making some changes. I did the cloning using wget. What I got is HTML pages. Now what software can I use to host this cloned website on localhost? Any ...
0
votes
0answers
12 views

getting the sci-hub url using the file title/URL

I have a .txt file that contains a list of 854 articles urls and titles (from the different journals and onferences sites). I want to have the list of sci-hub urls of thesse files. is there possible? ...
1
vote
1answer
12 views

SalesForce and Excel - Chrome Extension the Best Solution?

I work with a sales team who frequently enter OPPs into SalesForce. Part of this process requires the sales rep to manually: Google customer City+State to find Zip Codes Use an in-house spreadsheet ...
1
vote
0answers
17 views

Parse SEC Edgar on Pyhthon

I would like to scrape tables from the SEC such as this one. However, there is some JavaScript involved, and I don't know how to access specific tables, such as CONSOLIDATED BALANCE SHEETS. Is there ...
1
vote
0answers
17 views

Web Scraping for Coding Schools

Is there a fast and efficient way to create some web crawler or scraper to search for coding schools, their contact info, and location and put put them into an excel document or Google Sheet?
0
votes
0answers
15 views

Price comparison/history webscraping, processing, and then more scraping and processing

What I'm basically trying to do is create an automated way to compare a CSV/JSON file with keywords to sold listings on eBay. The overall flow would be something like this: Scrape data from a ...
1
vote
1answer
48 views

Web crawler that lets you specify depth of linked domains

I want to download every file (via HTTP, HTTPS and FTP, HTML, PHP as delivered, JS, CSS, linked PDF, images) from a domain, its subdomains, every domain linked from the previously mentioned pages and ...
2
votes
1answer
116 views

Foolproof way to scrape articles

I'm looking for a very, very simple way to scrape all articles released on several websites based on these criteria: I can't code, so it really has to be very straightforward (i.e. no Python or so) ...
2
votes
4answers
642 views

Web scraping app [closed]

I'd like a general purpose web page scraper app to create CSV files (XML or JSON could probably work too though as ultimately I just care that I can move it to some type of database somehow) without ...
0
votes
1answer
68 views

free, automatic, cloud-based, webscraper?

Years ago I used the Fetch Page module of Yahoo Pipes to turn some regularly-updated websites into RSS feeds. I could then use the rest of their tools to do what I want with it. I recently came ...
1
vote
2answers
74 views

Automate the download of the MP3 version of a show from a subscription-based RSS

I subscribe to a news site that frequently publish a page with a show in an enbeded video (the show) and a link to the MP3 of the show. The page is available to anyone but you need a valid ...
0
votes
1answer
294 views

Scraping information from email to workflow

I have an email delivered to a specific inbox which contains a number of fields I need to scrape. The fields will be subsequently fed into a workflow system. For example: From: example@other.com To:...
3
votes
0answers
47 views

Download Image from Keywords

I have around 50000 sentences, I want relevant images closely or partially related to those sentences. Before google image search APIv1 was best suited for my need as now it is deprecated need to find ...
2
votes
0answers
93 views

Mirroring a website like I see it after running JavaScript

My employer has created this wix website he wants me to replicate it from A to Z, I have done what I could, but some parts are still missing. He created the website and wants to move it to his server. ...
1
vote
0answers
306 views

Fast tool to mirror a website

I'm interested in downloading the entire contents of a website (mirroring). It took wget 8 hours to mirror a 3 GB website. I have also tried HTTrack, but the performance was the same. I've also ...
1
vote
2answers
49 views

Web scraping tool to scrape a dynamic page based on current day

I need to scrape a theatre webpage to know whether the bookings for next week has been opened. I am planning to write a scraper if the page has the required HTML element for the next Friday based on ...
0
votes
0answers
34 views

HTML link harvester

Can anyone suggest a program that can go through a list of URLs and scrape the URLs for a specific link? I have roughly about 1000 links that I want to find if it has a specified page which ideally ...
1
vote
0answers
61 views

A web scraper that recursively grabs all links on a page

I need to obtain all the links present in a group of web pages. Most of these pages are part of four websites, but some of these sites are not domain.com but rather domain.com/~site. Some of these ...
2
votes
1answer
74 views

Downloading all PDFs corresponding to a list of references

I am looking for a program, browser extension or website is that can download all PDFs corresponding to a list of references. Example: Given the input: Paoloni JA, Appleyard RC, Nelson J, ...
4
votes
2answers
169 views

Software to download all images from an entire website

I'm looking for software that will recursively crawl a website to download all of the images from the entire website. (Not just from a single webpage.) To illustrate: If the site is www.example.com ...
0
votes
1answer
264 views

Tool to extract text from multiple URLs and place in a file

I am in a situation where I have to analyze text data. I have URLs of numerous websites from where I need to rip text and place it in a file for further analysis. I tried few crawlers but they are ...
1
vote
3answers
32 views

Application or suite to compare site after updates

I am looking for a way to automatically scan our site for changes after we launch an update to our site. Of course, we have a service that scans for 404s, 500s, etc. But there are two problems we are ...
1
vote
0answers
127 views

Gratis data scraping tool to replace Import.io

I need to be able to scrape data from multiple websites regularly. Some coding is OK, but due to heavily constrained time the fewer things I have to learn like XPath, the better. I used to use Import....
2
votes
0answers
139 views

Check if HTML pages contain structured data, and output which syntax/vocabulary they use

I have a list of HTTP URLs and I want to know if the corresponding HTML pages contain structured data markup, and if yes, which syntaxes and which vocabularies they use. The tool can be anything: a ...
1
vote
0answers
42 views

Software to search for Youtube videos, given a search pattern

I wonder if exists a software that, given a search pattern like for example a song title, it can search for youtube videos and return the video urls of the search results. The application should ...
1
vote
0answers
47 views

Web scraping - appropriate programming language [closed]

Which language is the most adept at scraping the Web by parsing the page source in regards to reference material regarding the subject, i. e. official docs, and Stack Overflow questions. Also, ...
3
votes
1answer
110 views

Tool to extract text from HTML

Having a website, I'd like to extract content from it. So far I've been using Regex in Notepad++, but we all know we should not use Regex to parse HTML due to a redesign of the website, getting the ...
3
votes
1answer
69 views

Text-to-speech specific page element?

I'd like to convert a web page like this and have the program filter it for the comments, rather than include "permalink, source" or irrelevant page elements.
1
vote
0answers
200 views

Reliable website scraper working with URL parameters such as ?page=3

Abstract In my work I often have the need of document myself with many sources. Some of most useful ones rely on personal or corporate project blogs websites. Practically all of them are just simple ...
1
vote
0answers
108 views

Library to crawl Ajax-based websites fast

I would like to crawl Ajax websites with Java. I tried Selenium Webdriver and compared to Jsoup for example it is much slower (650ms vs 7.5s). Since Crawljax is based on Selenium Webdriver I haven't ...
1
vote
2answers
65 views

Application to compile a single book from fanfiction.net chapters

Needed: Software that can take a story posted in multiple chapters to fanfiction.net and then make it into a single text/book. Cost: free Desired technology: irrelevant (webapp, Windows, Linux, Perl/...
2
votes
2answers
636 views

Java library for scraping websites

I am looking for a good library to help me scrape information off of a certain website. The path the scraping has to follow can be rather dynamic, meaning that if the initial scrape is on a page of ...
2
votes
4answers
299 views

Whole website downloader

How do you download a static copy of a website? I mean, not loading scripts and using them later when you load the "offline" page, but saving the DOM as it is AFTER all the scripts made changes to the ...
4
votes
1answer
334 views

Scrape / Extract Wikipedia Category Taxonomies into UI

I would like to know how to build a visual taxonomy tree from the categories on Wikipedia as a seed; are there any existing tools which might be able to be used in order to extract this type of ...
1
vote
0answers
259 views

Pulling bank information: Alternatives to Yodlee API?

Yodlee has an Instant Account Verification API that allows a user to login into their bank account, answer security questions and return back a list of accounts + transactions. We need a similar API ...
5
votes
3answers
2k views

Python library for complex web scraping?

My goal is to interact with a website (Portfolio123.com) and download financial study results. To get to the content means logging in, then clicking buttons and/or filling forms. Some of these ...
1
vote
0answers
50 views

Bulk web scraper which supports login system

I have a website which has thousands of pages and I need to extract some data from those into a database file. The website has a login. How can I bulk scrape data I need from all of these pages? ...
1
vote
1answer
2k views

API that gives VIN from a (US) License Plate Number

I'm trying to write an app that gives basic anonymous car data (make/model/year) from a picture. I assumed the best way to do that would be to take a license plate number, look up the VIN from the ...
3
votes
2answers
76 views

What are good tools for automatically crawling websites, check for certain links, and extract data?

I would like to automatically crawl about 1 million URLs (on a CSV file) and check whether the sites provide a Facebook Login button (i.e., users can create accounts by connecting their Facebook ...
5
votes
3answers
235 views

Web service that serves as public proof that a given url contains something at a given time

I did something "amazing" on a website and my narcissistic side wants proof for posterity that I managed to actually do it, even if the website in question decides to change it's rules or goes off-...