Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [scraping]

The tag has no usage guidance.

1
vote
0answers
12 views

Parse SEC Edgar on Pyhthon

I would like to scrape tables from the SEC such as this one. However, there is some JavaScript involved, and I don't know how to access specific tables, such as CONSOLIDATED BALANCE SHEETS. Is there ...
1
vote
0answers
13 views

Web Scraping for Coding Schools

Is there a fast and efficient way to create some web crawler or scraper to search for coding schools, their contact info, and location and put put them into an excel document or Google Sheet?
0
votes
0answers
28 views

Bookmarklet that extracts content from web page (with XPath) and puts it into a URL

I need a bookmarklet that extracts the content of certain HTML elements. This content should then be included in an external URL’s query component, which the bookmarklet either redirects to or ...
0
votes
0answers
11 views

Price comparison/history webscraping, processing, and then more scraping and processing

What I'm basically trying to do is create an automated way to compare a CSV/JSON file with keywords to sold listings on eBay. The overall flow would be something like this: Scrape data from a ...
0
votes
0answers
14 views

Fast and Stable Web Scraper For JavaScript Generated Contents

I am trying to get the page source of a url i.e. DOM using Javascript. It is working fine when I request from chrome extension, I am able to post DOM with the API. I want to do same in JAVA using the ...
1
vote
1answer
37 views

Web crawler that lets you specify depth of linked domains

I want to download every file (via HTTP, HTTPS and FTP, HTML, PHP as delivered, JS, CSS, linked PDF, images) from a domain, its subdomains, every domain linked from the previously mentioned pages and ...
2
votes
1answer
111 views

Foolproof way to scrape articles

I'm looking for a very, very simple way to scrape all articles released on several websites based on these criteria: I can't code, so it really has to be very straightforward (i.e. no Python or so) ...
2
votes
4answers
406 views

Web scraping app [closed]

I'd like a general purpose web page scraper app to create CSV files (XML or JSON could probably work too though as ultimately I just care that I can move it to some type of database somehow) without ...
0
votes
1answer
66 views

free, automatic, cloud-based, webscraper?

Years ago I used the Fetch Page module of Yahoo Pipes to turn some regularly-updated websites into RSS feeds. I could then use the rest of their tools to do what I want with it. I recently came ...
1
vote
2answers
68 views

Automate the download of the MP3 version of a show from a subscription-based RSS

I subscribe to a news site that frequently publish a page with a show in an enbeded video (the show) and a link to the MP3 of the show. The page is available to anyone but you need a valid ...
0
votes
1answer
250 views

Scraping information from email to workflow

I have an email delivered to a specific inbox which contains a number of fields I need to scrape. The fields will be subsequently fed into a workflow system. For example: From: example@other.com To:...
3
votes
0answers
44 views

Download Image from Keywords

I have around 50000 sentences, I want relevant images closely or partially related to those sentences. Before google image search APIv1 was best suited for my need as now it is deprecated need to find ...
2
votes
0answers
91 views

Mirroring a website like I see it after running JavaScript

My employer has created this wix website he wants me to replicate it from A to Z, I have done what I could, but some parts are still missing. He created the website and wants to move it to his server. ...
1
vote
0answers
298 views

Fast tool to mirror a website

I'm interested in downloading the entire contents of a website (mirroring). It took wget 8 hours to mirror a 3 GB website. I have also tried HTTrack, but the performance was the same. I've also ...
1
vote
2answers
42 views

Web scraping tool to scrape a dynamic page based on current day

I need to scrape a theatre webpage to know whether the bookings for next week has been opened. I am planning to write a scraper if the page has the required HTML element for the next Friday based on ...
0
votes
0answers
29 views

HTML link harvester

Can anyone suggest a program that can go through a list of URLs and scrape the URLs for a specific link? I have roughly about 1000 links that I want to find if it has a specified page which ideally ...
1
vote
0answers
50 views

A web scraper that recursively grabs all links on a page

I need to obtain all the links present in a group of web pages. Most of these pages are part of four websites, but some of these sites are not domain.com but rather domain.com/~site. Some of these ...
2
votes
1answer
73 views

Downloading all PDFs corresponding to a list of references

I am looking for a program, browser extension or website is that can download all PDFs corresponding to a list of references. Example: Given the input: Paoloni JA, Appleyard RC, Nelson J, ...
4
votes
2answers
144 views

Software to download all images from an entire website

I'm looking for software that will recursively crawl a website to download all of the images from the entire website. (Not just from a single webpage.) To illustrate: If the site is www.example.com ...
0
votes
1answer
218 views

Tool to extract text from multiple URLs and place in a file

I am in a situation where I have to analyze text data. I have URLs of numerous websites from where I need to rip text and place it in a file for further analysis. I tried few crawlers but they are ...
1
vote
2answers
28 views

Application or suite to compare site after updates

I am looking for a way to automatically scan our site for changes after we launch an update to our site. Of course, we have a service that scans for 404s, 500s, etc. But there are two problems we are ...
1
vote
0answers
114 views

Gratis data scraping tool to replace Import.io

I need to be able to scrape data from multiple websites regularly. Some coding is OK, but due to heavily constrained time the fewer things I have to learn like XPath, the better. I used to use Import....
2
votes
0answers
133 views

Check if HTML pages contain structured data, and output which syntax/vocabulary they use

I have a list of HTTP URLs and I want to know if the corresponding HTML pages contain structured data markup, and if yes, which syntaxes and which vocabularies they use. The tool can be anything: a ...
1
vote
0answers
40 views

Software to search for Youtube videos, given a search pattern

I wonder if exists a software that, given a search pattern like for example a song title, it can search for youtube videos and return the video urls of the search results. The application should ...
1
vote
0answers
43 views

Web scraping - appropriate programming language

Which language is the most adept at scraping the Web by parsing the page source in regards to reference material regarding the subject, i. e. official docs, and Stack Overflow questions. Also, ...
3
votes
1answer
93 views

Tool to extract text from HTML

Having a website, I'd like to extract content from it. So far I've been using Regex in Notepad++, but we all know we should not use Regex to parse HTML due to a redesign of the website, getting the ...
3
votes
1answer
68 views

Text-to-speech specific page element?

I'd like to convert a web page like this and have the program filter it for the comments, rather than include "permalink, source" or irrelevant page elements.
1
vote
0answers
198 views

Reliable website scraper working with URL parameters such as ?page=3

Abstract In my work I often have the need of document myself with many sources. Some of most useful ones rely on personal or corporate project blogs websites. Practically all of them are just simple ...
1
vote
0answers
96 views

Library to crawl Ajax-based websites fast

I would like to crawl Ajax websites with Java. I tried Selenium Webdriver and compared to Jsoup for example it is much slower (650ms vs 7.5s). Since Crawljax is based on Selenium Webdriver I haven't ...
1
vote
2answers
63 views

Application to compile a single book from fanfiction.net chapters

Needed: Software that can take a story posted in multiple chapters to fanfiction.net and then make it into a single text/book. Cost: free Desired technology: irrelevant (webapp, Windows, Linux, Perl/...
2
votes
2answers
612 views

Java library for scraping websites

I am looking for a good library to help me scrape information off of a certain website. The path the scraping has to follow can be rather dynamic, meaning that if the initial scrape is on a page of ...
2
votes
4answers
271 views

Whole website downloader

How do you download a static copy of a website? I mean, not loading scripts and using them later when you load the "offline" page, but saving the DOM as it is AFTER all the scripts made changes to the ...
4
votes
1answer
312 views

Scrape / Extract Wikipedia Category Taxonomies into UI

I would like to know how to build a visual taxonomy tree from the categories on Wikipedia as a seed; are there any existing tools which might be able to be used in order to extract this type of ...
1
vote
0answers
249 views

Pulling bank information: Alternatives to Yodlee API?

Yodlee has an Instant Account Verification API that allows a user to login into their bank account, answer security questions and return back a list of accounts + transactions. We need a similar API ...
5
votes
3answers
2k views

Python library for complex web scraping?

My goal is to interact with a website (Portfolio123.com) and download financial study results. To get to the content means logging in, then clicking buttons and/or filling forms. Some of these ...
1
vote
0answers
46 views

Bulk web scraper which supports login system

I have a website which has thousands of pages and I need to extract some data from those into a database file. The website has a login. How can I bulk scrape data I need from all of these pages? ...
1
vote
1answer
2k views

API that gives VIN from a (US) License Plate Number

I'm trying to write an app that gives basic anonymous car data (make/model/year) from a picture. I assumed the best way to do that would be to take a license plate number, look up the VIN from the ...
3
votes
1answer
65 views

What are good tools for automatically crawling websites, check for certain links, and extract data?

I would like to automatically crawl about 1 million URLs (on a CSV file) and check whether the sites provide a Facebook Login button (i.e., users can create accounts by connecting their Facebook ...
5
votes
3answers
223 views

Web service that serves as public proof that a given url contains something at a given time

I did something "amazing" on a website and my narcissistic side wants proof for posterity that I managed to actually do it, even if the website in question decides to change it's rules or goes off-...
6
votes
1answer
82 views

Tool for Creating List of Businesses Located in Specific Suburb

What kind of software allows you to list all businesses located in a specific suburb? I am thinking of a web scraper that will scrape a google search (of businesses in a suburb) but maybe there is ...
3
votes
0answers
122 views

Create thumbnails (screenshots) of websites from list of URLs

I'm looking for software that will take my list of URLs and for each one create a thumbnail of the site. It must: Be free software I can run on Windows or Debian Linux Create thumbnails of equal ...
1
vote
1answer
81 views

Email Scraper that also scrapes Meta Data

I'm trying to compile a list of email addresses of Deans of universities in the U.S. This is to send them an important policy email to help shape U.S. higher-education policy. I'm looking for an ...
1
vote
0answers
65 views

Mirror Third Party Web Services Internally

When using third party web services one often runs into problems where the usage of the service over the network causes performance issues, usage is limited and web service structural changes ...
2
votes
2answers
109 views

Text or summary extractor from webpage

I look for a software or add-on (Firefox) to extract the main text of a webpage. For example for a Wikipedia article it must extract the article and omit the navigation bars, references or headers ...
2
votes
1answer
213 views

GUI scraper allowing the extract a specified element of a webpage for a list of URLs having a similar structure

I am looking for a program that can extract a specified element of a webpage for a list of pages having a similar structure. To put it otherwise, I would first specify the location of the text that ...
2
votes
0answers
74 views

Readability-like program/library able to extract content from forum-like websites?

Does anybody know if there's any program/library able to extract "main contents" (like the Readability experiment) from a forum-like HTML page? open-source; should be able to extract: posts contents ...
3
votes
1answer
54 views

Check a web page and send an email when a change is detected

I have a need to be able to automate the following: Submit a webform (login) Check the next page for given text (it’s a date) Send an email (if date has changed) I have found a few page watches but ...
3
votes
1answer
518 views

Retrieve Google Scholar results

I am looking for a library that can send a query to Google Scholar and retrieve all the results in some exploitable format (JSON, XML, etc.). A CLI program is OK too provided there is no interaction ...
1
vote
2answers
823 views

Bulk image downloader for anime image boards?

I am looking for a bulk image downloader for anime image boards. For example: grab/download all photos that are tagged under 'Uta+no☆prince-sama♪' from zerochan.net in bulk and have the ability to ...
2
votes
2answers
221 views

Extracting the core content from a webpage

I have a list of 1,000 URLs pointing to news articles. I am looking for a program (CLI if possible) or libraries (C++/Java or interpreted languages preferred) that can take a URL as input and output ...