The Stack Overflow podcast is back! Listen to an interview with our new CEO.

Questions tagged [scraping]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
-2
votes
0answers
5 views

Introduce me to data scraping please. I have a particular workflow in mind. How would you guys suggest I approach this? [on hold]

I appreciate you taking your valuable time to look through this question! I have a list of companies. I'd like to find a way to automate the process of looking these companies up on linkedIn, ...
0
votes
0answers
3 views

Small remote server for running a script that pings a website infrequently

I have a short script in Python that I would like to have ping a webpage every 30 minutes or so. The script then sends me an email if a particular node of page's HTML response has changed (essentially ...
0
votes
0answers
6 views

Scraping Wikipedia to build a database of musicians and their hometown/origin

I have to scrape Wikipedia to create a SQL database of musicians and producers along with their hometown or origin. What are some tools/programs I can use? Thanks so much!
0
votes
0answers
8 views

Scrapping different websites with different formats with Python

For a university project we are asked to scrap different portals that contains the same information in a different format. Some of the website have the information to be scrapped as PDF, HTML, XML or ...
0
votes
0answers
3 views

What is the fastest possible way to scrape large quantities of tweets? Is there a particular twitter library that excels with speed?

If I'd like to scrape a large quantity of tweets very often -- say 10000 a minute -- what would be the ideal library to achieve this? I've looked at Twint at Twitterscraper, but am not sure which one ...
0
votes
0answers
9 views

web scraping facebook and instagram

I want to scrap facebook using keywords and extract the pages which contains those keywords,same with instagram also. So any tool or script which would help in scraping?
1
vote
0answers
6 views

Great scraping tools/ data mining tools for bulk of websites, extracting few emails from each?

I have been looking for good open source tools / softwares that can read certain patterns and scrape a bulk of provided websites... My use case is like this: I have a list of ~900 company websites ...
0
votes
0answers
3 views

Looking for architecture to display in website ordered data stream from API

My goal is to gather similar data from different websites trough their API, sort them in real-time and display a chosen cell (typically highest or lowest) in real-time on a given website. What would ...
1
vote
0answers
18 views

Software package for scraping/crawling that resists well to website changes

I am writing scrapers for around 100 websites using alltheplaces, built on Scrapy. My main problem is the sites keep changing. I am extracting data either from JSON files directly or by parsing them ...
0
votes
0answers
10 views

Is there software to scrape specific tags for YouTube/Twitch channels?

I help manage a small 'media team' for a community. We hold our content creators to a schedule of uploading SOME form of our content to their channels every month on either Twitch or YouTube. ...
0
votes
0answers
15 views

extract data from website

I would like to extract comments or star ratings from a profile user of Amazon. Does anyone have an idea of a tool that can help me with that? I tried to use import.io but it didnt work well for me. ...
-1
votes
1answer
9 views

What Database type should I use for a scraped data comes from many websites?

I'm building a REST API using NodeJS storing a scraped data that comes from many websites ( the API I'm building is about selling - buying used cars ) There's a potential to scale in the future ...
0
votes
0answers
15 views

Batch word count tool for wikipedia

I have an excel sheet with wikipedia urls. They are stored in column A and every row has one url. The number of urls is around 1'000. I would like to get several statistics for every url, in ...
0
votes
0answers
8 views

how to web-scrape a page using a large list of keywords as a input for searches

I have a list of institutions (universities, approx 3500) in csv format and I want to extract the information on scientific articles, they've published in SCOPUS. To do so, names of the universities ...
0
votes
0answers
9 views

How to get my following post on Instagram

I will detail a little more my requirement, or my need. I follow on my personal Instagram account several pages and people all around the world. I would like to get all the people / page I follow, ...
0
votes
0answers
17 views

Best library for web scraping Python

I am trying to extract some information from the following website: https://reportcard.statesatrisk.org/report-card Specifically, I want to be able to retrieve the 'State Grade' column for each of ...
2
votes
1answer
20 views

PDF website crawler

I need an open source system that can crawl websites to gather PDF documents and then index them.
0
votes
0answers
15 views

Is there a lighter alternative to puppeteer for simulated clicking?

I'm writing an application that navigates to a website, clicks a button on the website (which updates it) and then extracts information from the website. The application requires a headless browser, ...
0
votes
0answers
5 views

API for Web Scraping from Google in Python

I wanna Code my own Virtual Assistant in Python, but I can’t find any free API to get data from Google. I wanna get answers to questions like: “How old is Tim Cook” or “Where is Tim Cook born”. Do ...
0
votes
0answers
13 views

Alternative to Diffbot

I am looking for an API that can accept a URL for a product and return a few details, like title, price, and a product image URL. Diffbot seems to be exactly what I am looking for, however, their ...
0
votes
0answers
8 views

Is there an API for notification whenever the content of a website or web page changes (like website-watcher service)?

I was writing a small software application which monitors websites for changes. If any change in the text, images or other files on a page occurs, the user should be notified. I was doing it by ...
0
votes
1answer
23 views

Competitors price tracking tool to automate price tracking for 100000 products?

I need to build a competitors prices tracking tool for 100,000 products from around 15 competitors? What would be the best way to do it? Please recommend?
0
votes
0answers
6 views

Crawling a JS-heavy web page with Chrome Headless, saving to html, convert to PDF

Is there a GUI- or commandline-based tool that can crawl a Javascript-heavy web site and save the ~50 pages encountered to .html files? Then I'd like to convert the HTML pages to PDF. Finally I'd like ...
0
votes
1answer
16 views

Recommendations for web scraping tools that require minimal installation

My work uses a custom database with addresses and other personal information. In my role I only have access to an html version. I frequently need to make letters based on this information, but ...
1
vote
0answers
15 views

Automation tool for getting information from various sites

I am searching for a software that I can use to get information from websites. I've seen some 'scraping' and 'RPA' tools that you can make rules based on CSS or XPaths. The thing is that there are ...
1
vote
0answers
13 views

wget alternative with headless browser (webscraper with rendered JS)?

Is there a recursive web crawler like wget except that it executes javascript (in a headless browser)? I just need to scrape some online documentation for a holiday, I don't want to write one myself (...
0
votes
1answer
16 views

please suggest me an API or suggestion to list facilities around an piece of land

I am building an android app for property listings for farmlands and vacant lands, it should include a feature to extract list of nearby facilities like, roads, highways, water-bodies, factories, ...
0
votes
0answers
9 views

Web Scraper that Crawls Links Within PDFs

I have an interesting crawler requirement. Essentially I have a site that hosts a TON of PDF files and many of these PDF files have links to other PDF files. Said links are not present within html ...
2
votes
1answer
35 views

internet data scraping tools

I'm a little distressed, for my job I need to get some info from a website, specifically http://www.museumsusa.org/museums/ , I need for every museum in America to get the name, address, phone number ...
1
vote
0answers
33 views

Scrape Facebook public event pages

I need to web scrape facebook for some data. I only need to visit 50 facebook pages every like 1 hour. I will be visting pages of events. I need to scrape the event name, the event time, and the ...
1
vote
1answer
54 views

Web scraping tool/software availabe for free?

I am looking for a free ready-to-use web scraping application. Any recommendations? Usage: Monitoring online product prices, vendors, top 20 online market places in Asia and comparing price ...
1
vote
1answer
29 views

How to scrape a dynamic webpage

I want to scrape webpages with dynamically loaded content. As a user browsing this webpage, I have to click on an HTML element before the content of interest is dynamically loaded into the page. I ...
0
votes
0answers
11 views

How to host a clone website on my local machine

I wanted to clone my website on localhost for making some changes. I did the cloning using wget. What I got is HTML pages. Now what software can I use to host this cloned website on localhost? Any ...
0
votes
0answers
12 views

getting the sci-hub url using the file title/URL

I have a .txt file that contains a list of 854 articles urls and titles (from the different journals and onferences sites). I want to have the list of sci-hub urls of thesse files. is there possible? ...
1
vote
1answer
14 views

SalesForce and Excel - Chrome Extension the Best Solution?

I work with a sales team who frequently enter OPPs into SalesForce. Part of this process requires the sales rep to manually: Google customer City+State to find Zip Codes Use an in-house spreadsheet ...
1
vote
0answers
18 views

Parse SEC Edgar on Pyhthon

I would like to scrape tables from the SEC such as this one. However, there is some JavaScript involved, and I don't know how to access specific tables, such as CONSOLIDATED BALANCE SHEETS. Is there ...
1
vote
0answers
23 views

Web Scraping for Coding Schools

Is there a fast and efficient way to create some web crawler or scraper to search for coding schools, their contact info, and location and put put them into an excel document or Google Sheet?
0
votes
0answers
18 views

Price comparison/history webscraping, processing, and then more scraping and processing

What I'm basically trying to do is create an automated way to compare a CSV/JSON file with keywords to sold listings on eBay. The overall flow would be something like this: Scrape data from a ...
1
vote
1answer
52 views

Web crawler that lets you specify depth of linked domains

I want to download every file (via HTTP, HTTPS and FTP, HTML, PHP as delivered, JS, CSS, linked PDF, images) from a domain, its subdomains, every domain linked from the previously mentioned pages and ...
2
votes
0answers
119 views

Foolproof way to scrape articles

I'm looking for a very, very simple way to scrape all articles released on several websites based on these criteria: I can't code, so it really has to be very straightforward (i.e. no Python or so) ...
1
vote
1answer
78 views

Automate the download of the MP3 version of a show from a subscription-based RSS

I subscribe to a news site that frequently publish a page with a show in an enbeded video (the show) and a link to the MP3 of the show. The page is available to anyone but you need a valid ...
0
votes
1answer
342 views

Scraping information from email to workflow

I have an email delivered to a specific inbox which contains a number of fields I need to scrape. The fields will be subsequently fed into a workflow system. For example: From: example@other.com To:...
3
votes
0answers
48 views

Download Image from Keywords

I have around 50000 sentences, I want relevant images closely or partially related to those sentences. Before google image search APIv1 was best suited for my need as now it is deprecated need to find ...
2
votes
0answers
97 views

Mirroring a website like I see it after running JavaScript

My employer has created this wix website he wants me to replicate it from A to Z, I have done what I could, but some parts are still missing. He created the website and wants to move it to his server. ...
1
vote
0answers
323 views

Fast tool to mirror a website

I'm interested in downloading the entire contents of a website (mirroring). It took wget 8 hours to mirror a 3 GB website. I have also tried HTTrack, but the performance was the same. I've also ...
1
vote
2answers
61 views

Web scraping tool to scrape a dynamic page based on current day

I need to scrape a theatre webpage to know whether the bookings for next week has been opened. I am planning to write a scraper if the page has the required HTML element for the next Friday based on ...
0
votes
0answers
34 views

HTML link harvester

Can anyone suggest a program that can go through a list of URLs and scrape the URLs for a specific link? I have roughly about 1000 links that I want to find if it has a specified page which ideally ...
1
vote
0answers
71 views

A web scraper that recursively grabs all links on a page

I need to obtain all the links present in a group of web pages. Most of these pages are part of four websites, but some of these sites are not domain.com but rather domain.com/~site. Some of these ...
2
votes
1answer
74 views

Downloading all PDFs corresponding to a list of references

I am looking for a program, browser extension or website is that can download all PDFs corresponding to a list of references. Example: Given the input: Paoloni JA, Appleyard RC, Nelson J, ...
4
votes
2answers
288 views

Software to download all images from an entire website

I'm looking for software that will recursively crawl a website to download all of the images from the entire website. (Not just from a single webpage.) To illustrate: If the site is www.example.com ...