2

I am looking for a program that can extract a specified element of a webpage for a list of pages having a similar structure.

To put it otherwise, I would first specify the location of the text that is of interest to me, e.g. the title of a StackExchange question:

enter image description here

then give a list of URLs pointing to other StackExchange questions (i.e. similar page structure). The scraper should extract the titles from those pages. It's okay if the information to be extracted must be in a specific div or span.

Free, Windows 7, GUI and open source are better.

  • Of course It's okay if the information to be extracted must be in a specific div or span. It's not just OK, it's a requirement. Text on a web page is/has no identifying characteristic, the software would have to analyze the HTML structure of your sample page, determine what HTML element the text is in, and can then extract the contents of the same HTML element on the other URLs you specified. – user416 Jul 19 '15 at 11:42
2

You can do exactly this sort of thing with python and scrapy.

Regarding your specific needs:

  • Free - Yes
  • Windows 7 - Cross platform so Yes
  • Open Source - Yes
  • GUI - Not inherently but there are a lot of python tools to allow you to roll your own GUI if you need them these include tk (comes with python), wxPython (my prefered) and lots of others.

There is plenty of online documentation and a stack overflow tag as one possible source of help & examples. An example of the command line usage, from the Scapy web page:

pip install scrapy
cat > myspider.py <<EOF

from scrapy import Spider, Item, Field

class Post(Item):
    title = Field()

class BlogSpider(Spider):
    name, start_urls = 'blogspider', ['http://blog.scrapinghub.com']

    def parse(self, response):
        return [Post(title=e.extract()) for e in response.css("h2 a::text")]

EOF
scrapy runspider myspider.py

When I run the above I get:

2014-10-22 06:18:04+0100 [blogspider] DEBUG: Scraped from <200 http://blog.scrapinghub.com>
        {'title': u'XPath tips from the web scraping\xa0trenches'} 2014-10-22 06:18:04+0100 [blogspider] DEBUG: Scraped from <200 http://blog.scrapinghub.com>
        {'title': u'Introducing Data Reviews'} 2014-10-22 06:18:04+0100 [blogspider] DEBUG: Scraped from <200 http://blog.scrapinghub.com>
        {'title': u'Extracting schema.org microdata using Scrapy selectors and\xa0XPath'} 2014-10-22 06:18:04+0100 [blogspider] DEBUG: Scraped from <200 http://blog.scrapinghub.com>
        {'title': u'Announcing Portia, the open source visual web\xa0scraper!'} 2014-10-22 06:18:04+0100 [blogspider] DEBUG: Scraped from <200 http://blog.scrapinghub.com>
        {'title': u'Optimizing memory usage of scikit-learn models using succinct\xa0Tries'} 2014-10-22 06:18:04+0100 [blogspider] DEBUG: Scraped from <200 http://blog.scrapinghub.com>
        {'title': u'Open Source at\xa0Scrapinghub'} 2014-10-22 06:18:04+0100 [blogspider] DEBUG: Scraped from <200 http://blog.scrapinghub.com>
        {'title': u'Looking back at\xa02013'} 2014-10-22 06:18:04+0100 [blogspider] INFO: Closing spider (finished)

Which are the header 2 elements from the referenced site.

| improve this answer | |
  • 3
    Could you elaborate a bit more on how this would be done? Reads like the OP needs to code this himself (rather than having a ready made tool) correct? – Angelo Fuchs Oct 21 '14 at 18:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.