Python plus one or more of Scrapy, Requests, Mechanize, etc.
- Free Gratis & Open Source
- Functional, Procedural or OO your choice Python does them all
- Cross Platform From a RaspberryPi to a super computer cluster
- Very fast to develop & debug Very succinct but clear language with a shallow learning curve and really good error reporting, stack trace, etc.
- Lots of data import, manipulation & export both in the standard libraries and the supporting libraries
- Great community support
Example using Scrapy, from the documentation site:
name = 'stackoverflow'
start_urls = ['http://stackoverflow.com/questions?sort=votes']
def parse(self, response):
for href in response.css('.question-summary h3 a::attr(href)'):
full_url = response.urljoin(href.extract())
yield scrapy.Request(full_url, callback=self.parse_question)
def parse_question(self, response):
'title': response.css('h1 a::text').extract(),
'votes': response.css('.question .vote-count-post::text').extract(),
'body': response.css('.question .post-text').extract(),
'tags': response.css('.question .post-tag::text').extract(),
This can be run as:
scrapy runspider stackoverflow_spider.py -o top-stackoverflow-questions.json
Which I have just tried on my computer and it took less than 5 seconds to produce a 47k json file which starts:
Simple Free IDE
There are a large number of free Python IDEs available with the preference being largely a matter of personal choice.
Python by default comes with idle which is workable but limited, personally I can recommend:
- Wing IDE, the 101 version is free,
- eric ide
The wiki lists a lot of development environments - you can use even Eclipse with the pyDev package for full IDE functionality.
As mentioned in the comments I should also mention Beautiful Soup for completeness (and still go with Scrapy ,-) For an IDE, some don't think that you can beat the community edition of PyCharm which can also be used commercially. – (Thanks Mawg)