Programming language to extract and analyse text and stats from online forums. MS Windows 7 -

Question

I've been retired for some years but I used to be an accomplished programmer. I've used, OOP and functional languages as well as Lisp and Prolog.

I'm using a high-spec Windows 7 desktop PC.

Question

I want to quick-hack some software that will automatically log in to a specified website, and extract text. It will need to be able to page through forums. I'll be pre-processing as I do the extraction and then post-processing whatever files I have created.

I'm very out of date with what languages are used these days. I could do it all in VBA for Excel but I don't think that's the best option. Also I'd like to be able to pass the software on to friends who may not have Excel.

Ideally I'm looking for a freely downloadable language with a simple IDE that's oriented towards text processing. I'd like a really quick learning curve so I'm avoiding heavy duty all-purpose languages that require me to faff about loading libraries etc.

All suggestions and reason for your choice welcome. Thanks.

Answer 1

Python plus one or more of Scrapy, Requests, Mechanize, etc.

• Free Gratis & Open Source
• Functional, Procedural or OO your choice Python does them all
• Cross Platform From a RaspberryPi to a super computer cluster
• Very fast to develop & debug Very succinct but clear language with a shallow learning curve and really good error reporting, stack trace, etc.
• Lots of data import, manipulation & export both in the standard libraries and the supporting libraries
• Great community support

Example using Scrapy, from the documentation site:

import scrapy

class StackOverflowSpider(scrapy.Spider):
    name = 'stackoverflow'
    start_urls = ['http://stackoverflow.com/questions?sort=votes']

    def parse(self, response):
        for href in response.css('.question-summary h3 a::attr(href)'):
            full_url = response.urljoin(href.extract())
            yield scrapy.Request(full_url, callback=self.parse_question)

    def parse_question(self, response):
        yield {
            'title': response.css('h1 a::text').extract()[0],
            'votes': response.css('.question .vote-count-post::text').extract()[0],
            'body': response.css('.question .post-text').extract()[0],
            'tags': response.css('.question .post-tag::text').extract(),
            'link': response.url,
        }

This can be run as:

scrapy runspider stackoverflow_spider.py -o top-stackoverflow-questions.json

Which I have just tried on my computer and it took less than 5 seconds to produce a 47k json file which starts:

Simple Free IDE

There are a large number of free Python IDEs available with the preference being largely a matter of personal choice.

Python by default comes with idle which is workable but limited, personally I can recommend: - Wing IDE, the 101 version is free, - SPE, - Spyder, - eric ide

The wiki lists a lot of development environments - you can use even Eclipse with the pyDev package for full IDE functionality.

Supplementary Information

As mentioned in the comments I should also mention Beautiful Soup for completeness (and still go with Scrapy ,-) For an IDE, some don't think that you can beat the community edition of PyCharm which can also be used commercially. – (Thanks Mawg)

Answer 2

If you're comfortable with Windows and VBA, then I assume that you will manage to write some glue code in C#.

In general, you need at least 3 things:

1. HttpClient class (and related classes), that will allow you to make http requests with http session support (by using cookies). Here's an example:

https://stackoverflow.com/questions/12373738/how-do-i-set-a-cookie-on-httpclients-httprequestmessage

2. Library to extract text from html. You can of course do it directly by converting html to DOM and extracting individual fragments from DOM object tree, but I suggest to use Readability library instead. Here's Readability port to C#:

https://github.com/marek-stoj/NReadability

3. As for free IDE for C#, look for Visual Studio Community 2013. It's free also for commercial use (for individuals, or up to 250 computers in single company).

Answer 3

Here is a vbscript I wrote that does that.

It checks the local weather.

It uses regex to analyze the text.

    Option Explicit

    Dim shell   : Set shell = CreateObject("WScript.Shell")
    Const url = "http://www.cdcc.usp.br/clima"
    Dim html, temp, umid, hora

    Function getHtml(byRef url)
        Dim xmlHttp : Set xmlHttp = CreateObject("MSXML2.XMLHTTP.6.0")
        xmlHttp.open "get", url, false
        xmlHttp.send
        getHtml = xmlHttp.responseText
    End Function

    Function parseRegEx(byRef sText, byRef regEx)
        Dim oRegx, matches, match, i
        Dim aResults : Redim aResults(0)
        Set oRegx = New RegExp
        With oRegx
            .Pattern    = replace(regEx, "`", chr(34) )
            .IgnoreCase = True
            .Global     = True
        End With

        Set matches = oRegx.Execute(sText)
        if (matches.Count>1) Then
            For Each match in matches
                'msgbox match.Value, 0, "Found Match"
                If match.SubMatches.Count > 0 Then
                    For i = 0 To match.SubMatches.Count-1
                        Redim Preserve aResults(UBound(aResults)+1)
                        aResults(UBound(aResults)-1) = match.SubMatches(i)
                    Next
                Else ' one reult only
                    aResults = match.Value
                End If

            Next
        ElseIf (matches.Count=1) then ' only one match found
            'msgbox matches(0).SubMatches(0)
            'aResults = matches(0).SubMatches(0)
            Redim aResults(1)
            aResults(0) = matches(0).SubMatches(0)
        end If 

        Set oRegx = nothing
        parseRegEx = aResults

    End Function


    html = getHtml(url)
    temp = parseRegEx(html, "<font face=`Arial` size=`5`>(.+)&deg;C<\/font><\/b><\/td>")(0)
    umid = parseRegEx(html, "umidade(?:.+\n).+(\d{2})%")(0)
    hora = parseRegEx(html, "Atualizada &agrave;s: <b>(\S+) <\/b>")(0)

    msgbox temp & "°C", "Umidade: " & umid & "%" & chr(9) & "(" & hora & ")"