1

I've been retired for some years but I used to be an accomplished programmer. I've used, OOP and functional languages as well as Lisp and Prolog.

I'm using a high-spec Windows 7 desktop PC.

Question

I want to quick-hack some software that will automatically log in to a specified website, and extract text. It will need to be able to page through forums. I'll be pre-processing as I do the extraction and then post-processing whatever files I have created.

I'm very out of date with what languages are used these days. I could do it all in VBA for Excel but I don't think that's the best option. Also I'd like to be able to pass the software on to friends who may not have Excel.

Ideally I'm looking for a freely downloadable language with a simple IDE that's oriented towards text processing. I'd like a really quick learning curve so I'm avoiding heavy duty all-purpose languages that require me to faff about loading libraries etc.

All suggestions and reason for your choice welcome. Thanks.

3

Python plus one or more of Scrapy, Requests, Mechanize, etc.

  • Free Gratis & Open Source
  • Functional, Procedural or OO your choice Python does them all
  • Cross Platform From a RaspberryPi to a super computer cluster
  • Very fast to develop & debug Very succinct but clear language with a shallow learning curve and really good error reporting, stack trace, etc.
  • Lots of data import, manipulation & export both in the standard libraries and the supporting libraries
  • Great community support

Example using Scrapy, from the documentation site:

import scrapy

class StackOverflowSpider(scrapy.Spider):
    name = 'stackoverflow'
    start_urls = ['http://stackoverflow.com/questions?sort=votes']

    def parse(self, response):
        for href in response.css('.question-summary h3 a::attr(href)'):
            full_url = response.urljoin(href.extract())
            yield scrapy.Request(full_url, callback=self.parse_question)

    def parse_question(self, response):
        yield {
            'title': response.css('h1 a::text').extract()[0],
            'votes': response.css('.question .vote-count-post::text').extract()[0],
            'body': response.css('.question .post-text').extract()[0],
            'tags': response.css('.question .post-tag::text').extract(),
            'link': response.url,
        }

This can be run as:

scrapy runspider stackoverflow_spider.py -o top-stackoverflow-questions.json

Which I have just tried on my computer and it took less than 5 seconds to produce a 47k json file which starts:

screenshot

Simple Free IDE

There are a large number of free Python IDEs available with the preference being largely a matter of personal choice.

Python by default comes with idle which is workable but limited, personally I can recommend: - Wing IDE, the 101 version is free, - SPE, - Spyder, - eric ide

The wiki lists a lot of development environments - you can use even Eclipse with the pyDev package for full IDE functionality.

Supplementary Information

As mentioned in the comments I should also mention Beautiful Soup for completeness (and still go with Scrapy ,-) For an IDE, some don't think that you can beat the community edition of PyCharm which can also be used commercially. – (Thanks Mawg)

0

If you're comfortable with Windows and VBA, then I assume that you will manage to write some glue code in C#.

In general, you need at least 3 things:

  1. HttpClient class (and related classes), that will allow you to make http requests with http session support (by using cookies). Here's an example:

https://stackoverflow.com/questions/12373738/how-do-i-set-a-cookie-on-httpclients-httprequestmessage

  1. Library to extract text from html. You can of course do it directly by converting html to DOM and extracting individual fragments from DOM object tree, but I suggest to use Readability library instead. Here's Readability port to C#:

https://github.com/marek-stoj/NReadability

  1. As for free IDE for C#, look for Visual Studio Community 2013. It's free also for commercial use (for individuals, or up to 250 computers in single company).
0

Here is a vbscript I wrote that does that.

It checks the local weather.

It uses regex to analyze the text.

    Option Explicit

    Dim shell   : Set shell = CreateObject("WScript.Shell")
    Const url = "http://www.cdcc.usp.br/clima"
    Dim html, temp, umid, hora

    Function getHtml(byRef url)
        Dim xmlHttp : Set xmlHttp = CreateObject("MSXML2.XMLHTTP.6.0")
        xmlHttp.open "get", url, false
        xmlHttp.send
        getHtml = xmlHttp.responseText
    End Function

    Function parseRegEx(byRef sText, byRef regEx)
        Dim oRegx, matches, match, i
        Dim aResults : Redim aResults(0)
        Set oRegx = New RegExp
        With oRegx
            .Pattern    = replace(regEx, "`", chr(34) )
            .IgnoreCase = True
            .Global     = True
        End With

        Set matches = oRegx.Execute(sText)
        if (matches.Count>1) Then
            For Each match in matches
                'msgbox match.Value, 0, "Found Match"
                If match.SubMatches.Count > 0 Then
                    For i = 0 To match.SubMatches.Count-1
                        Redim Preserve aResults(UBound(aResults)+1)
                        aResults(UBound(aResults)-1) = match.SubMatches(i)
                    Next
                Else ' one reult only
                    aResults = match.Value
                End If

            Next
        ElseIf (matches.Count=1) then ' only one match found
            'msgbox matches(0).SubMatches(0)
            'aResults = matches(0).SubMatches(0)
            Redim aResults(1)
            aResults(0) = matches(0).SubMatches(0)
        end If 

        Set oRegx = nothing
        parseRegEx = aResults

    End Function


    html = getHtml(url)
    temp = parseRegEx(html, "<font face=`Arial` size=`5`>(.+)&deg;C<\/font><\/b><\/td>")(0)
    umid = parseRegEx(html, "umidade(?:.+\n).+(\d{2})%")(0)
    hora = parseRegEx(html, "Atualizada &agrave;s: <b>(\S+) <\/b>")(0)

    msgbox temp & "°C", "Umidade: " & umid & "%" & chr(9) & "(" & hora & ")"

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.