6

My goal is to interact with a website (Portfolio123.com) and download financial study results.

To get to the content means logging in, then clicking buttons and/or filling forms. Some of these buttons load using 'body onload'. The resulting HTML often contains a link to a downloadable Excel file which is generated specifically for the logged in user.

Therefore some of the requirements are:

  • Cookies.
  • Wait for "body onload".
  • Find (and click) an element by id, name or XPath.
  • Import a given table from HTML.
  • Run JavaScript.
  • Download to file and/or to object.

Some preliminary choices that I have come up with so far:

  • Selenium.
  • RoboBrowser. EDIT: Blends Requests and BeautifulSoup. Won't click JavaScript buttons.
  • Mechanize + ExecJS. EDIT: Mechanize didn't work for me. See https://stackoverflow.com/questions/30655568/login-to-a-website-using-python-and-mechanize
  • Scrapy. This seems like it aims at web crawling. But will it do the job well?
  • PythonWebKit. This looked promising but when I tried installing it into Python using pip install PyhthonWebKit which didn't work. I got the message: "No matching distribution found for PythonWebKit".
  • dryscrape.

Do any of these work?

Improve this question
4
  • I got this list from searching the web but I don't even know if they do the job. I hope that those of you with experience with these libraries can fill me in on this.
    – ChaimG
    Commented Jun 4, 2015 at 19:34
  • From your question, a few more things are quite unclear: are you looking for a library (to code your own application) – or for some ready-to-use application which you'd just need to configure? Also, the title is misleading – and might tempt the one or other to mark it for closure (we do not cover howtos, manuals, tutorials, and the like).
    – Izzy
    Commented Jun 4, 2015 at 19:48
  • 1
    You forgot to list Beautiful Soup (crummy.com/software/BeautifulSoup). I am unsure if it can do what you want, but it rated way up there with Scrapy, so while you are trying out the others, you can try it too.
    – Mawg
    Commented Jun 5, 2015 at 9:59
  • @Mawg BeautifulSoup is a powerful tool to parse HTML. It can be used on top of some of the other choices above when needed.
    – ChaimG
    Commented Jun 5, 2015 at 15:47

3 Answers 3

Reset to default
2
  • PyQt is one that might help outworking complex JS stuffed sites.
  • Selenium is ok, see an example.
  • DryScrape is ok, though I've not used it that much.
Improve this answer
1

It is very easy when you use dryscrape python. On a Ubuntu PC, installation is as follows:

# apt-get install qt5-default libqt5webkit5-dev build-essential \
python-lxml python-pip xvfb

Usage as follows:

Import dryscrape as d
    Import time #for refreshing or waiting for page
    d.start_xvfb() #for using this in linux without Xserver
    br = d.Session() #creating new session
    br.visit('<Any Url>') #for open the page
    Input = br.at_xpath('//*[@name="email"]') #for find input
    Input.set('<input value>')
    Input.form().submit() #for submit
    time.sleep(5) #wait for page load

This works fine on Android.

Improve this answer
0

Selenium can do anything what user can do in a browser, in a cross-browser way. It is W3C standard for browser automation, and learning Selenium is good skill to have for your career (beyond this web-scraping project).

It is widely used for test automation of web-based applications.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.