My goal is to interact with a website (Portfolio123.com) and download financial study results.
To get to the content means logging in, then clicking buttons and/or filling forms. Some of these buttons load using 'body onload'. The resulting HTML often contains a link to a downloadable Excel file which is generated specifically for the logged in user.
Therefore some of the requirements are:
- Wait for "body onload".
- Find (and click) an element by id, name or XPath.
- Import a given table from HTML.
- Download to file and/or to object.
Some preliminary choices that I have come up with so far:
- Mechanize + ExecJS. EDIT: Mechanize didn't work for me. See https://stackoverflow.com/questions/30655568/login-to-a-website-using-python-and-mechanize
- Scrapy. This seems like it aims at web crawling. But will it do the job well?
- PythonWebKit. This looked promising but when I tried installing it into Python using pip install PyhthonWebKit which didn't work. I got the message: "No matching distribution found for PythonWebKit".
Do any of these work?