1

I need to get some content from a website. The problem is, when I use HTTrack Website Copier I get only the HTML. But the content I want is requested in PHP. Is there some tool which visits each link and can save the content (or the whole page, I can filter out later) I want to HTML?

Let me give you some more info about this website. It is a website with all educational institutions in my country. This educational institutions are categorized in provinces. When you click on a province, you got a list with places. When you click on a place, you can choose between levels (like primary school, secondary school etc.). Then the site redirect you to a php page inside a framework. This happens with this code:

<form name="x" action="schools.php" target="work" method="POST"></form>
<a class="mlink" href="javascript:document.x.submit();">

On this page you see the names of the schools. When you click on this, you will been redirected to a contact page.

<form name="x" action="details.php" target="work" method="POST"></form>
<a class="sort" href="javascript:document.x.submit();">

I need from this page the email adress, from all secondary schools in my country (about 1000).

So there are so many steps and I do not want to go 1 by 1. Because you need to click on province->place->type of education->schoolname.

Is there a solution?

1

PHP is actually a scripting language that is running on the server to produce the results that you need. Worse still at least some of the processing is performed on the local machine in JavaScript.

Without doing some digging I cannot say how much the JavaScript contributes to the final results - I can't tell what it is doing without looking at the specific web site but when you select the province it probably posts a PHP query to to populate the places part of the form and so on.

The first thing I would suggest is giving the Selenium web driver a go - with it you can automate your browser actions for any compatible browser driving it in one of several languages, (personally I would recommend Python), to direct it to the pages that you need and capturing them - this is reasonably sure to work as your browser would still be executing the JavaScript components for you.

Alternatively, if you have a specific list of province, place, type, schools to use you could explore using python plus the requests or mechanize libraries to populate and submit the form with all fields filled in. This would normally work OK as the server is not usually able to store the stages of filling in the form and relies on the form storing the information for it - be careful however about any alternatively spelt names or cases where the form has a plain text name and an ID that differs.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.