I want to scrape data from a website which has a text field and a button, on click of the button and according to the data in the text field a dynamic table is generated through AJAX.

I have used Jsoup for simple scraping but after researching found out that it does not support scraping of a dynamic page.

I would like to know some of the best scrapers around with the level of functionality and level of difficulty. I'm looking for free Java code; the final product is intended to run on Windows 7.

  • Can you code Python? If so, I have an answer for you. – Mawg says reinstate Monica Feb 19 '15 at 9:13
  • Selenium is free and it has a Java version. The proposed solution is in Python since I am more comfortable with it but the Java version is very similar. – aberna Feb 19 '15 at 9:45
  • ok il hav to try out selenium before ticking your answer right, i'l also wait for more answers, hw is HTMLunit? – Afroz Shaikh Feb 19 '15 at 9:48
  • This solution is for Python, but you can apply the principles in JS: blog.databigbang.com/web-scraping-ajax-and-javascript-sites – Mawg says reinstate Monica Feb 19 '15 at 10:18
  • @AfrozShaikh so did you try the proposed solution ? – aberna Feb 22 '15 at 7:56

The issue you are facing is quite common. Usually a basic scraper is not able to deal with ajax content loading. You should make usage of a library which let you wait for content loading, check if element are visible and so on.

So more than a basic scraper you would need a tool for automating browser navigation.

By my experience I would suggest you the usage of the framework Selenium Webdriver (http://docs.seleniumhq.org/projects/webdriver/).

It is very popular on Stack Overflow (https://stackoverflow.com/tags/selenium/info)

Selenium is a portable software testing framework for web browsers automation in a number of popular programming languages, including Java, C#, Ruby, Groovy, Python, PHP, and Perl. Test playback is possible in most modern web browsers. Selenium deploys on Windows, Linux, and Macintosh platforms.

A simple example of usage in python from documentation. Example with other languages here http://docs.seleniumhq.org/docs/03_webdriver.jsp#introducing-the-selenium-webdriver-api-by-example

from selenium import webdriver
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.support.ui import WebDriverWait # available since 2.4.0
from selenium.webdriver.support import expected_conditions as EC # available since 2.26.0

# Create a new instance of the Firefox driver
driver = webdriver.Firefox()

# go to the google home page

# the page is ajaxy so the title is originally this:
print driver.title

# find the element that's name attribute is q (the google search box)
inputElement = driver.find_element_by_name("q")

# type in the search

# submit the form (although google automatically searches now without submitting)

    # we have to wait for the page to refresh, the last thing that seems to be updated is the title
    WebDriverWait(driver, 10).until(EC.title_contains("cheese!"))

    # You should see "cheese! - Google Search"
    print driver.title


Note: In case you need an advanced scraper solution you can make usage of Scrapy(python library) joined with Selenium.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.