1

I have a website where I want to automate some tasks and make some "bot-like" program that will run on it. I am struggling to figure out the best option for this task. My initial thoughts have been to use Selenium with PHP or Javascript. I have used Selenium in the past for regression testing on applications I have worked on, so not quite the same purpose. I need something that can run headless on a Ubuntu server and that can also capture screen captures of the page (so that I can send updates to myself with pictures).

Goals of the tool:

  • Navigate across multiple HTML based pages that also rely on some Javascript rendering for their look
  • Send some requests to API's on the site (the API's are not enough, it still needs to go through multiple web pages and click on things via a mouse click, API is more for read-only access)
  • Take full page screenshots
  • Run headless on a Ubuntu server
  • Use either Javascript (can use things like node too), PHP, or Python.

I really am lost on which tool to use, and whether Selenium is the right tool to help me with this project, or if there are better tools designed for this purpose out there. I won't be doing anything rather complex, just trying to automate some repetitive clicking tasks on a webpage that cause my carpal tunnel issues.

Improve this question

3 Answers 3

Reset to default
1

Puppeteer is a Node-based tool for controlling headless (or full) Chrome. For example, it can

  • Generate screenshots and PDFs of pages.
  • Crawl a SPA (Single-Page Application) and generate pre-rendered content (i.e. "SSR" (Server-Side Rendering)).
  • Automate form submission, UI testing, keyboard input, etc. Create an up-to-date, automated testing environment.
  • Run your tests directly in the latest version of Chrome the latest JavaScript and browser features.
  • Capture a timeline of your site to help diagnose performance issues.
  • Test Chrome Extensions.
Improve this answer
1

Scrapy sounds like it might do the trick - it's written in Python and is really powerful. It is run headless, can perform navigation, take screenshots, call APIs, and much more.

Improve this answer
1
  • The screenshot taking looks to be a separate tool using Selenium. Do you think Scrappy is okay for this despite being designed more towards scraping?
    – ComputerLocus
    May 7, 2019 at 21:22
1

You could also use Laravel Dusk, it's also a headless Chrome browser like the other answer, but PHP and will work on cheap shared hosting. It's primary focus is testing, but can be used for automating tasks just as well.

Improve this answer
1
  • 1
    Interesting, I may give that a shot as I have used the Laravel framework and work more with PHP.
    – ComputerLocus
    May 8, 2019 at 15:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.