I am trying to run my web-scraping python code every day at a specific time without having to open my computer. I looked into AWS Lambda but when I tried to set up the layers to include selenium, pandas and chromedriver, the file size limit was exceeded. Is there an alternative approach that I could deploy my python app to run every day without high costs?
1
-
The question is how much CPU does your python need. Any linux has cron or some version of it that allows you to schedule things. To edit crontab -e . Depending on the resource required a Raspberry Pi4 or Pi 5 could be left on 24/7 on a shelf or etc with very low power requirements. Does this help?– cybernardCommented Oct 25, 2023 at 15:43
Add a comment
|
1 Answer
Let's stick to AWS since you will need a switched-on computer, even if not your own. So, you have a Python (the Pandas gave it away), Selenium and Chromium driver stack: Set up a small EC2 instance, whichever size suits you. But t2.micro is free if you are on the Free Tier, and t4g.small is on free trial until 31 Dec 2023.
Then either configure a cron job on the instance to start the scraping session or, if you are feeling adventurous, set up an AWS Lambda job to create the EC2 instance and destroy it again after the session. You may also want to consider the AWS Resource Scheduler service.
With Resource Scheduler, you can choose to automatically stop and start instances across multiple AWS Regions and AWS accounts according to a schedule you define. The Quick Setup configuration targets Amazon EC2 instances using the tag key and value that you specify.
