I need to build a competitors prices tracking tool for 100,000 products from around 15 competitors? What would be the best way to do it? Please recommend?


Read the web scraping chapter of the excellent (and free) eBook "Automate the boring stuff with Python".

That will tell you everything that you need to know.

  • This is insightful. But my problem is I don't have competitors URLs for each product. I just have product IDs that are consistent with competitors. The solution should search each individual product ID on the competitors' website, extract prices and keep track of prices. (For Example, I have QWER-1234 as a product id. The solution should search on Amazon, Alibaba,ebay etc. for QWER-1234 and should give me current prices). – Farhan Mohammed May 23 at 14:45
  • That's not going to happen without some human intervention – Mawg May 24 at 6:45
  • About the only good news is that it you have to do it manually once for each site then you can automate it aft er that. Since it's only 15 sites, that's not too bad, as I would expect them top have the same layout on every page. – Mawg May 24 at 6:46
  • Can you please elaborate a bit – Farhan Mohammed May 24 at 13:22
  • 1
    done I have coded it. – Farhan Mohammed Jun 2 at 3:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.