I am working on a piece of software using Python and Selenium to scrape content from a series of URLs in an effort to collect data on a specific market of products being sold.

Currently I am able to put that data into a CSV and then manipulate it with excel as needed. I would like to take this to the next level in terms of more robust reporting.

I would like to automate some of the reporting and in a sense create an initial template that could be used as an overview/summary that would be applicable to each of these subsets of data (as well as future data that I pull), as well as find a way to compare the different sets of data. My question here is what path(s) should I even be considering? I have begun to look into Pandas and it seems interesting, but are there other solutions I should look into?

If it helps to visualize, the data consists of a stores location, a list of products, prices, and other product related metrics.

In terms of the type of data I would like to show in the summary it would be fairly straight forward to start. Highest price / lowest price items per location, per category, per region, the ability to compare these data points across different locations and so on.

End goal for me is to be able to export some of this to excel when needed, but big picture I intend to write this information to a database and then display it on the web using Django.

I am still learning a lot and want to ensure Pandas is the best approach or if there are other routes that I should consider for my reporting/data analysis needs.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.