3

My open source software has volunteers who announce each new release (or other important news) on various channels (official Facebook page, official Twitter account, official mailing list, unofficial Reddit group, etc).

We would like to track these announcements, so that no channel is left behind, and so that we can get an idea of what type of news are popular.

Requirements:

  • Webapp
  • Each channel can be registered
  • For each news, link to the news on each channel
  • Popularity of each news on each channel (for instance on Facebook that could be number or like+comments, on a mailing list that would be number of replies)
  • Statistics, or raw data export
  • Free
  • Bonus if the webapp can be used by several admins, but everyone using the same login/password is also OK.
  • We don't mind the data being visible by anyone, it's OK.

It could look like this (or not): enter image description here ... with each link pointing to the relevant post. The admin interface would have a button to add a new announcement row, and a way to replace "not posted" with a link to the announcement on that channel.

Improve this question

2 Answers 2

Reset to default
0

I doubt some prepared tool for all this exist out of the box.

You face following problems:

  • each frontend should be scripted separately
  • you should change script on each change of the front-end (for example Linked-in changes frequently on the go)
  • you should save data into single database / output

I would recommend to implement scripts in automation testing tools. Selenium Webdriver and some programming language to "drive" the webdriver.(java, ruby, python) could be enough for this task.

You will have to employ a medium skilled programmer with knowledge of the chosen language, Xpath and (optionally) Selenium (it is very simple), but you will be able to crawl practically ANY content.

Improve this answer
0

I would suggest that you include a reviews page, possibly as markdown, in your repository that the volunteers can edit with their review links and commit just like any other source code. You can then have a python script that gets run daily/weekly to read from that markdown file and run through the links gathering the statistics on each link. You could even output to a .csv file against the scan date so as to be able graph trends.

That script could update the README.md or another page in the documents and post it to the repository. If you are using GitHub then updates to the README.md get rendered automatically on the repository home page - there is also support for Sphinx documentation to be automatically rendered - many/most source code repositories support this functionality.

Why Python

Python is especially well suited to this because:

  • the rich ecosystem includes tools such as:
    • requests-html which make gathering the data easy. You will possibly need a separate filter for gathering the data from each channel.
    • The Sphinx Documentation tool chain
    • Lots of graph generation tools
  • You can easily set python scripts to run as a service and/or with commit hooks
  • You don't need to worry what platform your script is run on
  • You could use a cloud service for this task and most have python as a major tool option.
  • This would make a good first project for someone learning python.

Additional points

  • Most repositories & version control systems allow you to select which authors can edit which files.
  • Many/most present a "home" page that automatically updates when changes to README.md or some other file are committed.
  • ReadTheDocs integration is available
  • Also worth a read is this
Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.