I have a Python script that appends data to a text file multiple times/second. The data is housed in JSON structure.

Currently, I use a program called Filebeat to insert data into an ELK stack. I like Filebeat because:

  1. It works "behind the scenes"
  2. It's run asynchronously/externally from the Python script
  3. It's easy to configure
  4. It can run on Windows or Linux

I would now like to insert the logs into a Postgres database, instead of an ELK stack. Let's assume the JSON data adheres to a set schema, and that the Postgres database uses that schema.

Is there a program out there that:

  • Monitors a directory or file for modification
  • Inserts the data appended into the Postgres database
  • Can do bulk insertions every X hours

Thank you in advance for your help!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.