I have a Python script that appends data to a text file multiple times/second. The data is housed in JSON structure.
Currently, I use a program called Filebeat to insert data into an ELK stack. I like Filebeat because:
- It works "behind the scenes"
- It's run asynchronously/externally from the Python script
- It's easy to configure
- It can run on Windows or Linux
I would now like to insert the logs into a Postgres database, instead of an ELK stack. Let's assume the JSON data adheres to a set schema, and that the Postgres database uses that schema.
Is there a program out there that:
- Monitors a directory or file for modification
- Inserts the data appended into the Postgres database
- Can do bulk insertions every X hours
Thank you in advance for your help!