Software Recommendations Stack Exchange is a question and answer site for people seeking specific software recommendations. It only takes a minute to sign up.
Sign up to join this community
Looking for some existing pipelining software/methods for moving the data from Logstash(using an ELK stack) to AWS Redshift.
ELK: Elasticsearch-Logstash-Kibana
I have already worked with Luigi, but I am looking for methods/software which are industry tested and widely used.
So, I have been dabbling with this problem, and finally architected a solution to the problem, and here is how you do it.
ElasticSearch has an output plugin which writes the log files to S3 in batches.
So, as soon as the file_size is filled, the file is pushed into a pre-specified S3 bucket and the next file is taken for writing.
And the S3 data can be written to Redshift, either through the COPY command, or write advanced cron jobs through a Luigi scripts.
You could write directly from logstash to S3 using S3 output plugin and then use AWS lambda loader to load the data into Redshift.
