Looking for some existing pipelining software/methods for moving the data from Logstash(using an ELK stack) to AWS Redshift.

ELK: Elasticsearch-Logstash-Kibana

I have already worked with Luigi, but I am looking for methods/software which are industry tested and widely used.


So, I have been dabbling with this problem, and finally architected a solution to the problem, and here is how you do it.

ElasticSearch has an output plugin which writes the log files to S3 in batches.

So, as soon as the file_size is filled, the file is pushed into a pre-specified S3 bucket and the next file is taken for writing.

And the S3 data can be written to Redshift, either through the COPY command, or write advanced cron jobs through a Luigi scripts.


You could write directly from logstash to S3 using S3 output plugin and then use AWS lambda loader to load the data into Redshift.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.