I'm working on a part of a project, that detects fraud from sensors' data using deep learning. Current steps are:

  1. Receiving sensors' data (I'm currently receiving data from Kafka).
  2. Predicting anomalies of data (Tensorflow + PySpark structured streaming + Pandas UDF - large latency in my opinion).
  3. Send the result back (to Kafka - using PySpark structured streaming).

The results are monitored by Kibana, but another team work on that part.

I don't think my solution is a good one, so I'm asking for suggestions on which platform/framework/tool should be used at each step. Any help/advice is appreciated.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.