I'm working on a part of a project, that detects fraud from sensors' data using deep learning. Current steps are:
- Receiving sensors' data (I'm currently receiving data from Kafka).
- Predicting anomalies of data (Tensorflow + PySpark structured streaming + Pandas UDF - large latency in my opinion).
- Send the result back (to Kafka - using PySpark structured streaming).
The results are monitored by Kibana, but another team work on that part.
I don't think my solution is a good one, so I'm asking for suggestions on which platform/framework/tool should be used at each step. Any help/advice is appreciated.