I am in the process of designing a simple but robust and scalable feature store architecture. So far, I've built a framework for creating custom feature pipelines in Scikit-Learn. Ideally, I want the users to build these custom pipelines (i.e. query from big query, apply sklearn transformations, return feature set) and then save their training data/feature datasets into the feature store for sharing/discoverability.
Firstly, what would be the best technology to use for processing these batch features and running ingestion jobs of these scikit-learn pipelines (i.e. I want to transmit the pipeline metadata and the output into the data warehouse)? Secondly, what would be the best platform for storing these features and also storing feature metadata and pipeline metadata as well?
Please kindly point me to some good resources to read through; that would be immensely helpful!