I know this has been asked many times and in many different ways. And there are tons of blog posts, articles, videos and courses addressing this and comparing hundreds of tools, libraries, frameworks… And that’s part of my problem: I am facing so many options that I feel like Buridan’s ass, dying of starvation for not knowing what to do.
Although I don’t want to write too much, I need to speak a little about our situation, in order to put the question in our context.
Our Team
Our Team is small. We have only four people, which could be qualified as beginner data scientists. One of us has a profile that is a little bit more “engineer”, so data engineer could be more suitable for him. Anyway, we don’t have much experience, neither in Python Projects nor in Machine Learning. What we have is passion and love for ML!
For a couple of years, we have been functioning with SAS, but now we plan to change to the Python landscape, as it is much more vivid and exciting. In the last year, we have made two projects in Python, but without using any good practices at all. Every step was made by hand and prone to error, models were neither monitored nor even deployed (they only were used for making some batch predictions), projects were not properly structured, documentation was painful…
So we know that we need to change it before it becomes unmanageable.
We don’t expect the size of the team to grow fast. Let’s say in a couple of years we can expect 10-12 people working with us (the organization knows the importance of Machine Learning, but economic issues can be an obstacle).
Our Projects
For the moment, we have only made “classical” Machine Learning. I mean: no Deep Learning. We have used Pandas and Scikit-Learn, XGBoost, etc. And only in Batch mode. But we expect it to change in less than a year, because we will need to train an image classifier to detect anomalies in customs packages, so it will need to be:
Trained using a deep learning convolutional network.
Integrated with other applications (that are coded in Java) and fast (real-time).
Other change we expect is to need more distributed computing, as we will need to manage some huge databases that simply do not fit in a pandas dataframe. This are the most important challenges we face.
Our Company
We work for a big company, which also imposes some restrictions to us. Mainly:
We do not have budget to spend in MLOps solutions, so everything has to be open and free.
We won’t hire data scientist / data engineers for the moment.
There are some tools, uses by other teams, that we should use as part of the MLOps stack, although they are not the best in the class.
Regarding the last item, a short list of this set of restrictions is the following:
We have a Cloudera Express installation. It’s the most basic and cheaper Cloudera option, so it does not come with any tool for Machine Learning management. It only gives to us HDFS, Impala, Spark and a set of nodes to run Python scripts on them.
We have Control-M as the orchestrator and workflow manager tool.
We have DataStage as the ETL tool.
We use SVN as the code version system (yes, no git).
We deploy our projects using a very simplified and self-made version of Docker. It’s a little bit awkward and I think that, if we push a little bit, we could convince the organization to let us use Docker. But if Docker is reachable, Kubernetes is out of our capabilities.
We have Jenkins for CI.
We have Visual Studio Code professional licenses.
Toolset
With this premises, I have two different and opposed concerns or even fears.
Fear of not using enough tools and good practices and arriving in a couple of years to a state where we cannot manage our own code, project and models.
Fear of using so many tools that they impose a burden our small team cannot bear.
It’s clear that we need some MLOps, but how much, I don’t know. I will review some things I have been reading, and I hope you can help me choosing the right tools.
Python Programming
It looks like we will program using Visual Studio. We will use a remote interpreter, because we will run things on the Cloudera Nodes, although we will program locally and integrate the code with a SVN repository.
Do we need tools for standardizing our code, like PyLint, Flake8, MyPy or Black? Would you recommend any of those?
CI and Deployment
We will use Jenkins. For deployment of our code, is Docker a no brainer, a minimum standard? I tend to think so from what I read, but I’d like to be sure and to have good arguments.
Do we need more tools?
Project Scaffolding
I have been reading about PyScaffold, CookieCutter and, best of all (from my point of view), Kedro. I think we will stick to Kedro template, because it offers much more functionality, and I like to think of each project as a set of pipelines to be run. What do you think of Kedro?
Documentation
Would you recommend having separate documents, or generating the documentation from the projects, using Sphinx or another similar tool? I tend to prefer the second option, because the first one very likely tend to generate obsolete docs. But I don’t know if the “burden” of the second is too big, and if the generated docs can suffice for a typical ML project.
Project registry
Is there any tool that could be used as a “project registry”, like a simple web app where we could navigate through our projects, read the docs and thinks like that? I don’t know. If not, the registry will be the SVN repo with all our projects as folders, and that’s all.
Data Exploration and Preparation
I think that Matplotlib, Seaborn and Pandas should suffice, and when things go big, we should use PySpark, Scala or even plain SQL in Impala. However, I know Dask exists, and newer tools like Koalas or Vaex. What do you think?
For creating data transformation pipelines, we will use Kedro, although there are lots of tools that look interesting, like Dagster.
When we enter the “deep learning” realm, can we keep using the same tools? Should we use another framework like TFX? I’d prefer not, cause learning one framework is hard, and two is worse. If a solution is valid for all our projects it’s better. Or TFX is valid for “classical” ML and Deep Learning?
Tests
I think unit testing can be too much burden for us. But I have come to Great Expectations library and think it’s well suited for ML projects. Would you recommend it as an important part of our MLOps stack?
By the way, there is a Kedro-Great Expectations plugin, so we could benefit from that.
Feature Store
Is it really needed, especially considering our team size and experience? If so, I have read about Feast and Snorkel.
Data Versioning
Is it really needed, especially considering our team size and experience? If so, I have read about DVC.
Experimenting
I think it’s an important piece, although I wonder if we really need a tool or we could use our own standard of reports and artifacts to follow what we have tried. But the risk that it goes unmanageable is high.
Kedro has a journal, I don’t know if it can suffice. Also it has a Kedro-MLFlow plugin, so that we could benefit from using MLFlow as the experiment tool.
I have also read about Guild, that seems really lightweigh and easy. I don’t know much more.
Training
I developed my own library for doing nested cross validation and, with the same function:
Optimizing hyperparameters (of model and pipeline).
Generating a report of the training to assess the quality of the model.
It’s build on top of Skopt. I did it pip installable, it’s here:
https://github.com/JaimeArboleda/nestedcvtraining
So we plan to use it with the typical models like XGBoost, LightGBM and all Scikit-Learn. And when we need other frameworks like TensorFlow or Keras, we will see.
Model Registry
I think it’s an important piece, although I don’t know if we even could build our own with an standard database. If not, MLFlow seems a mature option.
Model serving
I am not sure if it’s included in the previous point or not. Anyway, I have read about Streamlint and FastAPI. Would you recommend any of those?
Is Apache Kafka needed for real time predictions?
Visualization
With this I mean sharing with the organization basic web apps with customizable plots, explainable predictions and things like that. I have read about panel, which has the ability of transform a Jupyter Notebook into a simple web app. It might be interesting.
Model monitoring
Is there a good free tool for monitoring the models and detecting loss of accuracy, data drift and things like that? Or we should better generate our own script of monitoring to be run periodically?
BigData
As I said before, we plan to use mainly Spark when need.
I know it’s a lot of info. Maybe I have overcomplicated myself and I should use only 20% of what I think I should. Or maybe not. I have no idea. Any help will be GREATLY appreciated. Thanks in advance.
