ContentType Prediction

I am building the ContentType Prediction System from scratch, this it is more robust, flexible and scalable.

ML System Diagram for "ContentType Prediction System"

• I have created custom sklearn transformers to transform the datasets.
• I also implemented the model monitoring part using abstraction classes. I do monitoring using mlfow.
• I also write scripts for the reference about how to monitor, train and predict models, through this I want to give you some idea that how does this pipeline works.

Custom Transformers using sklearn

Yesterday, I have learned how to create a custom transformer using sklearn API.

I find it very useful and and ver elegant way to create pipelines with it. They are very simple to use and implement when you get it right.

A high level info about custom transformers.

• Create a class which inherit two sklearn classes from sklearn.base module TransformerMixin and BaseEstimator.
• Now, you have to define fit and transform methods in your class.
• And you are ready to use this custom transformer.

Remember this is not a fully pleged custom class because there are numerous things you have to keep in mind while making a custom transformer using sklearn API.

References

• Professional Preprocessing with Pipelines in Python - NeuralNine
• Developing a Custom Scikit-learn Transformer and Estimatior - Ploomber
• Developing custom scikit-learn transformers and estimators

Monitoring with mlflow

1st Draft

I have think a custom monitoring pipeline where you pass the model and params with the training and testing set. Then it calculate the score and log it with mlflow.

MLFlow UI showing scores of multiple trained models on bar chart.