Help Your Future-Self Succeed with ML Monitoring and Retraining

Seriously, your future self will be thanking you for setting up a well defined pipeline around this stuff when it inevitably comes time to retrain the model. In the same way that a CI/CD (Continuous Integration / Continuous Deployment) pipeline is helpful for reproducibility, a CI/CT/CD/CM (those new ones are Continuous Training and Continuous Monitoring) will ensure consistency in your model, minimize concept drift, increase reproducibility, and increase portability (should you need to change where you train or inference).

The first step is to make sure you have a system in place to monitor your model in production to catch data format issues and model performance degradation (how often is it making a bad prediction?). Once you have metrics, you can set up automation to trigger if a threshold is passed. For example, is your random number generator model suddenly returning lots of ones? Maybe there’s a problem there. Probably time for retraining (and a larger dataset). If you care about model accuracy you can use DVC (Data Version Control) so that you know when the dataset has changed and it’s time to retrain

Ideally, you’re training whenever your training data changes…but that’s also expensive. So setting a higher model performance threshold for conditions which warrant training and only then pulling in a new dataset is likely advisable if you are on a budget. When you’re ready to train it’s off to tools like Kubeflow, Vertex AI Pipelines, or Ray to orchestrate a training session, and handle the metadata collection.

Similar to Continuous Delivery and other aspects of DevOps, you’ll need to get buy-in from your organization to really make Continuous Training worthwhile. It’s also important to remember that automation isn’t everything – you’re still going to want to have someone on call who can take a look at things and see if the model needs to be changed (and not just retrained).

By implementing automation, you ensure organizational continuity, continued model relevance, and a better experience for your customers. It’s usually worth the cost!