Model Release Management Made Easy | Verta.ai

Managing model releases can be a challenging and time-consuming task for data science teams. However, by implementing a model catalog with release checklists, organizations can streamline their release process and ensure that their models are released to production quickly, safely and with little stress. In this post, we’ll review how ML teams and their partners can use customized release checklists integrated in Verta’s Model Catalog to simplify and streamline the model release and update process. 

What Is a Model Catalog?

A model catalog is a centralized repository for storing and managing ML models. It provides a platform for data scientists to share their work and enables other members of the team to discover, reproduce and build on top of existing models. Model catalogs can also help an organization to maintain version control, track model performance over time, and improve collaboration between team members.

When it comes to releasing new models into production, data science teams can realize several key benefits by using a well-organized and easily accessible model catalog:

1. Efficiency: By using a model catalog, data scientists can avoid duplicating work that has already been done by their colleagues. Instead, they can search the catalog for existing models that may be suitable for their project, saving them valuable time and effort.
2. Consistency: When models are stored in a central repository, it becomes easier to maintain version control and ensure consistency across different versions of the same model. This helps to avoid confusion and errors when it comes to releasing new models into production or replacing live models with new versions. 
3. Collaboration: Model catalogs streamline the handoff of model code between model creators and the ML and Platform engineers who deploy and manage the code. For example, a model catalog can provide tools for containerizing models and output a .tar file for download for deployment..

What is a Release Checklist?

A release checklist is a set of guidelines and best practices that Data Science teams follow when releasing new models into production. The checklist helps to ensure that all necessary steps are taken to prepare the model for release and minimize the risk of errors or bugs. With Verta, you can customize one or more release checklists and set controls on who can complete them and what actions can be taken with model versions with incomplete checklists. 

For example, a model release readiness checklist could include the following groups of steps:

• Model Handoff Requirements
• Responsible AI
• Model Risk Management
• Infrastructure
• Operations & Monitoring

Model Handoff Requirements might include discrete steps such as:

• Complete model description
• Classify model type and data type
• Complete model documentation
• Link to validation script
• List model owner
• Define model performance requirements
• Etc.

Responsible AI could be further broken down into steps such as:

• Link to model explainability script
• List features with protected classes
• Assign bias risk level

And so on for the other steps in the release process.

Why use a Release Checklist for Model Releases?

By breaking down the complex process of releasing a new model into production into discrete steps arranged into logical workflows, a release checklist helps ensure that all the stakeholders in the release process follow these steps consistently across different models and projects - and that no steps get missed in the process. 

The result is consistency of process, which helps avoid errors or bugs that can arise from missed steps or incomplete preparation. The release checklist-based process ensures that Data Science thoroughly tests and validates a model before release, improving the quality of the model.

A release checklist also helps to promote transparency and accountability within the organization. By documenting each step of the release process, data science teams can ensure that their work is easily auditable and that they are accountable for the models they release into production. (Model catalog also facilitate audits by providing a repository for all the documentation and artifacts related to a model, supporting Responsible AI.) 

How Model Catalogs and Release Checklists Work Together

When used together, model catalogs and release checklists help data science teams manage their model releases more efficiently and effectively. When a data scientist is preparing to release a new model into production, they can use their model catalog to search for existing models that may be suitable for their project, saving them valuable time and effort, since they may not need to start from scratch. This kind of reuse of existing assets increases the value of an organization’s overall investment in Data Science and ML.

Once a model has been built and trained, the data scientist can use the release checklist to ensure that all necessary steps are taken to prepare the model for release, including testing the model, validating the results, and ensuring that it is properly documented. Then, as noted above, a robust model catalog will containerize the model to prepare it for production.

Once the container is ready to go, the model can be released into production. At this point, the data scientist can use the model catalog to track the performance of the model over time and make any necessary updates or revisions. New versions of the model will go through the same checklist process, ensuring - again  consistency and quality across the lifecycle of the model. 

Learn more about Verta's Model Catalog here.