Adding GenAI to a Traditional ML Stack

This is part two of  a series, where we'll guide you through the essential elements your company needs for a successful GenAI initiative. Check our part 1 here. 

Many companies and teams have experienced success with traditional deterministic ML, whether through regressions, classifiers, object detection, or various other ML applications. Not all problems are well-suited for generative AI, but if there are programs or initiatives that you previously struggled with, GenAI might provide solutions.  But building with generative AI introduces key differences. Here’s what to keep in mind as you take the plunge:

Generative models are larger and more expensive to train, making the use of third-party or open-source models much more likely. Generative AI training (and even fine tuning) in-house does not make sense for the vast majority of organizations, but readily available APIs allow you to leverage third-party models. Open-source models present a great opportunity at a lower price point and on your familiar tech stack. 

The most significant difference in using generative models is the need for prompts. Instead of spending time training a model, you'll invest time in prompt engineering - that is, shaping the instructions the model receives on what to produce for your users. This might involve passing user data into the prompt as variables, or following any of the emerging prompting frameworks. You’ll be testing many prompts since it's the fastest and easiest way to influence the performance of your solution. Data scientists might not be the best at writing the most creative and high-performing prompts, so iterating quickly will involve cross-functional collaboration with any team member who is interested.

Your existing teams may benefit from upskilling with generative models, and it's important to allocate time and budget for them to become familiar with new technologies, which may be more different from what they're currently using than expected. Understanding when to use different techniques and strategies (like COT prompting or RAG) can save time and money.

Be cautious about cloud lock-in or being stuck with a single cloud provider. The GenAI world is evolving rapidly, and you don't want to become overly invested in one vendor. Expect to rely more on third-party tools with GenAI than with traditional ML. Due to the speed of the industry, continuous reevaluation of your tech stack will be crucial. Develop a strategy for staying nimble to new developments in the GenAI landscape.

Non-deterministic (generative) models outputs complicate the processes related to hardening, governance, releasing, and monitoring. Determining what is a “good” output is highly subjective and will be a team effort including not just developers and scientists but your customer success team, subject matter experts and your users. Make a plan for collecting real-world user feedback continuously so you can iterate on model selection and prompt engineering often. Brief your customer support team and any governance team members on the known and expected issues and the black-box nature of LLMs. Arrange playbooks for when a generated output creates customer service issues and ensure you’re not putting your organization in any compliance danger.

Simple, right?

Of course, if you need any help with this Verta is here to help. And our Workbench helps you with model selection, prompt engineering, model serving and monitoring with H-I-T-L feedback. Cheers.

Here’s a checklist you can use:

• Prepare and find the right problems for the solution
  • Understand Unique Challenges of Generative AI
  • Evaluate Previous Struggles through a GenAI lens
• Update existing development process
  • Embrace Prompt Engineering
  • Prepare for Continuous Upskilling and Tech Stack Evaluation
  • Select 3rd party tools, models and clouds carefully.
  • Make really good fall-back ans response plans
• Ship something!