The Challenge From Gen-AI Demo to Production: Why Manual Tuning Is a Nightmare

The journey from a generative AI (Gen-AI) demo to a fully functional, production-ready application is fraught with challenges. While showcasing an AI demo is relatively straightforward, making it robust, scalable, and efficient in real-world scenarios demands significant effort. A major bottleneck in this transition lies in the manual tuning of Gen-AI workflows—a labor-intensive a 6nd time-consuming process. 

Let’s say you’re building a customer support chatbot for your local hospital. Your workflow might involve:

1. Input Processing: Receiving the patient’s question in text.
2. Intent Recognition: Determining the patient’s intent using a language model.
3. Data Retrieval: Fetching relevant patient information from the patient’s medical record and a public medical knowledge base.
4. Response Generation: Crafting a text response using a language model to provide answers or direct to a doctor.

Optimizing this workflow manually is slow, expensive, and error-prone. You have to experiment with different models (e.g., which model shall I use for step 2 and what to use for step 4?), rewrite prompt templates (e.g., do I use few-shot examples and what examples to use?), optimize workflow pipelines (e.g., do I let the workflow go from step 2 again if step 4’s generation is not good? Shall I break step 2 into two steps?). After choosing an answer to these questions, you need to evaluate the workflow by running a test set with an evaluator. These tasks are performed over many iterations to refine and optimize the workflow. This often leads to months of development before the workflow is production-ready, or worse, months of development before finding out that you cannot make it to production. 

What if you could automate this process instead?

Introducing Cognify: The Future of Gen-AI Workflow Autotuning

Cognify is built to take the pain out of Gen-AI workflow tuning by employing three types of tuning methods (which are called “cogs”):

1. Architecture Cogs: These cogs rearrange the workflow—adding new steps, removing unnecessary ones, or changing the order.
2. Step Cogs: These cogs modify how each step works—swapping models or updating code functions.
3. Weight Cogs: Tweaking individual steps’ inputs, such as modifying prompts and adding few-shot examples.

Cognify executes with a single command line cognify optimize workflow.py, where `workflow.py` is a user gen-AI workflow program written in LangChain, LangGraph, DSPy, etc. Users can choose one or more metrics as Cognify’s tuning objectives (e.g., quality metrics like exact matching, workflow execution cost, and workflow execution monetary cost). Cognify automatically searches and applies appropriate cogs to the workflow and returns multiple autotuned workflow versions based on the user-specified objectives. 

The Autotuning Challenge: How to Use $5 to Cover a Search Space Worth of $168K?

A key challenge in designing an autotuning method is the vast search space and its associated cost. For example, for our customer support chatbot example, there are four tunable steps. Assuming we can apply three cogs, each with four options to these steps, the search space is (4³)⁴=4¹² possible workflows. Suppose each search uses GPT-4o and has 1000 output tokens; searching the entire space using brute force can cost around $168K and several weeks. 

Is it possible to autotune this workflow with a budget of $5 and 30 minutes (i.e., 0.003% of the brute-force search cost)? 

Cognify answers this question in the affirmative with a novel cog searching algorithm called AdaSeek, which we will discuss in Pt.2 of this blog post series, with the following suggested changes to the health chatbot example.

1. Architecture Changes

Cognify could make several suggestions to the workflow structure: 

• Model Ensembling: Calling an ensemble of "Intent Recognition" LLM calls in parallel and combining the results of these calls.
• Task Decomposition: Breaking the “Intent Recognition” step into two sub-steps (categorize the patient’s question, determining intent based on the category).

Such restructuring could significantly improve accuracy for multilingual users.

2. Step Changes

Cognify uses a larger language model (e.g., GPT-4o-mini) for the “Response Generation” step and smaller models (e.g., Llama-8B) for the rest language model steps, suggesting that response generation is most critical and challenging for the chatbot task. Using smaller models to less critical steps lowers the workflow execution time and cost.

3. Weight Changes

Cognify automatically adds few-shot examples to the "Intent Recognition" and “Response Generation” steps. These examples originate from the user-provided input dataset, and Cognify automatically filters out the effective ones for each step.

Cognify finishes the above autotuning process in 24 minutes with $5.4 of tuning cost (to execute the workflows in iterations).

The Takeaway

Cognify represents a significant advancement in automating and optimizing Gen-AI workflows. By systematically addressing architecture, step, and weight changes, it empowers machine learning engineers and data scientists to enhance gen-AI solutions without the burdens of manual tuning. Incorporating Cognify into your projects can lead to fast-time-to-market, more efficient, and more effective AI solutions.

Curious to dive deeper? Read Pt. 2 of this blog posts; check out the Cognify research paper; get Cognify from GitHub; read Cognify’s Documentation; and join our Discord.

‍