Databricks Generative AI Cookbook

TLDR; this cookbook and its sample code will take you from initial POC to high-quality production-ready application using Mosaic AI Agent Evaluation and Mosaic AI Agent Framework on the Databricks platform.

The Databricks Generative AI Cookbook is a definitive how-to guide for building high-quality generative AI applications. High-quality applications are applications that:

1. Accurate: provide correct responses

2. Safe: do not deliver harmful or insecure responses

3. Governed: respect data permissions & access controls and track lineage

Developed in partnership with Mosaic AI’s research team, this cookbook lays out Databricks best-practice development workflow for building high-quality RAG apps: evaluation driven development. It outlines the most relevant knobs & approaches that can increase RAG application quality and provides a comprehensive repository of sample code implementing those techniques.

Important

• Only have a few minutes and want to see a demo of Mosaic AI Agent Framework & Agent Evaluation? Start here.

• Want to hop into code and deploy a RAG POC using your data? Start here.

• Don’t have any data, but want to deploy a sample RAG application? Start here.

This cookbook is intended for use with the Databricks platform. Specifically:

• Mosaic AI Agent Framework which provides a fast developer workflow with enterprise-ready LLMops & governance

• Mosaic AI Agent Evaluation which provides reliable, quality measurement using proprietary AI-assisted LLM judges to measure quality metrics that are powered by human feedback collected through an intuitive web-based chat UI

Retrieval-augmented generation (RAG)

This first release focuses on retrieval-augmented generation (RAG). Future releases will include the other popular generative AI techniques: agents & function calling, prompt engineering, fine tuning, and pre-training.

The RAG cookbook is divided into 2 sections:

1. Learn: Understand the required components of a production-ready, high-quality RAG application

2. Implement: Use our sample code to follow an evaluation-driven workflow for delivering a high-quality RAG application

Code-based quick starts

Time required	Outcome	Link
🕧 30 minutes	Sample RAG app deployed to web-based chat app that collects feedback	RAG Demo
🕧🕧🕧 2 hours	POC RAG app with your data deployed to a chat UI that can collect feedback from your business stakeholders	Build & deploy a POC
🕧🕧 1 hour	Comprehensive quality/cost/latency evaluation of your POC app	- Evaluate your POC - Identify the root causes of quality issues

Table of contents

Overview

• GenAI Cookbook
• Retrieval-augmented generation (RAG)

Learn

• 1. Agents overview
• 2. Agent fundamentals
• 3. Agent quality knobs
• 4. Evaluating Agent quality
  • 4.1. Defining “quality”: evaluation sets
  • 4.2. Assessing performance: Metrics that Matter
  • 4.3. Enabling Measurement: Supporting Infrastructure
• 5. Evaluation-driven development workflow

Product demo

• 10 minute demo of Mosaic AI Agent Framework & Agent Evaluation
• Build & deploy the application
• 3/ Use Agent Evaluation to evaluate your application
• What’s next?

Implement

• Prerequisite: Gather requirements
• Step 1: Clone code repo & create compute
• Step 2: Deploy POC to collect stakeholder feedback
• Step 3: Curate an Evaluation Set from stakeholder feedback
• Step 4: Evaluate the POC’s quality
• Step 5: Identify the root cause of quality issues
• Step 6: Iteratively implement & evaluate quality fixes
• Step 6: Deploy & monitor