Build and Evaluate your AI Agent Systems with Databricks and MLFlow

Databricks makes it easy to create and register your own Agents and tools to manipulate your data, and also any other external system (from openai agent sdk, claude agents to MCP servers).

But building your agent is only the first step to deploy your production and drive business impact!

In this demo, we will be building an agent for your customer support team. It'll be able to process your customer support request, and provide guidance, ultimately **accelerating your time to resolution, increasing satisfaction and reducing costs!**

You will need the following steps, and Databricks makes it easier than ever:

1/ Create your tools

Let's start by creating our first tools. We'll create a few functions to help your customer support team to find customer information, and register them in your Catalog/Schema.

Remember, this could be any functions!

Open [01-create-tools/01_create_first_billing_agent]($./01-create-tools/01_create_first_billing_agent) to get started

2/ Deploy and Evaluate your agent

Now that our tools are ready, we'll create an Agent using Langchain and give equip the agent with the tools we previously created!

Once our agent created, we'll start running a first Evaluation round against an existing evaluation dataset, and improve our prompt to make it better and have better eval metrics!

Open [02-agent-eval/02.1_agent_evaluation]($./02-agent-eval/02.1_agent_evaluation) to deploy and evaluate your agent.

3/ Add a Knowledge base

Our agent is working well, but cannot answer questions on our internal product and procedure.

It's time to add a knowledge base, extracting information from PDF and adding a vector search to enrich our context with custom information in our prompt!

And of course, we'll close the loop by re-evaluating our agent, making sure we keep improving it.

Open [03-knowledge-base-rag/03.1-pdf-rag-tool]($./03-knowledge-base-rag/03.1-pdf-rag-tool) to add your Knowledge base.

4/ Deploy your agent using Databricks Application, capturing Customer Feedback in MLFlow

Now that our agent is ready, we can deploy it using Databricks Application.

For our demo, we'll release a simple GradIO chatbot application to show you how to capture user feedback within MLFlow, and use it to improve your eval set and your model performance over time!

Open [04-deploy-app/04-Deploy-Frontend-Lakehouse-App]($./04-deploy-app/04-Deploy-Frontend-Lakehouse-App) to get started.

5/ Add live, production monitoring to your agent

Our frontend application is now live! Our next step is to add a monitoring to evaluate all the incoming user requests, and make sure they respect our evaluation metrics over time.

Open [05-production-monitoring/05.production-monitoring]($./05-production-monitoring/05.production-monitoring) to add Production Monitoring to your agents!

6/ Track how your agent improves your business metrics

Out last step is to measure how your agent impact your business!

Thanksfully, Databricks let you manage your entire data stack, you can for example synchronize your CRM or Ticket support data and track how the CSAT score increases over time, benchmarking a team with and without agent for example!

Open [06-improving-business-kpis/06-improving-business-kpis]($./06-improving-business-kpis/06-improving-business-kpis) to track your Busines KPI with a ready to use dashboard!

Congratulation!
You're now ready to build and deploy your AI Agent with Databricks and MLFlow 3!

What's next: Discover ready to use Agent with Databricks Agent Bricks

Many AI tasks are common, and Databricks provides out of the box Agent that you can use and stich together:

- **Information Extraction**: Extract key information, classify content, or summarize text from documents into a structured JSON.
- **Custom LLM**: Specialize an LLM endpoint to perform custom text tasks (i.e. content generation, chat) aligned within your domain-specific guidelines.
- **Multi-Agent Supervisor**: Design an AI system that brings Genie Spaces and Agents together.
- **Knowledge Assistant**: Turn your docs into an expert AI chatbot.
- ...


Open [Databricks Agent Bricks]($./ml/bricks) and setup these agents in a few clicks!