Introduction to Databricks System Tables

System Tables are a Databricks-hosted analytical store for operational and usage data.

System Tables can be used for monitoring, analyzing performance, usage, and behavior of Databricks Platform components. By querying these tables, users can gain insights into how their jobs, notebooks, users, clusters, ML endpoints, and SQL warehouses are functioning and changing over time. This historical data can be used to optimize performance, troubleshoot issues, track usage patterns, and make data-driven decisions.

Overall, System Tables provide a means to enhance observability and gain valuable insights into the operational aspects of Databricks usage, enabling users to better understand and manage their workflows and resources.
- Cost and usage analytics
- Efficiency analytics
- Audit analytics
- GDPR regulation
- Service Level Objective analytics
- Data Quality analytics

Accessing your System tables With Unity Catalog

System Tables are available to customers who have Unity Catalog activated in at least one workspace. The data provided is collected from all workspaces in a Databricks account, regardless of the workspace's status with Unity Catalog. For example, if I have 10 workspaces and only one of them have Unity Catalog enabled then data is collected for all the workspaces and is made available via the single workspace in which Unity Catalog is active.

Enabling all system tables

All systems tables are not enabled by default. As an account admin, you can review the [_enable_system_tables]($./_enable_system_tables) notebook to enable them all.

Ask Genie insights on your Databricks usage through system tables

DBDemos installed a Genie Space for you, ready to answer your System Table questions in plain english!

The Genie space is ready with some example / instructions to improve your Q&A experience!

Feel free to open an issue in our [dbdemos git repository](https://github.com/databricks-demos/dbdemos) if you can't find the answer you're looking for, we'll improve it over time!

Open the System Table Genie space to deep dive into your system table data

System Table Dashboard - Leverage AI with Lakehouse

We installed a Dashboard to track your billing and Unity Catalog usage leveraging the System tables.

[Open the dashboard](/sql/dashboards/ab3b5298-e09e-4998-8ef3-ae456a7b888d) to review the informations available for you.

A note on Forecasting billing usage

Please note that this dashboard forecasts your usage to predict your future spend and trigger potential alerts.

To do so, we train multiple ML models leveraging `prophet` (the timeseries forecasting library).

**Make sure you run the `01-billing-tables/02-forecast-billing-tables` notebook to generate the forecast data.**

If you don't, data won't be available in the dashboard. `dbdemos` started a job in the background to initialize this data, but you can also directly run the notebook.

*For production-grade tracking, make sure you run your forecasting notebook as a job every day.*

Billing tables

Billing table contains all the information required to track and analyze your consumption in DBU (Databricks Unit) and by extension $USD.

* To get started with the billing sytem tables, open the [01-billing-tables notebook]($./01-billing-tables/01-billing-tables-overview) to see how to explore your billing data
* Leverage the Lakehouse capabilities to forecast your spend: open [02-forecast-billing-tables]($./01-billing-tables/02-forecast-billing-tables).

Audit Logs

The audit log tables let you track and monitor all operations within your Lakehouse

You can get information to understand your usage and diagnostic operations such as finding when a table was deleted, created...

Open the [02-audit-log notebook]($./02-audit-logs-tables/02-audit-log) to explore your audit logs.

Lineage

Lineage tables let you track and monitor all lineage around your data assets.

You can track at a Table and Column level to find dependencies and understand what your table is using including Workflows, SQL Queries, Dashboards, Notebooks...

This information is available directly within the Data Explorer UI, but also saved as system table to simplify automated analysis.

Open the [03-lineage notebook]($./03-lineage-tables/03-lineage) to explore your audit logs.

Conclusion

With Unity Catalog System tables, you can easily monitor and exploit all your Lakehouse operations, from Audit Logs, Lineage up to Billing forecast.

More system tables will be released soon, this demo will be updated accordingly, stay tuned!