What is the difference between the new compute metrics UI and Ganglia in terms of the resource usage they measure?
The new compute metrics UI in Databricks provides a more comprehensive view of a cluster’s resource usage compared to Ganglia. While Ganglia only measures Spark container consumption, the compute metrics UI includes Spark consumption and internal Databricks processes, giving you a more detailed overview of your cluster’s resource usage. This difference might result in discrepancies in the metric values between the two interfaces.