How can I ensure that each call to fmin() is logged to a separate MLflow main run when using SparkTrials and Hyperopt in Databricks Runtime ML?
To ensure that each call to fmin() is logged to a separate MLflow main run when using SparkTrials and Hyperopt in Databricks Runtime ML, Databricks recommends managing active MLflow runs by wrapping the call to fmin() inside a with mlflow.start_run(): statement. This way, each fmin() call is logged to a separate MLflow main run, and it becomes easier to log extra tags, parameters, or metrics to that run.