Skip to main content

Lakehouse Monitor

The Lakehouse Monitor at /infrastructure/lakehouse-monitor is the integration layer with Databricks' built-in Lakehouse Monitoring product. Clone-Xs reads the monitoring metrics it produces (data drift, profile drift, model drift) and surfaces them alongside Clone-Xs's native DQ Observability signals.

What Lakehouse Monitoring provides​

Databricks Lakehouse Monitoring computes — for every monitored Delta table — a set of profile and drift metrics on a schedule:

  • Profile metrics: row count, null %, distinct values, distribution stats per column
  • Drift metrics: comparison against a baseline (rolling window or fixed snapshot)
  • Custom metrics: arbitrary aggregations defined in a MonitorMetric spec

These land in companion _profile_metrics and _drift_metrics Delta tables alongside the source.

What this page adds​

  • Inventory — every monitored table across catalogs in one list, with per-table status
  • Sync to Clone-Xs — pull drift events into Anomalies so they correlate with other DQ signals
  • Comparison overlay — overlay Lakehouse Monitor metrics with Clone-Xs's Trust Scores and SLA results
  • One-click create — set up a monitor from Clone-Xs UI without leaving the portal

Create a monitor​

The wizard takes:

FieldNotes
Table FQNThe Delta table to monitor
Monitor typeTimeSeries / InferenceLog / Snapshot
Profile type configTime-series column, slicing exprs, granularity
BaselineOptional baseline table for drift comparison
Schedule cronWhen to refresh metrics
Output schemaWhere companion tables live

Creating a monitor calls Databricks' Lakehouse Monitoring API and registers the result in Clone-Xs's inventory.

Drift → Anomaly bridge​

When a monitor reports drift exceeding configured thresholds, Clone-Xs creates an Anomaly tagged with source: lakehouse_monitor. From there it follows the standard anomaly lifecycle: triage, correlation, incident promotion, remediation.

API​

GET   /infrastructure/lakehouse-monitor
POST  /infrastructure/lakehouse-monitor             # create
GET   /infrastructure/lakehouse-monitor/{table_fqn}
DELETE /infrastructure/lakehouse-monitor/{table_fqn}
POST  /infrastructure/lakehouse-monitor/{table_fqn}/refresh