Description:
Introduce a Source Health Dashboard that monitors the status, freshness, volume, and latency of incoming data sources (batch and streaming) in real time. Add alerting features for drops in ingestion, data schema mismatches, or unusual data volume patterns.
Why is this feature important?
Ensures data quality and trust across the platform by surfacing real-time issues with source integrations.
Reduces diagnostic time for ingestion failures that silently impact audience building or segmentation.
Proactively alerts technical and marketing teams to source failures, stale data, or delays, preventing downstream impacts on campaigns and decisioning.
Supports data governance and operational excellence in data-driven organizations.
Current Behavior:
Source connection status (e.g., for streaming or scheduled batch uploads) is not actively monitored in a centralized dashboard.
Failures may go unnoticed until audience segments look wrong or campaigns underperform.
There is no built-in alerting for low volume, schema drift, or ingestion latency beyond basic logging.
Use Case:
A key CRM source stops sending updates due to API auth failure - the dashboard shows a red status, and alerts are sent to the integration team via Slack or email.
A streaming source begins sending 50% less data than average over the last 24 hours - an alert flags this as a potential anomaly for investigation.
A newly added source has incorrect schema (e.g., missing a key identity field) - it’s flagged before it disrupts identity stitching or segment membership.
