New
Destination - add data to the manfest
When exporting data to a JSON file, the accompanying manifest control file currently includes the file SIZE in bytes. However, file size alone is not a reliable validation control for successfull data ingestion, as it doesn't verify whether all records were actually processed and loaded correctly.
Feature Request: We kindly request adding a record count (total number of records) field to the manifest file.
Why this is essential:
- Reliable Ingestion Validation: A record count provides a definitive checksum to ensure no data was truncated, dropped, or corrupted during export and ingestion pipelines.
- Automated Quality Control: Data engineering teams can easily automate reconciliation scripts by comparing the source record count against the ingested records.
Adding this metric will greatly improve data reliability and troubleshooting workflows.