Hi all,
We are encountering issues when ingesting a batch file for the Individual Profile class. The error message displayed is:
"Record size exceeded the limit".
Out of approximately 4.8M records, 7,450 records failed (~0.15%). According to Adobe documentation, the maximum size of a record class is 100 KB (hard). Based on this, the error seems to be valid, and we plan to apply changes to the ingestion process.
However, we previously had ingested a similar file without any issues. After analyzing that file, we identified 78 profiles (external_ids) exceeding the 100 KB threshold, as shown in the attached screenshot. This should have not occurred with a hard limit of 100KB per profile record.
Questions:
- What is the root cause of this behavior?
- Is Adobe rejecting the new ingestion because the number of oversized records is not residual?
- Does the hard limit of 100 KB per profile record apply strictly in all cases?
We need clarification on these points to prevent similar issues in future ingestions.
This is the query we have used to detected the records > 100KB:
WITH row_sizes AS (
SELECT length(
to_json(
struct(
identityMap,
_tenant)
)
) AS row_size_bytes, identityMap['externalID'][0].id AS external_id, _ACP_SYSTEM_METADATA.ingestTime AS ingest
FROM productos_cliente
)
SELECT external_id, MAX(row_size_bytes) AS max_row_size_bytes, COUNT(*) AS rows_over_threshold
FROM row_sizes
WHERE row_size_bytes > 100000 AND external_id IS NOT NULL AND CAST(from_unixtime(ingest / 1000) AS DATE) > '2025-11-01'
GROUP BY external_id
ORDER BY max_row_size_bytes DESC
LIMIT 500;
Thanks!
