Updating Metadata for 50L(5 Million) Assets in AEM 6.5 On-Prem (Append Without Overwrite)

Hi ​@Mystic_guide5225 

This requirement is feasible, but for 5 million assets and approximately 40 TB, I would not recommend using the Assets UI bulk editor or submitting one untested 5-million-row CSV directly in production. There is no honest 100% guaranteed solution at this scale—the implementation must be validated on a production-sized clone or representative performance environment first.  

Recommended approach

1. Define the update rule separately for every property

Append without overwrite - should not be treated as one global setting:

The Append option is mainly applicable to multi-value metadata such as tags. It should not be assumed to protect every single-value property from being replaced. Adobe demonstrates append behavior for existing multi-value metadata in the Assets bulk-edit workflow.

2. Prepare and validate the mapping file

Use an absolute AEM asset path as the unique identifier: /content/dam/brand/category/asset-001.jpg

Recommended validations before writing anything:

• Asset exists and is a valid dam:Asset.
• The path is case-sensitive and correctly normalized.
• All metadata property names and namespaces are valid.
• Data types match the metadata schema.
• Multi-value fields use the correct delimiter and stored values.
• Dropdown fields use the internal option values, not only the display labels.
• Duplicate asset rows are rejected or consolidated.
• Invalid rows are written to a separate rejection file.

Adobe's metadata import documentation recommends using assetPath as the first CSV column. It also confirms that missing metadata values for an asset are not updated, so the CSV generator should never send values for fields that are not intended to change.

3. Use the correct execution mechanism

For a small pilot or controlled batches, use the AEM Assets Metadata Import feature. It is asynchronous and supports CSV-based updates for existing assets. Start with a small batch; the current Assets documentation lists a default batch size of 50 and a maximum of 100 for the importer. Confirm the exact behavior against the customer's AEM 6.5 service pack before relying on those limits.

For the complete 5-million-asset operation, I would implement a resumable custom OSGi/Sling Job processor on AEM 6.5:

• Use a dedicated service user with least-privilege access.
• Process deterministic chunks rather than one large repository traversal.
• Keep a durable checkpoint containing the chunk ID, input row hash, status, and retry count.
• Commit after a controlled number of assets, initially around 25–50.
• Start with one worker and increase concurrency only after measuring the production-like environment.
• Make the job idempotent: rerunning a completed row must not create duplicate values.
• Store permanent failures in a dead-letter file for later correction and replay.
• Do not modify repository files, the datastore, renditions, or the Oak segment store directly.

The core merge logic should be equivalent to:

for each mapping row:
    read current metadata immediately before the update

    for each configured property:
        if incoming value is empty:
            skip

        if property is multi-valued:
            desired = unique(existing values + incoming values)
            write only if desired differs from existing values

        else if policy is PRESERVE_EXISTING and existing value is present:
            skip

        else if policy is REPLACE:
            write incoming value

        else if property is empty:
            add incoming value

    commit the asset update

Reading the current value immediately before writing is important. It protects metadata added by authors after the mapping file was created and avoids using an old export as the overwrite source.

4. Do not enable expensive processing unnecessarily

If the requirement is only to update metadata under jcr:content/metadata, do not trigger binary processing or XMP writeback for every asset unless there is a business requirement to update embedded metadata inside the original files.

Separate these two use cases:

• Repository metadata update — relatively lightweight and usually sufficient for AEM search and authoring.
• XMP writeback to the binary — significantly more expensive and should be run as a separate, controlled phase.

If XMP writeback is required, test it on representative file types first. Include the resulting workflow and binary-processing load in the capacity test.

5. Production rollout sequence

I would use this rollout:

• Take a verified backup and confirm that restoration works.
• Run the validator in read-only mode.
• Execute 100 assets covering different formats, folders, metadata types, and existing-value conditions.
• Verify that no unrelated properties changed and that multi-value values were merged without duplicates.
• Run a larger pilot, for example 1,000–10,000 assets.
• Measure CPU, heap/GC, disk I/O, repository write latency, Oak index lag, workflow queues, and author response times.
• Increase batch size or worker count only if the measurements remain within the agreed limits.
• Run the remaining population in off-peak windows.
• Pause automatically when queue depth, error rate, heap usage, disk latency, or author response time crosses the defined threshold.
• Perform a post-run reconciliation and replay only failed rows.

Adobe's AEM 6.5 guidance recommends performance testing, monitoring CPU, memory, disk, and network utilization, and measuring asset-processing behavior before relying on a large-scale workload. Adobe also recommends controlling workflow concurrency and using specific job queues for resource-intensive processing.

6. Verification and rollback

The process should produce these counters:

• Total input rows
• Assets found
• Assets not found
• Assets updated
• Assets skipped because existing values were preserved
• Duplicate values removed
• Validation failures
• Transient failures and retries
• Permanent failures

Before changing an asset, save the previous values of only the affected properties to a rollback file. This makes rollback much safer than trying to restore the entire repository:

assetPath,property,oldValue,newValue
/content/dam/brand/asset-001.jpg,my:tags,"summer|campaign","summer|campaign|2025"

If updated metadata must be visible on Publish, handle replication as a separate phase and monitor replication queues rather than combining it with the initial metadata write.

The safest design is: validated mapping file -> field-level merge policy -> resumable Sling Job/OSGi processor -> small commits ->throttling -> checkpoints -> dead-letter retries -> reconciliation and rollback file

Do not run a single unrestricted query, a single giant transaction, unrestricted parallel HTTP calls, or XMP writeback for all 5 million assets without a measured pilot.

https://experienceleague.adobe.com/en/docs/experience-manager-learn/assets/metadata/metadata-import-export

https://experienceleague.adobe.com/en/events

https://experienceleague.adobe.com/en/docs/experience-manager-65-lts/content/implementing/deploying/configuring/assets-performance-sizing

https://experienceleague.adobe.com/en/docs/experience-manager-65/content/implementing/developing/extending-aem/extending-workflows/workflows-best-practices

https://experienceleague.adobe.com/en/docs/experience-manager-65/content/implementing/deploying/configuring/monitoring-and-maintaining