Data Ingestion Issue

[random_guy_tech]

Hello Everyone,

I am performing data ingestion in AEP in my custom schemas but facing extreme slowness on data ingestion and my job is running from last 3 hours, can anyone please give any recommendations for same?

Regards

[MeitMedia]

Hello @random_guy_tech ,

Below are common issues which you need to check in data ingestion :-

• Problem: Data ingestion is slow or lagging.

  • Solution:
    1. Batch Size and Frequency: Adjust the batch size and frequency of data uploads to balance between performance and load. Smaller, more frequent batches may reduce lag.
    2. Network and Connectivity: Check for network issues or connectivity problems that could be causing delays. Use reliable, high-speed internet connections for large data uploads.
    3. Platform Health: Monitor AEP's health and performance metrics. High system load or scheduled maintenance may affect ingestion speed. Check the Adobe Status page for any reported outages or performance issues.

2. Schema and Data Modeling Issues:

• Problem: Schema mapping errors.

  • Solution:
    1. Schema Validation: Use AEP's schema validation tools to ensure that your schemas are correctly defined and that data conforms to these schemas.
    2. Correct Mapping: Verify that data fields are correctly mapped to the corresponding fields in the AEP schema. Use field names that match the schema and ensure data types are consistent (e.g., string, integer).
    3. Schema Updates: If you update a schema, ensure that all data sources and existing datasets are compatible with the new schema version. Mismatches can cause errors.

• Problem: Issues with Experience Data Model (XDM) integration.

  • Solution:
    1. Use XDM Best Practices: Follow Adobe's guidelines and best practices for defining and using XDM schemas. This includes proper use of classes, mixins, and data types.
    2. XDM Validation: Run validation checks on your XDM schemas to ensure compliance with AEP's requirements. Use the Adobe Experience Platform interface to test XDM schema integrations.

3. Identity Management and Profile Issues:

• Problem: Customer profiles are not merging correctly.

  • Solution:
    1. Identity Graph Configuration: Ensure that the Identity Graph is correctly configured to handle identity stitching. This involves defining identity namespaces and matching rules.
    2. Identity Fields: Verify that identity fields (e.g., email, phone number, customer ID) are correctly mapped and populated. Inconsistent or missing identity data can prevent profiles from merging.
    3. Profile Merge Policies: Review and adjust profile merge policies to control how profiles are stitched together based on identity fields. Adjust settings to prioritize certain identities or events.

Best regards,

MEIT MEDIA (https://www.meitmedia.com)

Find us on LinkedIn

Contact Us: infomeitmedia@gmail.com

[DavidRoss91]

Hi @random_guy_tech 

How are you ingesting the data? Batch or streaming?

David

[Rite18 · Accepted solution]

Adding to prior comments @random_guy_tech  if it is a large data set consider using batch over segmentation service 

-  debug each layer / items as suggested in prior comments 

-  if streaming consider checking API rate limit as well, as it would slowdown ingestion

[kautuk_sahni]

@random_guy_tech Did you find the suggestions helpful? Please let us know if you require more information. Otherwise, please mark the answer as correct for posterity. If you've discovered a solution yourself, we would appreciate it if you could share it with the community. Thank you!