AEP: CSV Ingestion into Object Array Schema – Overwriting Issue + Mapping/Transformation Clarification (Snowflake too) | Community
Skip to main content
R
RS195
Level 2
July 17, 2025
Solved

AEP: CSV Ingestion into Object Array Schema – Overwriting Issue + Mapping/Transformation Clarification (Snowflake too)

  • July 17, 2025
  • 2 replies
  • 819 views

Hi all,
I’m working with an XDM Individual Profile schema in AEP that includes a custom array of objects. This array is defined in a custom namespace and sits one level under the profile, for example:

{ "_tenant": { "customArrayObject": [ { "fieldA": "value", "fieldB": "value" }, { "fieldA": "value", "fieldB": "value" } ] } }

My source file is in CSV format, with multiple rows for the same identity (same ID). Each row represents one object that should go into the array. Example:

ID FieldA FieldB 123 value A value B 123 value C value D 123 value E value F

Issue:
When I ingest this into AEP (via UI upload or dataset), only the last row is retained, the earlier ones are overwritten. The array is not constructed.

My Questions:

  1. Is this behavior expected when using CSV to populate object arrays?

  2. Are any transformations or mapping logic available during CSV ingestion that would allow grouping these rows into an array under one profile?

  3. I also have a Snowflake connection set up in AEP, if I ingest the same structured data using that source, would the array transformation behave differently? Or would it still require JSON-style structure before ingestion?

  4. Is JSON ingestion still the only reliable way to preserve nested arrays like this?

Best answer by Parvesh_Parmar

Hello @rs195 ,

 

As I understand, you're looking to store data in the form of an array of objects at the profile level in Adobe AEP.

Before diving into how this could be done, I want to highlight an important point: even if you manage to store the data in that format, it may become challenging to use it effectively later on—for example, for segmentation, personalization, or exporting to third-party systems. So, I recommend carefully considering how you plan to use this data after ingestion. In my experience, storing array-type structures at the profile level often leads to complications.

Now, regarding CSV ingestion, there are limitations:

  • CSV files are typically structured in rows, and you cannot directly map multiple rows from a CSV to a single profile record in JSON array format.

  • The Adobe UI and data mapping tools don’t support this type of transformation natively.

This means that the ingestion process would need to change.

Possible Alternatives:

  1. Use of APIs (Recommended for Small Data Sets):
    If the data volume is manageable, you might consider ingesting via the AEP API. This allows you to send data in a proper JSON format, which may give you more flexibility with mapping to a schema containing arrays.

  2. Flatten the Structure on Profile Schema:
    Another option is to define multiple attributes on the profile itself, such as:

    • fieldA1, fieldB1

    • fieldA2, fieldB2, etc.
      This is a manual workaround and not scalable, but it works if the maximum number of array elements is known and limited. However, you'd still need to modify your CSV format accordingly.

  3. Use Event Schema Instead (Recommended Approach):
    The best and most scalable option may be to store this data on an event schema instead of the profile schema. Here's why:

    • You don’t need to predefine complex structures.

    • Just include fields like profileID, fieldA, fieldB, etc.

    • Then, map your CSV to the event schema and ensure the same namespace is used for the profileID in both the profile and event schemas.

    This approach is typically easier to manage and aligns well with AEP’s strengths—especially if your use case can be supported through event-level data.

I hope this helps clarify the options. Let me know if you’d like to go deeper into any of these approaches.

Best regards,
Parvesh

2 replies

Parvesh_Parmar
Community Advisor
Parvesh_ParmarCommunity AdvisorAccepted solution
Community Advisor
July 18, 2025

Hello @rs195 ,

 

As I understand, you're looking to store data in the form of an array of objects at the profile level in Adobe AEP.

Before diving into how this could be done, I want to highlight an important point: even if you manage to store the data in that format, it may become challenging to use it effectively later on—for example, for segmentation, personalization, or exporting to third-party systems. So, I recommend carefully considering how you plan to use this data after ingestion. In my experience, storing array-type structures at the profile level often leads to complications.

Now, regarding CSV ingestion, there are limitations:

  • CSV files are typically structured in rows, and you cannot directly map multiple rows from a CSV to a single profile record in JSON array format.

  • The Adobe UI and data mapping tools don’t support this type of transformation natively.

This means that the ingestion process would need to change.

Possible Alternatives:

  1. Use of APIs (Recommended for Small Data Sets):
    If the data volume is manageable, you might consider ingesting via the AEP API. This allows you to send data in a proper JSON format, which may give you more flexibility with mapping to a schema containing arrays.

  2. Flatten the Structure on Profile Schema:
    Another option is to define multiple attributes on the profile itself, such as:

    • fieldA1, fieldB1

    • fieldA2, fieldB2, etc.
      This is a manual workaround and not scalable, but it works if the maximum number of array elements is known and limited. However, you'd still need to modify your CSV format accordingly.

  3. Use Event Schema Instead (Recommended Approach):
    The best and most scalable option may be to store this data on an event schema instead of the profile schema. Here's why:

    • You don’t need to predefine complex structures.

    • Just include fields like profileID, fieldA, fieldB, etc.

    • Then, map your CSV to the event schema and ensure the same namespace is used for the profileID in both the profile and event schemas.

    This approach is typically easier to manage and aligns well with AEP’s strengths—especially if your use case can be supported through event-level data.

I hope this helps clarify the options. Let me know if you’d like to go deeper into any of these approaches.

Best regards,
Parvesh

Parvesh Parmar – Adobe Community Advisor https://www.linkedin.com/in/parvesh-parmar/
    K
    Adobe Employee
    KevinFosterDXAdobe Employee
    Adobe Employee
    July 18, 2025

    I've also found that sometimes arrays are the only valid data structure that I can use for my data, e.g. lists of things per profile.

     

    But instead of using CSV format, it's easy to get a sample JSON record for your target dataset by going to the Schemas > Browse page, selecting the target schemas, and then clicking:


    "Download sample file

    Generate a sample JSON object that conforms to your chosen schema's structure. This object can serve as a template to ensure your data is correctly formatted for ingestion into datasets that employ that schema. The sample JSON file will be downloaded by your browser."

     
    I've found that it's been better for me to use CSV only for ad hoc testing, and to stick with JSON for my automated production ingestion.
    John_Man
    Community Advisor
    John_ManCommunity Advisor
    Community Advisor
    August 5, 2025

    Hi @rs195 

     

    In your CSV upload, AEP will take them as individual records with the same ID, and thus the last row will overwrite the previous ones.

     

    You can strcuture your data in CSV as an JSON structure, in your case it will be array of objects. Then use the json_to_object function to assign it to the schema. Below is a sample (which involves array of array of objects)

     

     

     

    If JSON structure is difficult, can also use delimited fields in CSV as well, see below link for reference,

     

    https://experienceleaguecommunities.adobe.com/t5/adobe-experience-platform/data-not-getting-ingested-into-array-type-objects-using-csv-file/m-p/703537#M5574

     

    Hope it helps,

    John

     

      Terms & ConditionsAccessibility statement

      Loading footer...