AEP: CSV Ingestion into Object Array Schema – Overwriting Issue + Mapping/Transformation Clarification (Snowflake too)

[RS195]

Hi all,
I’m working with an XDM Individual Profile schema in AEP that includes a custom array of objects. This array is defined in a custom namespace and sits one level under the profile, for example:

{
  "_tenant": {
    "customArrayObject": [
      {
        "fieldA": "value",
        "fieldB": "value"
      },
      {
        "fieldA": "value",
        "fieldB": "value"
      }
    ]
  }
}

My source file is in CSV format, with multiple rows for the same identity (same ID). Each row represents one object that should go into the array. Example:

ID       FieldA     FieldB
123      value A    value B
123      value C    value D
123      value E    value F

Issue:
When I ingest this into AEP (via UI upload or dataset), only the last row is retained, the earlier ones are overwritten. The array is not constructed.

My Questions:

1. Is this behavior expected when using CSV to populate object arrays?

2. Are any transformations or mapping logic available during CSV ingestion that would allow grouping these rows into an array under one profile?

3. I also have a Snowflake connection set up in AEP, if I ingest the same structured data using that source, would the array transformation behave differently? Or would it still require JSON-style structure before ingestion?

4. Is JSON ingestion still the only reliable way to preserve nested arrays like this?

[Parvesh_Parmar]

Hello @RS195 ,

As I understand, you're looking to store data in the form of an array of objects at the profile level in Adobe AEP.

Before diving into how this could be done, I want to highlight an important point: even if you manage to store the data in that format, it may become challenging to use it effectively later on—for example, for segmentation, personalization, or exporting to third-party systems. So, I recommend carefully considering how you plan to use this data after ingestion. In my experience, storing array-type structures at the profile level often leads to complications.

Now, regarding CSV ingestion, there are limitations:

• CSV files are typically structured in rows, and you cannot directly map multiple rows from a CSV to a single profile record in JSON array format.

• The Adobe UI and data mapping tools don’t support this type of transformation natively.

This means that the ingestion process would need to change.

Possible Alternatives:

1. Use of APIs (Recommended for Small Data Sets):
   If the data volume is manageable, you might consider ingesting via the AEP API. This allows you to send data in a proper JSON format, which may give you more flexibility with mapping to a schema containing arrays.

2. Flatten the Structure on Profile Schema:
   Another option is to define multiple attributes on the profile itself, such as:

   • fieldA1, fieldB1

   • fieldA2, fieldB2, etc.
     This is a manual workaround and not scalable, but it works if the maximum number of array elements is known and limited. However, you'd still need to modify your CSV format accordingly.

3. Use Event Schema Instead (Recommended Approach):
   The best and most scalable option may be to store this data on an event schema instead of the profile schema. Here's why:

   • You don’t need to predefine complex structures.

   • Just include fields like profileID, fieldA, fieldB, etc.

   • Then, map your CSV to the event schema and ensure the same namespace is used for the profileID in both the profile and event schemas.

   This approach is typically easier to manage and aligns well with AEP’s strengths—especially if your use case can be supported through event-level data.

I hope this helps clarify the options. Let me know if you’d like to go deeper into any of these approaches.

Best regards,
Parvesh

[KevinFosterDX]

I've also found that sometimes arrays are the only valid data structure that I can use for my data, e.g. lists of things per profile.

But instead of using CSV format, it's easy to get a sample JSON record for your target dataset by going to the Schemas > Browse page, selecting the target schemas, and then clicking:

"Download sample file

Generate a sample JSON object that conforms to your chosen schema's structure. This object can serve as a template to ensure your data is correctly formatted for ingestion into datasets that employ that schema. The sample JSON file will be downloaded by your browser."

https://experienceleague.adobe.com/en/docs/experience-platform/xdm/ui/sample#generate-sample-data-fo...

I've found that it's been better for me to use CSV only for ad hoc testing, and to stick with JSON for my automated production ingestion.

[RS195]

Hi @Parvesh_Parmar, thank you, this was extremely helpful.

You're right, my schema currently uses an object array under the profile (defined in a custom namespace), and I ran into exactly what you described: only the last row is retained during CSV ingestion, which makes sense now in hindsight.

I really appreciate the way you outlined the trade-offs across options. I had a few follow-ups just to make sure I fully grasp things:

1. For the event schema approach, can I still create meaningful profile-level segments (e.g., “has at least one record with activeLoan = true”) by linking the event data to the profile using customerIdentifier as the identity?

2. If I go with flattening the structure on the profile (e.g., field1_account1, field1_account2), I assume ingestion via CSV works, but segmentation becomes clunky and capped by the number of fields defined, correct?

3. Just confirming: even if I were to ingest the same data through a Snowflake source connector, unless the data is already pre-structured as an array in JSON-like format, AEP would still treat multiple rows as overwrites, right?

Thanks again for such a thoughtful response, this really helped clarify things. I’ll wait to hear back on the above before aligning the design with the team.

[Sukrity_Wadhwa]

Hi @Parvesh_Parmar,

Can you please help @RS195 further with their follow up queries?

Thanks!

[John_Man]

Hi @RS195 

In your CSV upload, AEP will take them as individual records with the same ID, and thus the last row will overwrite the previous ones.

You can strcuture your data in CSV as an JSON structure, in your case it will be array of objects. Then use the json_to_object function to assign it to the schema. Below is a sample (which involves array of array of objects)

If JSON structure is difficult, can also use delimited fields in CSV as well, see below link for reference,

https://experienceleaguecommunities.adobe.com/t5/adobe-experience-platform/data-not-getting-ingested...

Hope it helps,

John