Overwriting the Array Object

[nikhil_719]

Hi,

I am  writing into an array inside a data set using query service daily against a profile , coming next run we wish previous values of array to get replaced with the new values. As we notice right now values inside array fields are getting appended rather getting replaced.

Example -

Day 1 :

Data ingested using query (batch)

ID	Product Viewed Recently (array)
123	‘A','B','C'

In Profile we see Product Viewed Recently as ‘A’,’B’,’C’ across 0,1,2, array placement

Day 2

Data ingested using query (batch)

ID	Product Viewed Recently (array)
123	‘D’,’E’,’F’

In Profile we see Product Viewed Recently as ‘A’,’B’,’C’ ,’D’,’E’,’F’ across 0,1,2, 3,4,5 array placement. Whereas only want to see ‘D’,’E’,’F’.

Is there a way for specific dataset we can set possibility of replacing an array field instead of appending ?

[Danny-Miller]

Can you tell us a little more about:

1. How you are sending the data in and the payload?
2. Where you are sending the data in (one dataset or many)?
3. What the schema looks like?
4. How you are verifying what the data looks like after Day 2 (Query Service or Profile Viewer, etc.)?

By default the array should be replaced, assuming:

1. We are talking about the Profile, not Data Lake
2. All the data is in one dataset

[nikhil_719]

Hi Danny-Miller,

Here are my comments , thanks in advance 

Can you tell us a little more about:

1. How you are sending the data in and the payload? - We are sending data in the dataset via query service using a struct command to convert in expected object array 
2. Where you are sending the data in (one dataset or many)? - In a single dataset 
3. What the schema looks like? - Dataset is created referring the schema
4. How you are verifying what the data looks like after Day 2 (Query Service or Profile Viewer, etc.)? - In profile , inside data set we have multiple rows  which looks fine 

By default the array should be replaced, assuming:

1. We are talking about the Profile, not Data Lake - yes we are talking about profile and I have similar understanding that data should be replaced 
2. All the data is in one dataset - Yes these records are ingested in same dataset 

[Danny-Miller]

Interesting.  It might help to see the actual code/data, but sounds like you are doing the right thing.

[nikhil_719]

I tried inserting array values for the same profile via data flow (batch) as well skipping the query service to test if previous array is replaced or appended. In this case also I see new array values getting added to array list rather than replacing the previous one. Could this be any property setting of dataset / schema 

[Danny-Miller]

Not that I know of.  I'd try and reproduce it in a new sandbox, schema, dataset, etc. and simplify it down as much as possible.  Then open a ticket &/or come back here with actual screenshots and queries.

[arijitg · Accepted solution]

Hi Nikhil,

Little curious on how "Product Viewed Recently (array)" is used in a profile class and not in event. 

Can you try the mapper function upsert_array_replace and see if it does the overwrite for you? (If your dataset is upsert enabled)

Ref : Data Prep Mapping Functions | Adobe Experience Platform

[nikhil_719]

Yes dataset is upsert enabled, this makes sense. I think when dataset is upsert enabled Array enrichment is considered as append rather than replacing current array (which I think could make sense ). Possible solution could be to either disable upsert true