Expand my Community achievements bar.

SOLVED

Overwriting the Array Object

Avatar

Level 1

Hi,

I am  writing into an array inside a data set using query service daily against a profile , coming next run we wish previous values of array to get replaced with the new values. As we notice right now values inside array fields are getting appended rather getting replaced.

Example -

Day 1 :

   

Data ingested using query (batch)

ID

Product Viewed Recently (array)

123

‘A','B','C'

In Profile we see Product Viewed Recently as ‘A’,’B’,’C’ across 0,1,2, array placement

 

Day 2

 

Data ingested using query (batch)

ID

Product Viewed Recently (array)

123

‘D’,’E’,’F’

 

In Profile we see Product Viewed Recently as ‘A’,’B’,’C’ ,’D’,’E’,’F’ across 0,1,2, 3,4,5 array placement. Whereas only want to see ‘D’,’E’,’F’.

Is there a way for specific dataset we can set possibility of replacing an array field instead of appending ?

1 Accepted Solution

Avatar

Correct answer by
Moderator

Hi Nikhil,

 

Little curious on how "Product Viewed Recently (array)" is used in a profile class and not in event. 

Can you try the mapper function upsert_array_replace and see if it does the overwrite for you? (If your dataset is upsert enabled)

Ref : Data Prep Mapping Functions | Adobe Experience Platform

View solution in original post

7 Replies

Avatar

Employee Advisor

Can you tell us a little more about:

  1. How you are sending the data in and the payload?
  2. Where you are sending the data in (one dataset or many)?
  3. What the schema looks like?
  4. How you are verifying what the data looks like after Day 2 (Query Service or Profile Viewer, etc.)?

By default the array should be replaced, assuming:

  1. We are talking about the Profile, not Data Lake
  2. All the data is in one dataset

Avatar

Level 1

Hi Danny-Miller,

Here are my comments , thanks in advance 

Can you tell us a little more about:

  1. How you are sending the data in and the payload? - We are sending data in the dataset via query service using a struct command to convert in expected object array 
  2. Where you are sending the data in (one dataset or many)? - In a single dataset 
  3. What the schema looks like? - Dataset is created referring the schema
  4. How you are verifying what the data looks like after Day 2 (Query Service or Profile Viewer, etc.)? - In profile , inside data set we have multiple rows  which looks fine 

By default the array should be replaced, assuming:

  1. We are talking about the Profile, not Data Lake - yes we are talking about profile and I have similar understanding that data should be replaced 
  2. All the data is in one dataset - Yes these records are ingested in same dataset 

Avatar

Employee Advisor

Interesting.  It might help to see the actual code/data, but sounds like you are doing the right thing.

Avatar

Level 1

I tried inserting array values for the same profile via data flow (batch) as well skipping the query service to test if previous array is replaced or appended. In this case also I see new array values getting added to array list rather than replacing the previous one. Could this be any property setting of dataset / schema 

Avatar

Employee Advisor

Not that I know of.  I'd try and reproduce it in a new sandbox, schema, dataset, etc. and simplify it down as much as possible.  Then open a ticket &/or come back here with actual screenshots and queries.

Avatar

Correct answer by
Moderator

Hi Nikhil,

 

Little curious on how "Product Viewed Recently (array)" is used in a profile class and not in event. 

Can you try the mapper function upsert_array_replace and see if it does the overwrite for you? (If your dataset is upsert enabled)

Ref : Data Prep Mapping Functions | Adobe Experience Platform

Avatar

Level 1

Yes dataset is upsert enabled, this makes sense. I think when dataset is upsert enabled Array enrichment is considered as append rather than replacing current array (which I think could make sense ). Possible solution could be to either disable upsert true