Expand my Community achievements bar.

Duplicates removal using Fusion?

Avatar

Level 4

Hello,

 

Is there a way to remove duplicates coming from Parse CSV?

CSV file is coming from another system, so we do not have access to it before it's been received by Fusion. We need to make sure that we have only 1 line per employee, and exclude any duplicates, if they happen to be there before the scenario processes it further.

Are there any examples how to implement this?

 

Could you please advise?

Thank you!

8 Replies

Avatar

Level 10

Hello Viktoriia,

 

Without knowing the structure of your csv file, I would use this workflow in Fusion.

As I do not have a corresponding csv file, I used a data store with contact names in the column name:

 

1. use the module Set variable with the variable value emptyarray, which you find under keywords on the array functions tab

2. parse your csv records into an Array aggregator 

3. add another Set variable module with the variable name from step 1 and the value map(2. <Array[]>; data.name)

4. add another Set variable module with the variable name from step 1 and the value deduplicate(<3. [])

 

Regards

Lars

Avatar

Level 4

Lars,

 

I tried this flow, but so far I was not able to make it work and not sure what I'm missing:

 

For example, I send file with 4 entries, 2 of them duplicates.

  • Array Aggregator module gets all 4 bundles from Parse CSV as collections and in Output is 1 Bundle as Array with 4 Collections
  • But next Set Variable module, where I map array from Array Aggregator is emptyViktoriia_0-1700677360530.png
  • Then next Set Variable module that is supposed to deduplicate array is throwing an error

Viktoriia_1-1700677451096.png

What might be a problem?

 

 

Avatar

Community Advisor

As an alternative approach, you could place an array aggregator right after the parse CSV, and group it by whatever key you need to ensure is only processed once.

Avatar

Level 4

Thank you, Chris! I tried that but for some reason the other values (except for the Group by field) were empty and did not carry over values to the following modules (I have more than 15 columns in CSV file total). I guess I'm just missing something there.

Avatar

Community Advisor

You have to select which fields to pass thru, did you do that?

Avatar

Level 4

I tried, but they were not available at that time (aggregator did not see the parse module).

Avatar

Community Advisor

Sometimes you have to run it once for Fusion to pick up what available options are. Sometimes even that doesn't work, in which case you can create a set multiple variables module, map your csv fields into that, and then pick up the variables with your aggregator.