Duplicates removal using Fusion?

[Viktoriia]

Hello,

Is there a way to remove duplicates coming from Parse CSV?

CSV file is coming from another system, so we do not have access to it before it's been received by Fusion. We need to make sure that we have only 1 line per employee, and exclude any duplicates, if they happen to be there before the scenario processes it further.

Are there any examples how to implement this?

Could you please advise?

Thank you!

[lgaertner]

Hello Viktoriia,

Without knowing the structure of your csv file, I would use this workflow in Fusion.

As I do not have a corresponding csv file, I used a data store with contact names in the column name:

1. use the module Set variable with the variable value emptyarray, which you find under keywords on the array functions tab

2. parse your csv records into an Array aggregator 

3. add another Set variable module with the variable name from step 1 and the value map(2. <Array[]>; data.name)

4. add another Set variable module with the variable name from step 1 and the value deduplicate(<3. [])

Regards

Lars

[Viktoriia]

Thank you, Lars, I'll try this approach.

[Viktoriia]

Lars,

I tried this flow, but so far I was not able to make it work and not sure what I'm missing:

For example, I send file with 4 entries, 2 of them duplicates.

• Array Aggregator module gets all 4 bundles from Parse CSV as collections and in Output is 1 Bundle as Array with 4 Collections
• But next Set Variable module, where I map array from Array Aggregator is empty
• Then next Set Variable module that is supposed to deduplicate array is throwing an error

What might be a problem?

[ChrisStephens]

As an alternative approach, you could place an array aggregator right after the parse CSV, and group it by whatever key you need to ensure is only processed once.

[Viktoriia]

Thank you, Chris! I tried that but for some reason the other values (except for the Group by field) were empty and did not carry over values to the following modules (I have more than 15 columns in CSV file total). I guess I'm just missing something there.

[ChrisStephens]

You have to select which fields to pass thru, did you do that?

[Viktoriia]

I tried, but they were not available at that time (aggregator did not see the parse module).

[ChrisStephens]

Sometimes you have to run it once for Fusion to pick up what available options are. Sometimes even that doesn't work, in which case you can create a set multiple variables module, map your csv fields into that, and then pick up the variables with your aggregator.