Exclude event columns in Adobe DataStream set up

[Cristhian_AlexisNa]

Hello everyone,

I understand that in DataStream set up you can apply some rules for filtering event data that comes from Web/Mobile SDK and then it send data to event dataset. But, Is there a way to send some columns to event dataset enabled for profile and the rest of them, send to other dataset that isn't enabled for profile? In other words, I want to exclude some columns for the first dataset (I don't want to send all of them) and send it to other dataset.

The purpose of this, is to reduce the usage of Total Storage License.

Any advice is grateful.

Thank you.

[Harveer_SinghGi1]

Hi @Cristhian_AlexisNa ,

You cannot split a single event and send it to multiple datasets in AEP using datastreams data prep. However, you can track the same event twice using sendEvent and send it to different datasets with respective columns needed in that dataset.

What you'll have to do is trigger the sendEvent twice and route data to different datasets which can be achieved through data collection following below steps,

• Create two datastream,
  • DataStream 1:
    • Configure it with dataset 1 that needs to be profile enabled
    • Add rule in data prep to drop columns that are not needed in profile
  • DataStream 2:
    • Configure it with dataset 2 that is not required for profile
    • Add rule in data prep to drop columns that are needed in profile

• In Adobe Launch use datastream 1 as default datastream for WebSDK configuration
• Prepare the payload for dataset 1 (populate only needed columns/fields)
• Send it to AEP using sendEvent.
• Prepare the payload for dataset 2 (populate only needed columns/fields)
• Send it to AEP using sendEvent while using the datastream override option to select datastream 2 for this event.

This approach however comes with few caveats as you might have noticed,

• It causes doubling the sendEvent calls, so it won't be an ideal solution if you need to split the data on every AEP tracking call
• Same AEP event will be split in two places so you won't be able to breakdown the attributes from one dataset with other as they will behave as two different events

I'll suggest exploring other options within AEP where you track the events as it is from your property in a single event dataset and then using AEP capabilities like a CTAS or data distiller create another dataset with only the columns required for profile and have this dataset profile enabled.

Cheers!

[Harveer_SinghGi1]

[Update]

You need to perform only one of below steps for each dataset and not both,

• Add rule in data prep to drop columns that are needed in profile
• Prepare the payload for dataset (populate only needed columns/fields)

I'd suggest to use the second option where you prepare the relevant payload in Launch itself because using the sendEvent. Other reason to use this option is that I have noticed data prep to fail sometimes when working with large amounts of data which means the transformation might be missed on some events. Adobe Launch approach on the other hand doesn't have any such issues.

Cheers!

[Cristhian_AlexisNa]

Hi @Harveer_SinghGi1 ,

I have 2 questions:

1. When you mention Rule in DataPrep, do you mean Dynamic Setup or Edit Mapping?

2. With the second option, I must create two differents datastream for each dataset?

Thank you!

[Harveer_SinghGi1 · Accepted solution]

Hi @Cristhian_AlexisNa ,

The two options work as below,

• Drop the unwanted columns/fields through DataPrep (Edit Mapping) in DataStream: Steps to implement this solution are,
  • Create two DataStream,
    • DataStream 1:
      • Configure it with dataset 1 that needs to be profile enabled
      • Add rule in data prep to drop columns that are not needed in profile
    • DataStream 2:
      • Configure it with dataset 2 that is not required for profile
      • Add rule in data prep to drop columns that are needed in profile
  • In Adobe Launch use DataStream 1 as default DataStream for WebSDK configuration
  • Send entire payload to AEP using sendEvent (This sends data to dataset 1 using DataStream 1).
  • Send entire payload to AEP using sendEvent while using the datastream override option to select DataStream 2 and dataset 2 for this event
  • 
• Drop the unwanted columns/fields through Adobe Launch: Steps to implement this solution are,
  • Create one DataStream,
    • DataStream 1: 
      • Configure primary event dataset as dataset 1 that needs to be profile enabled
      • Configure secondary dataset as dataset 2 that is not required for profile
      • 
  • In Adobe Launch use DataStream 1 as default DataStream for WebSDK configuration
  • Prepare the payload for dataset 1 (populate only needed columns/fields)
  • Send dataset 1 specific payload to AEP using sendEvent. (This sends data to dataset 1 using DataStream 1)
  • Prepare the payload for dataset 2 (populate only needed columns/fields)
  • Send dataset 2 specific payload to AEP using sendEvent while using the datastream override option to select DataStream 1 and dataset 2 for this event
  • 

So, to answer your questions, DataPrep is "Edit Mapping" option only in DataStream. And in second option you only need to create one DataStream with both event datasets added to it and choose the datasets for respective sendEvents.

Cheers!