Launch Web SDK Extension - Define a custom _id attribute in Experience Event calls or a deduplication key for identical events from different sources

[#1]

Description - These days in Launch's Web SDK extension, it is not possible to manually override the "_id" attribute in Web SDK XDM object data elements.

As a workaround, if one wanted to manually define this _id, one would have to code an alloy "sendEvent" and specify the _id there.

The approach / idea resembles very much the hybrid tracking approach for client- and server-side, that many 3rd party vendors suggest these days, where a deduplication key can be specified that either drops or merges events with duplicate ids.

Why is this feature important to you

Completeness of data and fastest possible data availability:

• Web SDK data relies on cookie consent which may not have been given by the users i.e., an order event may or may not be captured by the Web SDK
• this lack of order data significantly impacts the number of addressable members in an audience, that may be built upon CRM and commerce data
• while this commerce data can also be streamed into the Platform from the commerce or CMS backend through an HTTP API call, this will still likely be slower than the Web SDK data if you need it for immediate personalization (e.g., recommendation)
• but due to the immutability of Experience Events, sending order data from two sources, Web SDK + Commerce will affect the order events and revenue numbers in the CDP and connected analytics sources.
• right now, there is no deduplication key like other vendors provide in their Conversion API
• now, if we could manually specify the same _id attribute for all data sources, which ever event comes in first wins, and the other(s) with the same _id will be ignored in the CDP profile (while they are still floating around in the datalake)

How would you like the feature to work

Either implement a deduplication key feature like other Conversion APIs or allow the _id to be manually overridden (with a clear warnings about the possible implications)

Current Behaviour

The UI shows a big warning "The value for this field will be auto-populated when this data element is provided as the XDM object for a Send event action. This value cannot be overwritten."

[#2]

Hi @bjoern__koth ,

To make sure I am understanding you, you want to be able to handle situations like this:

• A company is ingesting purchase data from two different systems (web sdk page events and a backend commerce system). Web SDK has speed advantages. The backend commerce system has thoroughness advantages, since it includes purchases from users who block Web SDK calls from their browsers.
• Since both systems will often have data for the same purchase, you want to have a way to deduplicate the two, separately recorded purchase events.

My initial reaction is that this deduplication should be handled by the purchaseID. So perhaps something to declare uniqueness of an event namespace, similar to what we have now for identity namespaces. You could declare a purchaseID as unique namespace, so any additional events arriving with the same purchaseID would be dropped.

Or maybe it would be important to be more sophisticated like dedupe the combo of purchaseID and eventType, in case there was a return processed for the purchaseID?

I am very familiar with Adobe Target's approach to deduplicating orders. There was built-in deduplication of any order with identical values in the "orderID" field. But the system would actually record both events and just filter the duplicates out in the reporting. You could open the Audit Report and view the full list of order events, including those flagged as duplicates. Perhaps it is better to record all of the events (not block any at ingestion), but to have mechanisms that do this when the data is being accessed? I assume Adobe Analytics must have had a similar feature. Maybe CJA does too?

Anyway, interesting idea.

[#3]

Hi @dwright, 

thanks for your feedback and very interesting views indeed!

I have been digging and could not find any conclusive evidence that commerce.order.purchaseID is in fact being used as deduplication key. 

In the classic AA world, this has been the case, in CJA it can be set up using metric deduplication, but do you know how this is being handled in the context of RT-CDP or AJO? Maybe it is already supported and an undocumented feature.

Alternatively, something built in that does not require creative workarounds to make it work.

Like an Adobe field group similar to "Profile test details" that introduces a single dedupeKey.

Bottom line is, that it should be possible to ingest identical purchase data from multiple systems without affecting revenue, without the need to use Query Service.

This is almost more of an AEP product idea as seems xD

Cheers
Björn