Streamlining Data Integration: Edge Network, AEP and Google Campaign Manager 360

[#1]

Streamlining Data Integration:  Edge Network, AEP and Google Campaign Manager 360
Kritika Pareek, Julien Piccini

                       

Integrating Adobe Experience Platform (AEP) with Google Campaign Manager 360 (CM360) to track and send conversion events from the Adobe Edge Network was a challenging but rewarding project. CM360, unlike AEP, isn't optimized for direct event streaming, necessitating creative solutions to bridge the gap.

Business Use Case

The objective was to establish a reliable method to forward conversion events from Adobe Edge Network to CM360. We tried reduce reliance on client-side scripts and improve data accuracy. The solution utilized the Campaign Manager 360 API to upload conversions in batch format, ensuring seamless integration while aligning with CM360’s attribution models and reporting standards.

Project Context

The project focused on forwarding the conversion events from AEP to Google CM360 through a server-side integration. However, CM360 could not process real-time event streams, which meant that the conversion events needed to be transformed into batch uploads.

This presented a challenge, as AEP typically sends profile-related data to destinations, whereas CM360 expects event data in a batch format. To meet the requirements, we needed to apply additional processing to transform AEP’s event data into the format CM360 could handle.

Our mission was clear: transform the data, apply the necessary filters, and ensure efficient delivery to Google’s Floodlight API. The goal was to accomplish this without overhauling the entire system architecture every time a new field needed to be incorporated, ensuring flexibility and scalability for future enhancements.

Implementation: Architecture Overview

                                      

Data Transformation and Filtering

We implemented a rule-based mechanism to filter and forward conversion events from the Adobe Web SDK on the client website to a designated AEP dataset. This ensures only relevant events are processed efficiently.

                                                      

Structuring the Payload

One of the most critical aspects of the project was ensuring that the payload sent to CM360 could accommodate new fields without necessitating a complete system overhaul. To achieve this, we structured the schema to include a dedicated field that contained the entire payload as a JSON string, allowing us to easily adapt the payload by adding new elements in future updates, without affecting the core architecture. Most of the payload components were set up as Data Elements in Launch Server Side, which further enhanced maintainability. 

                                                                                         

Sending the data to AEP

To send data to AEP, we leveraged HTTP Ingestion, an efficient method for integrating data streams into the platform. Since the process operated entirely within Adobe’s ecosystem on the server side, we avoided the need for explicit authentication, simplifying the data ingestion process.

In this setup, streaming data was transformed into micro-batches, with each batch landing in the designated dataset approximately every 15 minutes. We avoided aggregating these smaller batches from AEP in AppBuilder, as it would have added unnecessary complexity. Instead, the batching process was delegated to AEP, simplifying the AppBuilder code and enhancing maintainability by eliminating additional batch aggregation logic. This approach provided a structured way to handle continuous data flow while aligning with the platform’s ingestion capabilities.

A key design choice was not enabling the dataset for profile, which offered several critical advantages:

1. Profile License Optimization: The identity used (e.g. gclid) did not count against the client’s licensed profile limits, preserving valuable capacity.
2. Identity Graph Integrity: Excluding the dataset from profiles ensured that the client’s identity graph remained unpolluted, avoiding unintended relationships or noise in identity resolution.
3. Profile Weight Management: By keeping extra event data separate from the profiles, we prevented unnecessary increases in the average profile weight, maintaining the overall system efficiency.

Extracting and Processing Data: AWS and Adobe AppBuilder

Once the data was ingested into AEP, it was exported to an AWS S3 bucket at regular intervals. Currently, the extraction occurs every 3 hours as an incremental load, which is the minimum export interval allowed by AEP. This ensures that each data extract contains only new records that need to be processed. To handle the data processing, we used AppBuilder, which read and processed the exported files. AppBuilder efficiently tracked the last read timestamp, ensuring it only processed new data and optimized resource usage.

Sending Data to Google Campaign Manager 360

Once processed, the data was sent to CM360 via the Floodlight API. The batching system grouped records in sets of 1,000 to ensure optimal delivery performance. Given the API’s reliance on OAuth tokens, we also implemented an automatic token renewal system, as the tokens generated from the Google service account expired every 60 minutes.

Monitoring and Sending Status to AEP

A key component of the implementation was establishing a feedback loop to ensure visibility into the data processing status. After sending the conversion data to Google CM360, we used the AEP Data Ingestion API from AppBuilder to forward Google's response back to a dedicated AEP dataset. This dataset was crucial for monitoring and analyzing record ingestion success and failure, enabling ongoing analysis and improved data processing oversight.

The status dataset was integrated with Customer Journey Analytics, a native UI data analysis tool within the Adobe ecosystem. This integration provided the customer with real-time insights into the success rate of each record processed by CM360, enabling them to identify potential issues or failures at a granular level.

By monitoring the data flow and pinpointing any anomalies or issues in processing, the client gained valuable insights to optimize their campaigns and improve overall data quality, ensuring a smoother, more efficient conversion tracking process.

Challenges and Lessons Learned

While the project was a success, it wasn’t without its challenges:

• Permissions: Ensuring that the correct access permissions were granted to the Google service account was a hurdle. The association of multiple profile IDs added to the complexity, requiring careful management.
• Schema Flexibility: Structuring the payload as an entire JSON string provided flexibility for adding new fields, allowing the system to be future-proofed without major reconfigurations. While this approach is effective, it is important to note that change management must be handled externally, as AEP does not audit changes to the payload.

Conclusion

Working with the customer on this integration has been a valuable experience, blending the robust capabilities of Adobe Experience Platform with the specific needs of Google Campaign Manager 360. Our solution enabled seamless conversion tracking and reporting, and the flexibility built into the system ensures it will scale efficiently as business needs evolve.

This approach is not limited to CM360 but can be applied to any batch data ingestion support API where the requirement is to convert streaming data into batch data with minimal data processing. For scenarios requiring more extensive data processing, a similar solution can be implemented on any other cloud platforms, offering adaptability and scalability across diverse business needs.

This project serves as an excellent example of how well-thought-out data processing architectures can bridge operational gaps between platforms while delivering accurate and timely results.

Questions? Feedback? Connect us on kpareek@adobe.com, piccini@adobe.com

See you on the next blog!

References

CM360 Profile list : https://developers.google.com/doubleclick-advertisers/rest/v4/userProfiles/list
Google API : https://developers.google.com/doubleclick-advertisers/rest/v4/conversions/batchinsert