Adopting Modern CI/CD Practices for Adobe Experience Platform Pipeline
Authors: Pranay Kumar and Douglas Paton
In this post, we continue our look at Adobe Experience Platform Pipeline. In the first two parts of this series, we looked at in-flight processing in Pipeline, as well as how we used Kafka to create Pipeline. This time, we look at modern CI/CD practices adopted by Adobe Experience Platform Pipeline including our choice of implementation, and how they've helped us so far. We also look at how automation has helped streamline the entire process.
Adobe Experience Platform Pipeline is the messaging bus that uses Kafka to connect solutions. It provides geographically distributed managed Kafka clusters with an HTTP API wrapper. It serves over a dozen different locations, some with multiple clusters, and enables us to route data from one to another based on certain rules and topic metadata.
These geo-locations are a heterogeneous mix of infrastructure, including AWS, Azure, and our own private data centers. These systems are made up of virtual machines,
