This post highlights how the Adobe Experience Platform engineering organization is working to improve efficiency and maximize the performance of our services while consistently reducing our impact on the environment.
As we continuously build, release, and leverage new technologies, it's easy to overlook how these affect the world we live in. From the electricity powering our computers to the data centers processing millions of requests per second, in just one year these technologies generate the equivalent of 26.5 million cars in carbon pollution.
At Adobe, we recognize that our products are not only changing the way our customers work, they are also changing the environment. Our long-held culture of conservation is woven into everything we do, and we believe that sustainability in our industry begins with the lines of code we write. This means that as technologists it's our responsibility to build solutions that consistently reduce our impact on the planet.
This post details such a solution developed by the Adobe Ethos team who promotes resource efficiency across Adobe Experience Platform teams through empowering them to run high-performing applications at a lower cost for both Adobe and the environment.
The problem with provisioning
The technology industry prioritizes performance, and it's common practice for engineers to over-provision their services to accommodate varying workloads. While doing this certainly lowers the risk of outages, it also results in excessive energy usage, higher overall costs, and a growing carbon footprint.
Adobe has long acknowledged that our industry is increasingly resource-intensive and is constantly looking to reduce energy consumption and stabilize costs. So far, we have moved to the cloud, invested in efficiency-driven technology (e.g., schedulers and microservices), and even designed our buildings to produce just as much energy as they consume to achieve net-zero energy usage.
Now, the Adobe Ethos team is extending this sustainability effort by taking a closer look at how our applications are using their resources and spreading awareness about the costly effects of over-provisioning.
Sending a message to provision more efficiently
Our initial effort to reduce the cost of over-provisioned services running on Adobe Ethos – a multi-cloud container platform that supports Adobe Experience Platform with self-service provisioning, pipelines, tooling, compliance, and security controls.
Our initiative was simple:
Measure how each application is performing with its allocated resources per specific metrics: CPU utilization, memory, total network traffic.
Make an "efficiency recommendation" based on utilization over the previous six months. This is sent in the form of a pull request (PR) to their source-code repository.
If the PR is approved, automatically test, merge, and re-deploy their application with the recommended resource allocation.
To measure the performance of applications running on Adobe Ethos, we partnered with the business intelligence team to generate a dashboard that visualized the amount of allocated resources per application, how much it was costing Adobe, and what percentage of resources was actually being used.
In the case of one team, their application was running on 50 cores with only 9% utilization. Based on their previous usage pattern, we sent a PR with the appropriate vertical container size for their application. Once the engineer reviewed the PR and approved the recommendation, it triggered a series of automated tests before re-deploying their application with the recommended container size.
The following graph illustrates their utilization after applying this efficiency effort:
Figure 1: CPU utilization after right-sizing the application’s container size.
After right-sizing their containers in April, the application increased its utilization to 95% running on just 20 cores, saving a considerable amount of resources. The Adobe Ethos platform also ensured the application was sufficiently provisioned during a load spike in May (see graph above) by automatically scaling the containers horizontally – freeing the team from scrambling during an outage.
Another notable efficiency improvement belongs to Adobe Experience Platform. Teams merged almost a dozen PRs into their repositories, which saved a total of 1,256 cores and reduced Adobe's monthly spend by an average of 58%. This outcome highlights yet again how efficiency not only maximizes our utilization but also leads to significant savings.
Achieving efficiency while maintaining performance
To further prove that it was possible to save resources without impacting an application's performance, we embarked on a more progressive initiative. Based on the feedback from our PR messages, it became clear that not all teams can efficiently scale their containers based on set metrics, such as CPU utilization or network traffic. As a result, our second initiative aims to provide automatic horizontal scaling based on custom metrics.
This practice is known as "queue-based scaling," which we implemented using both internal and open-source technologies.
To demonstrate queue-based scaling, we enlisted the following technologies:
In the demo, we simulate a load increase by sending multiple messages to the client queue on Azure Queue Storage – overwhelming the service. The exporter, which is configured using client-provided parameters, reads the number of messages in the client queue and passes that number to HPA via Prometheus. Since the number of messages exceeds the set limit (in this case: six), HPA automatically scales up the container, enabling the service to run smoothly once again. Furthermore, once the queue returns to zero, the additional containers are automatically terminated to free the cores and save resources.
While still a work in progress and with known limitations, our goal is to encourage Adobe engineers to trust in the platform to sufficiently provision their applications, as well as enhance their role in helping Adobe run a sustainable business.
Leading the way with sustainable solutions
Adobe is known for transforming the industry with innovative products and outstanding services. Now we want to lead the way in building sustainable solutions that help customers do amazing things while protecting the environment.
As the digital world continues to flourish, efficiency improvements play a key role in pacifying the growing impact of energy usage. With the Adobe Ethos platform, we empower our engineers to curb the climbing costs of over-provisioning simply by trusting the platform to responsibly manage their resources. Our next step is to migrate our applications to a new tech stack that supports queue-based scaling. Additionally, we will encourage application owners to be mindful of their resources, provision responsibly, and code with conservation in mind.
While running a sustainable business is everyone's responsibility, as developers, architects, data scientists, and engineers, we have an opportunity to build and promote cleaner technology that allows us to continue innovating at Adobe while doing our part for the planet.
*Special thanks to Laurent Rouquette for developing the exporter used in this demo.