How to Improve your Optimization Program

[#1]

By @surebee and @vinay_sodha 

With optimization programs becoming essential for businesses, a question we often see in consulting from clients is how they can improve their existing program.  

Improving your optimization program can be a daunting task, but it doesn’t have to be! The ideal state for an optimization program can look different based on your organization’s resources, testing objectives and KPIs - there is no one-size-fits-all approach. We’ll review different aspects of optimization programs and what we’ve seen helps to improve them. 

Assess current state 

The first step is to establish a ‘baseline’ where you will want to assess where you are currently in your program. Having a good understanding of your current state will reveal areas to improve to effectively scale your optimization program. 

Start by taking an inventory of your resources, testing workflows, and processes. 

Adobe technology stack 

Get a good understanding of the technology stack you are working with. This will help assess and categorize the program limitations into technical, people, and process buckets. For example: 

• What Adobe products are you utilizing, for example: Target, Analytics or CJA?  
• Have you migrated to WebSDK? If not, is there a migration plan in place for the near future? 
• Are you using Data Collection with WebSDK? 

Teams involved 

• Do you have a dedicated team for your testing program?  
• Are the team members limited to internal parties and Adobe Consulting Services, or are other external technology partners involved? 
• Is there generally a good sense of alignment between the business and tech stakeholders internally and externally?  

Roles and responsibilities 

• Are there established roles and responsibilities between team members?  
• How are test results communicated across the team? 

Identify knowledge baseline 

• Does your team understand Adobe Target’s capabilities (e.g., A/B, XT, Auto-Target, Recommendations)?
• Are they using advanced targeting capabilities such as profile scripts?  
• If not, are there any knowledge gaps that need to be addressed? 

Testing ecosystem 

Goals:  

• Do you have a clear set of testing goals and KPIs?  
• How up to date are these goals and are they suitable for your current testing needs? 

Workflow: 

• What does the process look like from test ideation to development to go live?  
• Time to market per test: How long does it take to launch a test? Is this timeline ideal for your program or is it limiting your testing volume? 

Type of tests: 

• What kinds of tests do you typically run? A/B, Multivariate (MVT), Automated Personalization (AP), Recommendations? 
• Are you using a good mix of manual testing and machine learning? 

Test result analysis: 

• When analyzing test results, are you analyzing your KPIs, additional metrics, and applying the right segmentation in your analysis? 
• Does the data align with your expectations? 

Taking Action 

If you have made it this far, congratulations on doing much of the heavy lifting! Now comes the fun part - start on a plan to action and address the gaps identified in the initial assessment. 

Adobe technology stack 

Technology is constantly evolving with product features, enhancements, and bug fixes. Make sure that the stack you are on supports the features that you are looking for. For example: Are you... 

Interested in machine learning or Recommendations?...Ensure you have a Target Premium license 

Looking for reporting parity between Target and Analytics?...Make sure you are provisioned for Analytics for Target (A4T) 

While it is always recommended to use the latest and greatest Adobe stack, we understand that this is not the reality for most customers. Adobe Consulting can help perform an in-depth implementation audit and compose a step-by-step roadblock mitigation and migration plan to upgrade your stack and achieve your ideal implementation state. Major upgrades, such as migration from at.js to WebSDK, require careful planning and investments in time and resources.  

People and process 

Team sizes and structures vary by organization. For example, your dedicated optimization team could include:  

• Business stakeholders for test ideation 
• Project Manager to drive the testing program 
• Adobe Business Consultant to assist with test ideation and analysis 
• Adobe Technical Consultant for test development, implementation and troubleshooting 
• QA resource to help validate end user experiences and tracking 

Here is an example of a program structure with a strong center of excellence and Adobe Consulting's support.

Here's an example of a mature program model with multiple business units and strong leadership visibility.

Test types and KPIs 

Aligning your testing goals and KPIs with the current business needs and relevance to the problem you are trying to solve is also important. For example: 

Are you a... 

Bank trying to increase engagement with credit cards or loans? 

Retail site trying to increase revenue through orders for certain products? 

At minimum you’ll want to make sure you’re tracking the essential conversions, such as when a user gets to a confirmation page and click-throughs on relevant calls to action. 

Ensure your business stakeholders and consultants have a detailed test plan so that you don't have overlapping campaigns and conflicts between testing activities. This will help you stay focused on your main objectives.  

Choosing the test type best suited for your use case is equally important. For example: 

• A/B: Ideal for testing/optimizing single variables per variant (from designs, messaging, page layout, or page structure) 
• XT: Provides personalized / curated experiences to specific groups of users 
• Machine Learning: Scale personalization and dynamically serve the most relevant experience to each visitor based on their profile and behavior 

For further reading on choosing the best activity type, we have a good resource available on the Experience League. Link here. 

When reviewing test results, make sure that you are doing an apples-to-apples comparison, especially when applying additional segments and metrics. If you have never tested a particular page, try running a baseline test to establish data before running additional tests to ensure that the data aligns with your expectations. 

Conclusion

This chart summarizes some of the ways you could improve your Optimization Program.

Thank you for reading along! We hope this blog gives you some pointers to help review and improve your optimization program. Also, once you have gotten your optimization program to where you want, don’t stop there. Optimization itself is a continual process, so your program should continue to evolve as well. Just remember as long as you have the right people and processes in place everything else should fall into place. Stay tuned for another post in the future! 

***  
Are you looking to optimize your website's performance and drive conversions?  
Look no further than Adobe - the industry leader in A/B testing. With more A/B tests under our belt than any other organization in the world, we have the expertise and knowledge to take your online presence to the next level. Adobe Target is unrivaled in the industry, and with Adobe Consulting, you can have access to this powerful tool and the expertise behind it. Don't miss out on the opportunity to unlock the full potential of your website. Contact us today and discover how Adobe can help your business thrive. If you would like to understand how Adobe can help your business reach out to Adobe Consulting Services.