Adobe Experience Platform

NimashaJain · 9/15/21

Author: Jaemi Bremner (@Jaemi_Bremner), Baubak Gandomi.

This is an article about a view we have adopted at Adobe Campaign regarding the tests we write, and how this helps writing end-to-end tests in enterprise software.

At first, we set out like most people to write a set of tests that tested our application. The problem we discovered is that, at least in our case, applications no longer work alone, and in many enterprise applications your product is usually part of a bigger whole. This means that there are other teams that will need to write scenarios that pass through your product. We decided on adopting a design and methodology, we call Test-as-a-Product, that allows people outside of your team to use your tests in their scenarios.

This approach had a lot of consequences, which led to a new discipline we now call “Test-as-a-Product” in writing tests. This article will share these insights. I hope it will help you in designing your frameworks and libraries.

Design over Technology

I can’t count the number of times I have met managers and QA who quickly in a conversation introduce a framework they wrote or one they are promoting. This is usually done without any knowledge of how you perform your tests or what challenges you are facing.

We are relying more and more on technology before considering the problems that need to be solved in the industry. I think the choice of technology for testing is quite complex. The questions I am most interested in are the following:

What technical challenges are we faced with, when testing our application?
What are the applications our system is integrating with?
What are the frameworks they are using to test?
How do we manage test data?
Who will be writing tests?
Who will be using the tests?

There are of course other requirements, but the ones above are the most relevant to me. Although technology could provide answers to the first item, the interfacing/integrating systems is just as important in enterprise software. In this case, the technology (unless using the same) is not the answer, and you can only solve such problems through design.

The design will encapsulate the functionality and problems you are trying to solve, so it is a good starting point. Once you have defined a design you can see which technology can help you best with it.

Although I think that design principles can be applied to most technologies, there are frameworks that force a design. We need to have a clear understanding of the requirements as well as a general understanding of the design before we pick the perfect technology.

As you can see in the requirements, many of the topics are human-related. For example, we need to identify roles and personas in the test system.

Personas

Test Personas is something that is not often discussed when testing. Most testing involves few personas:

QA Team
Development Team (Ideal)

In a Test-as-a-Product mindset, we expand the personas to also include:

IT Department: Validate a deployment or predicting problems before migrations.
Support Team: Reproduce an issue.
Product Manager(s): Write use cases and validate the hypothesis
Consulting: Perform data migrations and ETL operations with customers.

Our test libraries have been used by system integrators, and IT teams for environment validation. In some cases, they have also been used by product managers to test their hypotheses.

Test Product

Our tests can be used in two ways:

An SDK allowing others to write their own scenarios.
An application that lets the user run a specific scenario.

This has been made possible through design. We will share the design principles involved in this. These principles need to be made possible when we select the technology.

At Adobe Campaign, when we speak about the product, we usually mean the SDK. However, we have consumers of our tests that use the tests as an application in itself.

Since our tests are supposed to both test our product and help run end-to-end tests, we prefer using the term test library rather than test “framework”. Another reason for which I think a library is better suited is that apart from some design principles, we do not force a language nor do we abstract the code into a meta language.

Many of the design principles are generally good sense, but it is good to raise them in the context of testing.

Test Artifact

When viewing your tests as a product you are stating that your tests will be executed by someone else. As such, you need to start thinking of what is the artifact that is executed?

Do your clients download your code and execute it? At Adobe Campaign, since our code is in java we have two test artifacts:

A maven dependency library
An executable jar

Naturally, the artifact may be different for you depending on the language you are using.

Test SDK

Our libraries are available within Adobe for whoever wants to use them in their scenarios. To be able to create a test SDK you need to separate your code into two separate parts:

Product Manipulators
Scenarios

Product Manipulators

These are methods or APIs that either manipulate the product you are testing or gives you insight into the state of a product. Product manipulators need to be functional and should hide the inner workings of the product they manipulate such as the product API or the DOM.

Scenario

This is an assembly of a set of Product Manipulators.

Figure 1: Process Test System and System Under Test

What we propose is quite similar to the standard Page Object model. However, there are two main differences. First, the PageObject model is often (but not as a rule) technical whereas what we emphasize is the functional aspect of the Product manipulators. Second, the difference is that we generalize this approach to all types of tests and not just UI Tests.

In short, as long as you separate your scenarios from the manipulators you can easily create an SDK. At Adobe Campaign our Test SDK is a library made available through our artifactory.

In our environment, we have a series of projects that have adopted this design. We are now able to include their manipulator code in our scenarios.

Figure 2: Test Scenarios at Adobe Campaign

Data Independence

If you want to have your tests travel you need to provide proper data independence. This means that your design should allow your tests to run independently of what is stored in the test environment.

If your tests require certain prerequisite data in the system you are testing, it will be very hard for a third party to execute or even use them.

Execution Properties

Execution properties even in other test views is a topic that is rarely talked about. When your test is a product you need to pay special attention to them.

In our experience we have classified the execution properties in two main categories:

Dynamic Properties that change at most executions
Static Properties that usually remain the same between executions

The main difference is how the properties are stored.

Dynamic Properties

Dynamic properties tend to change between executions. Because of this, they need to be passed at execution time. We don’t usually store default values for these.

We also categorize secrets in this category, because they should never be stored in the project.

Static Properties

Static properties are execution values that could usually remain the same between executions.

No matter how you store the values, static properties come with certain requirements:

You should be able to override them at run time
These properties need to be included in the artifact so that your users do not need to set them manually
No secrets should be stored

Document your Tests

I cannot emphasize the importance of comments and documentation in your tests. Your manipulators need to be fully commented on and have legible code documentation, like Javadoc.

The same goes for the scenarios. Having proper test documentation, allows the users of your tests to really use your test product.

Other Requirements

Once your tests travel outside your work unit you need to make them maintainable. This is partially for putting on a good show, but also, like any good code, you want the users (The personas using your tests) to understand your code without needing to contact you all the time. For this, we set a few high-level guidelines

Enforce Coding Standards

Enforce proper coding standards in your projects. Treat the test code in the same standard as the code you are testing.

Test your Tests

There is nothing worse than errors in tests. Especially when the error is in code that a lot of tests use. It is often best to extract code that can be unit tested, and test them as such.

A lot of our internal shared libraries have been extracted, and are libraries in their own repositories that are thoroughly tested.

Implement a logging Strategy

Once your tests are used by others, you need to provide the user with the means to understand what your tests are doing.

Like the product you are testing you need to:

Ensure that the logs contain enough information for debugging problems.
Are legible by users outside your team.

Accepting Authentication

Your Test Product/SDK needs to accept authentication as a parameter in one way or the other. This not only allows you to test various user roles, but it also allows your users to execute their tests on environments and authentications of their own.

Conclusion & Challenges

The main take-ways we like you to have after finishing this posting are:

When selecting a framework, look beyond your own system. See what frameworks are used by other teams/ with which you will integrate with.
Your tests can and should be used by other teams. Make it easier for others to integrate and use your test code.
Treat your tests with love. Tests should have the same quality and expectations as the product they are testing.

Like most designs, there are, however, technological challenges that need to be better addressed, and which I for one would like to see solutions for.

Language Freedom

Many of these ideas work quite easily within single language libraries. If you include the test SDK of a different language than yours, it gets a bit more complicated to implement.

So far we have, with little effort, be able to run cross-product tests, using test libraries written in languages such as Kotlin and Scala, but once you move to other languages it could get a bit more tricky.

Change Management

Like all libraries we need to be able to inform our users about changes in our Test Product, and in the underlying tested application. This is an item, to which we do not currently have a satisfying solution.

Switching between UI Test Libraries

This is currently one of our biggest challenges. When using Selenium it is possible to reuse test libraries as long as they are in the same language family, but it is very hard to switch from a Webriver in Java to one in javascript. It gets even harder to switch from a Selenium-based system to a non-Selenium-based test framework.

Solving this would make this concept go much further, and allow for better collaboration between teams.

Follow the Adobe Experience Platform Community Blog for more developer stories and resources, and check out Adobe Developers on Twitter for the latest news and developer products. Sign up here for future Adobe Experience Platform Meetup.

Related Blogs

Automated Upgrade Testing: A Process to Test and Validate Software Upgrades, 3/11/2021
Transactional Coverage — a Functional Approach to Detecting Test Gaps, 12/10/2020
Problems with Code Coverage when Assessing Functional Tests, 12/3/2020

Originally Published: May 27, 2021