Simulation Mode Experience not working as intended
Adding this product enhancements request for a customer -
Background:
Journey testing has been a significant hurdle for our organization. We are unable to use dry run fully as we have API calls to manage our opt outs and dry run stops working when there is an API call. Test mode is very limited because we, as a product team, do not have access to build test profiles in AEP to match the journey strategy scenario - business rule decision. It requires one of our engineers to manually build them for us, and we need to put that request in far ahead and we are unable to adjust test profiles in the midst of a testing cycle. With this, we have limited access to test mode.
Simulation Experience and Requests:
When simulation mode came out, we were very excited as it "solved" all our issues. It is a valuable tool to streamline the testing process and also to ensure we are testing all potential scenarios. With test mode (prior to simulation mode), we would require 2-3 sprints of time (4-6 weeks) to have test profiles developed, and we were not able to pivot on the fly. With simulation mode, we are able to test faster and in more real time. Simulation mode also allowed us to see the reason "why" each test profile did what they did. The idea of simulation mode is extremely valuable; however, could use some enhancements to ensure the AI tool does not hallucinate (see support case details below).
-
MOST IMPORTANT: Allow for the client to describe what the journey "should do," what each node is supposed to do, or even what outcomes the AI tool should build a profile for. Without this feature, the AI tool created profiles that automatically passed all scenarios even when, in real life, they all failed. We should be able to produce a "testing prompt" to ensure we are testing what we want to have happened.
-
Provide more helplets to understand that if you choose not to have all test emails come to your email that they will go to an @example.comaddress. We received notes of concern from our Adobe Deliverability manager that we had a high rate of non-deliveries to the @example.comdomain which was coming from our simulation tests.
-
Also, there is a false understanding that these are temporary test profiles. Simulation did not create temporary profiles and were logged into our schema. We were able to query them and all the engagement data. This could cause an issue with schema size limits.
Overall, this tool is very important for the future of efficient and effective journeys and it will come in very handy as we become more sophisticated with our Decisioning features, but before we start using it again, we will need to be able to control the simulation more.
Support case detail
-
Case number: E-002454700
-
Details:
In testing our journeys in AJO we have started using the AI assisted simulation mode. We use this prior to launching tests live in production and primarily do this as dry run does not work for us with our API calls. Simulation mode does offer a great benefit of being able to develop, on the fly temporary test profiles, that match the needs of our journey. However, we have encountered a series of issues with the tool that is rendering it unusable for our organization. We need your assistance fixing these issues:
We recently ran out new journeys through simulation mode with success. Each test profile made it through the journey successfully and so we launched. After launching we had to revert and close all out journeys are they were not working. Here are the issues we encountered:
-
we tested all of our journeys which includes nodes tied to API calls and email validation checks. Each of these checks passed the simulation mode and then immediately failed in the live environment and cause all our profiles to exit the journey.
-
We received our weekly deliverability assessment and found that we had a large qty (~270) emails being delivered to @example.com. This is the email domain that the simulation tool sends to if you choose not to use your own email address. Our Ultimate Success team member said that simulation mode needs to be run in a testing environment with test IP addresses but that AJO does not have a dedicated testing environment.
-
Also, for our implementation we are not seeing these test profiles show up as temporary. We are able to query off for the data in AEP.
These three issues are indicating that simulation mode is not behaving as intended. We are asking to help to see why the new feature is not working.
