How to handle historical data when combining multiple report suites into one global report suite?

[Tadtman]

If we are merging a whole bunch of report suites into one global instance (using virtual report suites to parse out different BUs/websites), what are my options in ensuring that historical data stays as intact as possible? During this re-implementation, we'll be cleaning up and combining report suites, focusing on creating 1 consistent implementation framework across many global properties.

We're aware that as we are re-imagining our report suite structure to a single suite, this data migration process would dovetail nicely with a transition to CJA. But until that happens, we plan on using the new Analytics suite as a data source sending to CJA (as opposed to a Web SDK/data streams direct-to-CJA data collection). That way, the analytics rework is useful and we won’t need to re-do anything on potential tool integration.

Are these my options for combing and retaining historical data?

1. Housing it with Adobe somehow (what costs are associated with this option)

2. Keeping that data in-house and manually stitching together.

[kayawalton · Accepted solution]

Hi @Tadtman! When we did this at my organization, we had an overlap where both the global and site-level report suites were collecting the same data. This allowed us to QC the virtual report suites with the site-level data and gave us some historical data in the global report suite. 

As for the historical data before the global report suite has been built, we pulled those into our data warehouse but only for a set number of time and for key KPIs. Our data decay is relatively quick being in the news business so we were not too concerned about having to go back other than for baseline creation. You might need to ask those kinds of questions about your organization in making the decision about whether to retain the data in Adobe or just keep a limited amount elsewhere. 

[Jennifer_Dungan]

This can also be tricky.. the same s object is used by both suites, and if not careful, it can corrupt data between the suites...

We tried that when we migrated one site into the existing Global... however, that was before there was a "wait then" option in the rules... (there was only "then" and depending on how long some things took to process, the objects were overwritten incorrectly before sending the suites...)

Though, it does sound like most of the dimensions will be the same, which could reduce corruption... but those variants could cause some problems

Unless you are just doing one to one data, you can send calls to multiple servers in the same call... but this really depends on the current state and the future state.

[kayawalton]

D'oh! Thanks for the reminder, @Jennifer_Dungan! So we actually completed a dimensions and metrics cleanup right before so everything aligned.  

[Jennifer_Dungan]

Yeah ^ Cleanups before a merge definitely help... it all depends on how different the suites are to start... If it means completely re-coding the sites, that likely won't happen

The big merge I did was a completely different profile.. so I had to see what was good and what was bad in the site that was moving to the global... make new dimensions to maintain stuff that we wanted to keep (and add it to the rest of the sites in the global suite), then had to teach people how to find the equivalent data (in the new format) in the global suite...

I've had other migrations where despite being different suites, the sites were on the same code base, so merging them was snap, cause all the dimensions and events lined up....

Migrations can be on either extreme, or anywhere in between

[Hemang35]

When merging multiple report suites into one global instance and ensuring historical data stays intact, you have a few options:

1. Data Warehouse: You can export historical data from each individual report suite using Adobe Analytics Data Warehouse. This will allow you to store the data in-house or in any data storage solution you prefer. You can then use this data for historical comparisons or to stitch together reports manually.

2. Virtual Report Suites: As you mentioned, you can use virtual report suites to parse out different BUs/websites. This will allow you to apply specific filters or segments to the global report suite without affecting the underlying data. This can help you preserve the historical context of your data while still allowing for a consistent implementation framework.

3. Data Migration using Adobe Customer Care: Adobe Customer Care can help you merge historical data from multiple report suites into one global instance. This is a paid service, and the cost will depend on the complexity of the migration and the amount of data involved. Contact Adobe Customer Care to discuss the specifics of your situation and obtain a quote for this service.

It is important to note that while these options can help you preserve historical data when merging report suites, they may not be perfect solutions. There might still be discrepancies in data when you combine report suites, especially if there were differences in tracking implementations or data collection methods.

As for your plan to use the new Analytics suite as a data source for Customer Journey Analytics (CJA), that is a viable approach. You can send the data from the global report suite to CJA, allowing you to leverage the reworked implementation framework and avoid redoing tool integration. Just keep in mind that when you transition to CJA, you may need to adjust your reporting and analysis processes to accommodate the different data model and capabilities provided by CJA.

[Jennifer_Dungan]

Your suites should have at least a 2 year retention policy... potentially more depending on your contract. The historical data in your existing suites will remain, and slowly truncate month by month as always.

You do not have to pay extra for those existing suites... Adobe is based on server calls, and if the calls move from suite a, b, c, etc over to the new suite, you should be using the same number of server calls.

While Adobe may be able to copy data from your existing suites over to the new suite, since you will likely have some changes (to support a global suite architecture) as well as changes based on your cleanup... also, not all your suites may be set up identically...  this would be a heavy processing ETL, and probably not worth the cost....

I've been through similar shifts, generally I just take the hit and create multiple panels in Workspace for anything that needs to see historical data in comparison with the current (until the current has fully taken over and can support all the YOY / MOM, etc comparison reports....)

You could also look at temp reports built using Excel Report Builder, where you take traffic before and after the global suite, and build out simple reports in one graph.... (note that Visits and Visitors could be inflated in the "before" if you are adding the suites together - you may have to have a disclaimer for that, since the same users could have visited multiple sites)

If you already have a process for Raw Data Feeds, you could look at using that to build some of the reports (to de-dup the users based on ECID, and potentially de-dup visits by creating logic around the time that the same users interacts with multiple sites)... but Raw Data Feeds are a big under taking to get the all the processing rules down... so if you don't already have that, then this may not be an efficient effort (unless you are planning to have Data Feeds anyway, and this work is already planned)

So basically, only you and your team can really make the final decision, after weighing the time, effort and costs associated to these paths... but I can say in my experience, we went simple... balancing between multiple suites until were fully into one global suite (using multiple panels in Workspace and Report Builder to create reports)