Rapid re-purposing of dimensions: Alternative to deleting data?

[#1]

Issue

Deleting data is incredibly problematic (and expensive!) in Adobe analytics.

One of the impacts is on our agility - we have several dimensions that are tied up as they contain historic data that is no longer relevant and we are waiting for that data to disappear through the standard data retention deletion process before we reuse - This is something we really struggle with.

Requirements

The requirement is to be able to quickly re-purpose variables/dimensions, thus increasing the flexibility/agility of the Adobe Analytics tool.

For this scenario the requirement is to prevent end users seeing the historical data (rather than deleting the data).  Therefore I was wondering if there was an alternative approach which would allow us to free up dimensions quickly for new purposes.

Possible solution?

The problem with deleting data seems to be unpicking all the processing that has happened.  My thinking is this unpicking could be avoided by having virtual variables (much like a "virtual track" on a digital multi-track recorder)

Each prop/eVar would have say 3 "virtual" versions.  The admin console would allow admins to switch between which virtual version of a prop/eVar is in use.

This would mean only one virtual version of a prop/eVar would be available at one time to record new data into and retrieve data from.

At a time when the original purpose for a prop or eVar is no longer relevant the admin could simply cut across to a new/fresh version of the prop/eVar - allowing immediate re-purposing.

This would be different to deleting data as all virtual versions would continue to exist in the background but only one available at any one time (i.e. no data is actually deleted)

From a storage/infrastructure perspective it should also be different to just giving us loads more props/eVars as only one virtual variable could be recording new data at any one time (so storage requirement should be less)

NOTE: this sort of already exists for eVars in the way that switching between different attribution types (linear/first touch/last touch) cuts across to a different instance/version of the eVar

[#2]

Hi Andy,

This is a very interesting proposal.

One challenge that would need to be overcome is the fact that dimension (props/evars) references are persisted in Reports & analytics bookmark definitions, Analysis Workspace projects, ReportBuilder workbooks, Alerts, and even coded into API requests made by customers. In the Analytics engineering world we call anything that contains a persisted reference to a dimension or metric an "asset".

If the Admin user switched the virtual version that was active on a given dimension, would it be reasonable for all the assets that reference that dimension to start displaying the new data rather than the old? What if the new data doesn't make any sense in the context of the asset? One simple example is alerts. If you have an hourly alert configured for, say, low values of a metric that is filtered by the value of a given dimension (e.g. alert on low page views for the "myAwesomeProduct" page) and we switch what values are actually being reported on in that dimension, the recipient of the alert is likely to get a false alert every hour.

A heavy-handed approach would be to disable all assets that used an inactive virtual version of a dimension but there are complications with doing this (e.g. dimension references are in ReportBuilder workbooks which are stored on customer computers which Analytics can't access, stored in customer API requests that are hardcoded, etc.) and I'm not sure the experience would be much better. For example, an executive is used to receiving a daily dashboard and all of a sudden he stops getting it because it used a virtual dimension version that is now inactive.

I'm not bringing these examples up as a way of saying, "it's too hard". The Analytics product and engineering teams can certainly work on solutions for these complications. I'm bringing these examples up as a way of furthering the discussion and soliciting additional input and feedback from users on what they would think are acceptable solutions if we implemented a feature like this. I would love to hear your feedback on what you think the best experience for your users would be.

Thanks!

[#3]

Hi Brian, thanks for the response and some interesting questions!

"Would it be reasonable for all the assets that reference that dimension to start displaying the new data rather than the old?"

I would be absolutely OK with all the assets that reference that dimension displaying the new data rather than the old - in fact this is the behaviour I would have expected.  (I believe this is how an eVar attribution change works today)

"What if the new data doesn't make any sense in the context of the asset?"

In the way I was envisaging this working this would be a non-issue.  If I was looking to re-purpose a dimension, as the administrator of the tool I would be making sure the dimension was not being used before I made the switch (including auditing usage, comms with stakeholders etc). 

…which neatly brings me on to another area which I think you could do with enhancing.

The tool could really benefit with the audit capabilities being significantly strengthen - it's far too difficult to find out what dimensions are being used where and by who.

However, that's probably another idea for another day :-)