Expand my Community achievements bar.

SOLVED

Do Processing Rules effect Data Warehouse?

Avatar

Level 3

Do Processing Rules effect Data Warehouse? I once had a processing rule which filtered out all numeric digits from all clicks.  We created this rule to avoid collecting account numbers (PII).  This processing rule worked as designed as no numeric digits appeared in UI data.  I was horrified to find account numbers in Data Warehouse as Data Warehouse ignored my processing rules.

Topics

Topics help categorize Community content and increase your ability to discover relevant content.

1 Accepted Solution

Avatar

Correct answer by
Employee Advisor

Processing Rules are applied before Data Warehouse extracts. It's likely something was overlooked. Could you please check or run tests again?

View solution in original post

6 Replies

Avatar

Level 6

Hi @erichami , how did you wind up filtering the numbers? Did you just create a "Does Not Contain" condition for every number 0-9? Just curious, but not sure about the actual answer. It is processed separately then the Analytics Reports/API so it seems plausible that it could have data that was filtered through normal processing.

You could try to filter before it reaches processing (e.g. custom code in your tag management -- even if the variable is set directly, you could copy it to a separate context data variable, use custom code to filter, then be sure to clear the analytics variable before sending the beacon, and, finally, set the processing rules to set the variable from the new, pre-filtered context data variable).

Avatar

Level 3
This was years ago, but the rule blocked any text that "contained" 0 or 1 or 2 or 3 or 4 or 5 or 6 or 7 or 8 or 9. @Jacob-DDdev

Avatar

Level 6
Thanks, @erichami . Yea, curious about this myself to know how normal processing impacts Data Warehouse.

Avatar

Level 6

Now that I think about it, Data Warehouse must be available only after processing, since it includes values that are only set during processing (e.g. context data). It seems most likely that the conditional rule is failing. You may also want to do a Data Feed export on the variable(s) in question to compare the pre/post variable values to see how they were processed and if things are making it past the filtering processing rule.

Avatar

Correct answer by
Employee Advisor

Processing Rules are applied before Data Warehouse extracts. It's likely something was overlooked. Could you please check or run tests again?

Avatar

Level 10
Do any of the answers below answer your initial question? If so, can you select one of them as the correct answer? If none of the answers already provided answer your question, can you provide additional information to better help the community solve your question?