Do Processing Rules effect Data Warehouse?

[jkm-disco]

Do Processing Rules effect Data Warehouse? I once had a processing rule which filtered out all numeric digits from all clicks.  We created this rule to avoid collecting account numbers (PII).  This processing rule worked as designed as no numeric digits appeared in UI data.  I was horrified to find account numbers in Data Warehouse as Data Warehouse ignored my processing rules.

[jkm-disco]

Hi @erichami , how did you wind up filtering the numbers? Did you just create a "Does Not Contain" condition for every number 0-9? Just curious, but not sure about the actual answer. It is processed separately then the Analytics Reports/API so it seems plausible that it could have data that was filtered through normal processing.

You could try to filter before it reaches processing (e.g. custom code in your tag management -- even if the variable is set directly, you could copy it to a separate context data variable, use custom code to filter, then be sure to clear the analytics variable before sending the beacon, and, finally, set the processing rules to set the variable from the new, pre-filtered context data variable).

[erichami]

This was years ago, but the rule blocked any text that "contained" 0 or 1 or 2 or 3 or 4 or 5 or 6 or 7 or 8 or 9. @jkm-disco

[jkm-disco]

Thanks, @erichami . Yea, curious about this myself to know how normal processing impacts Data Warehouse.

[#5]

Processing Rules are applied before Data Warehouse extracts. It's likely something was overlooked. Could you please check or run tests again?

[#6]

Now that I think about it, Data Warehouse must be available only after processing, since it includes values that are only set during processing (e.g. context data). It seems most likely that the conditional rule is failing. You may also want to do a Data Feed export on the variable(s) in question to compare the pre/post variable values to see how they were processed and if things are making it past the filtering processing rule.