Can Data Distiller preserve attribute values across scheduled executions (stateful processing)? | Community
Skip to main content
R
RNA_Rohan
Level 2
July 24, 2026
Question

Can Data Distiller preserve attribute values across scheduled executions (stateful processing)?

  • July 24, 2026
  • 1 reply
  • 9 views

Hi everyone,

I'm working on a scheduled Data Distiller job that computes an eligibility flag for entities and writes the results to a Profile-enabled dataset.

One of the output attributes is a timestamp that should represent when the entity first became eligible.

The desired behavior is:

  • If an entity transitions from not eligible → eligible, set the timestamp to CURRENT_TIMESTAMP.
  • If the entity remains eligible in subsequent scheduled runs, preserve the original timestamp.
  • If the entity becomes ineligible, reset the timestamp (or set it to NULL).

Conceptually, the logic would be similar to:

CASEWHEN current.isEligible = TRUEAND (previous.isEligible = FALSE OR previous.isEligible IS NULL)
    THEN CURRENT_TIMESTAMPWHEN current.isEligible = TRUEAND previous.isEligible = TRUETHEN previous.EligibilityTimestamp

    ELSE NULL END

My questions are:

  1. Is it supported to read the previous state from a Profile-enabled dataset and write the updated results back to the same dataset within a scheduled Data Distiller job?
  2. If not, what is the recommended Adobe pattern for maintaining stateful attributes (such as a "first eligible timestamp") across scheduled executions?
  3. Has anyone implemented a similar state-preservation pattern using Data Distiller or Query Service?

Any guidance or best practices would be greatly appreciated.

Thanks!

 

1 reply

Devyendar
Devyendar
Level 7
July 24, 2026

​@RNA_Rohan 

 

Yes, but not as true mutable stateful processing.

Data Distiller / Query Service jobs are effectively stateless between runs unless you persist state yourself. Query Service is append-only for data manipulation: INSERT INTO is supported, but UPDATE / DELETE are not. So the pattern is to insert new records that represent the desired current state, rather than updating existing rows.

You can use the same Profile-enabled dataset as an append-only state/history table, as long as you are querying the Data Lake table behind that dataset.

The pattern would be:

Step 1: Read the latest prior row per entity from the dataset, using RANK / ROW_NUMBER to get the most recent appended version of the record.

Step 2: Compute the current eligibility status using your business logic.

Step 3: Insert a new row back into the same Profile-enabled dataset with the updated status and timestamp logic.

Important caveat: you need a reliable timestamp or sequence value to determine which row is the most recent. I would recommend writing your own run timestamp into the dataset, for example ingestionTimestamp.

Alternatively, you can use Query Service system metadata fields such as:

- _ACP_BATCHID
- _acp_system_metadata.acp_sourceBatchId
- _acp_system_metadata.ingestTime

But having a dedicated timestamp within the dataset gives more flexibility and makes the logic easier to reason about.

Also, to query system metadata fields, use:

SET drop_system_columns=false;

So overall, yes, this pattern can work, but it is append-only state preservation rather than mutable stateful processing.

Terms & ConditionsAccessibility statement

Loading footer...