Data Ingestion ETL options for Encrypted | Community
Skip to main content
LizaFrancis
LizaFrancis
Level 2
May 22, 2025
Solved

Data Ingestion ETL options for Encrypted

  • May 22, 2025
  • 3 replies
  • 537 views

We have .pgp encrypted files being ingested from Data Landing Zone via API. We would like to reference a source field attribute and update an existing target attribute that does not exist in the schema (or source file). Is this possible via Data Prep? Or would this require a post ingestion ETL with something like Data Distiller?

This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.
Best answer by Asheesh_Pandey

@lizafrancis  Interesting question, as target field isn't in the schema or the source file, Data Prep can’t handle it alone. You’ll need a post-ingestion step using Data Distiller or another ETL to generate or populate that field based on your logic.

3 replies

Asheesh_Pandey
Community Advisor
Asheesh_PandeyCommunity AdvisorAccepted solution
Community Advisor
May 23, 2025

@lizafrancis  Interesting question, as target field isn't in the schema or the source file, Data Prep can’t handle it alone. You’ll need a post-ingestion step using Data Distiller or another ETL to generate or populate that field based on your logic.

- Asheesh
    AnkitJasani29
    AnkitJasani29
    Level 6
    May 23, 2025

    Hi @lizafrancis ,

    Data Prep is strictly a map‐and‐transform‐during‐ingestion step and only knows about the XDM attributes you've already defined in your schema.

    You can:

    • Pass through existing source fields
    • Create calculated fields
    • Map those to XDM attributes
    • Validate against XDM so only valid attributes get ingested

    Reference: https://experienceleague.adobe.com/en/docs/experience-platform/data-prep/home

     

    However, if the target attribute you need to update doesn’t exist in the schema (and thus isn’t in the source), Data Prep has no “upsert new XDM attribute” capability. To add or enrich with a brand-new field you have two options:

    1. Extend your XDM schema to include that attribute, then revise your Data Prep mapping.
    2. Use a post‐ingestion ETL - Data Distiller lets you run SQL‐based batch queries against your ingested data to create Derived Datasets with new or updated attributes and then write them back into the Data Lake. Reference: https://experienceleague.adobe.com/en/docs/experience-platform/query/data-distiller/overview

    Thanks

    Ankit

     

      LizaFrancis
      LizaFrancisAuthor
      Level 2
      May 23, 2025

      Thank you @asheesh_pandey and @ankitjasani29  I figured as such but it's nice to see the validation and instructions.   I appreciate your quick feedback. 

        Terms & ConditionsAccessibility statement

        Loading footer...