Seeing negative values for Integer datatype in AEP CDP | Community
Skip to main content
A
AEPuser16
Level 2
May 20, 2025
Solved

Seeing negative values for Integer datatype in AEP CDP

  • May 20, 2025
  • 2 replies
  • 983 views

Hi,

 

I have a schema in prod and there is an ID field in it(like CRMID for example). The source datatype given to us in INT64 and I have used 'integer' datatype in AEP. After doing data ingestion, I see few negative values and few positive values and all values look incorrect compared to source values for this ID. Example 723456789123 is the value in source file and I'm seeing -1234567890 and 1234567890 values in AEP after data ingestion.The mapping is done correctly and I don't see any errors in my dataflow. What could be the reason for this? Any suggestions how to proceed with this will be helpful. 

 

Also, for our use case, we have not enabled Schema and Dataset for profile. So, incase if I update datatype, what are the next steps I can take, to get the old incorrect data also corrected? Or first to just test this process, how can I copy/ duplicate schema. 

 

Regards,

Ramyasri 

    Best answer by TylerKrause

    That sounds like it may be experiencing an overflow issue when it's converting it into the 32 bit integer type, would it be possible to adjust the Schema to use a Long or a String instead? Both would be a better match for a CRMID. You will need to do a backfill of this data into the new dataset, as the data you digested is now incorrect.

    If you've already ingested data into the dataset/schema, this would be considered a breaking change and you'll need to remake the Schema to update the data type and subsequently create a new dataset as well.

    Let me know if you have any other questions!
    Tyler Krause

    2 replies

    TylerKrause
    Adobe Champion
    TylerKrauseAdobe ChampionAccepted solution
    Adobe Champion
    May 20, 2025

    That sounds like it may be experiencing an overflow issue when it's converting it into the 32 bit integer type, would it be possible to adjust the Schema to use a Long or a String instead? Both would be a better match for a CRMID. You will need to do a backfill of this data into the new dataset, as the data you digested is now incorrect.

    If you've already ingested data into the dataset/schema, this would be considered a breaking change and you'll need to remake the Schema to update the data type and subsequently create a new dataset as well.

    Let me know if you have any other questions!
    Tyler Krause

    AnkitJasani29
    AnkitJasani29
    Level 6
    May 21, 2025

    Hi @aepuser16 ,

    723456789123 - far exceeds this limit, leading to data corruption or overflow, hence the negative/incorrect values you are seeing.

    In your AEP schema, you used the 'integer' data type, which maps to a 32-bit signed integer in many systems, including Adobe's internal mapping logic.

    Reference:

    https://experienceleague.adobe.com/en/docs/experience-platform/xdm/schema/field-constraints

     

    Go to Schemas > Your Schema > Edit

    Locate the field and change the type from integer to long

    Save the schema

     

    Go to Sources > Your Dataflow > Edit Mapping

    Make sure field is mapped as int64 (long) in AEP

    Save and re-enable the dataflow

     

    If you want to correct old data then re-ingest corrected values using same keys or create new dataset

    Thanks

    Ankit

    A
    AEPuser16Author
    Level 2
    May 21, 2025

    Thank you @ankitjasani29  and @tylerkrause  for your answers!

    From both the responses I understand that I need to change the datatype from Integer to either long or string datatype in the schema.If schema is not that complex I can change the datatype in UI directly correct?

    I have a dataflow setup via API. We have source data files stored in Azure Storage Explorer. These are incremental data files, a new file is received every day, and each file is retained in Azure Blob Storage for 7 days before it gets deleted. 

    So, In this case, what I understood is,

    • Disable and delete the existing dataset which has incorrect data and create a new dataset with updated schema.
    • Create a new incremental dataflow via API which points to a new dataset and updated schema.
    • Re-ingest the deleted files/older data again if needed, to have correct data.
    • Since this via API, I don't need to change any datatype in the mapping set correct? Please see attached screenshot.

     

    Let me know if my understanding or process that I'm going to follow is correct or needs any changes.

     

    Thank you!

      TylerKrause
      Adobe Champion
      TylerKrauseAdobe Champion
      Adobe Champion
      May 21, 2025

      If you've already loaded data to the dataset built off the schema you won't be able to edit the existing schema via the UI. You can make a new one via the UI or follow the steps I outlined to create the new one and append the new field within the schema.

      All your other bullets are correct, but make sure your new mapping set is consistent with your new schema (updating the value of int into long/string).

      Tyler Krause

      Terms & ConditionsAccessibility statement

      Loading footer...