Customer Attributes: Backlog & Schema Mis-match (CSO #2018073005 & #2018082407)

Avatar

Avatar

cjensen-federat

Employee

Avatar

cjensen-federat

Employee

cjensen-federat
Employee

24-08-2018

=================================

Update as of Wednesday, August 29, 2018

What is Customer Attributes?

Customer Attributes allows for customers to upload additional visitor information to the Experience Cloud, which is then made available in Adobe Target and Adobe Analytics.

How does it work?

Customer creates a “Data Source” within the Experience Cloud user interface (UI).  The Customer is then able to upload visitor information from a third-party system to this data source via FTP (up to 4GB .csv files) or directly in the UI (up to 100MB .csv files).  Within the Data Source, customers define which solutions will be able to consume this information.  Currently Target and Analytics are the only options.

When visitors go to the website and identify themselves (typically by logging in), the Experience Cloud ID is tied to the CustomerID that was uploaded in the Customer Attribute data file.  This is how the data is stitched to the Adobe solutions.

What are the problems?

  • File Backlog (2018073005): Files uploaded were queued between July 30, 2018 and August 27, 2018.
  • Schema Mis-match (2018082407): File upload attempts result in schema mis-match error beginning August 24, 2018.

Who is Affected?

  • Backlog: Customers using the FTP method to upload data files to the Pacific Northwest and London Data Centers. If you look at the FTP configuration on your data source and look at the hostname.  If it is ftp5.adobe.net (Pacific Northwest - OR1) or ftp3.adobe.net (London – LON5), these files would be affected.
  • Schema Mis-match: Any customers attempting to upload files via both the FTP and/or HTTP method post August 24, 2018.

What does this Mean?

  • Backlog: Queue is currently being managed and there are no files backlogged.  Additional hardware coming online.
  • Schema Mis-Match: Technical teams continue to investigate with top priority.  Impacted customers should subscribe to CSO # 2018082407 for ongoing email updates.  To subscribe, click here.

=============================

Update as of Friday, August 24, 2018

Which Files have been affected?

The backlog includes the files from the last two to three weeks. You can navigate to the Customer Attributes UI, and click on a data source.  If you scroll down below the graph, you should the files that have been uploaded for the last 30 days.  If you look at the status column, you will see “queued” or “failed”.  You may also see previously "queued" files move to the "failed" status.

Resolution

There are a few steps that need to happen to return back to full service with Customer Attributes:

  1. Adobe TechOps/Engineering teams collaborating to increase throughput and processing resources for the Customer Attributes system
  2. Adobe Engineering will remove the backlogged files over this coming weekend (August 25th-26th) from the processing queue, and push them to a “failed” state.
  3. Starting Monday (August 27th) morning at 7am Mountain Time or after, we ask that customers begin to upload their full customer data sets for each of their Customer Attribute Data Sources – Please see Required Action: Re-upload Full Data File(s) on or After Mon, Aug 27 below for additional details
    • This will reduce backlog times from 2-3 weeks, down to a day or less and ensure that any data that may or may not have been processed is up to date.
  4. After this full customer upload, customers may return the their regular file upload schedule and method – Please see Best Practices below for additional information that will help streamline processing below

Required Action: Re-upload Full Data File(s) on or After Mon, Aug 27

  1. Determine which Data Source(s) have files that are in this backlog
  2. Determine how far back to pull visitor data in your third-party system and generate the .csv file(s)
  3. How large is the file?
  4. Upload this files following the Standard Process
    • Upload .csv file to root folder of FTP
    • Wait for it to completely upload to FTP (may take more than a few seconds, depending on size of file)
    • Once .csv has been uploaded, create .fin file.  This is an empty text file with a .fin extension (case-sensitive).  The name of the .fin file must match the name of the .csv file exactly (case-sensitive).
    • Upload .fin file.
      • NOTE – A brand new .fin file MUST be used. Using an existing .fin file and just renaming it will not work!
    • Repeat the process for each .csv file
    • Reference: https://marketing.adobe.com/resources/help/en_US/mcloud/crs_data_file.html

Best Practices

  • Upload 3GB files (max is 4GB)
  • upload .csv, wait until file upload is complete, then upload brand new .fin file
  • Upload full set of customer information on Monday, then subsequent files should only include new/changed records.  We have seen customers upload many gigs of data per day, when there are only two to three thousand new/changed records.  That takes a long time to process, and occupies resources unnecessarily.

How Will this Issue be Mitigated in the future?

A cross-functional team has been working on this around the clock to bring the processing back in line and ensure this does not happen again.  These are the actions taking place:

  • Additional hardware and processing capabilities are being added to handle increased volume and scale to future needs.
  • Proactive alerts are being added on the back-end to notify Adobe Engineers of backlogged files, and resource consumption to better address additional load and spikes in incoming volume.
  • The architecture of the Customer Attributes back-end is being reviewed to enable it to scale better in response to unanticipated load increases.

Please feel free to comment here with any additional questions.

Replies

Avatar

Avatar

settytb1

Employee

Avatar

settytb1

Employee

settytb1
Employee

21-09-2018

Hi Kevin,

Yes any import of any file would be impacted by this event (FTP or UI imports).  We are getting pretty close to being caught up either today or early this weekend by our estimates.

In Target, you can validate what values Target has for your current visitor profile via an Mbox trace.  In the trace you'll see the profile parameters in there and the customer attribute values will be included in that mbox trace.  I believe the parameters are like crs.(data source name or id).attribute something like that.  In the Adobe debugger it might be in the mbox parameters or you may have to do a full mbox trace to get that level of detail.

I hope that helps,

Seth

P.S. I've been working with our agent on that ticket as well.

Avatar

Avatar

settytb1

Employee

Avatar

settytb1

Employee

settytb1
Employee

21-09-2018

Update 9/21:

Hi folks we are caught up!  Our recommendation would be to stagger any imports that you have outstanding.  Meaning don’t upload all of them the same moment Monday morning.  Also, smaller number of large files are preferred (ex. combine daily uploads into one weekly import etc.) We will continue to monitor the health of the system closely over the next few weeks to make sure all the changes that have been made continue to work as expected. We will keep this thread updated if anything arises.  Thank you for your patience with us.

Avatar

Avatar

settytb1

Employee

Avatar

settytb1

Employee

settytb1
Employee

28-09-2018

Hi all, final update.  The Customer Facing statement has been published via the CSO.  If you need a copy please reach out to your CSM or Customer Care.

Thanks,

Seth