Customer Attributes: Backlog & Schema Mis-match (CSO #2018073005 & #2018082407)

[#1]

=================================

Update as of Wednesday, August 29, 2018

What is Customer Attributes?

Customer Attributes allows for customers to upload additional visitor information to the Experience Cloud, which is then made available in Adobe Target and Adobe Analytics.

How does it work?

Customer creates a “Data Source” within the Experience Cloud user interface (UI).  The Customer is then able to upload visitor information from a third-party system to this data source via FTP (up to 4GB .csv files) or directly in the UI (up to 100MB .csv files).  Within the Data Source, customers define which solutions will be able to consume this information.  Currently Target and Analytics are the only options.

When visitors go to the website and identify themselves (typically by logging in), the Experience Cloud ID is tied to the CustomerID that was uploaded in the Customer Attribute data file.  This is how the data is stitched to the Adobe solutions.

What are the problems?

• File Backlog (2018073005): Files uploaded were queued between July 30, 2018 and August 27, 2018.
• Schema Mis-match (2018082407): File upload attempts result in schema mis-match error beginning August 24, 2018.

Who is Affected?

• Backlog: Customers using the FTP method to upload data files to the Pacific Northwest and London Data Centers. If you look at the FTP configuration on your data source and look at the hostname.  If it is ftp5.adobe.net (Pacific Northwest - OR1) or ftp3.adobe.net (London – LON5), these files would be affected.
• Schema Mis-match: Any customers attempting to upload files via both the FTP and/or HTTP method post August 24, 2018.

What does this Mean?

• Backlog: Queue is currently being managed and there are no files backlogged.  Additional hardware coming online.
• Schema Mis-Match: Technical teams continue to investigate with top priority.  Impacted customers should subscribe to CSO # 2018082407 for ongoing email updates.  To subscribe, click here.

=============================

Update as of Friday, August 24, 2018

Which Files have been affected?

The backlog includes the files from the last two to three weeks. You can navigate to the Customer Attributes UI, and click on a data source.  If you scroll down below the graph, you should the files that have been uploaded for the last 30 days.  If you look at the status column, you will see “queued” or “failed”.  You may also see previously "queued" files move to the "failed" status.

Resolution

There are a few steps that need to happen to return back to full service with Customer Attributes:

1. Adobe TechOps/Engineering teams collaborating to increase throughput and processing resources for the Customer Attributes system
2. Adobe Engineering will remove the backlogged files over this coming weekend (August 25^th-26^th) from the processing queue, and push them to a “failed” state.
3. Starting Monday (August 27^th) morning at 7am Mountain Time or after, we ask that customers begin to upload their full customer data sets for each of their Customer Attribute Data Sources – Please see “Required Action: Re-upload Full Data File(s) on or After Mon, Aug 27” below for additional details
   • This will reduce backlog times from 2-3 weeks, down to a day or less and ensure that any data that may or may not have been processed is up to date.
4. After this full customer upload, customers may return the their regular file upload schedule and method – Please see “Best Practices” below for additional information that will help streamline processing below

Required Action: Re-upload Full Data File(s) on or After Mon, Aug 27

1. Determine which Data Source(s) have files that are in this backlog
2. Determine how far back to pull visitor data in your third-party system and generate the .csv file(s)
3. How large is the file?
   • Customer attributes has a file size limit for FTP uploaded files.  Documentation states 4GB is the maximum size.  We are asking that you batch your data files into 3GB files maximum for this exercise. (https://marketing.adobe.com/resources/help/en_US/mcloud/crs_data_file.html)
4. Upload this files following the Standard Process
   • Upload .csv file to root folder of FTP
   • Wait for it to completely upload to FTP (may take more than a few seconds, depending on size of file)
   • Once .csv has been uploaded, create .fin file.  This is an empty text file with a .fin extension (case-sensitive).  The name of the .fin file must match the name of the .csv file exactly (case-sensitive).
   • Upload .fin file.
     • NOTE – A brand new .fin file MUST be used. Using an existing .fin file and just renaming it will not work!
   • Repeat the process for each .csv file
   • Reference: https://marketing.adobe.com/resources/help/en_US/mcloud/crs_data_file.html

Best Practices

• Upload 3GB files (max is 4GB)
• upload .csv, wait until file upload is complete, then upload brand new .fin file
• Upload full set of customer information on Monday, then subsequent files should only include new/changed records.  We have seen customers upload many gigs of data per day, when there are only two to three thousand new/changed records.  That takes a long time to process, and occupies resources unnecessarily.

How Will this Issue be Mitigated in the future?

A cross-functional team has been working on this around the clock to bring the processing back in line and ensure this does not happen again.  These are the actions taking place:

• Additional hardware and processing capabilities are being added to handle increased volume and scale to future needs.
• Proactive alerts are being added on the back-end to notify Adobe Engineers of backlogged files, and resource consumption to better address additional load and spikes in incoming volume.
• The architecture of the Customer Attributes back-end is being reviewed to enable it to scale better in response to unanticipated load increases.

Please feel free to comment here with any additional questions.

[#2]

Are only file uploads via FTP affected? Or are Customer Attributes uploaded via web interface also experiencing latency?

[#3]

Hi Anthony,

Regarding the delay of processing files, that is limited to FTP files only.  However the schema mis-match issue was occurring with both FTP and UI uploaded files. Engineering just rolled out a release in the last day to resolve the schema mis-match issues.

[#4]

I had problems 09/03, Do you know somenthing abou it?

[#5]

test

[#6]

Update to the two current issues:

• Batch import process (queued files): We have made multiple changes to the system to mitigate the issue.  The current backlog is falling and we expect to be fully caught up early next week.  To do that we have had to purposely queue the small frequent uploads.  Any file that is less than 2 MB's and uploaded within 30 minutes of the last file are being queued currently. It is easier for Adobe to process less frequent but larger imports with the way the system is designed today.
• Visitor Matching process: This process is the piece that takes a Customer ID or Alias ID and matches it to an Experience Cloud Visitor ID. We are pushing out some code changes today and tomorrow that should significantly increase the throughput of the system. 

We will keep you updated as more information is available.  Thank you for your patience.

[#7]

Update 9/17:

• Batch processing: we are trending down in the right direction.  Timing is difficult to tell but we should by caught up by the following weekend (9/22-9/23). We are still queuing small files (less than 2 MB's and more frequent than once per 30 minutes).  There are some files that are older than 7 days.  We are going to mark those as failed as to not overwrite anything more recently updated.
• Visitor Matching process: We are adding 2 more nodes today which will significantly increase our throughput.  Once these are in place and the processing stabilizes we can give a better estimate until we're fully caught up.

Thanks again for your patience.

Seth

[#8]

Question for Users of Client attributes from someone who is suspecting synching issues:

- I Upload new attributes file

- status goes from loading, publishing, to complete

- This update changed client attribute "Reward Level" from "X" (as established in a prior file upload) to a new value of "Y" for a particular customer ID

- I use the "ID Lookup" and it, as hoped, shows the Reward Level of "Y"

Does that mean everything is complete? or is there further synching down the line of adobe architecture that needs happening? Is there further time to get that new value truly synched into other "production tables" so to speak? I'm seeing behavior in some that'd lead me to believe that - even though the ID Lookup shows an updated value, that doesn't mean that the final production source tables for adobe have that value replicated to it.

[#9]

Hi Kevinob,

That's a good question.  There is some time delay.  What you illustrated is that you updated the value in the Customer Attributes system.  From there, once the change is realized.  It then pushes the changes to the relevant subscriptions (think solutions).  It can take a few hours to push the data into Analytics and Target.  Once that push happens then the solutions should see the change reflected (Reward Level Y) etc.

I hope that helps,

Seth

[#10]

Got it. Extremely helpful. I've always assumed immediate update of downstream solutions once Customer Attributes said "complete"...and we've always experienced very quick practical 'realization' of attribute updates in Production experiences... except for this past week.

Is there a chance that's related to this 'backlog/synch' event? If so then it'd be pertinent to update the impact statements as I'm not using FTP (just the browser drag/drop function for fairly small csv files with a very small set of attributes).

Next question that might be dumb: We're referencing Client Attributes in a Target Audience. Is there anyway to surface/ confirm "what value did Target see in Client Attributes as it ran this test"? We're trying to confirm that a Target experience is wonky because the Client attribute must not yet be up to date with the most recently uploaded value (as seen in ID Lookup)