Expand my Community achievements bar.

Customer Attributes: Backlog & Schema Mis-match (CSO #2018073005 & #2018082407)

Avatar

Employee

=================================

Update as of Wednesday, August 29, 2018

What is Customer Attributes?

Customer Attributes allows for customers to upload additional visitor information to the Experience Cloud, which is then made available in Adobe Target and Adobe Analytics.

How does it work?

Customer creates a “Data Source” within the Experience Cloud user interface (UI).  The Customer is then able to upload visitor information from a third-party system to this data source via FTP (up to 4GB .csv files) or directly in the UI (up to 100MB .csv files).  Within the Data Source, customers define which solutions will be able to consume this information.  Currently Target and Analytics are the only options.

When visitors go to the website and identify themselves (typically by logging in), the Experience Cloud ID is tied to the CustomerID that was uploaded in the Customer Attribute data file.  This is how the data is stitched to the Adobe solutions.

What are the problems?

  • File Backlog (2018073005): Files uploaded were queued between July 30, 2018 and August 27, 2018.
  • Schema Mis-match (2018082407): File upload attempts result in schema mis-match error beginning August 24, 2018.

Who is Affected?

  • Backlog: Customers using the FTP method to upload data files to the Pacific Northwest and London Data Centers. If you look at the FTP configuration on your data source and look at the hostname.  If it is ftp5.adobe.net (Pacific Northwest - OR1) or ftp3.adobe.net (London – LON5), these files would be affected.
  • Schema Mis-match: Any customers attempting to upload files via both the FTP and/or HTTP method post August 24, 2018.

What does this Mean?

  • Backlog: Queue is currently being managed and there are no files backlogged.  Additional hardware coming online.
  • Schema Mis-Match: Technical teams continue to investigate with top priority.  Impacted customers should subscribe to CSO # 2018082407 for ongoing email updates.  To subscribe, click here.

=============================

Update as of Friday, August 24, 2018

Which Files have been affected?

The backlog includes the files from the last two to three weeks. You can navigate to the Customer Attributes UI, and click on a data source.  If you scroll down below the graph, you should the files that have been uploaded for the last 30 days.  If you look at the status column, you will see “queued” or “failed”.  You may also see previously "queued" files move to the "failed" status.

Resolution

There are a few steps that need to happen to return back to full service with Customer Attributes:

  1. Adobe TechOps/Engineering teams collaborating to increase throughput and processing resources for the Customer Attributes system
  2. Adobe Engineering will remove the backlogged files over this coming weekend (August 25th-26th) from the processing queue, and push them to a “failed” state.
  3. Starting Monday (August 27th) morning at 7am Mountain Time or after, we ask that customers begin to upload their full customer data sets for each of their Customer Attribute Data Sources – Please see Required Action: Re-upload Full Data File(s) on or After Mon, Aug 27 below for additional details
    • This will reduce backlog times from 2-3 weeks, down to a day or less and ensure that any data that may or may not have been processed is up to date.
  4. After this full customer upload, customers may return the their regular file upload schedule and method – Please see Best Practices below for additional information that will help streamline processing below

Required Action: Re-upload Full Data File(s) on or After Mon, Aug 27

  1. Determine which Data Source(s) have files that are in this backlog
  2. Determine how far back to pull visitor data in your third-party system and generate the .csv file(s)
  3. How large is the file?
  4. Upload this files following the Standard Process
    • Upload .csv file to root folder of FTP
    • Wait for it to completely upload to FTP (may take more than a few seconds, depending on size of file)
    • Once .csv has been uploaded, create .fin file.  This is an empty text file with a .fin extension (case-sensitive).  The name of the .fin file must match the name of the .csv file exactly (case-sensitive).
    • Upload .fin file.
      • NOTE – A brand new .fin file MUST be used. Using an existing .fin file and just renaming it will not work!
    • Repeat the process for each .csv file
    • Reference: https://marketing.adobe.com/resources/help/en_US/mcloud/crs_data_file.html

Best Practices

  • Upload 3GB files (max is 4GB)
  • upload .csv, wait until file upload is complete, then upload brand new .fin file
  • Upload full set of customer information on Monday, then subsequent files should only include new/changed records.  We have seen customers upload many gigs of data per day, when there are only two to three thousand new/changed records.  That takes a long time to process, and occupies resources unnecessarily.

How Will this Issue be Mitigated in the future?

A cross-functional team has been working on this around the clock to bring the processing back in line and ensure this does not happen again.  These are the actions taking place:

  • Additional hardware and processing capabilities are being added to handle increased volume and scale to future needs.
  • Proactive alerts are being added on the back-end to notify Adobe Engineers of backlogged files, and resource consumption to better address additional load and spikes in incoming volume.
  • The architecture of the Customer Attributes back-end is being reviewed to enable it to scale better in response to unanticipated load increases.

Please feel free to comment here with any additional questions.

Topics

Topics help categorize Community content and increase your ability to discover relevant content.

12 Replies

Avatar

Level 1

Are only file uploads via FTP affected? Or are Customer Attributes uploaded via web interface also experiencing latency?

Avatar

Employee

Hi Anthony,

Regarding the delay of processing files, that is limited to FTP files only.  However the schema mis-match issue was occurring with both FTP and UI uploaded files. Engineering just rolled out a release in the last day to resolve the schema mis-match issues.

Avatar

Level 2

I had problems 09/03, Do you know somenthing abou it?

Avatar

Employee

Update to the two current issues:

  • Batch import process (queued files): We have made multiple changes to the system to mitigate the issue.  The current backlog is falling and we expect to be fully caught up early next week.  To do that we have had to purposely queue the small frequent uploads.  Any file that is less than 2 MB's and uploaded within 30 minutes of the last file are being queued currently. It is easier for Adobe to process less frequent but larger imports with the way the system is designed today.
  • Visitor Matching process: This process is the piece that takes a Customer ID or Alias ID and matches it to an Experience Cloud Visitor ID. We are pushing out some code changes today and tomorrow that should significantly increase the throughput of the system. 

We will keep you updated as more information is available.  Thank you for your patience.

Avatar

Employee

Update 9/17:

  • Batch processing: we are trending down in the right direction.  Timing is difficult to tell but we should by caught up by the following weekend (9/22-9/23). We are still queuing small files (less than 2 MB's and more frequent than once per 30 minutes).  There are some files that are older than 7 days.  We are going to mark those as failed as to not overwrite anything more recently updated.
  • Visitor Matching process: We are adding 2 more nodes today which will significantly increase our throughput.  Once these are in place and the processing stabilizes we can give a better estimate until we're fully caught up.

Thanks again for your patience.

Seth

Avatar

Level 2

Question for Users of Client attributes from someone who is suspecting synching issues:

- I Upload new attributes file

- status goes from loading, publishing, to complete

- This update changed client attribute "Reward Level" from "X" (as established in a prior file upload) to a new value of "Y" for a particular customer ID

- I use the "ID Lookup" and it, as hoped, shows the Reward Level of "Y"

Does that mean everything is complete? or is there further synching down the line of adobe architecture that needs happening? Is there further time to get that new value truly synched into other "production tables" so to speak? I'm seeing behavior in some that'd lead me to believe that - even though the ID Lookup shows an updated value, that doesn't mean that the final production source tables for adobe have that value replicated to it.

Avatar

Employee

Hi Kevinob,

That's a good question.  There is some time delay.  What you illustrated is that you updated the value in the Customer Attributes system.  From there, once the change is realized.  It then pushes the changes to the relevant subscriptions (think solutions).  It can take a few hours to push the data into Analytics and Target.  Once that push happens then the solutions should see the change reflected (Reward Level Y) etc.

I hope that helps,

Seth

Avatar

Level 2

Got it. Extremely helpful. I've always assumed immediate update of downstream solutions once Customer Attributes said "complete"...and we've always experienced very quick practical 'realization' of attribute updates in Production experiences... except for this past week.

Is there a chance that's related to this 'backlog/synch' event? If so then it'd be pertinent to update the impact statements as I'm not using FTP (just the browser drag/drop function for fairly small csv files with a very small set of attributes).

Next question that might be dumb: We're referencing Client Attributes in a Target Audience. Is there anyway to surface/ confirm "what value did Target see in Client Attributes as it ran this test"? We're trying to confirm that a Target experience is wonky because the Client attribute must not yet be up to date with the most recently uploaded value (as seen in ID Lookup)

Avatar

Employee

Hi Kevin,

Yes any import of any file would be impacted by this event (FTP or UI imports).  We are getting pretty close to being caught up either today or early this weekend by our estimates.

In Target, you can validate what values Target has for your current visitor profile via an Mbox trace.  In the trace you'll see the profile parameters in there and the customer attribute values will be included in that mbox trace.  I believe the parameters are like crs.(data source name or id).attribute something like that.  In the Adobe debugger it might be in the mbox parameters or you may have to do a full mbox trace to get that level of detail.

I hope that helps,

Seth

P.S. I've been working with our agent on that ticket as well.

Avatar

Employee

Update 9/21:

Hi folks we are caught up!  Our recommendation would be to stagger any imports that you have outstanding.  Meaning don’t upload all of them the same moment Monday morning.  Also, smaller number of large files are preferred (ex. combine daily uploads into one weekly import etc.) We will continue to monitor the health of the system closely over the next few weeks to make sure all the changes that have been made continue to work as expected. We will keep this thread updated if anything arises.  Thank you for your patience with us.

Avatar

Employee

Hi all, final update.  The Customer Facing statement has been published via the CSO.  If you need a copy please reach out to your CSM or Customer Care.

Thanks,

Seth