Data Feed re-triggering automation | Community
Skip to main content
k6ivupuu_k
k6ivupuu_k
Level 2
May 2, 2024
Solved

Data Feed re-triggering automation

  • May 2, 2024
  • 1 reply
  • 967 views

Is there any way for me to programatically retrigger Adobe Data Feed, when there is a downstream failure?

 

E.g.

 

1. Data feed has been set up succesfully, it runs daily (and 99% of times it runs just fine)

2. Data feed zip file is received successfully over FTP

3. Something goes sour downstream in zip processing.

4. Now I want an automatic message to Adobe (something something involving API?) to retrigger the datafeed for this particular day to resend the zip file (because for whatever the reason, I know it is silly but because of downstream dependencies I need the data feed zip to "arrive again in the correct folder")

 

Thanks in advance!

 

 

This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.
Best answer by Jennifer_Dungan

I haven't heard any any sort of automated re-send process (though that would be a really nice feature). 

 

Unfortunately, I think you will have to manually create a feed file for the one that ended up failing, and choose to send it to the same folder... if you don't have the authentication keys for that location, someone may have to manually get the file and upload it... 

 

Just curious, how often are you getting failures? I don't think that this is something that happens often for most people, so that's likely why no such feature exists... but I understand... while a different type of job, I would love the ability to re-send scheduled projects due issues; rather than having to spend time manually running it... 

 

But if you feel strongly about this, please log an idea with details about how you would like it to work at https://experienceleaguecommunities.adobe.com/t5/adobe-analytics-ideas/idb-p/adobe-analytics-ideas

 

1 reply

Jennifer_Dungan
Community Advisor and Adobe Champion
Jennifer_DunganCommunity Advisor and Adobe ChampionAccepted solution
Community Advisor and Adobe Champion
May 2, 2024

I haven't heard any any sort of automated re-send process (though that would be a really nice feature). 

 

Unfortunately, I think you will have to manually create a feed file for the one that ended up failing, and choose to send it to the same folder... if you don't have the authentication keys for that location, someone may have to manually get the file and upload it... 

 

Just curious, how often are you getting failures? I don't think that this is something that happens often for most people, so that's likely why no such feature exists... but I understand... while a different type of job, I would love the ability to re-send scheduled projects due issues; rather than having to spend time manually running it... 

 

But if you feel strongly about this, please log an idea with details about how you would like it to work at https://experienceleaguecommunities.adobe.com/t5/adobe-analytics-ideas/idb-p/adobe-analytics-ideas

 

    k6ivupuu_k
    k6ivupuu_kAuthor
    Level 2
    May 5, 2024

    This is pretty much what I expected. I'll add some background if anybody is interested. Because I was a bit unclear with the "downstream part". Pure corporate gold though 🙂

     

    In our implementation the datafeed feeds the daily zip to a folder and that triggers downstream processing (obv a separate tool to allow for incoming files). So basically if a zip file arrives, then this will trigger an automation to process, transform and feed the data in the zip file into a data warehouse.

     

    So now, if this processing and/or data load to DW fails, for whatever the reason, then we have the ZIP file (perfectly sent through datafeed), but (allegedly by the team dealing with data loads) there are no ways to retrigger the zip processing "just out of thin air", i.e. whitout re-triggering the datafeed.

     

    Then, after getting the new datafeed ZIP file, the processing rules for the DW will clean up any bad data from DW for the day and the new datafeed data is going to be reprocessed.

     

    Now it is just the time it takes for me to be notified, then I retrigger datafeed and this means this data can not be used in an agile way for calculations. The two clicks to resend are in essence absolutely not a problem.

     

    Sidenote. What I also discovered - offline app data can have huge impact on datafeed. We also wondered why the data in the feed is so different when resending and that pretty much was the case.

     

     

    Jennifer_Dungan
    Community Advisor and Adobe Champion
    Jennifer_DunganCommunity Advisor and Adobe Champion
    Community Advisor and Adobe Champion
    May 5, 2024

    Thank you for sharing. 

     

    I am not sure why there is no way to re-trigger your job in your process.. it seems like some error handling should be able to do that (even if on failure, the file is copied to another folder, and new files dropped there trigger the re-process), but I am no expert on that...

     

    Ah yes, offline data... if you have imported data, or just data that processes from offline apps when users come back online, that definitely becomes troublesome for data feeds.

     

    Adobe also has something called "livestream" (which used to be called "Firehose").. I am not sure if it contains all delta information or not... our team investigated it a few years ago, but ultimately dropped it... so I don't know all the capabilities or current costs.


    Since we don't have offline data, we aren't at a high risk of losing data, but if this can support sending delta updates as they are received, this might be something to check out?

     

    https://developer.adobe.com/analytics-apis/docs/2.0/guides/endpoints/livestream/ 

    Terms & ConditionsAccessibility statement

    Loading footer...