Hi Vinay,
To answer your questions:
1 Does anyone have any idea why this happened?
A: 504 errors are network timeout errors that occur when something is taking too long or just not working right. There is extensive online resources if you want to deep dive into various causes of 504s.
2 As this workflow was running perfectly fine till now, what might have caused this to fail after so many days?
A: Given that the workflow was functional and there does not appear to have been any changes made based on your description, it is likely an issue at the network level and not actually with Campaign.
3 Can this be once in a lifetime failure?
A: Could be but unlikely. Networks can experience issues for numerous reasons including breakdowns in hardware, and power failures. So as long as these errors are very few and far between, I would not stress too much about them.
4 What are the required steps to fix this?
A: Unfortunately there is not much you can do at this point outside of enlisting your network administrators to review the server/network logs at the time to see if they notice anything odd. Typically, if Campaign is being asked to analyze these issues we need to enable network logging using tools, such as Wireshark and Fiddler to monitor and capture a TCPDump when the issue occurs. Only then can you really isolate the root cause.
5 Should I wait till tomorrow and check this again?
A: That is certainly reasonable if the workflow has been stable all this time. I would also suggest adding workflow alerting on this workflow by adding yourself as a supervisor of the workflow. That way you can be notified if it should fail.
Documentation on adding yourself as a supervisor:
https://experienceleague.adobe.com/docs/campaign-classic/using/automating-with-workflows/monitoring-workflows/monitoring-workflow-execution.html?lang=en
Regards,
Craig