CL01 Frequent Performance Issues

[KatherineHa1]

We've been experiencing frequent outages and/or performance issues on CL01. 3 major incidents in the past 3 weeks. Not receiving WHY this is occurring from WF. Are the other clusters experiencing the same?

[EricLu2]

I’m on CL01 and my users are grumbling on a regular basis. I don’t know if the other clusters are experiencing trouble. Eric

[DagmaraGa]

We are CL01 and keep experiencing performance issues lately.

[SusanPf]

We are on CL01 and have experienced extreme delayed responses. I would recommend you call your Workfront Rep to discuss and put in a trouble ticket.

[KatherineHa1]

Thanks, we have put in tickets and called our ASE.

[SusanPf]

We just had a meeting with ours and that is what she recommended. It's important they know.

[KatherineHa1]

Also want to mention that for us, on CL01, last Thursday March 9 when my.workfront.com was down, the attask.com domain WAS working. Very Interesting.

[SusanPf]

Hmmm. Not good. We are planning the switch next week.

[EricLu2]

Hi: Katherine - let me share what happened last week (I think I am using the right words in the right order): 1) Many companies use Microsoft Azure to authenticate the user and allow them to get to the WorkFront URL; 2) To use this, WorkFront had Microsoft create an Azure Active Directory App called “AtTask” years ago; 3) Many companies use that Azure AD app to validate something and direct them to the appropriate URL; 4) Last week, as part of moving to the companyname.my.workfront.com URL, WorkFront asked Microsoft to create an Azure AD (WorkFront) app that points to the new URL; 5) I think Microsoft responded to that request Thursday afternoon; 6) Microsoft SHOULD have left the old Azure AD (AtTask) alone. It was in production and lots of people were using it; 7) What Microsoft did, to the amazement of many, was modify the existing Azure AD app (AtTask). I think, but have been unable to validate, Microsoft changed the name and URL and keys and whatever; 8) Thursday afternoon, I started getting people saying they couldn’t get into WorkFront; 9) I directed them to use the old URL, and they could get in - the old URL no longer went through the Azure AD App because of what Microsoft did; 10) As a result, we could not get to WorkFront Thursday afternoon and for a few hours Friday morning; 11) My Azure/Microsoft expert and I spent just under an hour with Conway at the help desk. We figured out how to fix it in the first thirty or so minutes, then Conway documented it. Conway did fabulous. He took it as a fun challenge and that takes the edge off the fact my corporation is off the air, WorkFront-wise; 12) Conway said, at the end of the call, that there were tickets coming in for this problem and so he was going to share how to fix the problem with the other techs. I gather there were other companies with the same problem; We had a Sev 1 outage of WorkFront for a number of hours Friday morning, because someone at Microsoft was cavalier with making changes to a production application. This may not be the cause of what you experienced on Thursday, but I’m wondering, you know? Thanks, Eric

[EricLu2]

Hi: Unnamed sources at WorkFront might have said: The performance degradation you saw today was caused by a customer's API call(s) with one of their custom integrations. We killed the calls, limited them to 5 concurrent connections, and are having some discussions, both internally and with the client, for more guardrails to prevent these kinds of things for those types of integrations. Our ops team is working very hard to ensure stability, and we've had a code freeze as well to mitigate any other chances of interruptions of any kind. If you see any problems, please report it to us because chances are if you're seeing it, then others must also be seeing it. I hope the WorkFront team working the performance issues is feeling the love for their honesty. Eric

[LauraRa1]

Hi Eric Not sure of all the techy details but we tried to modify our SSO setup last week (we use Azure) per WF instructions so it was ready for the new URL. It didn't work and we had to engage WF support. They managed to get us back to original state and told us there was an error on their end. To date we haven't heard from them..... We've had lots of slow down issues but only in the late afternoon for us (GMT) but we've always experienced somewhat of a slowdown about 3pm when we figured the USA gets online :)

[LouiseRe2]

We also have been victims of the continual performance issues over the last 3 weeks and this is taking up time that I don't have to monitor the situation, update users and deal with queries as a result.

[DavidCo3]

We're also on CL01 and have experienced the various slow-downs and outages over the past few weeks.

I report them each time they occur and on the whole, Workfront Support have been pretty good, but it sometimes takes a while to get through the basics and get them to accept that it is not a customer-end issue.

Speaking of which - does anyone else get "Internal Server Error - Read" messages? We get a couple of these per week. I've been told by Support that this is an Akamai-related error but that it is a customer infrastructure issue. This doesn't really make sense to me as it is an internal server error....and Akamai is used widely by other web-based systems and we don't get these errors with anything other than Workfront.

Regards, David

[EricLu2]

Hi: Are those the pink error messages at appear at the top, in the middle. We get them all the time, especially during these “slow downs”. Tech Support used to immediately say it was our network, I’d ask them for facts, they had none. I, on the other hand, had TRACERT dumps that show where the slow downs are occurring and surprise surprise it is with their servers. ☺

[JaimesonWe]

We've had the same issues, and same user feedback about reliability. Similar conversations with our ASE have been escalated. Certainly not ideal to have outages as we're getting more and more users into the rhythm of using Workfront daily.

[DavidCo3]

Hi Eric, We do get those 'eclipselink' red/pink error messages too, but these Read Errors are different...the whole screen goes white and you just have the title and error code at the top. Neither of the types of errors are very frequent fortunately (I am aware of a couple a week, on average), and I log these occurrences with Support, who generally forward them onto Tech Support for investigation of the underlying causes. Regards, David.