archive playback suddenly not working

[adam_k]

i've been working all day and playing back the same archives successfully.  suddenly, they're not loading and i'm not getting any kind of error messages.  the archives do exist on my webdav server, new archives are being stored there successfully. i just can't play them back now, and i have no information on why they're not playing back. 

the trace (with my account details redacted) is as follows.  i can send a private message to an adobe employee with my account details if tha'ts required to look into this.

first i need to find out why this is happening, then i need to know if there's a way an error message can be reported back to my app.

one thing i did do was modify a node through the LCCS Air app.  it was a simple custom node where i changed persistItems from true to false.

19:27:29 GMT-0500    #TicketService# ticket received: 8bnh0sd8nx3k

19:27:29 GMT-0500    Getting FMS at https://na2.collaboration.adobelivecycle.com/fms?ticket=8bnh0sd8nx3k&playback=adam_roxanne_3&proto=r..., attempt #1/3

19:27:29 GMT-0500    result: <fms>

  <origin>fms7.acrobat.com</origin>

  <proto_ports>rtmfp:1935,rtmps:443</proto_ports>

  <retry_attempts>2</retry_attempts>

</fms>

19:27:29 GMT-0500    protocols: [object ProtocolPortPair],[object ProtocolPortPair]

19:27:29 GMT-0500    [attempt 1 of 2] Connecting to 0/1: rtmfp://fms7.acrobat.com/playback/na2-sdk-3073a7e4-fa08-4353-a662-e05f0e45cf15/[REDACTED]/[REDACTED] #startProtosConnect#

19:27:29 GMT-0500    tempNetStatusHandler 0/2,NetConnection.Connect.Success

19:27:29 GMT-0500    isTunneling? false

19:27:29 GMT-0500    is using RTMPS? false

19:27:29 GMT-0500    #SessionManagerPlayback 889 fms connected: [Event type="connected" bubbles=false cancelable=false eventPhase=2]

nothing happens after this point.

[Raff_]

Congratulations! You found a bug

Basically a process that manages playbacks on one of the FMS server has died and I suspect the FMS itself doesn't realize that and keeps pretending that can server your playback.

I will need to restart the service on that machine but it may take a while (don't want to disrupt other meetings). One suggestion is to try to wait 10 / 15 minutes for the service to "forget" that your playback was running on fms7 and hopefully when you retry you'll get reassigned to another server.

Again, the solution is for me to restart the service: that would clear up the problem but we may wait until later tonight to do it.

Sorry for the inconvenience

[adam_k]

thanks for the super fast response raff.

no rush on restarting today, i can wait til tomorrow as this is not a production scenario. but it would be good to have an event sent to the client side so that i can report this to the user, and for my own server monitoring purposes.

what are your stats on the various server uptimes? how often does a machine go down?

[Raff_]

once a year

again, this is a strange case where a bug caused one particular process to fail. We'll have to fix the problem and add an extra monitoring for this case. It just went under the radar...

[Raff_]

ok, I found the problem and it's affecting both recording and server to server API. I am working on a fix and we'll restart the servers as soon as possible (but it may be a few hours away).

sorry for the inconvenience.

[Raff_ · Accepted solution]

The problem has been solved. Sorry it tooks longer than expected but we wanted to fix the root cause instead of spending the day restarting servers (and of course got caught into a bunch of unrelated problems)

[adam_k]

thanks for resolving this raff.  let me take this opportunity to vote for an error event to be made available to the client when this situation arises in the future.  either way, thanks for the quick response and action.