Room suddenly crashes, stopping streams & messages

[tracedwax]

Our room crashed last night, where all the video streams suddenly stopped and chat messages couldn't be sent.

This seems similar to what we reported in http://forums.adobe.com/post!reply.jspa?message=3674477 - did you ever reproduce or resolve the room crashing issue documented there?

This crash seemed different, however, as in the previous room crashes the current streams would continue, you just couldn't add or retract items, and the room would stay in a corrupt state where items could be added but not modified or retracted.  This time, the room recovered when we checked it again this morning.

Switching to a new room, everything worked fine, but we lost a lot of users in the process.  

Testing circumstances as similarly as possible, we weren't able to reproduce the crash today.  There were 70 users in the room at the time of the crash, perhaps 15 of them streaming.

The crash occurred at approximately 9:02pm ET last night in room yrtv17.  Could you please investigate any logs from your side and see what might have caused the crash?  Anything we can do to avoid similar room crashes in the future?

Thanks,

-Trace

[tracedwax]

Hi Nigel,

Our room broke just now, same as before - it's room yrtv26.  Are you available to log in real quick and look at the logs, and if there's anything else you can learn related to a potential fix, or if this corroborates the issues that you've seen previously?

There were no problems with rooms on Monday evening.

Thanks,

-Trace

[tracedwax]

Just saw a similar issue in room yrtv31, where we couldn't retract the singleton item in the private message node within our 'message collection' collection node.  This happened twice tonight and has happened in the past.

This time, when users tried to enter the room, they weren't able to synchronize with it, so everyone experienced multiple runtime errors.  Haven't seen that before.

Also, usually we can still post a new message even in a broken room that tells everyone in the room to leave.  We weren't even able to get that working in yrtv31 just now.

Please let us know what you find out related to these broken rooms.

Thanks again,

-Trace

[tracedwax]

At the end of tonight's session, the same issue recurred on room yrtv32.  It's seeming to be increasingly reproducible - we'll see if we can post some sample code that reproduces the problem, that we can send you.

[Former Community Member]

Hi Trace,

I just took a look at yrtv26 at 22:31, and see nothing unusual at that

time. Actually, the room shut down normally at 22:20. I see a gap in the

logs between 21:55:06 and 22:03:16. Did the freeze up happen during this 8

minute period?

nigel

[Former Community Member]

Ah, ignore that last mail, I see now that forum timestamps are different

than the times in my inbox (I didn't receive your post until 22:31, but it

seems as though you wrote it at 16:18). Could you write exact times in the

post as a favor to me, to avoid confusion? I'm still digging around.

nigel

[tracedwax]

Our activity ends at 10pm (9:59 to be exact).  That's when we noticed the behavior we weren't expecting.

Will see if we can create the reproducible use case for you on Monday.

Thanks again,

-Trace

[tracedwax]

Hi Nigel,

We reproduced the problem again just now -- seems to be consistently reproducible now.  The room that's currently corrupted is room yrtv34, and it became corrupt at 5:38pm ET today. (Fri Jun 24 2011 17:38:01 GMT-0400 (EDT) on our server).

The action that's breaking it is setting an item in the text of the a node called 'privateMessage' inside the collection node called 'MESSAGE_COLLECTION' node:

var messItem:MessageItem = new MessageItem("PrivateMessage");

messItem.publisherID =  this.MyID

messItem.collectionName = _msgCollection.sharedID;

messItem.body = {

note: "logoutAll"

}

_msgCollection.publishItem(messItem);

This instructs all the users(who are subscribed to updates on that node) to simultaneoulsy leave the room.  There were perhaps 5 users in this particular instance.

The room will be broken until the session resets.  Once the session resets, the room will work normally.

Please let us know if you see anything interesting in your logs.  Now that the room is reliably breaking, shall we schedule a time with you where we can break it again and you can observe the problem as it happens?

Thanks,

-Trace

[davisballen]

Hi Nigel,

Thanks again for taking the time to look into this issue.

As promised, we've set up an environment where you can consistently reproduce the issue yourself.  It's a copy of our staging environment, so we sent you the credentials and instructions via private message.

In our testing to set this up, we also just broke rooms yrtv100 and yrtv101.

Very much looking forward to what you find out and are able to fix -- we have some high profile usage coming up quite soon!

Cheers,

Davis

[tracedwax]

Hi Nigel,

Looks like another room (yrtv38) just broke, through different means than when we were testing earlier this week (that problem had been fixed).  We're seeing an inability to retract users and items through the LCCS console again (though we can retract them through the server-server API).  We can't chat.  New streams aren't being published, and current streams are frozen.

We're investigating the properties of the broken room to see if we can learn anything while it's in that state, but meanwhile, are you available to investigate the currently broken room?

Kind regards,

-Trace

[tracedwax]

Hi Nigel & co,

Tonight, we had a large number of people on and the room broke 5 times.  Rooms yrtv45, yrtv46, and yrtv47 all broke.  The times when the rooms broke are at 9:01 ET, 9:04:ET, 9:23ET, and 9:40ET.  As of present, yrtv45 and yrtv46 have righted themselves, but yrtv47 is still broken.

This is the first time when rooms have broken with a large number of live users present.  Do your logs reveal any interesting information about what happened in these rooms right before they broke?

Kind regards,

Trace & Davis