LCCS Audio/Video Limitations

[__DEBUG__]

The goal of our project is to have an Audio/Video chat of 10-20 users (most of the users in the chat stream low-quality video and have audio muted, while two groups with up to 5 users all streaming full audio and video). We decided to use the latest LCCS with Flash 10.1 as a platform for our solution, because this combination seemed that it will provide us with a transparent way to achieve p2p communication with a fail-over mechanism.

But in our attempts to reach 10 users with full Audio/Video chat we run into some issues that look like we've hit the current technology limitations. We tried all kinds of options, but whatever we've tried our current achievement is just 5 users in an AV chat. Trying to join another user to the conversation results either in browser hangup or crash, we even managed to get our local network wi-fi router to get sunk when we reached 6 users.

I'll give you a brief explanation of the 3 main options we've tried and I'll also supply you with a link from where you can download an isolated example we've used in our tests, so everyone can check it. I'll be happy to get any feedback on whether I'm using the LCCS framework components in the correct and intended way.

1. Using the default settings of LCCS, no changes to the StreamManager properties.

On a single Mac computer the limit is 3-5. On Macs it tends to crash with fewer users in comparison with Windows. We tested it also on the local network with different machines - again Safari on Mac is crashing when we try to join a 6th user to a chat with 5.

Almost identical results we're getting with the other options.

2. Increasing the number of streamManager.maxP2PStreamPublish to 20.

3. Using streamManager.maxP2PStreamPublish=20 and session.streamManager.streamMulticast=true

From the link below you can download the Flash Builder project. You can import it and start testing.

DOWNLOAD THE TEST APPLICATION

Some notes on the Test Application:

The test application consists of two mxml application - LccsLimitationsOwner and LccsLimitationsPublisher. You should modify LccsLimitationsOwner with your correct username and email. And you should run it before the publishers applications, because the owner sets the StreamManager properties. Make sure you edut the correct room URLs in both applications.Also make sure that you check Auto Promote Users from the Room Console of your room. The users should be at least publishers.

When you start the publisher app, you will be prompted for a username - here you can put whatever you want, later you'll refer to these usernames when you want to connect to someone with the test app.

The testing is basicaly connecting several users and then each of them adds the rest by typing a username and clicking the GO-button.

Before running the owner's application you can set in the mxml what kind og configuration you want to test, by changing the value of the lccsConfiguration-property to one of the following values:

[Bindable] public static var DEFAULT_CONFIGURATION:String = "default";
[Bindable] public static var HUB_SPOKE_CONFIGURATION:String = "hub-spoke";
[Bindable] public static var MULTICAST_CONFIGURATION:String = "multicast";

Let me know if you have any questions on the test application!

[Former Community Member]

I haven't yet got time to look at that but I will take a look at it sometimes this week and update you the details.

Thanks

Hironmay Basu

[Former Community Member]

Hi Antonio,

I got time to look at the issue now. Yes, you were right. Multiple group multicasting wasnot working and I had to change few files to get it to work.

Thanks for pointing this out. Since the 10.1 SDK is still in kind of beta, some features like this sometimes needs more testing. And best way to get it done is through the community like in this case.

There are still some optimizations I would do before next release so that it works even smoother.

If we would have given out the source, I would have given you the source files to fix but in the current case, I will pass on the swc. Normally, we don't give the swc mid of release but since its a blocking issue for you and you helped me unearth this , so I will pass you the 10.1 swc. Just remember, its an intermediary build and  when we release the next version of SDK soon, this will automatically go in.

Try it out and let me know if things work for you.

Thanks

Hironmay Basu

[aj-runtimelogic]

Thanks Hironmay,

I'm glad you were able to track down the problem and I'm looking forward to trying out the fix.

The file you attached seems to be some kind of project folder, though, not the lccs.swc.  When you fix the attachment can you make sure to include the flash only build of the SWC too please (lccsFlash.swc) since that is what my project is compiling against.  Thanks again!

[Former Community Member]

Hi Antonio,

The file is indeed a swc. I attached it in the forum link from the browser so it always gets into this fornat called lccs.swc.zip which if you open, you get the swc. I just downloaded and verified. So, go to the forum post link in browser and download it.

In case you still have problems, send me your email id. I will mail both the swcs.

I am also attaching the flash only swc with this post.

Hope this helps

Thanks

Hironmay Basu

[aj-runtimelogic]

Ah, I see what's happening now.  When I double clicked the ZIP to decompress it (I'm on Mac OS X 10.6.4), it unzipped and then subsequently expanded the SWC wrapper itself, so I was left with just a folder containing all the SWC internals.  Not having looked at the internals of a SWC library before I thought it was the wrong attachment.  Heh. 

I opened with a different ZIP utility and all is well now.  Will report back after I've tested your fix.  Thanks.

[Former Community Member]

Definitely let us know how your custom groups work with multicast. And I agree that its really weird how forums puts every file in a zip format.

Thanks

Hironmay Basu

[aj-runtimelogic]

Hi Hironmay,

I just wanted to give you some initial feedback on the fixes.  I am now able to successfully create NetGroups with streamMulticast set to true.  We built a call management architecture into our existing app and have been able to audio chat with other users within a custom NetGroup.  I have yet to test with multiple simultaneous "calls" - IE. multiple custom NetGroups with multiple published audio streams in each group, but will reply back to this thread when I do.

One issue I've run into is with the NET_STATUS NetGroup.Connect.Success event we get back from the StreamManager after we get the session sync event back from ConnectSession.  Before I had created any custom groups I would get one event and see the following displayed in my trace window:

01:05:05 GMT-0700    mainNetStatusHandler: NetGroup.Connect.Success

01:05:05 GMT-0700    NetGroup Creation Successful

01:05:05 GMT-0700    NetGroup Creation Successful

I always assumed that was because I was automatically connecting to the default NetGroup when logging in to the room.  Now that there are multiple NetGroups in the room, it seems like I get one of these events for each NetGroup in the room and I get a whole page worth of those traces repeated.  This is somewhat confusing to me and seems to contradict my assumption.  The user who has logged into an LCCS room with multiple NetGroups isn't automatically connected to them all, are they?  Is this behavior what you would expect with multiple NetGroups?

One other issue is that we are creating a unique NetGroup each time we place people into a "call".  Pretty quickly the room becomes littered with NetGroups.  Is there a way to handle cleaning up inactive NetGroups on the server?  What are your recommendations for handling this issue?  I don't want to rely on the client app to clean up after itself since a user could simply close the browser window and immediately exit the client.

Thanks!

[Former Community Member]

Hi Antonio,

With multiple groups, you may have in your app let's say 2 webcamsubscribers each having a different group and two audioSubscribers each having a different group.

So, your StreamManager which is just one(per room) will have the bookkeeping of all NetGroups but your streams will flow only through the group that your subscriber belongs to because when a stream is group, it is created with a particular group specifier. So, even though the StreamManager create the groups, you are publishing only in the group that your audio/camera subscriber belongs to.

You may argue that, why create groups before. Rather, when you use any subscriber, it should create groups then and similarly when you unsubscribe all subscribers from a group, it should get deleted. I haven't yet done the dynamic management of the groups in StreamManager with respect to multiple subscribers but I will definitely keep that in radar before the release after the coming one in few weeks. Just creating the groups won't affect your performance because even dynamic creation and deletion is an expensive operation.

Regarding those traces, as I said, I was using a debug build with traces , I will remove them for the release SDK.

Hope this helps

Thanks

Hironmay Basu

[aj-runtimelogic]

Hi Hironmay,

It's been awhile since I've checked in.  We have completed our audio chat "call management" code and UI that works in coordination with LCCS so now I can finally do extensive testing of multiple simultaneous audio chat sessions within multiple NetGroups in a room.  So far I've been able to confirm that multiple NetGroup sessions with streamMulticasting enabled is working well.

The does seems to be quite a bit of latency when the streams are first published (4+ second delays), but that does seem to improve progressively and after about the first 20 seconds of the call it seems to have stabilized at about .5 seconds.  I'd like to enable the directed routing capability, and I know I can call

_currentSession.streamManager.getMulticastGroupSpec(_currentGroupName);

which returns a GroupSpecifier instance but can I set routingEnabled to true and pass this in to another call?  If so, what is that call and should this be set before creating audio pub/sub or can it be changed dynamically?

Also, I wanted to touch base about a new issue I've encountered which may require a fix.  A group of users can have a successful audio chat with our system, and they can then exit the chat session, but when attempting to start a new audio chat I'm getting back a null pointer exception from within the LCCS SDK.  I use the following code to create and dispose the streams:

private function startStreams(): void

{

     _audioSubscriber = new AudioSubscriber();

     _audioSubscriber.connectSession = _currentSession;

     _audioSubscriber.groupName = _currentGroupName;

     _audioSubscriber.addEventListener(StreamEvent.CONNECTION_TYPE_CHANGE, this.audioConnectionTypeChanged);

     _audioSubscriber.subscribe();

     _audioPublisher = new AudioPublisher();

     _audioPublisher.connectSession = _currentSession;

     _audioPublisher.groupName = _currentGroupName;

     _audioPublisher.codec = SoundCodec.SPEEX;

     _audioPublisher.subscribe();

     _audioPublisher.publish();

}

private function disposeStreams(): void

{

     _audioSubscriber.close();

     _audioSubscriber.removeEventListener(StreamEvent.CONNECTION_TYPE_CHANGE, this.audioConnectionTypeChanged);

     _audioSubscriber = null;

     _audioPublisher.stop();

     _audioPublisher.close();

     _audioPublisher = null;

}

The first time through everything works fine, but the second time I call startStreams. I get back the following exception on the _audioSubscriber.subscribe() :

TypeError: Error #1009: Cannot access a property or method of a null object reference.

at com.adobe.rtc.collaboration::AudioSubscriber/createNetStream()[C:\work\branches\connect\1010\SDKApp\payload\libs\flashOnly\player10.1\src\com\adobe\rtc\collaboration\AudioSubscriber.as:888]

at com.adobe.rtc.collaboration::AudioSubscriber/playStream()[C:\work\branches\connect\1010\SDKApp\payload\libs\flashOnly\player10.1\src\com\adobe\rtc\collaboration\AudioSubscriber.as:610]

at com.adobe.rtc.collaboration::AudioSubscriber/playStreams()[C:\work\branches\connect\1010\SDKApp\payload\libs\flashOnly\player10.1\src\com\adobe\rtc\collaboration\AudioSubscriber.as:544]

at com.adobe.rtc.collaboration::AudioSubscriber/subscribe()[C:\work\branches\connect\1010\SDKApp\payload\libs\flashOnly\player10.1\src\com\adobe\rtc\collaboration\AudioSubscriber.as:518]

at kazoo::LiveCycleServer/startStreams()[/Projects/Runtime Logic/Clients/Qimera/Projects/Flash Trunk/kazoo/LiveCycleServer.as:363]

I should note that the user is still logged into the LCCS server and that no other calls to LCCS have been made between when disposeStreams and startStreams is called.  I'm just attempting to maintain the user session in _currentSession and recreate the audio pub / sub when they want to start a new audio chat session.  I also tried updating to the latest release of the SDK (I was using the custom build you had provided up to this point) but I'm still encountering this error.  Am I doing something wrong with this sequence of calls?

Thanks.

[Former Community Member]

Hi Antonio,

So we don't turn on Direct routing be default i.e. groupSpecifier.routingEnabled = false by default. However, you may turn on the routingEnabled on a particular group instance returned from getMulticastGroupSpec call. I am not sure if this needs to be done before audio pub/sub creation or can be changed dynamically. Try it out with the getMulticastGroupSpec call and if it doesn't work, I may add an API where you can specify whether you want to turn this on or not when groups are created.

Regarding the other error you mentioned, I will run your dispose code on my side and get back to you investigating whether it’s a problem in your dispose code or we have issues on our side.

Thanks for trying this out. You have made our 10.1 NetGroup feature more stable.

Regards

Hironmay Basu

[aj-runtimelogic]

Hi Hironmay,

Just following up to see if you had had a chance to test the component destroy / recreate code and see what is causing the error?

Thanks.

[Former Community Member]

Hi Antonio,

Sorry, it had just slipped among many issues but I will run it soon and let you know.

Thanks

Hironmay Basu

[Former Community Member]

Hi Antonio,

I got some time to take a look into the issue you mentioned. It only happened for me using your code when I use custom groups and I have users logged in and then start sharing and stop. It was kind of a race condition. The reason you were getting exception was because the second user didn't have the group in his list when the first user creates it after both have logged in.

Anyways, I got a fix and I am providing you with a temp build. Let me know if it works for you or not.If everything is fine, this will go into the patch update we plan to have next week.

Thanks

Hironmay Basu

[aj-runtimelogic]

Hi Hironmay,

I tested the temp build and it appears to fix the error I had experienced previously on re-instantiation of the pub / sub classes.  I say "appears" because I compile against the lccsFlash.swc version of the lib and had to include /locale/en_US/framework_rb.swc to get it to compile without errors and I'm not sure if that may have affected my build in other ways.  Please attach the lccsFlash.swc version and I'll test again tomorrow to give a more definitive answer.

Thanks!

[Former Community Member]

Hi Antonio,

I am attaching the 10.1 lccsFlash.swc for your convenience. We are anyways going to have a full update next week.

Thanks

Hironmay Basu

[aj-runtimelogic]

Hi Hironmay,

Things seem to be working well in regard to that latest fix.  Thanks!

I have one more issue to resolve before we can go live with this system: When having an audio chat between any number of clients when using stream multicasting, there is a large transmission delay (4-6 seconds) between when user A speaks and when user B will hear them.  This problem starts immediately after both clients begin publishing their streams and generally lasts for 10-20 seconds.  At that point the noticeable transmission delay drops down sharply to the .1-.5 second range, and is generally usable.  The addition of the AEC feature that will be in the FP 10.2 release will make a transmission delay in that range hardly noticeable.

The problem is that for that first 10-20 seconds it's impossible to have a conversation.  If I turn off multicasting for the room, then this initial performance lag is almost non existent.  I wanted to rule out that I was doing something in my code to cause the problem, so I did the same test with the MulticastTest sample app from the latest LCCS SDK release and can reproduce the problem there.

Is this a known issue?  Something that is being worked on?  We have considered putting in an elaborate work-around where we remotely mute stream playback on the other clients and wait a default amount of time after connecting to allow these initial latency issues to resolve, but it's not a very good work around since the usability will be poor and I expect to run into technical issues getting that to work in a reasonable manner.  I'd much rather know that there is a fix pending from Adobe that will address this issue.

Thanks!

[Former Community Member]

Hi Antonio,

No, this first 15 -20 seconds is not a known issue to me at least. My take is , when its trying to form the multicast mesh there are some lags there. However, I have done certain optimizations in general for audio/video which is supposed to go out in jan. I am attaching a temporary version of the optimized swc.

Run it and see if it helps or not. Regarding the first 15-20 secs, i need to do deeper investigation and talk to player team if they are aware of this issue if the optimized swc too behaves the same. Let me know how this swc behaves.

Regards

Hironmay Basu

[aj-runtimelogic]

Can I trouble you for the Flash-only version of the SWC please? 

[Former Community Member]

Hi,

Attached is the flash swc for 10.1

Thanks

Hironmay Basu

[aj-runtimelogic]

We tested the custom build a couple of times tonight and unfortunately there was no improvement with the startup lag for multicast streams.  In fact, at no point during a 5 minute conversation did the round-trip delay (time between when I speak and when I hear my echo) get below 3 seconds, so figure a minimum delay of 1.5 seconds in one direction.

[Former Community Member]

I might have to talk to the player team if there are aware of such issue

on startup. Unfortunately, adobe's holiday break starts tomorrow and I

will be able to get back to you with more details sometimes only in Jan.

Thanks

Hironmay Basu