Spiked CPU when viewing multiple streams one at a time then publishing

[tracedwax]

Hi all,

We're finding the Flash Player's CPU process 'ratchets' slightly higher each time a user views a new webcam stream one at a time in a WebcamSubscriber.

For example, I'm watching the stream for User A, then that stream is deleted and I then watch the stream for user B, then that stream is deleted and I watch the stream A, B, or C, and so on.  It seems to 'ratchet' higher by the same amount regardless of whether I've seen that particular user's stream before.

It's only a few percentage points of CPU usage each time, but in the aggregate it can get to 100% usage and crash the flash player quickly. 

Some details of our tests:

- We've managed to contain it so this only happens when the user is also publishing.

- If a user has been watching for a long time without publishing, then starts publishing, the CPU usage will suddenly spike as soon as they start publishing.  In our tests it spikes as much and more as if they had been publishing the entire time. 

- If the user starts publishing before they start watching, they're not affected by this spike. 

- If a user starts publishing but hasn't been watching the streams, the publishing CPU usage is normal. 

- Refreshing the browser page and publishing again, CPU usage is normal. 

- Calling System.gc(); while running in the flash player debugger seems to have no effect on the CPU spikes.  Whatever streams are being kept around must still have something pointing to them so they won't be garbage collected. 

To subscribe to each successive stream, we're doing that by setting

webcamSubscriber.publisherIDs = [ newStreamID ];

We've been experimenting for a long time with different settings that might reduce the CPU spikes and ratcheting, but haven't been able to resolve the issue.

How can we prevent this CPU ratcheting and spking?

Thanks very much,

-Trace

[aponnusa]

Hi Trace,

From your observations, watching or subscribing to streams seems to spike your CPU.

I have tested apps that have a publisher and 5 other streams (or subscribers) and the CPU wasn’t choking. I will try to recreate the scenario you mentioned.

In the mean time as a workaround, can you close the existing WebcamSubscriber and null it, and then create a new instance of WebcamSubsriber and use it to subscribe to the new stream.

Thanks

Arun

[tracedwax]

Hi Arun,

Thanks very much for your response.  We've tried that, and the CPU still ratchets and spikes the same as if there's just one camera.  It's almost as if something is hanging on to a reference to the stream and it's not getting garbage collected properly after being set to null, but I couldn't find that reference anywhere in the LCCS source code.

Anything else we can do?

Thanks,

-Trace

[aponnusa]

Is it possible that you have other components that are interfering or spiking your CPU. Also if possible can you share your code.

Would you be able to check this link and see it hogs your CPU for more subscribers. - http://blogs.adobe.com/arunpon/files/2011/05/WebCameraFinal31.swf

Code for the app in the link

<?xml version="1.0" encoding="utf-8"?>
<mx:Application xmlns:mx="http://www.adobe.com/2006/mxml" layout="absolute" xmlns:rtc="http://ns.adobe.com/rtc">
     
     <mx:Script>
          <![CDATA[
               import com.adobe.coreUI.controls.CameraUserBar;
               import com.adobe.rtc.collaboration.WebcamSubscriber;
               import com.adobe.rtc.events.CollectionNodeEvent;
               import com.adobe.rtc.events.SessionEvent;
               import com.adobe.rtc.events.SharedPropertyEvent;
               import com.adobe.rtc.events.StreamEvent;
               import com.adobe.rtc.events.UserEvent;
               import com.adobe.rtc.messaging.UserRoles;
               import com.adobe.rtc.sharedManagers.StreamManager;
               import com.adobe.rtc.sharedManagers.descriptors.StreamDescriptor;
               import com.adobe.rtc.sharedModel.SharedProperty;
               
               protected var _camSubscribers:Object;
               protected var _currentSubscriber:WebcamSubscriber ;
               protected var _sharedProperty:SharedProperty ;
               
               /**
                *  Handler for the stop and start buttons.
                */ 
               protected function startBtn_clickHandler(event:MouseEvent):void
               {
                    if ( startBtn.label == "Start" ) {
                         webCamPub.publish();
                         startBtn.label = "Stop" ;
                         
                         if (_camSubscribers && _camSubscribers[cSession.userManager.myUserID]) {
                              var webcamSubscriber:WebcamSubscriber = _camSubscribers[cSession.userManager.myUserID];
                              smallSubscriberContainer.addChild(webcamSubscriber);
                         }
                    }else if (startBtn.label == "Stop" ){
                         webCamPub.stop();
                         startBtn.label = "Start" ;
                    }
               }
               
               /**
                * SynchronizationChange event handler. Initialize the Shared property used to sync the Subscriber info
                * who would be the centre of the app.
                */
               protected function cSession_synchronizationChangeHandler(event:Event):void
               {
                    if (cSession.isSynchronized) {
                         _sharedProperty = new SharedProperty();
                         _sharedProperty.isSessionDependent = true ;
                         _sharedProperty.sharedID = "webcamShare2" ;
                         _sharedProperty.connectSession = cSession ;
                         _sharedProperty.subscribe();
                         _sharedProperty.addEventListener(SharedPropertyEvent.CHANGE,onChange);
                         
                         _camSubscribers = new Object();
                         cSession.streamManager.addEventListener(StreamEvent.STREAM_RECEIVE,onStreamRecieved);
                         cSession.streamManager.addEventListener(StreamEvent.STREAM_DELETE,onStreamDelete);
                         addExistingStreamers();
                    }
               }
               
               /**
                *  Set up a thumbnail subscriber for every new camera stream
                */
               protected function onStreamRecieved(p_evt:StreamEvent):void
               {
                    if (p_evt.streamDescriptor.type == StreamManager.CAMERA_STREAM) {
                         setUpfromDescriptor(p_evt.streamDescriptor);
                    }
               }
               
               /**
                * Clicking a subscriber updates the shared value, which in turn enlarges the thumbnail after getting updated
                */ 
               protected function onClick(p_evt:MouseEvent):void
               {
                    if ( (p_evt.currentTarget is WebcamSubscriber) &&  !(p_evt.target.parent is CameraUserBar)) {
                         _sharedProperty.value = (p_evt.currentTarget as WebcamSubscriber).publisherIDs;
                    }
               }
               
               /**
                * Clean up when a user stops publishing his camera or exits his app.
                */ 
               protected function onStreamDelete(p_evt:StreamEvent):void
               {
                    if (p_evt.streamDescriptor.type == StreamManager.CAMERA_STREAM) {
                         if ( _camSubscribers[p_evt.streamDescriptor.streamPublisherID]) {
                              var webcamSubscriber:WebcamSubscriber = _camSubscribers[p_evt.streamDescriptor.streamPublisherID];
                              if (webcamSubscriber) {
                                   smallSubscriberContainer.removeChild(webcamSubscriber);     
                              }
                              if (p_evt.streamDescriptor.streamPublisherID != cSession.userManager.myUserID) {
                                   webcamSubscriber.removeEventListener(UserEvent.STREAM_CHANGE,onCameraPause);
                                   webcamSubscriber.removeEventListener(UserEvent.USER_BOOTED,onUserBooted);
                                   delete _camSubscribers[p_evt.streamDescriptor.streamPublisherID];
                                   webcamSubscriber.close();
                                   webcamSubscriber = null;
                              } else {
                                   if (_currentSubscriber && _currentSubscriber.publisherIDs[0] == cSession.userManager.myUserID) {
                                        _sharedProperty.value = null;
                                   }
                              }
                         }
                    }
               }
               
               /**
                * Logic for handling the Pause event on CameraUserBar on every Subscriber
                */ 
               protected function onCameraPause(p_evt:UserEvent):void
               {
                    var userStreams:Array = cSession.streamManager.getStreamsForPublisher(p_evt.userDescriptor.userID,StreamManager.CAMERA_STREAM);
                    
                    if (userStreams.length == 0) {
                         trace("onCameraPause: no userStreams");
                         return;
                    }
                    
                    for (var i:int = 0; i< userStreams.length ; i++ ) {
                         if (userStreams[i].type == StreamManager.CAMERA_STREAM ) {
                              break;
                         }
                    }
                    
                    var streamDescriptor:StreamDescriptor = userStreams[i];
                    if ( streamDescriptor.streamPublisherID == cSession.userManager.myUserID ) {
                         cSession.streamManager.pauseStream(StreamManager.CAMERA_STREAM,!streamDescriptor.pause,streamDescriptor.streamPublisherID);
                    }     
               }
               
               /**
                * Initial set up of all users who are streaming when this app launches
                */ 
               protected function addExistingStreamers():void
               {
                    var streamDescritpors:Object = cSession.streamManager.getStreamsOfType(StreamManager.CAMERA_STREAM);
                    for (var i:String in streamDescritpors) {
                         setUpfromDescriptor(streamDescritpors[i]);
                    }
               }
               
               /**
                * Helper method to create a thumbnail subscriber.
                */ 
               protected function setUpfromDescriptor(p_descriptor:StreamDescriptor):void
               {
                    if (! _camSubscribers[p_descriptor.streamPublisherID]) {
                         var webCamSubscriber:WebcamSubscriber = new WebcamSubscriber();
                         webCamSubscriber.connectSession = cSession ;
                         webCamSubscriber.addEventListener(UserEvent.STREAM_CHANGE,onCameraPause);
                         webCamSubscriber.addEventListener(UserEvent.USER_BOOTED,onUserBooted);
                         webCamSubscriber.webcamPublisher = webCamPub;
                         webCamSubscriber.subscribe();
                         webCamSubscriber.sharedID = p_descriptor.streamPublisherID;
                         webCamSubscriber.publisherIDs = [p_descriptor.streamPublisherID];
                         webCamSubscriber.height = webCamSubscriber.width = 180;
                         webCamSubscriber.addEventListener(MouseEvent.CLICK, onClick);
                         smallSubscriberContainer.addChild(webCamSubscriber);
                         _camSubscribers[p_descriptor.streamPublisherID] = webCamSubscriber;
                    }
               }
               
               /**
                * This method is the listener to SharedPropertyEvent.CHANGE event. It updates the centred subscribes as its value
                * changes.
                */ 
               protected function onChange(p_evt:SharedPropertyEvent):void
               {
                    if ( _currentSubscriber != null ) {
                         _currentSubscriber.removeEventListener(UserEvent.USER_BOOTED,onUserBooted);
                         _currentSubscriber.removeEventListener(UserEvent.STREAM_CHANGE,onCameraPause);
                         centeredSubscriber.removeChild(_currentSubscriber);
                         _currentSubscriber.close();
                         _currentSubscriber = null ;
                    }
                    
                    if ( _sharedProperty.value == null || _sharedProperty.value.length == 0 ) {
                         return ;
                    }
                    
                    
                    _currentSubscriber = new WebcamSubscriber();
                    _currentSubscriber.connectSession = cSession ;
                    _currentSubscriber.subscribe();
                    _currentSubscriber.webcamPublisher = webCamPub ;
                    _currentSubscriber.publisherIDs = _sharedProperty.value ;
                    _currentSubscriber.addEventListener(UserEvent.USER_BOOTED,onUserBooted);
                    _currentSubscriber.addEventListener(UserEvent.STREAM_CHANGE,onCameraPause);
                    _currentSubscriber.width = _currentSubscriber.height = 500;
                    centeredSubscriber.addChild(_currentSubscriber);
               }
               
               /**
                * Logic for handling the Close event on CameraUserBar on every Subscriber
                */ 
               protected function onUserBooted(p_evt:UserEvent=null):void
               {
                    var tmpFlag:Boolean = false;
                    if (_currentSubscriber && _currentSubscriber.publisherIDs[0] == p_evt.userDescriptor.userID) {
                         if (_currentSubscriber.parent) {
                              _currentSubscriber.removeEventListener(UserEvent.USER_BOOTED,onUserBooted);
                              _currentSubscriber.removeEventListener(UserEvent.STREAM_CHANGE,onCameraPause);
                              _currentSubscriber.close();
                              _currentSubscriber.parent.removeChild(_currentSubscriber);
                              _currentSubscriber = null;
                              _sharedProperty.value = null;
                         }
                         tmpFlag = true;
                    }
                    
                    if ( _camSubscribers[p_evt.userDescriptor.userID]) {
                         var webcamSubscriber:WebcamSubscriber = _camSubscribers[p_evt.userDescriptor.userID];
                         tmpFlag = true;
                    }
                    
                    if (tmpFlag) {
                         webCamPub.stop();
                         startBtn.label = "Start";
                    }
               }
               
               
          ]]>
     </mx:Script>
     
     <!-- 
     You would likely use external authentication here for a deployed application; 
     you would certainly not hard code Adobe IDs here.
     -->
     <rtc:AdobeHSAuthenticator
          id="auth" 
          userName="Your Username" 
          password="Your password" />
     <rtc:ConnectSessionContainer id="cSession" authenticator="{auth}" width="100%" height="100%" roomURL="Your RoomUrl">
          <mx:VBox id="rootContainer" width="100%" height="800" horizontalAlign="center">
               <rtc:WebcamPublisher width="1" height="1" id="webCamPub"/>
               <mx:VBox width="500" height="500" id="centeredSubscriber" horizontalAlign="center" verticalAlign="middle"/>
               <mx:Label text="Click on a Subscriber thumbnail to make it bigger." />
               <mx:HBox width="100%" height="200" horizontalAlign="center" verticalAlign="top" id="smallSubscriberContainer" creationComplete="cSession_synchronizationChangeHandler(event)"/>
               <mx:Button  id="startBtn" label="Start"  click="startBtn_clickHandler(event)" height="20"/>
          </mx:VBox>
     </rtc:ConnectSessionContainer>
</mx:Application>

Thanks

Arun

[tracedwax]

Hi Arun,

Appreciate your setting up this example app, it's a great way to hone in on this issue.

When I ran this code with 2 cameras and switched between them in one of the views, I still noticed the ratcheting effect.  Upon each switch, the Flash process's CPU would spike by a few percent temporarily, then settle to a level that was about 1% higher than before per switch if I waited a few seconds before switching again, and about 5% higher than before per switch if I switched between the cameras rapidly. 

I added a third, nonpublishing computer to the room to just observe the switching, and it had a ratcheting effect as well, but less than the 2 publishing computers.

I was running this on Firefox 4 on a mac, but I've noticed this effect on all browsers on both Mac and Windows.

When I went to your link, the swf loaded by my video didn't appear, so I compiled the code into a new swf configured with a new developer account in a new room with the default configuration.

When following the steps above and running this code, are you able to notice the CPU ratcheting effect?  If not, please email me at tracedwax (at) gmail.com and I'll share my screen and demonstrate what we're seeing.

Thanks again,

-Trace

[aponnusa]

Hi Trace,

We ran a few tests at our end, and the ratcheting effect wasn’t consistent.. It was only consistent on my Win XP machine that had less RAM.

So I was wondering if you could share your machine details too, just to be sure.

So we tested on

Mac - Chrome

Mac - Safari

Win 7 - FireFox

Win 7 - IE

Win XP - Chrome

Win XP - IE

Win XP - FireFox.

As mentioned earlier, the ratcheting effect was consistent only on a WinXP & Firefox combination. We are looking into this issue, and update you if we have some information.

Thanks

Arun

[tracedwax]

I've been running on 3 Macs, one new with 8GB ram, one 2 years old with 2GB ram, and one 4 years old with 3GB ram.  The first two ran Firefox 4, the latter Firefox 3.  All of them ratcheted consistently.  Were you logging in to the swf and room from 2 computers at once, then clicking back and forth about once per second to reproduce the greatest ratcheting effect? 

Can also test on some additional machines.  If you repair the blog link so I can turn on my video, I can test again with as similar a configuration as possible.  I've noticed that the video doesn't turn on consistently.

Thanks again,

-Trace

[aponnusa]

Hi Trace,

The link works for me. (I did test it on a Mac machine)... Can I know the problem you are facing ? (But then I am running the same code, I pasted on this post)

Also I tested the app the way you mentioned. I had 4 subscribers & and switching between subscribers every second. We couldnt observe Ratcheting consistently. Is it possible for you to share your code to reproduce the issue.

Thanks

Arun

[tracedwax]

Hi Arun,

Very much appreciate your continuing to test this issue.

I was able to load http://blogs.adobe.com/arunpon/files/2011/05/WebCamera_Prod.swf just now, and again noticed a significant ratcheting effect on my two macs, again both running Snow Leopard, one with 2GB Ram and one with 8GB.

The code I ran locally to reproduce the issue is identical to the code you posted; I only changed the authentication settings to be the default 'myFirstRoom' room on a brand new LCCS developer account.

Thanks,

-Trace

[aponnusa]

Hi Trace,

Just uploading a snapshot of the CPU performance at my end. I did change the subscribers for every second for the last 10 minutes. It didnt spike the CPU

I am still looking into this issue, but just updating the test results at my end. (I tested on FF & Mac and used Flash debug 10.3 player)

Thanks

Arun

[aponnusa]

Hey Trace,

We did a minor optimization, and would like you to check if it fixes your ratcheting issue.

I am attaching new swc's for your reference.

Thanks

Arun

[tracedwax]

Hi Arun,

Thanks very much for this - it seems to be performing better.  There's still a significant ratcheting effect.  After 20 switches once per second, my 8GB mac is at 25% CPU and the 2GB memory mac hovers at 45% CPU using your sample app.  After another 20, my 2GB mac is at 110% and my 8gb mac hovers at 75%CPU.

Are there any further quick optimizations that might reduce the ratcheting just a bit further?

-Trace

[tracedwax]

Arun - are you able to reproduce the ratcheting issue? If not, let's set up a screenshare so you can see exactly how I'm reproducing it and how the CPU behaves over time.

-Trace

[aponnusa]

Hi Trace,

For your usecase, switching publisherID's (as you initially mentioned) might yield better performance. - http://blogs.adobe.com/arunpon/files/2011/05/WebCameraFinalTrace2.swf

We are still looking into this issue, and would update you for any performance optimization we could do.

Thanks

Arun

[Former Community Member · Accepted solution]

Hi Trace,

As Arun is alluding to, we've found a bug in publisherIDs that, now that we

fixed it, seems to alleviate the issue significantly. It still persists when

you're removing and recreating webcamSubscribers, which we're looking into.

Could you try the swf Arun has published? What we're seeing is that it

takes a large (100+) clicks to get to 100%, and that waiting for a bit will

often then clean that up (we've seen it bounce from 100%+ down to 10%).

try the swf and let us know if you're seeing similar differences. We're not

completely done yet, but it's definitely much better on this end.

thanks

nigel

[tracedwax]

Thanks again, very much.  Things seem much better.  You guys are rock stars.

-Trace

[tracedwax]

Running more detailed tests, we still did see some ratcheting, just much less than before.  Might be worth double checking if you haven't already, could there be a similar but much lesser issue in the AudioSubscriber publisherIDs?

[tracedwax]

The LCCS Navigator seems to ratchet more than anything else, if looking at that helps further debugging.

Thanks,

-Trace