Expand my Community achievements bar.

Dive into Adobe Summit 2024! Explore curated list of AEM sessions & labs, register, connect with experts, ask questions, engage, and share insights. Don't miss the excitement.

Still a significant CPU ratcheting effect when changing PublisherIDs

Avatar

Level 3

Hi Arun, Nigel, & co,

Thanks again for making the bug fixes last week -- seems to have improved the ratcheting issue some. 

Running a battery of further tests this morning, I noticed however that the improvement wasn't as much as I'd originally hoped: there was still a large ratcheting effect with a significant impact.

To isolate the problem, I ran your swf again at http://blogs.adobe.com/arunpon/files/2011/05/WebCameraFinalTrace2.swf.  My standard test has been to start the app on 4 computers, turn on cameras for the first 3 computers, and take an initial reading of CPU usage.  Then I measured the CPU each time I clicked on someone to put them on the larger WebcamSubscriber, then switched once per second between them, in order, ten times each.  Then I turned the fourth computer's camera on and measured its CPU usage.  I then waited 10 minutes and measured everyone's CPU usage, then switched 10 more times over an additional 30 seconds.

We ran experiments like this multiple times and found noticed significant CPU ratcheting.  CPU usage would recover sometimes, but the recovery took a long time.  The fourth computer showed no CPU increase until its camera turned on, at which point it showed a very large CPU increase.  Putting the fourth computer on the large WebcamSubscriber crashed Flash Player.  Waiting for 10 minutes, the CPU hadn't decreased for any of the computers.  Repeating the switching, however, the computers' CPU initially decreased by a lot (perhaps garbage collection kicked in), but then increased again as before.

Here are the details.  Computer A is a MacBookPro '10 i5 4GB, Computer B is a MacBook '09 CoreDuo 2GB, Computer C is a MacBook Pro '10 i7 8GB, and Computer D is a '07 MacBook Pro CoreDuo 3GB.  They're all Macs, but PCs showed similar ratcheting effects.  A, B, and C were running Firefox 4 and D was running Firefox 3, but we've seen similar effects in other browsers.

The numbers below are the CPU percentages of the Flash Player process while running your SWF above that includes your publisherIDs bug fix:

Before starting cams

A - 1.6%

B - 2.1%

C - 2.5%

D - 6.5%

A,B,C start their cams

A - 10%

B - 10%

C - 9%

D - 22%

A is published to big screen

A - 19%

B - 12.5%

C - 12%

D - 33%

B is published to big screen

A - 13.5%

B - 22%

C - 12%

D - 32%

C is published to big screen

A - 14%

B - 13%

C - 19%

D - 32%

Switch between A,B,C 9 times each, such that A is on big screen

A - 36%

B - 17%

C - 24%

D - 33%

Switch to B on the big screen

A - 19%

B - 36%

C - 25%

D - 32%

Switch to C on the big screen

A - 20%

B - 19%

C - 63%

D - 33%

D turns on Cam

A - 26%

B - 24%

C - 64%

D - 113%

D to big screen

A - 26%

B - 24%

C - 37%

D - 125%

Wait 10 minutes, D freezes, put A on big Screen

A - 28%

B - 17%

C - 20%

put C on big screen

A - 18%

B - 17%

C - 48%

Switch 40 times in 30 seconds, put A on big screen

A - 85%

B - 37%

C - 14%

Put B on big screen

A - 38%

B - 92%

C - 14%

put C on big screen

A - 39%

B - 40%

C - 22%

Thanks very much,

-Trace

30 Replies

Avatar

Level 3

When you can, would it also be possible to provide the source code for WebCameraFinalTrace2.swf and the source code from the lccs.swc containing those new bug fixes?  That will help with our further testing and debugging efforts.

Thanks,

-Trace

Avatar

Employee

Code for WebCameraFinalTrace2

<?xml version="1.0" encoding="utf-8"?>
<mx:Application xmlns:mx="http://www.adobe.com/2006/mxml" layout="absolute" xmlns:rtc="http://ns.adobe.com/rtc" xmlns:authentication="com.adobe.rtc.authentication.*" xmlns:session="com.adobe.rtc.session.*" xmlns:collaboration="com.adobe.rtc.collaboration.*">
     
     <mx:Script>
          <![CDATA[
               import com.adobe.coreUI.controls.CameraUserBar;
               import com.adobe.rtc.collaboration.WebcamSubscriber;
               import com.adobe.rtc.events.CollectionNodeEvent;
               import com.adobe.rtc.events.SessionEvent;
               import com.adobe.rtc.events.SharedPropertyEvent;
               import com.adobe.rtc.events.StreamEvent;
               import com.adobe.rtc.events.UserEvent;
               import com.adobe.rtc.messaging.UserRoles;
               import com.adobe.rtc.sharedManagers.StreamManager;
               import com.adobe.rtc.sharedManagers.descriptors.StreamDescriptor;
               import com.adobe.rtc.sharedModel.SharedProperty;
               
               protected var _camSubscribers:Object;
               protected var _currentSubscriber:WebcamSubscriber ;
               protected var _sharedProperty:SharedProperty ;
               
               /**
                *  Handler for the stop and start buttons.
                */
               protected function startBtn_clickHandler(event:MouseEvent):void
               {
                    if ( startBtn.label == "Start" ) {
                         webCamPub.publish();
                         startBtn.label = "Stop" ;
                         
                         if (_camSubscribers && _camSubscribers[cSession.userManager.myUserID]) {
                              var webcamSubscriber:WebcamSubscriber = _camSubscribers[cSession.userManager.myUserID];
                              smallSubscriberContainer.addChild(webcamSubscriber);
                         }
                    }else if (startBtn.label == "Stop" ){
                         webCamPub.stop();
                         startBtn.label = "Start" ;
                    }
               }
               
               /**
                * SynchronizationChange event handler. Initialize the Shared property used to sync the Subscriber info
                * who would be the centre of the app.
                */
               protected function cSession_synchronizationChangeHandler(event:Event):void
               {
                    if (cSession.isSynchronized) {
                         _sharedProperty = new SharedProperty();
                         _sharedProperty.isSessionDependent = true ;
                         _sharedProperty.sharedID = "webcamShare2" ;
                         _sharedProperty.connectSession = cSession ;
                         _sharedProperty.subscribe();
                         _sharedProperty.addEventListener(SharedPropertyEvent.CHANGE,onChange);
                         
                         _camSubscribers = new Object();
                         cSession.streamManager.addEventListener(StreamEvent.STREAM_RECEIVE,onStreamRecieved);
                         cSession.streamManager.addEventListener(StreamEvent.STREAM_DELETE,onStreamDelete);
                         addExistingStreamers();
                    }
               }
               
               /**
                *  Set up a thumbnail subscriber for every new camera stream
                */
               protected function onStreamRecieved(p_evt:StreamEvent):void
               {
                    if (p_evt.streamDescriptor.type == StreamManager.CAMERA_STREAM) {
                         setUpfromDescriptor(p_evt.streamDescriptor);
                    }
               }
               
               /**
                * Clicking a subscriber updates the shared value, which in turn enlarges the thumbnail after getting updated
                */
               protected function onClick(p_evt:MouseEvent):void
               {
                    if ( (p_evt.currentTarget is WebcamSubscriber) &&  !(p_evt.target.parent is CameraUserBar)) {
                         //                    changeCurrentSub((p_evt.currentTarget as WebcamSubscriber).publisherIDs);
                         _sharedProperty.value = (p_evt.currentTarget as WebcamSubscriber).publisherIDs;
                    }
               }
               
               /**
                * Clean up when a user stops publishing his camera or exits his app.
                */
               protected function onStreamDelete(p_evt:StreamEvent):void
               {
                    if (p_evt.streamDescriptor.type == StreamManager.CAMERA_STREAM) {
                         if ( _camSubscribers[p_evt.streamDescriptor.streamPublisherID]) {
                              var webcamSubscriber:WebcamSubscriber = _camSubscribers[p_evt.streamDescriptor.streamPublisherID];
                              if (webcamSubscriber) {
                                   smallSubscriberContainer.removeChild(webcamSubscriber);
                              }
                              if (p_evt.streamDescriptor.streamPublisherID != cSession.userManager.myUserID) {
                                   webcamSubscriber.removeEventListener(UserEvent.STREAM_CHANGE,onCameraPause);
                                   webcamSubscriber.removeEventListener(UserEvent.USER_BOOTED,onUserBooted);
                                   delete _camSubscribers[p_evt.streamDescriptor.streamPublisherID];
                                   webcamSubscriber.close();
                                   webcamSubscriber = null;
                              } else {
                                   if (_currentSubscriber && _currentSubscriber.publisherIDs[0] == cSession.userManager.myUserID) {
                                        _sharedProperty.value = null;
                                   }
                              }
                         }
                    }
               }
               
               /**
                * Logic for handling the Pause event on CameraUserBar on every Subscriber
                */
               protected function onCameraPause(p_evt:UserEvent):void
               {
                    var userStreams:Array = cSession.streamManager.getStreamsForPublisher(p_evt.userDescriptor.userID,StreamManager.CAMERA_STREAM);
                    
                    if (userStreams.length == 0) {
                         trace("onCameraPause: no userStreams");
                         return;
                    }
                    
                    for (var i:int = 0; i< userStreams.length ; i++ ) {
                         if (userStreams[i].type == StreamManager.CAMERA_STREAM ) {
                              break;
                         }
                    }
                    
                    var streamDescriptor:StreamDescriptor = userStreams[i];
                    if ( streamDescriptor.streamPublisherID == cSession.userManager.myUserID ) {
                         cSession.streamManager.pauseStream(StreamManager.CAMERA_STREAM,!streamDescriptor.pause,streamDescriptor.streamPublisherID);
                    }
               }
               
               /**
                * Initial set up of all users who are streaming when this app launches
                */
               protected function addExistingStreamers():void
               {
                    var streamDescritpors:Object = cSession.streamManager.getStreamsOfType(StreamManager.CAMERA_STREAM);
                    for (var i:String in streamDescritpors) {
                         setUpfromDescriptor(streamDescritpors[i]);
                    }
               }
               
               /**
                * Helper method to create a thumbnail subscriber.
                */
               protected function setUpfromDescriptor(p_descriptor:StreamDescriptor):void
               {
                    if (! _camSubscribers[p_descriptor.streamPublisherID]) {
                         var webCamSubscriber:WebcamSubscriber = new WebcamSubscriber();
                         webCamSubscriber.connectSession = cSession ;
                         webCamSubscriber.addEventListener(UserEvent.STREAM_CHANGE,onCameraPause);
                         webCamSubscriber.addEventListener(UserEvent.USER_BOOTED,onUserBooted);
                         webCamSubscriber.webcamPublisher = webCamPub;
                         webCamSubscriber.subscribe();
                         webCamSubscriber.sharedID = p_descriptor.streamPublisherID;
                         webCamSubscriber.publisherIDs = [p_descriptor.streamPublisherID];
                         webCamSubscriber.height = webCamSubscriber.width = 180;
                         webCamSubscriber.addEventListener(MouseEvent.CLICK, onClick);
                         smallSubscriberContainer.addChild(webCamSubscriber);
                         _camSubscribers[p_descriptor.streamPublisherID] = webCamSubscriber;
                    }
               }
               
               protected function changeCurrentSub(p_publishIDs:Array):void
               {
                    
                    if ( _currentSubscriber != null ) {
                         _currentSubscriber.close();
                         _currentSubscriber.removeEventListener(UserEvent.USER_BOOTED,onUserBooted);
                         _currentSubscriber.removeEventListener(UserEvent.STREAM_CHANGE,onCameraPause);
                         centeredSubscriber.removeChild(_currentSubscriber);
                         _currentSubscriber = null ;
                    }

                    if (_currentSubscriber==null) {           
                         _currentSubscriber = new WebcamSubscriber();
                         _currentSubscriber.connectSession = cSession ;
                         _currentSubscriber.subscribe();
                         _currentSubscriber.webcamPublisher = webCamPub ;
                         _currentSubscriber.addEventListener(UserEvent.USER_BOOTED,onUserBooted);
                         _currentSubscriber.addEventListener(UserEvent.STREAM_CHANGE,onCameraPause);
                         _currentSubscriber.width = _currentSubscriber.height = 500;
                         centeredSubscriber.addChild(_currentSubscriber);
                         _currentSubscriber.publisherIDs = p_publishIDs ;
                    }
               }
               
               
               /**
                * This method is the listener to SharedPropertyEvent.CHANGE event. It updates the centred subscribes as its value
                * changes.
                */
               protected function onChange(p_evt:SharedPropertyEvent=null):void
               {
                    changeCurrentSub(_sharedProperty.value);
                    /*               
                    if ( _currentSubscriber != null ) {
                    
                    _currentSubscriber.removeEventListener(UserEvent.USER_BOOTED,onUserBooted);
                    _currentSubscriber.removeEventListener(UserEvent.STREAM_CHANGE,onCameraPause);
                    centeredSubscriber.removeChild(_currentSubscriber);
                    _currentSubscriber.close();
                    _currentSubscriber = null ;
                    }
                    
                    if (p_evt!=null && _sharedProperty.value == null || _sharedProperty.value.length == 0 ) {
                    return ;
                    }
                    
                    
                    _currentSubscriber = new WebcamSubscriber();
                    _currentSubscriber.connectSession = cSession ;
                    _currentSubscriber.subscribe();
                    _currentSubscriber.webcamPublisher = webCamPub ;
                    _currentSubscriber.publisherIDs = _sharedProperty.value ;
                    _currentSubscriber.addEventListener(UserEvent.USER_BOOTED,onUserBooted);
                    _currentSubscriber.addEventListener(UserEvent.STREAM_CHANGE,onCameraPause);
                    _currentSubscriber.width = _currentSubscriber.height = 500;
                    centeredSubscriber.addChild(_currentSubscriber);
                    */
               }
               
               /**
                * Logic for handling the Close event on CameraUserBar on every Subscriber
                */
               protected function onUserBooted(p_evt:UserEvent=null):void
               {
                    var tmpFlag:Boolean = false;
                    if (_currentSubscriber && _currentSubscriber.publisherIDs[0] == p_evt.userDescriptor.userID) {
                         if (_currentSubscriber.parent) {
                              _currentSubscriber.removeEventListener(UserEvent.USER_BOOTED,onUserBooted);
                              _currentSubscriber.removeEventListener(UserEvent.STREAM_CHANGE,onCameraPause);
                              _currentSubscriber.close();
                              _currentSubscriber.parent.removeChild(_currentSubscriber);
                              _currentSubscriber = null;
                              _sharedProperty.value = null;
                         }
                         tmpFlag = true;
                    }
                    
                    if ( _camSubscribers[p_evt.userDescriptor.userID]) {
                         var webcamSubscriber:WebcamSubscriber = _camSubscribers[p_evt.userDescriptor.userID];
                         tmpFlag = true;
                    }
                    
                    if (tmpFlag) {
                         webCamPub.stop();
                         startBtn.label = "Start";
                    }
               }
               
               
          ]]>
     </mx:Script>
     
     <!--
     You would likely use external authentication here for a deployed application;
     you would certainly not hard code Adobe IDs here.
     -->
     <authentication:AdobeHSAuthenticator
          id="auth"
          userName="guest"
          />
     <mx:UIComponent width="500" height="500" id="centeredSubscriber" x="200"/>
     <session:ConnectSessionContainer id="cSession" authenticator="{auth}" width="100%" height="100%" roomURL="https://collaboration.adobelivecycle.com/userName/roomName">
          <collaboration:WebcamPublisher width="1" height="1" id="webCamPub"/>
          <mx:Label text="Click on a Subscriber thumbnail to make it bigger." />
          <mx:HBox width="100%" height="200" horizontalAlign="center" verticalAlign="top" id="smallSubscriberContainer" creationComplete="cSession_synchronizationChangeHandler(event)"/>
          <mx:Button  id="startBtn" label="Start"  click="startBtn_clickHandler(event)" height="20"/>
     </session:ConnectSessionContainer>
</mx:Application>

I have attached the src files of player 10 & 10.1

Avatar

Employee

Hey Trace,

We are still looking into this issue, infact some of the best minds within the team are trying to optimize and solve the ratcheting problem. I would update you if we have any updates.

Thanks

Arun

Avatar

Level 3

Thanks again - looking forward to what you all find out!

-Trace

Avatar

Former Community Member

Hi Trace,

Thanks for the ongoing reports - we'd noticed last week that somehow the

impact of the ratcheting was affected by turning the publisher on (worse) or

off (better). But I haven't yet seen the level of problems you're seeing -

on our side, when using publisherIDs rather than replacing the

WebcamSubscriber, the impact takes 100+ clicks to be even in the 30% range.

Waiting for a time seems to reduce itself though.

We're still looking into this, especially the impact of the publisher and

why destroying and creating new subs makes the difference it does.

nigel

Avatar

Level 3

Thanks again for all the work you're doing to debug this.

The CPU consistently ratchets quite quickly here on the sample app, on all our computers.  Trying to think through what might be different between your setup and ours.  Are you closer to the LCCS servers than us?  If so, could that potentially be exacerbating the issue in our case?

If it's worthwhile for your testing, we can set up some of our computers here with a screen share while you're debugging so you can reproduce, analyze and fix the problem more easily.

-Trace

Avatar

Level 3

Hi Nigel & co.,

Thanks again for all the great improvments to date.

Will there be any additional fixes to reduce the remaining significant CPU ratcheting/spiking effects before the upcoming release, including the ratcheting associated with creating and destroying publishers & subscribers but especially the ratcheting that occurs with simple switching such as in your sample app as documented by our above experiment?

Kind regards,

Trace

Avatar

Level 3

Hi there,

We installed the swc from the latest release, and ran additional tests, still seeing the significant ratcheting effect.  We had 4 cameras on in a room, along with audio.  The first one was displayed on a webcamsubscriber to the other 3, the entire time.  For the other 3 cameras, we switched bewteen them on a second WebcamSubscriber, switching once per second 10 times each.  When displayed on the WebcamSubscriber, the CPU had spiked to 90% for each of them.  The first time they were displayed on the WebcamSubscriber, the CPU was at 50%. 

So 30 switches (10 times each) ratcheted their CPUs up by 40%.

This is still causing our users' flash players to crash much more often than we'd like.  You mentioned a few times you guys were working on fixes; now that your release is out, is there another SWC we could download with just enough of a fix to this issue so that our users' flash players would crash less?

Thanks very much,

-Trace

Avatar

Employee

Hi Trace,

All the optimizations we found, was included in our last release 1.5. We are yet to discover newer performance optimizations, and would update you as soon as we have one. Thanks for your patience on this issue.

Thanks

Arun

Avatar

Former Community Member

Hi Trace,

We're definitely still looking at this - nothing looks too incriminating so

far, and I'm starting to worry you're encountering a Player bug. As you

note, we're done with this release cycle, and our time is more freed up for

further research here.

Are you guys using publisherIDs alone for switching, without re-creating and

destroying webcamsubscribers as you go? If you're creating/destroying, the

issue is still apparent, but not really much at all for just re-assigning

publisherIDs. Let us know if you're seeing something different.

Could you explain this pair of sentences further?

When displayed on the WebcamSubscriber, the CPU had spiked to 90% for each of

them.  The first time they were displayed on the WebcamSubscriber, the CPU was

at 50%. 

Are you saying that the CPU spiked to 50% right away? For whom? I'm

guessing the 90% is after 10 switches.

thanks for your patience,

nigel

Avatar

Level 3

Hi Nigel,

I think I may be seeing this exact same problem with the 10.3 library posted last week. I ran a test last night with about 10 users publishing in a room and it seemed like all of the flash players were crashing.

I just ran a test today and saw the same problem with 4 cameras publishing to the same room.

Seems like it's fine if I"m just publishing two or three cams, but I saw the CPU usage on one of my clients go up to 95% today on a 4 cam test.

In case it helps... each publishing user can only be subscribing to one of the cams at a time. When the subscriber side switches cams I am calling the .close() method so there shouldn't be any leak in how I'm using the API from what I can see.

I can try the same test by reverting back to the old library if it helps.

Thanks,

Barry

Avatar

Level 3

Hi Nigel,

Just to clarify for my case the questions you asked in the previous post.

Each client has a subscriber to each of the other clients, but only one is active at any given time. When a subscriber becomes inactive, the .close() method is called before the client starts subscribing to the other publisher.

Also... I have one additional subscriber that switches between the publishers by changing publisher Ids. I never call the .close() method on that one because it's always subscribing to someone.

Hope this helps,

Barry

Avatar

Former Community Member

Hi guys,

I think I may finally have gotten this one... Could you please help validate?

Go to :

http://blogs.adobe.com/collabmethods/files/2011/06/WebcamRatchet.swf

with as many cams as you can. Click on each thumbnail back and forth over and over again, as per usual.

prior results : I could drive the CPU past 100%, and get it to stay there.

after the fix : I can't get the CPU past 30%, and it drops down to 11% when done clicking about.

  Let me know if you're seeing the same results, and we'll get a new SWC build for more testing.

  thanks!

  nigel

Avatar

Level 3

Hi Nigel,

Awesome. I'll run my reproducible case tomorrow afternoon.

I owe you a Boddingtons :-)

Barry

Avatar

Former Community Member

Just be sure to stick to this particular swf - it's got the magic fix (it's

one line, which shouldn't make a difference, but really does somehow).

nigel

Avatar

Level 3

A million thanks, Nigel!  There was no ratcheting using this SWF.  Please provide the swc and we'll test it out with our codebase.

We're using the 10.3 swc with AEC locally, but can use any SWC to test this today.

Kind regards,

-Trace

Avatar

Level 3

Hi Nigel,

We just tried it with the 5 cams we've had problems with in our app, plus an additional 6th cam and we got no racheting either. There was also an extra naked dude cam in there... you might have a new Chatroulette on your hands if you don't take that down soon :-)

When you are rebuilding the swc, will it be the version that doesn't do peer assisted networking? That's the swc we are currently using in our app. If you want us to test with a patch swc, we can do this test any time.

Thanks for the fix. Looking forward to hearing about that line of code.

Barry

Avatar

Level 3

Hi Nigel,

If you have a swc or source code handy I can test this with, I'm currently in our lab; we have multiple cams and computers here and can provide some detailed data of how the new code performs in the wild.

Thanks again - very excited for these changes!

-Trace

Avatar

Former Community Member

Ok.. I'm no build master, so I'm struggling a little with ANT hell to get a SWC ready. Should be a couple more minutes...

  nigel

Avatar

Former Community Member

Alright, this SWC seems to have the goods. It's based off of the Player 10.0 LCCS.SWC - we'll get you a 10.3 version on Monday.

Let me know how it goes.

  nigel