User Synchronization vs StorageResourceProvider API

[Gdubz-57m2mu]

Hi all, just wanted to get everyone’s opinion on User Synchronization[1] (using Sling Content Distribution[2] (SCD)) in AEM 6.2.

Right now in AEM 5.6.1 we’re using some custom Forward Replication/Reverse Replication (FR/RR) for all UGC, including user profile preferences. From our experience thus far, this is probably the worst thing in the world, and we’ve only got about 15-20k active users.

I went ahead and setup User Synchronization in my local dev environment (1 author, 2 publishers) and it’s working out pretty well for any preference modifications made under the rep:User node. I’m concerned when we scale back up to the number of active users we have in PROD right now, if SCD will encounter any of the performance problems that FR/RR had. I can only hope that the answer is no but I don’t know for sure. My guess is that SCD doesn’t have all of the overhead that FR/RR has from the AEM interface using replication agents, workflow instances, leaving behind var/audit logs, etc.

I also wanted to mention that we’re planning to implement MSRP[3], a MongoDB server for the rest of our UGC needs (Comments, Blogs, Notifications, etc) and wanted to get your thoughts on possibly storing the profile preferences there. Would that work? What would it look like? Has anyone tried it or had experience with it? etc.

From my limited understanding of the StorageResourceProvider API[4] (SRP API), reading and writing to MongoDB UGC structure is on a per-component basis, right? So:

/content/my-project/en/home/profile/jcr:content/profile-details-component

...would then become:

/content/usergenerated/asi/mongo/content/my-project/en/home/profile/jcr:content/profile-details-component

...and each individual entry beneath that would have a properties that tie it to a specific user? I don’t imagine there’s a way to actually have it be a shadow node of the user (instead of the component) would there? e.g., Actual Profile Node: /home/users/t/test-user vs. Mongo Shadow Node: /content/usergenerated/asi/mongo/home/users/t/test-user

Thanks so much for any and all input!
- Greg

[1] https://docs.adobe.com/docs/en/aem/6-2/administer/security/security/sync.html
[2] https://github.com/apache/sling/blob/trunk/contrib/extensions/distribution/README.md
[3] https://docs.adobe.com/docs/en/aem/6-2/administer/communities/srp/msrp.html
[4] https://docs.adobe.com/docs/en/aem/6-2/develop/communities/essentials/srp.html

[JK_Kendall]

Hi,

I'm moving your post to the AEM Communities topic for more visibility.

- JK

[Bertrand_de_Coa]

hello Greg, great questions/thoughts.

SCD has been designed and tested with much larger number of "members" (as we tend to name publish side users). It's much more lighweight than before, and only uses the author instance as a pivot (content is never stored as such on author). Also, please note that compared to previous versions of AEM Communities, the user node is quite lightweight also since we "moved" all the associated UGC into SRP storage (activities, messages, notifications especially), so you're down to user node + profile preferences + avatar maybe. 

Why would you want profile preferences to go in SRP also? This will cause some issue with some of our components (like profile), but also prevents other AEM features to work (segments, context hub...)

Cheers

Bertrand

[Jitendra_S_Toma]

Hi,

In my personal opinion, Sling content distribution does not solve major problem client faces however could be useful for syncing lighter content like user profile syncing. But if you are looking for havy UGC content syncing then external storage would be better option.

Sling Content distribution does not seem feasible option in the case of content needs to be pushed back to author. concept seems more like same as replication but in better form.

Regards,

Jitendra

[Gdubz-57m2mu]

Bertrand de Coatpont wrote...

hello Greg, great questions/thoughts.

SCD has been designed and tested with much larger number of "members" (as we tend to name publish side users). It's much more lighweight than before, and only uses the author instance as a pivot (content is never stored as such on author). Also, please note that compared to previous versions of AEM Communities, the user node is quite lightweight also since we "moved" all the associated UGC into SRP storage (activities, messages, notifications especially), so you're down to user node + profile preferences + avatar maybe. 

Why would you want profile preferences to go in SRP also? This will cause some issue with some of our components (like profile), but also prevents other AEM features to work (segments, context hub...)

Cheers

Bertrand 

Thanks for the reply Bertrand. The reason I was even considering SRP as an option over UserSync/SCD is because right now in PROD we have a completely custom implementation of comments, blogs, notifications, and more (we wrote it all from scratch for 5.5/5.6.1 before AEM provided Communities as it's known today). On the user profile node, we're storing things like:

• Personal bookmarks to important pages (yes, in-site instead of browser, I know how bad that is, but that's what the client wanted).
• Blog subscriptions (to receive emails on new blog posts or comment replies)
• General site preferences (how frequently to receive updates/emails, etc).
• User homepage configuration (we have a fairly dynamic homepage which users can drag and drop elements around and we save that customization in a json string)
• Profile details from LDAP (department, location, phone, etc)

We're currently storing their profile photos/avatars in "/content/dam/usergenerated/profile-images/*" so that part could probably be done via SRP. The point I'm trying to make though is that our user profile node isn't exactly super lightweight and a single user may frequently update their profile node: homepage customization changes causes much of our FR/RR headache right now.

We would like to get to a point where we're using all of the OOTB Communities social components instead of this completely custom implementation, but that's going to take some time. Honestly, if Adobe hadn't deprecated and removed the com.day.cq.collab.commons package, we probably wouldn't even be upgrading this code right now (luckily, they did).

Jitendra S.Tomar wrote...

Hi,

In my personal opinion, Sling content distribution does not solve major problem client faces however could be useful for syncing lighter content like user profile syncing. But if you are looking for havy UGC content syncing then external storage would be better option.

Sling Content distribution does not seem feasible option in the case of content needs to be pushed back to author. concept seems more like same as replication but in better form.

Regards,

Jitendra

Thank you so much for your input Jitendra!

So we should definitely try to move some of this user-specific data off of the user profile node into something readable/writable in Mongo (MSRP)?

1. What do you consider to be lightweight content?
2. What do you consider to be heavy UGC?

Just trying to get an idea if you're talking about a simple json string like { 'department' : 'human resources' } or if you're talking about many nodes/properties, maybe even files?

[dwalling · Accepted solution]

As general guidance, I would recommend using the profile tree in JCR only for attributes that change infrequently like passwords, address, possibly site preferences. These are things that are unlikely to change during a given login session and perhaps only change a few times per month or year. This will keep the load on SCD light and avoid an potential bottleneck problems. Other attributes that will change during a login session should be stored through the SRP APIs into the configured SRP storage. This strategy is demonstrated within the product by the approach we're taking with the gamification data (scores, badges), messages, activitystreams, subscriptions and notifications, etc.

I'm not sure if we have sufficient samples for this strategy, but if you continue to ask questions here we'll try to help out and continue to improve the samples. Some possibly useful links:

• How to develop an SCF+SRP component from scratch: https://github.com/Adobe-Marketing-Cloud/aem-communities-todomvc-sample
• A repo of SRP tools, currently just containing a way to clean out UGC: https://github.com/Adobe-Marketing-Cloud/aem-communities-srp-tools
• The beginnings of a UI for browsing UGC in SRP: https://github.com/Adobe-Marketing-Cloud/aem-communities-srp-browser

HTH,

Don

[Gdubz-57m2mu]

dwalling wrote...

As general guidance, I would recommend using the profile tree in JCR only for attributes that change infrequently like passwords, address, possibly site preferences. These are things that are unlikely to change during a given login session and perhaps only change a few times per month or year. This will keep the load on SCD light and avoid an potential bottleneck problems. Other attributes that will change during a login session should be stored through the SRP APIs into the configured SRP storage. This strategy is demonstrated within the product by the approach we're taking with the gamification data (scores, badges), messages, activitystreams, subscriptions and notifications, etc.

I'm not sure if we have sufficient samples for this strategy, but if you continue to ask questions here we'll try to help out and continue to improve the samples. Some possibly useful links:

• How to develop an SCF+SRP component from scratch: https://github.com/Adobe-Marketing-Cloud/aem-communities-todomvc-sample
• A repo of SRP tools, currently just containing a way to clean out UGC: https://github.com/Adobe-Marketing-Cloud/aem-communities-srp-tools
• The beginnings of a UI for browsing UGC in SRP: https://github.com/Adobe-Marketing-Cloud/aem-communities-srp-browser

HTH,

Don

This is extremely helpful and satisfies my initial concerns/questions from the original post, thank you Don!

One thing that would be helpful is if the samples could be revisited when the Communities API is updated, not just the documentation. Or maybe even branch the sample projects on github for each major/minor version (not necessarily patch versions), so you don't lose the working samples for people who haven't upgraded yet.

Just a thought!