How many Sling Jobs is too much? Or how to tune performance for a backend script that loops over /content/mysite.

[jeremylanssiers]

I have a long running script that loops over multiple country pages, their language roots and hundreds of pages below each language root. It runs scheduled overnight, but it can also be run manually during the day for one language root at a time.

I would like to transform this into a small script that spawns Jobs for each country, each language and perhaps even each page. 

But I am worried I might end up with too many Sling Jobs, making matters worse. Hoe many Sling Jobs is too much? What would be a good approach for this kind of scenario?

[arunpatidar]

The optimal number of active Sling Jobs in Adobe Experience Manager (AEM) depends on various factors, and there is no fixed number that suits all scenarios. The number of active Sling Jobs that is considered "good" depends on factors such as system resources, job complexity, and performance requirements.

Here are some considerations to help you determine the optimal number of active Sling Jobs:

1. System Resources: Consider the available system resources, including CPU, memory, and disk I/O. If the system is resource-constrained, running too many concurrent jobs might lead to performance issues.

2. Job Complexity: The complexity of the jobs being executed is an important factor. Some jobs may be computationally intensive or involve resource-intensive operations. Consider the nature of the jobs and how they impact system resources.

3. Job Queues: Sling Jobs are often processed through Job Queues in AEM. Each queue has a maximum parallel setting that limits the number of jobs processed concurrently. Adjust the maximum parallel setting based on system performance and requirements.

I think you should start with less umbr of jobs and rely on JCR Queries to filter the pages and perform operations only on those pages.


We had similar kind of requirement where we had to check all of the pages for link type and adjusted those like via a sling job.

We had only 1 sling job for three language roots that runs a query and return only the impacted subset pages and perform adjustment.

[Jörg_Hoh · Accepted solution]

As already mentioned by Arun, the system can be a guiding principle. But for the granularity the overall system resources do not play that much a role, because you always restrict the number of concurrent executions of these jobs by the queue configuration.

I would consider these factors when designing the granularity:

1. what is the logical granularity of the tasks to be executed? Is the scope just the page or rather a site? Can you implement improvements when you are handling multiple pages in a single job (for example because you need to resolve other information, and you can do that once for each page, or once for a larger set of pages).
2. What is the expected error rate? Whenever you retry, the job starts from Zero. Of course your implementation can make sure, that it can benefit from results of previously executed, but eventually failed job, but that's not always easy to implement. So design your jobs, that you don't waste the work of hours, when it fails on the 2nd last page.
3. Another factor: If your jobs are too large, every restart of the system executing this job will restart the job itself. For example in AEM CS instances can live long (> 6h) but also only for a few minutes.

*