LuceneBinariesCleanup task in "Daily maintenance job" is taking too long.

[prsn1034]

Hi Community,

The task named "Lucene Binaries Cleanup" in Daily Maintenance job is taking lot of time to complete. Below is the statistics extracted from useraudit.log file for same. 

Weekly Job Run	Daily Job Run	Time Taken by LuceneBinariesCleanup Task
	24/07/2024, Wed	9.31Hrs
	25/07/2024, Thu	10.16Hrs
	26/07/2024, Fri	9.30Hrs
27/07/2024, Sat - Weekly Maintenance Task	27/07/2024, Sat	7.09Hrs
	28/07/2024, Sun	7.43Hrs
	29/07/2024, Mon	14.48Hrs
	30/07/2024, Tue	15:20Hrs
	31/07/2024, Wed	16.20Hrs
	01/08/2024, Thu	05:03Hrs
	02/08/2024, Fri	05:41Hrs
03/08/2024, Sat - Weekly Maintenance Task	03/08/2024, Sat	05:46Hrs
	04/08/2024, Sun	08.30Hrs
	05/08/2024, Mon	09.34Hrs
	06/08/2024, Tue	7.40Hrs
	07/08/2024, Wed	9.20Hrs
	08/08/2024, Thu	6.47Hrs
	09/08/2024, Fri	1.50hrs
10/08/2024, Sat - Weekly Maintenance Task	10/08/2024, Sat	1.47hrs
	11/08/2024, Sun	2.24hrs
	12/08/2024, Mon	8.56Hrs

Below is my case,

* AEM on-premise

* Happens only on PROD Author environment, but works normally on lower environment

* Also reproducible on Cloned PROD environment, just to check if traffic is the issue.

* VersionPurge task is running weekly, but Health reports says it has to be executed daily. However, tried scheduling it on daily basis on Cloned PROD env, but in vain.

Most importantly, this job generates the .tmp files in the underlying /tmp folder and causes overflow issues 

ref : https://experienceleaguecommunities.adobe.com/t5/adobe-experience-manager-assets/tmp-folder-is-full/...

Question : 

1. Is this normal for the given task to take hours?

2. Why sometime it finishes in 1-2hrs and sometime it lasts for 10-15+ hours?

3. Last but not the least, any ideas how I can optimize it?

4. Even after successful Lucene Binaries cleanup, why the next job run takes huge amount of time?

5. Gone through documentation, but didn't help much. 

Because, this job consumes /tmp folder's 100% space, other tasks like package creation using /crx/packmgr OR content authoring or content publishing gets blocked.

Help suggestions are welcomed.

KR,

Prasanna

[kautuk_sahni]

@Tad_Reeves @aanchal-sikka @Raja_Reddy @BrianKasingli Could you kindly take a look at this question and provide your thoughts? Your insights would be greatly appreciated.

[PrasannaMo]

Hi @kautuk_sahni Thanks for looking into it. I got a fix from Adobe Support(Rohith Venati) on this..Default value of the property "blobTrackSnapshotIntervalInSecs" from "org.apache.jackrabbit.oak.segment.SegmentNodeStoreService.config"  was the root cause. I was advised to set it to ZERO, tested this on multiple times on multiple Clones PROD AUthor environments and moved to PROD last week.