oak-run Compaction Failing for Lack of Diskspace

[jforney]

Trying to run TarMK compaction on AEM v6.2.

Here is the command I'm giving it:

java -Dtar.memoryMapped=true -Dupdate.limit=5000000 -Dcompress-interval=20000000 -Dcompaction-progress-log=1500000 -Dlogback.configurationFile=logback.xml -Xmx12g -jar oak-run-1.4.5.jar compact /opt2/crx/author/crx-quickstart/repository/segmentstore

Here is what I'm getting...

      Tue Sep 27 22:30:58 EDT 2016, data00393a.tar
        Fri Jan 13 23:25:52 EST 2017, data03226a.tar
        Tue Sep 27 22:31:33 EDT 2016, data00394a.tar
        Fri Jan 13 21:20:47 EST 2017, data02785a.tar
        Thu Dec 08 14:54:56 EST 2016, data01235a.tar
        Fri Dec 09 12:22:51 EST 2016, data01363a.tar
        Tue Jan 10 03:30:28 EST 2017, data01894a.tar
    size 975.5 GB (975502680735 bytes)
    -> compacting
13:31:29.273 [TarMK disk space check [/opt2/crx/author/crx-quickstart/repository/segmentstore], active since Mon Jan 16 13:31:29 EST 2017, previous max duration 0ms] WARN  o.a.j.o.p.segment.file.FileStore - Available disk space (75.2 GB) is too low, current repository size is approx. 974.6 GB
13:31:29.273 [main] WARN  o.a.j.o.p.segment.file.FileStore - TarMK GC #0: compaction canceled: Not enough disk space available
    -> cleaning up
13:32:31.924 [TarMK disk space check [/opt2/crx/author/crx-quickstart/repository/segmentstore], active since Mon Jan 16 13:32:31 EST 2017, previous max duration 0ms] WARN  o.a.j.o.p.segment.file.FileStore - Available disk space (75.2 GB) is too low, current repository size is approx. 974.6 GB
Compaction failure stack trace:
java.io.IOException: No space left on device
        at java.io.RandomAccessFile.writeBytes(Native Method)
        at java.io.RandomAccessFile.write(RandomAccessFile.java:525)
        at org.apache.jackrabbit.oak.plugins.segment.file.TarWriter.writeEntry(TarWriter.java:235)
        at org.apache.jackrabbit.oak.plugins.segment.file.TarWriter.writeEntry(TarWriter.java:221)
        at org.apache.jackrabbit.oak.plugins.segment.file.TarReader.cleanup(TarReader.java:816)
        at org.apache.jackrabbit.oak.plugins.segment.file.FileStore.cleanup(FileStore.java:895)
        at org.apache.jackrabbit.oak.run.CompactCommand.execute(CompactCommand.java:106)
        at org.apache.jackrabbit.oak.run.Mode.execute(Mode.java:63)
        at org.apache.jackrabbit.oak.run.Main.main(Main.java:49)
Compaction failed in 13.71 min (822s).
bash-4.1$

So what space is it talking about.  Is it java.io temp space?  I tried adding -Djava.io.tmpdir=/opt3/temp to the JVM parms for the oak-run to volume with loads of space but it had no effect on the outcome.  I know that the volume where the repository resides is very low on disk space; that is why I want to do the compaction.

[andrewj40704587]

Hello,

It's talking about actual diskspace on your local filesystem. Your repository is nearly 1 TB, and there is only 72 GB of free disk space. Oak run needs disk space to work with as part of the compaction - I don't remember exact numbers, but I think you need at least 10% or 20% of your repository size of free disk space or it won't even run. 

I hope that helps,

Andrew

[jforney]

Yep we added more disk and then it stopped complaining about not having enough disk to operate with but then the darn thing failed saying it couldn't read one of the .tar files.  Might you also know about this.  I was figuring on giving up on it for now and putting in a ticket with Adobe.

[‎1/‎16/‎2017 2:58 PM] Forney, James D:

[root@aw01emcpappp013 opt3]# tail -f jcr-compact1.txt

[root@aw01emcpappp013 opt3]# tail -f jcr-compact1.txt

        Fri Jan 13 23:10:32 EST 2017, data03183a.tar

        Tue Sep 27 22:30:58 EDT 2016, data00393a.tar

        Fri Jan 13 23:25:52 EST 2017, data03226a.tar

        Tue Sep 27 22:31:33 EDT 2016, data00394a.tar

        Fri Jan 13 21:20:47 EST 2017, data02785a.tar

        Thu Dec 08 14:54:56 EST 2016, data01235a.tar

        Fri Dec 09 12:22:51 EST 2016, data01363a.tar

        Tue Jan 10 03:30:28 EST 2017, data01894a.tar

    size 997.0 GB (997044939423 bytes)

    -> compacting

Compaction failure stack trace:

java.io.IOException: Failed to open tar file /opt4/crx/author/crx-quickstart/repository/segmentstore/data02443b.tar

        at org.apache.jackrabbit.oak.plugins.segment.file.TarReader.openRO(TarReader.java:168)

        at org.apache.jackrabbit.oak.plugins.segment.file.FileStore.<init>(FileStore.java:449)

        at org.apache.jackrabbit.oak.plugins.segment.file.FileStore.<init>(FileStore.java:91)

        at org.apache.jackrabbit.oak.plugins.segment.file.FileStore$ReadOnlyStore.<init>(FileStore.java:1430)

        at org.apache.jackrabbit.oak.plugins.segment.file.FileStore$ReadOnlyStore.<init>(FileStore.java:1425)

        at org.apache.jackrabbit.oak.plugins.segment.FileStoreHelper.openReadOnlyFileStore(FileStoreHelper.java:226)

        at org.apache.jackrabbit.oak.plugins.segment.FileStoreHelper.checkFileStoreVersionOrFail(FileStoreHelper.java:184)

        at org.apache.jackrabbit.oak.plugins.segment.FileStoreHelper.openFileStore(FileStoreHelper.java:211)

        at org.apache.jackrabbit.oak.run.CompactCommand.execute(CompactCommand.java:79)

        at org.apache.jackrabbit.oak.run.Mode.execute(Mode.java:63)

        at org.apache.jackrabbit.oak.run.Main.main(Main.java:49)

Compaction failed in 4.075 min (244s).

[andrewj40704587]

Last time I had issues like that, they had me delete the index, run a consistency check to find the last good revision, edit the journal.log to revert the repository to that revision, then run compaction again. You'll probably want to open an Adobe ticket.