Expand my Community achievements bar.

SOLVED

TarMK GC: Failed to remove old file

Avatar

Level 5

Hi all,

Left my local dev instance running overnight and now every 5 seconds the following 3 lines are spammed in my error log.

16.11.2016 12:41:24.368 *WARN* [TarMK flush thread [C:\adobe\aem62publish\crx-quickstart\repository\segmentstore], active since Wed Nov 16 12:41:24 CST 2016, previous max duration 91775ms] org.apache.jackrabbit.oak.plugins.segment.file.FileStore TarMK GC: Failed to remove old file C:\adobe\aem62publish\crx-quickstart\repository\segmentstore\data00017a.tar. Will retry later. 16.11.2016 12:41:24.368 *WARN* [TarMK flush thread [C:\adobe\aem62publish\crx-quickstart\repository\segmentstore], active since Wed Nov 16 12:41:24 CST 2016, previous max duration 91775ms] org.apache.jackrabbit.oak.plugins.segment.file.FileStore TarMK GC: Failed to remove old file C:\adobe\aem62publish\crx-quickstart\repository\segmentstore\data00015a.tar. Will retry later. 16.11.2016 12:41:24.368 *WARN* [TarMK flush thread [C:\adobe\aem62publish\crx-quickstart\repository\segmentstore], active since Wed Nov 16 12:41:24 CST 2016, previous max duration 91775ms] org.apache.jackrabbit.oak.plugins.segment.file.FileStore TarMK GC: Failed to remove old file C:\adobe\aem62publish\crx-quickstart\repository\segmentstore\data00012a.tar. Will retry later.

Would manually deleting these files clear this up? I don't want to risk losing data but if they're just trying to be deleted by AEM, then I might as well. Also, how do I stop this from happening again? Is it a desktop file permissions issue?

Thanks for any help!

1 Accepted Solution

Avatar

Correct answer by
Community Advisor

You could enable debug log on the org.apache.jackrabbit.oak.plugins.segment to see what it was attempting to do with your files

May be something in your log would suggest you whether it is safe to delete these (AEM might have tried to delete them, you would simply help to delete these manually)

But it does not feel right to delete files manually, better check permissions or anything else that can prevent file.delete() from execution.

Suspect, you are hitting this code:

// remove all obsolete tar generations Iterator<File> iterator = pendingRemove.iterator(); while (iterator.hasNext()) { File file = iterator.next(); log.debug("TarMK GC: Attempting to remove old file {}", file); if (!file.exists() || file.delete()) { log.debug("TarMK GC: Removed old file {}", file); iterator.remove(); } else { log.warn("TarMK GC: Failed to remove old file {}. Will retry later.", file); } }

In which case it would be fairly safe to delete files.

As they are within the list of:

/** * List of old tar file generations that are waiting to be removed. */ private final LinkedList<File> toBeRemoved = newLinkedList();

P.S. really good talk from this years adaptto[1] [2] might help too

[1] https://adapt.to/content/dam/adaptto/production/presentations/2016/adaptTo2016-Into-the-tar-pit-a-Ta...

[2] https://adapt.to/content/dam/adaptto/production/presentations/2016/adaptTo2016-Into-the-tar-pit-a-Ta...

Regards,

Peter

View solution in original post

3 Replies

Avatar

Correct answer by
Community Advisor

You could enable debug log on the org.apache.jackrabbit.oak.plugins.segment to see what it was attempting to do with your files

May be something in your log would suggest you whether it is safe to delete these (AEM might have tried to delete them, you would simply help to delete these manually)

But it does not feel right to delete files manually, better check permissions or anything else that can prevent file.delete() from execution.

Suspect, you are hitting this code:

// remove all obsolete tar generations Iterator<File> iterator = pendingRemove.iterator(); while (iterator.hasNext()) { File file = iterator.next(); log.debug("TarMK GC: Attempting to remove old file {}", file); if (!file.exists() || file.delete()) { log.debug("TarMK GC: Removed old file {}", file); iterator.remove(); } else { log.warn("TarMK GC: Failed to remove old file {}. Will retry later.", file); } }

In which case it would be fairly safe to delete files.

As they are within the list of:

/** * List of old tar file generations that are waiting to be removed. */ private final LinkedList<File> toBeRemoved = newLinkedList();

P.S. really good talk from this years adaptto[1] [2] might help too

[1] https://adapt.to/content/dam/adaptto/production/presentations/2016/adaptTo2016-Into-the-tar-pit-a-Ta...

[2] https://adapt.to/content/dam/adaptto/production/presentations/2016/adaptTo2016-Into-the-tar-pit-a-Ta...

Regards,

Peter

Avatar

Level 10

We will also look at adding this subject to AEM Ask the experts webinar as this comes up a lot. 

Avatar

Level 5

Thanks for the advice PuzanovsP, I'll give it a try next time it happens. Shutting down and restarting my local instance seemed to resolve the log spam part of the issue (for now). Maybe there was some "can't delete, file still in use" situation going on.

Sounds good, Scott!