Dear Developers/Architects,
I was trying to create packages programatically in AEM, it fetchs given page's path and page's reffrences like xf paths and dampaths(more than 1500 dampaths), i combined content path and xf path in seprate package(package-0). when comes to dampath i splited the dampath in to 300 dampath/package and tried to create packages(package-1,package-2 & viceversa..). when i ran this code in my local aem instance, i was able to create spilt of packages(combined (xf & content paths) package and split of dampath packages) without any issues. when i deploy the code on AEMasCS, im getting gateway timeout issue. Kindly let us know your suggestion on this...
java code if the dampath more than 300
Topics help categorize Community content and increase your ability to discover relevant content.
Views
Replies
Total Likes
Please check the permission in user admin for that user.if not,apply the permission to the user which ever required.
Hi @Raja_Reddy , thanks for you're response. the code doesn't required any user permission, as im creating package only.not using any system user. So i feel permission may not issue.
@raja-thannasi : How much of content is there on your AEMaaCS instance? If the size is too much, package creation can timeout, please refer this: https://experienceleague.adobe.com/docs/experience-manager-cloud-service/content/implementing/develo...
If possible, try to have less content for your testing to rule out if this is due to content size. In your case, you can try to reduce this to 50 or 100 from current 300 and see if it works.
Hi @Kamal_Kishor , as per adobe recomendation, we should not create large set of packages, is there any limits Adobe have given as i dont see the package size limit on the link you sent., Also the user have full access rights to /etc/packages folder.
@raja-thannasi can you try to save and close the session object in your finally block.
Hi @yashp0808 ,thanks for your response, the gateway timeout issue happened on the line below higlighted in the screenshot attached, where createPackage method. so saving the session will not work in the finally block.
@Kamal_Kishor , the size of the package was not able to size on package manager wizard,as it was incomplete state.please find the screenshot.
Also here , on my Local AEM instance, the same code were working as expected able to package /content path. issue was with on AEMasCS environment.
@raja-thannasi : do you have same content on your local and AEMaaCS instance?
Since your package creation failed, you will not see the size. Are you seeing anything in your logs related to this error?
thanks.
yes @Kamal_Kishor , in my local instance its 280MB. same was not working in AEMascs. In error logs, im not seeing gateway timeout,when i see browser console im getting gateway timeout for the servlet call. logs, i can see it stopped executing on the line packagehelper.createpackage method.
@raja-thannasi : Thanks for providing the details.
Can you try calling your servlet with a user which is part of admin group on AEMaaCS? Please try this and if it works with admin user, then you may have to create the packages with a service-user with permission to create/edit/delete permission under your content paths.
My understanding is based on this link where it says gateway error could be due to user permission or proxy issues.
Please try and let me know.
thanks.
@raja-thannasi Did you find the suggestions from users helpful? Please let us know if more information is required. Otherwise, please mark the answer as correct for posterity. If you have found out solution yourself, please share it with the community.
Views
Replies
Total Likes
Hi @raja-thannasi
A good approach all we should have is a system user with required permissions. Here is the below code which achieves our requirement.
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.util.HashMap;
import java.util.Map;
import javax.jcr.RepositoryException;
import javax.jcr.Session;
import javax.servlet.Servlet;
import org.apache.jackrabbit.vault.fs.api.PathFilterSet;
import org.apache.jackrabbit.vault.fs.api.ProgressTrackerListener;
import org.apache.jackrabbit.vault.fs.config.DefaultWorkspaceFilter;
import org.apache.jackrabbit.vault.packaging.JcrPackage;
import org.apache.jackrabbit.vault.packaging.JcrPackageDefinition;
import org.apache.jackrabbit.vault.packaging.JcrPackageManager;
import org.apache.jackrabbit.vault.packaging.PackageException;
import org.apache.jackrabbit.vault.packaging.PackagingService;
import org.apache.jackrabbit.vault.util.DefaultProgressListener;
import org.apache.sling.api.SlingHttpServletRequest;
import org.apache.sling.api.SlingHttpServletResponse;
import org.apache.sling.api.resource.ResourceResolver;
import org.apache.sling.api.resource.ResourceResolverFactory;
import org.apache.sling.api.servlets.HttpConstants;
import org.apache.sling.api.servlets.SlingAllMethodsServlet;
import org.osgi.framework.Constants;
import org.osgi.service.component.annotations.Component;
import org.osgi.service.component.annotations.Reference;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@Component(service = Servlet.class, property = {
Constants.SERVICE_DESCRIPTION + "= API to download packages based on path",
"sling.servlet.methods=" + HttpConstants.METHOD_GET, "sling.servlet.paths=" + "/bin/api/createpackage" })
public class PackagecreateServlet extends SlingAllMethodsServlet {
private static final long serialVersionUID = 1L;
private transient Logger log = LoggerFactory.getLogger(PackagecreateServlet.class);
@Reference
private transient ResourceResolverFactory resolverFactory;
private void closePackageStream(InputStream packageStream) {
if (packageStream != null) {
try {
packageStream.close();
} catch (IOException e) {
log.error("Error while closing InputStream", e);
}
}
}
private InputStream getPackageStream(JcrPackage pack) throws Exception {
if (pack != null && pack.getNode() != null && pack.getNode().hasProperty("jcr:content/jcr:data")) {
return pack.getNode().getProperty("jcr:content/jcr:data").getBinary().getStream();
}
return null;
}
@Override
protected void doGet(SlingHttpServletRequest request, SlingHttpServletResponse response) throws IOException {
String packageName = request.getParameter("packageName");
String packagePath = request.getParameter("packagePath");
JcrPackageManager jpm = null;
if (packageName == null || packageName.isEmpty() || packagePath == null || packagePath.isEmpty()) {
response.getWriter().write("failed: URL parameters 'packageName' or 'packagePath' are missing");
return;
}
String[] paths = packagePath.replace("[", "").replace("]", "").split(",");
ResourceResolver resourceResolver = null;
Session session = null;
JcrPackage pack = null;
InputStream packageStream = null;
try {
Map<String, Object> param = new HashMap<>();
param.put(ResourceResolverFactory.SUBSERVICE, "subservice");
resourceResolver = resolverFactory.getServiceResourceResolver(param);
session = resourceResolver.adaptTo(Session.class);
jpm = PackagingService.getPackageManager(session);
pack = jpm.create("testPackage", packageName, "1.0.0");
JcrPackageDefinition definition = pack.getDefinition();
DefaultWorkspaceFilter filter = new DefaultWorkspaceFilter();
for (String path : paths) {
PathFilterSet pathFilterSet = new PathFilterSet();
pathFilterSet.setRoot(path.trim());
filter.add(pathFilterSet);
}
boolean autoSave = true;
if (definition != null)
definition.setFilter(filter, autoSave);
ProgressTrackerListener listener = new DefaultProgressListener();
jpm.assemble(pack, listener);
packageStream = getPackageStream(pack);
if (packageStream != null) {
response.setContentType("application/zip");
response.setHeader("Content-Disposition", "attachment; filename=test-dynamic-package-1.0.0.zip");
OutputStream out = response.getOutputStream();
byte[] buffer = new byte[4096];
int bytesRead;
while ((bytesRead = packageStream.read(buffer)) != -1) {
out.write(buffer, 0, bytesRead);
}
out.flush();
} else {
response.getWriter().write("failed: Package content is not available");
}
} catch (PackageException e) {
log.error("Package Exception: " + e.getMessage(), e);
response.getWriter().write("failed: PackageException - " + e.getMessage());
} catch (Exception e) {
log.error("Error while processing request", e);
response.getWriter().write("failed: " + e.getMessage());
} finally {
closePackageStream(packageStream);
if (pack != null) {
try {
jpm.remove(pack);
} catch (RepositoryException e) {
log.error("Package Exception: " + e.getMessage(), e);
}
}
if (resourceResolver != null && resourceResolver.isLive()) {
resourceResolver.close();
}
if (session != null && session.isLive()) {
session.logout();
}
}
}
}
URL how it looks like : http://localhost:portnumber/bin/api/createpackage?packageName=testpackage&packagePath=[/content/abcp... After hitting this URL in browser a package gets downloaded into your local. we can add 'n' number of filters in the array of URL and customize the logic as per our requirement version we used is AEM 6.5. If we comment this piece of code jpm.remove(pack); the package is visible in /crx/packmgr/index.jsp.
Hi @Raja_Reddy ,
as suggested i updated the code but still not working and showing in error log as node path does not exist though its valid path.
Hi @raja-thannasi
Have you created a system user and provided permissions?
Hi @Raja_Reddy , i have given full permission for user like /content,/etc,/conf.
when i ran the code, im getting below error.
9 *INFO* [[0:0:0:0:0:0:0:1] [1706079271697] POST /bin/triggerPackageCreator HTTP/1.1] org.apache.jackrabbit.vault.packaging.PackagingService JcrPackageManager acquired w/o service! Alternate package roots will not be respected.
java.lang.IllegalStateException: null
Hi @Raja_Reddy , Please find the java file.
import java.io.IOException;
import java.io.PrintWriter;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
import java.util.HashMap;
import java.util.HashSet;
import java.util.Iterator;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import java.util.Set;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.stream.Collectors;
import java.util.stream.Stream;
import javax.jcr.RepositoryException;
import javax.jcr.Session;
import javax.servlet.Servlet;
import org.apache.commons.lang3.StringUtils;
import org.apache.jackrabbit.vault.fs.io.AccessControlHandling;
import org.apache.jackrabbit.vault.packaging.JcrPackage;
import org.apache.jackrabbit.vault.packaging.JcrPackageDefinition;
import org.apache.jackrabbit.vault.packaging.JcrPackageManager;
import org.apache.jackrabbit.vault.packaging.PackageException;
import org.apache.jackrabbit.vault.packaging.Packaging;
import org.apache.sling.api.SlingHttpServletRequest;
import org.apache.sling.api.SlingHttpServletResponse;
import org.apache.sling.api.resource.Resource;
import org.apache.sling.api.resource.ResourceResolver;
import org.apache.sling.api.resource.ResourceResolverFactory;
import org.apache.sling.api.servlets.SlingAllMethodsServlet;
import org.apache.sling.servlets.post.JSONResponse;
import org.jetbrains.annotations.NotNull;
import org.osgi.service.component.annotations.Component;
import org.osgi.service.component.annotations.Reference;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import com.adobe.acs.commons.packaging.PackageHelper;
import com.adobe.granite.rest.Constants;
import com.day.cq.wcm.api.Page;
import com.day.cq.wcm.api.PageManager;
import com.google.gson.Gson;
import com.google.gson.JsonObject;
/**
* @author Raja
* <p>
* Servlet that facilitates the creation of content packages. This
* servlet allows clients to create content packages by providing
* relevant details. It extends the SlingAllMethodsServlet, allowing it
* to handle various HTTP methods, primarily POST.
*/
@Component(service = { Servlet.class }, property = { "sling.servlet.paths=" + "/bin/triggerPackageCreator",
"sling.servlet.methods=POST" })
public class PackageCreatorServlet extends SlingAllMethodsServlet {
/**
*
*/
private static final long serialVersionUID = 1L;
// Logger instance for this class
private static final Logger LOGGER = LoggerFactory.getLogger(PackageCreatorServlet.class);
// Constants for package creation messages, default values and resource paths
// List to store paths of child pages for the package
private final List<String> childPagePathList = new ArrayList<>();
// Reference to a helper for package creation
@Reference
private transient PackageHelper packageHelper;
// Reference to the packaging service
@Reference
private transient Packaging packaging;
@Reference
private ResourceResolverFactory resourceResolverFactory;
/**
* Handles the POST request to create a content package. Processes the input
* parameters, interacts with the packaging service, and returns a JSON
* response.
*
* @param request The Sling HTTP request object.
* @param response The Sling HTTP response object.
*/
@Override
protected void doPost(SlingHttpServletRequest request, SlingHttpServletResponse response) {
response.setContentType(JSONResponse.RESPONSE_CONTENT_TYPE);
response.setCharacterEncoding(Constants.DEFAULT_CHARSET);
JsonObject jsonResponse = new JsonObject();
String packageName = request.getParameter(PageConstants.PACKAGE_NAME);
String packageDescription = request.getParameter(PageConstants.PACKAGE_DESCRIPTION);
String pathsList = request.getParameter(PageConstants.PACKAGE_LIST);
String getChildren = request.getParameter(PageConstants.CHILD_PAGE_EXIST);
int damNodeCount = Integer.parseInt(request.getParameter(PageConstants.DAM_NODE_COUNT));
int getPageLevel = request.getParameter(PageConstants.CHILD_PAGE_EXIST).equals(PageConstants.CHILD_PAGE_NO) ? 0
: Integer.parseInt(request.getParameter(PageConstants.PAGE_LEVEL));
List<String> packageArray = new ArrayList<>();
if (StringUtils.isNotEmpty(pathsList)) {
packageArray = collectReferences(request.getResourceResolver(), pathsList, packageName, packageDescription,
getChildren, getPageLevel, damNodeCount);
}
String msg;
if (!packageArray.isEmpty()) {
msg = "Package creation is Successful and path is";
for (String resourcePackagePath : packageArray) {
msg = msg.concat("<br><a href=\"" + resourcePackagePath + "\">" + resourcePackagePath + "</a>");
}
} else {
msg = "Error";
}
jsonResponse.addProperty(PageConstants.MESSAGE, msg);
try (PrintWriter out = response.getWriter()) {
out.print(new Gson().toJson(jsonResponse));
} catch (IOException e) {
LOGGER.error("Unable to process package creation ...", e);
}
}
/**
* Collects references from a given set of paths and creates a package from
* them.
* <p>
* This method identifies content references within the provided page paths and
* aggregates them into different sets (e.g., experience fragments, digital
* assets). Depending on the input parameters, it might also include child
* resources. After collecting all the necessary paths, it triggers package
* creation.
* </p>
*
* @param resourceResolver The resource resolver used to access the content
* repository.
* @param pagePaths A comma-separated list of page paths to collect
* references from.
* @param packageName The desired name of the resulting content package.
* @param packageDescription A brief description for the resulting content
* package.
* @param getChildren A flag indicating whether to include child
* resources. Expected values are "YES" or "NO".
* @param pageLevel The depth level to which child pages should be
* considered.
* @return The path to the created package or an empty string if an exception
* occurred during package creation.
*/
private List<String> collectReferences(ResourceResolver resourceResolver, String pagePaths, String packageName,
String packageDescription, String getChildren, int pageLevel, int damNodeCount) {
Set<String> xfPaths = new HashSet<>();
Set<String> damPaths = new HashSet<>();
ContentVisitor contentVisitor = new ContentVisitor();
Set<String> contentPagePaths = getResourceSet(resourceResolver, pagePaths, pageLevel, getChildren);
if (!contentPagePaths.isEmpty()) {
iterateContentPaths(resourceResolver, xfPaths, damPaths, contentVisitor, contentPagePaths);
return packagePaths(resourceResolver, contentPagePaths, xfPaths, damPaths, packageName, packageDescription,
damNodeCount);
}
return new ArrayList<>();
}
/**
* Generates a list of package paths based on the provided content paths, XF
* paths, DAM paths, package name, package description, and DAM node count. The
* DAM paths will be split into multiple packages if their size exceeds the
* specified damNodeCount.
*
* @param resourceResolver the ResourceResolver to resolve resources
* @param contentPaths a set of content paths to be included in the
* package
* @param xfPaths a set of experience fragment paths to be included
* in the package
* @param damPaths a set of DAM paths to be included in the package
* @param packageName the name of the package
* @param packageDescription the description of the package
* @param damNodeCount the maximum number of DAM nodes allowed in a single
* package
* @return a list of package paths created based on the provided parameters
*/
private List<String> packagePaths(ResourceResolver resourceResolver, Set<String> contentPaths, Set<String> xfPaths,
Set<String> damPaths, String packageName, String packageDescription, int damNodeCount) {
Map<String, String> packageDefinitionProperties = new HashMap<>();
packageDefinitionProperties.put(JcrPackageDefinition.PN_AC_HANDLING,
AccessControlHandling.OVERWRITE.toString());
packageDefinitionProperties.put(JcrPackageDefinition.PN_DESCRIPTION, packageDescription);
Session session = null;
ResourceResolver resolver = null;
Resource thumbnailPath;
Set<Resource> combinedPaths = null;
List<String> packagePaths = new ArrayList<>();
try {
thumbnailPath = resourceResolver.getResource(PageConstants.QUERY_PACKAGE_THUMBNAIL_RESOURCE_PATH);
final Map<String, Object> authInfo = Collections.singletonMap(ResourceResolverFactory.SUBSERVICE,
"demoSystemUser");
resolver = resourceResolverFactory.getServiceResourceResolver(authInfo);
session = resolver.adaptTo(Session.class);
// Combine content and XF paths
combinedPaths = Stream.concat(contentPaths.stream(), xfPaths.stream()).map(resourceResolver::resolve)
.collect(Collectors.toSet());
// Create the main package if DAM paths are within limits or empty
if (damPaths.isEmpty() || damPaths.size() <= damNodeCount) {
combinedPaths.addAll(damPaths.stream().map(resourceResolver::resolve).collect(Collectors.toSet()));
packagePaths.addAll(createAndReturnPackagePath(session, thumbnailPath, combinedPaths, packageName,
packageDefinitionProperties));
} else {
LOGGER.info("1111111111111 QAQAQA#### combinedPaths.size=={}", combinedPaths.size());
LOGGER.info("2222222222222 QAQAQA#### contentPaths.size=={}", contentPaths.size());
LOGGER.info("3333333333333 QAQAQA#### xfPaths.size== {}", xfPaths.size());
LOGGER.info("4444444444444 QAQAQA#### damPaths.size== {}", damPaths.size());
// Create the main package with content and XF paths
packagePaths.addAll(createAndReturnPackagePath(session, thumbnailPath, combinedPaths, packageName,
packageDefinitionProperties));
int packageCount = 1;
for (Set<Resource> damPackageResources : damPaths.stream().map(resourceResolver::resolve)
.collect(Collectors.groupingBy(it -> new AtomicInteger().getAndIncrement() / damNodeCount))
.values().stream().map(HashSet::new).collect(Collectors.toList())) {
String subPackageName = packageName + "-" + packageCount;
packagePaths.addAll(createAndReturnPackagePath(session, thumbnailPath, damPackageResources,
subPackageName, packageDefinitionProperties));
packageCount++;
}
}
} catch (Exception e) {
LOGGER.error("Exception occured in packagePaths method...", e);
} finally {
try {
// Close the Session if it's not null
if (null != session) {
session.logout(); // Or the appropriate closing method
}
} catch (Exception e) {
LOGGER.error("Error closing Session: ", e);
} finally {
try {
// Close the ResourceResolver if it's not null
if (null != resourceResolver) {
resourceResolver.close();
}
if (null != resolver) {
resolver.close();
}
} catch (Exception e) {
LOGGER.error("Error closing ResourceResolver: ", e);
}
}
}
LOGGER.info("combinedPaths.size=={}", combinedPaths.size());
LOGGER.info("contentPaths.size=={}", contentPaths.size());
LOGGER.info("xfPaths.size== {}", xfPaths.size());
LOGGER.info("damPaths.size== {}", damPaths.size());
return packagePaths;
}
/**
* Resolves a set of page paths to their corresponding resources. Based on the
* getChildren flag, it can either directly add the page paths or retrieve the
* children of the pages.
*
* @param resourceResolver The resource resolver to be used for resolving the
* page paths.
* @param pagePaths String containing newline-separated page paths.
* @param pageLevel Depth level for retrieving child pages.
* @param getChildren Flag to determine whether to retrieve children of the
* pages.
* @return Set of resolved resources.
*/
private @NotNull Set<String> getResourceSet(ResourceResolver resourceResolver, String pagePaths, int pageLevel,
String getChildren) {
Set<String> childSet = new HashSet<>();
Set<String> pageSet = new HashSet<>(Arrays.asList(StringUtils.split(pagePaths, '\n'))).stream()
.map(String::trim).collect(Collectors.toSet());
for (String pagePath : pageSet) {
PageManager pageManager = resourceResolver.adaptTo(PageManager.class);
if (null != pageManager && null != pageManager.getPage(pagePath)
&& pageManager.getPage(pagePath).isValid()) {
Page startPage = Objects.requireNonNull(pageManager).getPage(pagePath);
if (StringUtils.equalsIgnoreCase(PageConstants.CHILD_PAGE_YES, getChildren)) {
List<String> childPathList;
childSet.add(pagePath);
int renderDepth = startPage.getDepth() + pageLevel;
childPathList = iteratePageAndChildren(startPage, renderDepth);
childSet.addAll(childPathList);
} else {
childSet.add(pagePath + PageConstants.JCR_CONTENT);
}
} else {
LOGGER.info("{} is not Valid Path.", pagePath);
}
}
return childSet;
}
/**
* Iterates over the children of a given page up to a specified depth.
*
* @param page The starting page for the iteration.
* @param depth The depth up to which the children should be retrieved.
* @return List of child page paths up to the specified depth.
*/
private List<String> iteratePageAndChildren(Page page, int depth) {
Iterator<Page> childPageIterator = page.listChildren();
while (childPageIterator.hasNext()) {
Page childPage = childPageIterator.next();
int childDepth = childPage.getDepth();
if (childDepth <= depth) {
childPagePathList.add(childPage.getPath() + PageConstants.JCR_CONTENT);
}
iteratePageAndChildren(childPage, depth);
}
return Collections.unmodifiableList(childPagePathList);
}
/**
* Iterates over a set of paths and processes each path with a content visitor.
*
* @param resourceResolver The resource resolver to be used for resolving the
* paths.
* @param xfPaths Set to store XF paths obtained from the content
* visitor.
* @param damPaths Set to store DAM paths obtained from the content
* visitor.
* @param contentVisitor The content visitor to process each path.
* @param pages Set of page paths to iterate over.
*/
private void iterateContentPaths(ResourceResolver resourceResolver, Set<String> xfPaths, Set<String> damPaths,
ContentVisitor contentVisitor, Set<String> pages) {
for (String page : pages) {
contentVisitor.accept(resourceResolver.resolve(page));
xfPaths.addAll(contentVisitor.getXfPaths()); // contains experience frament paths
damPaths.addAll(contentVisitor.getDamPaths());// contains dam paths
}
}
/**
* Creates a package with the given resources, package name, and package
* definition properties, and returns the package path in a list.
*
* @param session The ResourceResolver to be used for
* accessing resources
* @param thumbnailPath Thumbnail of the package
* @param packageResources A set of resources to be included in the
* package
* @param packageName The name of the package to be created
* @param packageDefinitionProperties A map of properties to be added to the
* package definition
* @return A list containing the path of the created package
*/
private List<String> createAndReturnPackagePath(Session session, Resource thumbnailPath,
Set<@NotNull Resource> packageResources, String packageName,
Map<String, String> packageDefinitionProperties) {
PackageHelper pckHelper = this.packageHelper;
Packaging lPackaging = this.packaging;
List<String> packagePathArray = new ArrayList<>();
JcrPackageManager packageManager;
JcrPackage jcrPackage = null;
try {
if (session == null) {
return Collections.emptyList(); // Return empty list if session is not available
}
jcrPackage = pckHelper.createPackage(packageResources, session, PageConstants.DEFAULT_GROUP_NAME,
packageName, PageConstants.DEFAULT_VERSION, PackageHelper.ConflictResolution.Replace,
packageDefinitionProperties);
packageHelper.addThumbnail(jcrPackage, thumbnailPath);
packageManager = lPackaging.getPackageManager(session);
packageManager.assemble(jcrPackage, null);
packagePathArray.add(Objects.requireNonNull(jcrPackage.getNode()).getPath());
} catch (IOException | RepositoryException | PackageException e) {
LOGGER.error("Exception occurred during package createAndReturnPackagePath method ", e);
} finally {
if (null != jcrPackage) {
jcrPackage.close();
}
}
return packagePathArray;
}
}
Views
Likes
Replies
Views
Likes
Replies
Views
Like
Replies