How can I enable private/gated content crawling in AEM ? Hi Team
We have a third-party search engine that will crawl the pages in AEM, index them, and then provide the search results. There are a few pages that are behind authentication. We would need the crawler to crawl these pages. We can achieve this by skipping the SAML authentication for these pages. I tried to write some Servlet Filters, so that, the request will come to Filter before authentication. As per https://github.com/Adobe-Consulting-Services/acs-aem-samples/issues/63 this feature no more works in the latest AEM versions. I am looking for a better solution.
Update:- When I say behind authentication, it is the OOTB Authentication Enabled checkbox in the page properties.
My requirement
1. Third-party Search engine will crawl the pages.
2. It will have an identifier in the User Agent which will allow us to identify if the request is from the crawler or not.
3. If the User Agent has the identifier, I need to skip the S