Exporting Content into 3rd party search | Community
Skip to main content
Sean-McK
Sean-McK
Level 4
February 8, 2023
Solved

Exporting Content into 3rd party search

  • February 8, 2023
  • 3 replies
  • 1269 views

Hey All,

 

I am trying to figure out the best way to export pages and their content into a 3rd party search company. The data needs to look like this :

 

{ "op": "add", "path": "/path/to/page", "value": { "attributes": { "title": "Page Title", "url": "https://www.website.com/path/to/page.html", "description": "This is basiclly all of the content on the page. So if there is 2 different text area's on the page it should put that content inside this description." } } }

 

 

I know I can ask the page for everything including a description, but that doesn't account for things on the page. For example, lets say I put a new text component on the page and I added text that I want searched on, then it wouldn't pull that data. I started to look into jSoup (and httpclient since I am a SPA), to crawl the page, but is that the best option?

 

Thanks for anyone who has an opinion

This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.
Best answer by Sean-McK

This is only good if you are running outside of AEM. I realized a couple of things...one of them was using HTMLUnit and Jsoup to scrape and parse. I decided to use content services and the json model to parse. 

3 replies

Kiran_Vedantam
Community Advisor
Kiran_VedantamCommunity Advisor
Community Advisor
February 8, 2023

Hi @sean-mck 

 

You can use the AEM JSON Exporter functionality that comes OOTB from core components to expose the content as a service (JSON).

 

https://experienceleague.adobe.com/docs/experience-manager-64/developing/components/json-exporter.html?lang=en#:~:text=JSON%20Exporter%20with%20Content%20Fragment,json%20extension.

 

Hope it helps!

Thanks,
Kiran Vedantam

    Sean-McK
    Sean-McKAuthor
    Level 4
    February 8, 2023

    Hey Kiran,

     

    Thanks for the request. I think the issue with that option is finding out what components and what fields to pull. For example, in a text component the field name is text (easy), but what if I create a custom component? So lets say my custom component has 3 fields (title, subtitle, and botteminfo)...now when I do that json I will have to know to bring in those 3 fields? I guess I can create a config that takes field names to "index" and when it gets the page it can look for all those field names

     

      Nitin_laad
      Community Advisor
      Nitin_laadCommunity Advisor
      Community Advisor
      February 8, 2023

      Hi @sean-mck 

      In your scenario, you have two options to consider: the Push model and the Pull model. The Push model involves sending updated data to the search service upon page publication  or using EventHandler from AEM. On the other hand, the Pull model involves configuring a crawler at the search service side, which periodically(ex 1 hour) retrieves data from AEM.

        kaikubad
        Community Advisor
        kaikubadCommunity Advisor
        Community Advisor
        February 9, 2023

        We faced similiar kind of scenario. We wanted to search on our websites but not using AEM.

        So we used typesense and scrapy to get it done. 
        Scrapy scraps all the data that are available on web pages and store it on typesense.

         

          Sean-McK
          Sean-McKAuthorAccepted solution
          Level 4
          February 10, 2023

          This is only good if you are running outside of AEM. I realized a couple of things...one of them was using HTMLUnit and Jsoup to scrape and parse. I decided to use content services and the json model to parse. 

            Terms & ConditionsAccessibility statement

            Loading footer...