How to store large amounts of data? | Community
Skip to main content
F
ForrestLi
Level 3
July 30, 2021
Solved

How to store large amounts of data?

  • July 30, 2021
  • 3 replies
  • 1745 views

I have a requirement to store the change records of some specified Node. And I can display these records on the page

Example :

NameUserTime
Node1admin2021-7-31

 

I have evaluated some, I may need to store more than 100,000 data. Where and how can I store this data in the AEM？After saving, I can query, and the query speed is not very slow.

 

Thanks,

Forrest

This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.
Best answer by joerghoh

Hi,

 

if I understand you correctly you want to store a kind of audit information along with each page on each change, is this correct?

 

While technically it's doable, I don't expect that it will be ever used as such, because noone needs all the details in a change log for a website, even in legally binding cases (I worked with AEM users in the financial industry). In AEM a version is created whenever a page is published, and you can easily revert to that. On top of that you have permission and activation workflows. If that is not sufficient you need to have such a fine-grained controlled, there are probably better solutions out there.

Especially because all that comes with some considerations you should do:

* What do you do with this audit log if the page is deleted? Do you need to keep it?

* Do you need to guarantee that these audit entries are immutable?

* Is the audit enough or do you need to store the page (at that point in time) as well?

 

(I don't want to talk about the performance implications if you have thousands of pages with thousands of such entries.)

3 replies

Dipti_Chauhan
Community Advisor
Dipti_ChauhanCommunity Advisor
Community Advisor
July 30, 2021

Hi @forrestli 

 You want to import this data from other source or these are AEM node change log?

 

Thanks

Dipti

    F
    ForrestLiAuthor
    Level 3
    August 2, 2021

    Hi @dipti_chauhan 

     

    I am importing products data from other system to AEM.  If a product node is changed (add, change, delete), I need to record the change, then I can view the change record on the Dashboard page.

     

    Thanks,

    Forrest

      arunpatidar
      Community Advisor
      arunpatidarCommunity Advisor
      Community Advisor
      July 30, 2021

      You can store the data as aem node and property but make sure, do not create all node within a parent.

      Distribute the nodes in a structure, could be based on number or count.

       

      Arun Patidar
        joerghoh
        Adobe Employee
        joerghohAdobe EmployeeAccepted solution
        Adobe Employee
        August 1, 2021

        Hi,

         

        if I understand you correctly you want to store a kind of audit information along with each page on each change, is this correct?

         

        While technically it's doable, I don't expect that it will be ever used as such, because noone needs all the details in a change log for a website, even in legally binding cases (I worked with AEM users in the financial industry). In AEM a version is created whenever a page is published, and you can easily revert to that. On top of that you have permission and activation workflows. If that is not sufficient you need to have such a fine-grained controlled, there are probably better solutions out there.

        Especially because all that comes with some considerations you should do:

        * What do you do with this audit log if the page is deleted? Do you need to keep it?

        * Do you need to guarantee that these audit entries are immutable?

        * Is the audit enough or do you need to store the page (at that point in time) as well?

         

        (I don't want to talk about the performance implications if you have thousands of pages with thousands of such entries.)

          F
          ForrestLiAuthor
          Level 3
          August 2, 2021

          Hi @joerghoh 

           

          I am importing products data from other system to AEM and stored them as the nodes.  If a product node is changed (add, change, delete), I need to record the change, then I can view the change record on the Dashboard page.

           

          Thanks,

          Forrest

            Terms & ConditionsAccessibility statement

            Loading footer...