Expand my Community achievements bar.

Dive into Adobe Summit 2024! Explore curated list of AEM sessions & labs, register, connect with experts, ask questions, engage, and share insights. Don't miss the excitement.
SOLVED

How to store large amounts of data?

Avatar

Level 3

I have a requirement to store the change records of some specified Node. And I can display these records on the page

Example :

NameUserTime
Node1admin2021-7-31

 

I have evaluated some, I may need to store more than 100,000 data. Where and how can I store this data in the AEM?After saving, I can query, and the query speed is not very slow.

 

Thanks,

Forrest

1 Accepted Solution

Avatar

Correct answer by
Employee Advisor

Hi,

 

if I understand you correctly you want to store a kind of audit information along with each page on each change, is this correct?

 

While technically it's doable, I don't expect that it will be ever used as such, because noone needs all the details in a change log for a website, even in legally binding cases (I worked with AEM users in the financial industry). In AEM a version is created whenever a page is published, and you can easily revert to that. On top of that you have permission and activation workflows. If that is not sufficient you need to have such a fine-grained controlled, there are probably better solutions out there.

Especially because all that comes with some considerations you should do:

* What do you do with this audit log if the page is deleted? Do you need to keep it?

* Do you need to guarantee that these audit entries are immutable?

* Is the audit enough or do you need to store the page (at that point in time) as well?

 

(I don't want to talk about the performance implications if you have thousands of pages with thousands of such entries.)

View solution in original post

7 Replies

Avatar

Community Advisor

Hi @ForrestLi 

 You want to import this data from other source or these are AEM node change log?

 

Thanks

Dipti

Avatar

Level 3

Hi @Dipti_Chauhan 

 

I am importing products data from other system to AEM.  If a product node is changed (add, change, delete), I need to record the change, then I can view the change record on the Dashboard page.

 

Thanks,

Forrest

Avatar

Community Advisor
Hi @ForrestLi We had same requirement where we wanted to keep Product Import (From other system) log data to check if product added, deleted or updated for that we created a node in AEM and wrote cleanup scheduler to clean old logs .I am looking forward to suggestion if anyone have better approachx.

Avatar

Community Advisor

You can store the data as aem node and property but make sure, do not create all node within a parent.

Distribute the nodes in a structure, could be based on number or count.

 



Arun Patidar

Avatar

Correct answer by
Employee Advisor

Hi,

 

if I understand you correctly you want to store a kind of audit information along with each page on each change, is this correct?

 

While technically it's doable, I don't expect that it will be ever used as such, because noone needs all the details in a change log for a website, even in legally binding cases (I worked with AEM users in the financial industry). In AEM a version is created whenever a page is published, and you can easily revert to that. On top of that you have permission and activation workflows. If that is not sufficient you need to have such a fine-grained controlled, there are probably better solutions out there.

Especially because all that comes with some considerations you should do:

* What do you do with this audit log if the page is deleted? Do you need to keep it?

* Do you need to guarantee that these audit entries are immutable?

* Is the audit enough or do you need to store the page (at that point in time) as well?

 

(I don't want to talk about the performance implications if you have thousands of pages with thousands of such entries.)

Avatar

Level 3

Hi @Jörg_Hoh 

 

I am importing products data from other system to AEM and stored them as the nodes.  If a product node is changed (add, change, delete), I need to record the change, then I can view the change record on the Dashboard page.

 

Thanks,

Forrest

Avatar

Employee Advisor
HI @ForrestLi: Why do you need to track these changes in AEM and not in the other system where you get that data from? That feels much more natural to me.