Your achievements

Level 1

0% to

Level 2

Tip /
Sign in

Sign in to Community

to gain points, level up, and earn exciting badges like the new
BedrockMission!

Learn more

View all

Sign in to view all badges

SOLVED

How to store large amounts of data?

ForrestLi
Level 2
Level 2

I have a requirement to store the change records of some specified Node. And I can display these records on the page

Example :

NameUserTime
Node1admin2021-7-31

 

I have evaluated some, I may need to store more than 100,000 data. Where and how can I store this data in the AEM?After saving, I can query, and the query speed is not very slow.

 

Thanks,

Forrest

1 Accepted Solution
Jörg_Hoh
Correct answer by
Employee
Employee

Hi,

 

if I understand you correctly you want to store a kind of audit information along with each page on each change, is this correct?

 

While technically it's doable, I don't expect that it will be ever used as such, because noone needs all the details in a change log for a website, even in legally binding cases (I worked with AEM users in the financial industry). In AEM a version is created whenever a page is published, and you can easily revert to that. On top of that you have permission and activation workflows. If that is not sufficient you need to have such a fine-grained controlled, there are probably better solutions out there.

Especially because all that comes with some considerations you should do:

* What do you do with this audit log if the page is deleted? Do you need to keep it?

* Do you need to guarantee that these audit entries are immutable?

* Is the audit enough or do you need to store the page (at that point in time) as well?

 

(I don't want to talk about the performance implications if you have thousands of pages with thousands of such entries.)

View solution in original post

7 Replies
Dipti_Chauhan
Community Advisor
Community Advisor

Hi @ForrestLi 

 You want to import this data from other source or these are AEM node change log?

 

Thanks

Dipti

ForrestLi
Level 2
Level 2

Hi @Dipti_Chauhan 

 

I am importing products data from other system to AEM.  If a product node is changed (add, change, delete), I need to record the change, then I can view the change record on the Dashboard page.

 

Thanks,

Forrest

Dipti_Chauhan
Community Advisor
Community Advisor
Hi @ForrestLi We had same requirement where we wanted to keep Product Import (From other system) log data to check if product added, deleted or updated for that we created a node in AEM and wrote cleanup scheduler to clean old logs .I am looking forward to suggestion if anyone have better approachx.
Arun_Patidar
Community Advisor
Community Advisor

You can store the data as aem node and property but make sure, do not create all node within a parent.

Distribute the nodes in a structure, could be based on number or count.

 

Jörg_Hoh
Correct answer by
Employee
Employee

Hi,

 

if I understand you correctly you want to store a kind of audit information along with each page on each change, is this correct?

 

While technically it's doable, I don't expect that it will be ever used as such, because noone needs all the details in a change log for a website, even in legally binding cases (I worked with AEM users in the financial industry). In AEM a version is created whenever a page is published, and you can easily revert to that. On top of that you have permission and activation workflows. If that is not sufficient you need to have such a fine-grained controlled, there are probably better solutions out there.

Especially because all that comes with some considerations you should do:

* What do you do with this audit log if the page is deleted? Do you need to keep it?

* Do you need to guarantee that these audit entries are immutable?

* Is the audit enough or do you need to store the page (at that point in time) as well?

 

(I don't want to talk about the performance implications if you have thousands of pages with thousands of such entries.)

View solution in original post

ForrestLi
Level 2
Level 2

Hi @Jörg_Hoh 

 

I am importing products data from other system to AEM and stored them as the nodes.  If a product node is changed (add, change, delete), I need to record the change, then I can view the change record on the Dashboard page.

 

Thanks,

Forrest

Jörg_Hoh
Employee
Employee
HI @ForrestLi: Why do you need to track these changes in AEM and not in the other system where you get that data from? That feels much more natural to me.