How to store large amounts of data?

[ForrestLi]

I have a requirement to store the change records of some specified Node. And I can display these records on the page

Example :

Name	User	Time
Node1	admin	2021-7-31

I have evaluated some, I may need to store more than 100,000 data. Where and how can I store this data in the AEM？After saving, I can query, and the query speed is not very slow.

Thanks,

Forrest

[Dipti_Chauhan]

Hi @ForrestLi 

 You want to import this data from other source or these are AEM node change log?

Thanks

Dipti

[ForrestLi]

Hi @Dipti_Chauhan 

I am importing products data from other system to AEM.  If a product node is changed (add, change, delete), I need to record the change, then I can view the change record on the Dashboard page.

Thanks,

Forrest

[Dipti_Chauhan]

Hi @ForrestLi We had same requirement where we wanted to keep Product Import (From other system) log data to check if product added, deleted or updated for that we created a node in AEM and wrote cleanup scheduler to clean old logs .I am looking forward to suggestion if anyone have better approachx.

[arunpatidar]

You can store the data as aem node and property but make sure, do not create all node within a parent.

Distribute the nodes in a structure, could be based on number or count.

[Jörg_Hoh · Accepted solution]

Hi,

if I understand you correctly you want to store a kind of audit information along with each page on each change, is this correct?

While technically it's doable, I don't expect that it will be ever used as such, because noone needs all the details in a change log for a website, even in legally binding cases (I worked with AEM users in the financial industry). In AEM a version is created whenever a page is published, and you can easily revert to that. On top of that you have permission and activation workflows. If that is not sufficient you need to have such a fine-grained controlled, there are probably better solutions out there.

Especially because all that comes with some considerations you should do:

* What do you do with this audit log if the page is deleted? Do you need to keep it?

* Do you need to guarantee that these audit entries are immutable?

* Is the audit enough or do you need to store the page (at that point in time) as well?

(I don't want to talk about the performance implications if you have thousands of pages with thousands of such entries.)

[ForrestLi]

Hi @Jörg_Hoh 

I am importing products data from other system to AEM and stored them as the nodes.  If a product node is changed (add, change, delete), I need to record the change, then I can view the change record on the Dashboard page.

Thanks,

Forrest

[Jörg_Hoh]

HI @ForrestLi: Why do you need to track these changes in AEM and not in the other system where you get that data from? That feels much more natural to me.