How to list ALL the data of a table which has over 1 million in the Javascript? | Community
Skip to main content
Level 3
April 11, 2019

How to list ALL the data of a table which has over 1 million in the Javascript?

  • April 11, 2019
  • 3 replies
  • 5532 views

I have table has over millions of records, I wanted to use the queryDef to list all the records, but the queryDef can only fetch 1000 records one time. If I use the loop syntax to use the queryDef for many times, how can I make sure it will stop after I list all the records? Is there any other functions I can use except the queryDef? Thanks!

This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.

3 replies

geoffreyl240454
Level 3
April 11, 2019

Hi felixs51477447​,

I don't know if another function exists but what I would do:

- Set a lastId parameter to 0

- Sorting your records by Id in your queryDef

- Adding a condition tag "id > lastId" in the where tag of your queryDef

- Getting the last id recovered for each queryDef execution

- Exit the loop when the result of the queryDef equal to 0

Geoffrey.

Venu_Reddy
Level 6
April 12, 2019

Hi felixs51477447​,

When you use Javascript to process large amounts of data you must take some precautions to avoid

memory leaks. Otherwise the server may become unresponsive after a while.

Queries in Javascript (queryDef) results should be limited in number. Do not create a querydef result

of more than 10,000 (ten thousand) records. Use the “lineCount” attribute when loading a large

table. If you have more than 10,000 records to process, it's possible, but requires a special technique.

Exit the Javascript interpreter every 10,000 lines, then start again from the line where you stopped.

Use parseInt(), parseFloat() and toString() to cast the XML types to normal Javascript types before

using them.

For instance : var iRecipientId = parseInt(ctx.recipient.@id); process(iRecipientId).

Otherwise it could create errors on comparison of values, because they are not of the same type.

Regards,

Venu

Jonathon_wodnicki
Community Advisor
Community Advisor
April 12, 2019

Hi,

The for-loop will terminate when the db cursor reaches the end of the result set, i.e. when all records have been fetched.

I think Venu is speaking to the Tomcat memory limit, which defaults to 2 mb per js process in Campaign installations. Avoid buffering the full table to memory if possible.

Thanks,

-Jon