Fusion Datastore Search Module - better way to search for strings over certain length?

Question

I have a route in my scenario where the objective is to truncate all strings in a column that are 2000 characters or greater, and I'm hoping someone can suggest a more elegant solution.
Right now, I use a Search module as the first module in the route and it is searching just for ANY entry in the datastore that has a unique key (i.e. EVERY row in the data store).

Then I tried to use an iterator module to go through - what I'd thought was - every search result, and only if the string is 2000+ chars - depending on the row type - a type-specific variable would be set to create a truncated string to ultimately overwrite the value in the original string field of the datastore.

However this approach has produced very problematic results. I learned that the iterator module processed each search result in an array separately, creating individual bundles. And the bundles kept on proliferating so that even if there were 161 rows in the datastore, it was processing 12,000+ bundles. According to Adobe, right now, the module is set to return all records from the data store as long as the unique key exists. This means that on the first pass, one bundle was returned, on the second pass, two bundles, on the third, three, and so on—until operation 161 returns 161 bundles.

Is there a better way for me to do the initial search of the data store so my results are only the rows where the length of the string field is 2000+ and then to go row-by-row of those results without the problem of the bundles proliferating to the degree they did? Someone suggested capturing the protocolNumber in a set variable and retrieving it just before module 166 I'd be allowed to refine the filter to only considers the most recent record. This would then reset with the next module and avoid the proliferation. I haven't tried that approach yet, and before I begin, thought to ask if the community had any other ideas?

Thank you for your thoughts, and let me know if I can clarify my problem any better for you.

Sven-iX · Accepted Answer

Hi @mylah_d

I think you're on the right track here.

All "Search" modules have an implicit iterator, ie their output is a list of bundles, not an array. So you don't actually need an iterator here.

Remove that and the rest looks good to me. You may want to uncheck "insert missing record" though.

Since it's 4 specific fields you could reduce the set you operate on by adding filters:

length(field1) >= 2000 OR length(field2) >= 2000 OR length(field3) >= 2000 OR length(field4) >= 2000
AND protocolNumber exists

Mylah_D · Answer

I had meant to also share the screenshot of the truncating logic downstream of the Data Store Search module, which runs "for each" search result:  This is the route in its entirety:

Sign up

Login with SSO

Login to the community

Login with SSO

Scanning file for viruses.

This file cannot be downloaded