Help with Search Module limits

[BrandonNW]

Happy New Year!!

I have a scenario where I am connecting to HR data to search for terminated users and deactivate them in WF. The process itself is very simple, but I am running into an issue with the 2000 maximum search limit. We have over 2000 users, so basically the functionality works, but only for the first 2000 users. I am guessing I am missing something silly that will allow me to search all users, but I am not sure what I am missing.

Use Case:

Pull all HR user records and then search WF for user records that do not match and deactivating those users.

Thanks!!

[William--]

You might want to get familiar with paginated API responses and add a repeater module with that logic.

Alternatively, as part of these kinds of scenarios I like to tag the user's Workfront profile with the fact that the user has been checked against the other platform, so I do an update record module early in the scenario where I update a custom field on the user's profile with the date that the other platform was last checked. This enables filtering/reporting in Workfront ("How long since this profile was last checked against HR?") and also helps with the scenario because I can add a filter on the search module "DE:Other Platform Date Last Checked = null" (or less than today). This way when I run the scenario a second/third/fourth/... time, it skips all the users whose profiles have a value recorded in that custom field on their profile, until all users have a value recorded.

[BrandonNW]

Thanks Will. That sounds like it might do the trick! Now I just gotta figure out how to use the paginated response. 😋

[William--]

Sure thing, Brandon.

To be clear, the approach I described is a workaround to having to use paginated responses. If you tag the profile as you check the user, you can use that tag as a filter on the search module of your scenario so it only looks for users who haven't been tagged/checked. So the first run will be limited to your first 2,000 users, but the next time you run it, the search module will skip those users and move on to the next 2,000, and so on.

[AndyHess · Accepted solution]

Using paginated responses in Fusion, require usage of several API features. $$LIMIT and $$COUNT along with the Fusion Repeater module.

1. objcode/count? Using this URL as a custom api call along with the needed filters that align to your eventual call will provide a number of records
2. use the repeater module, initial value = 0 and the Repeats a formula of the count {{ceil(6.body.data.count / 200)}}. The integer shown is the value of the limit you plan to use in #3
3. objcode/search? Now do your search for records that can be higher than the 2K limit. use the same filter in #1. The keys here are the $$LIMIT filter and the $$FIRST filters. $$LIMIT is your records/search and $$FIRST is the repeat# times the limit or {{(7.i * 200)}}

I placed a simple blueprint of this method. It's fairly straightforward.

[BrandonNW]

Thanks Andy! The blueprint was super helpful. I am still very much so a beginner in the Fusion world and being able to see the modules already built out made it quite easy to tweak them for my particular scenario. This isn't perfect yet, and I still have a couple filters to work out, but the repeater module with the paginated calls got me passed that hurdle at least. 😊 Very much appreciated sir!

[ToddCr]

Andy, would you be able to go into more detail? I'm really new at this and having the same problem. I have almost no experience with API calls so I really don't understand what you're talking about. In our case we need all the user account records, even disabled accounts. This comes out to 3222 records as of today. Currently I'm using the WF Search going into a CSV module, and then into the SFTP module. I'm guessing I put the repeater between the WF Search and CSV? Thank you for any help you can provide.

[AndyHess]

Todd,

If you are using Fusion, there is a sample blueprint you can import into an empty scenario that will explain the process.

1. Don't use the WF Fusion Search module, use the Custom API module and the URL will be <objectCode>/count. (user/count). This will return the number of records. Under the query section add the necessary search criteria. If you want all objects, its not needed
   1. Establish what your processing limit per loop will be, 100/500/1000/2000. and then use a Fusion function of upper() to get the number of loops needed to process all records
   2. This number is your repeats number for the repeater module
   3. Next, make a limited call to your WF module. user/search. with a query that contains the limits below.
      1. 

If handling via direct API calls then some of this does not pertain as these are Fusion functions. Your API developer would need to integrate these functions into their code.

Hope this helps & good hunting,

[ToddCr]

Thank you Andy, this helps a great deal and also thank you for the quick response. I was thinking I needed the Fusion search but now I understand a bit more about what I need to try next. This'll be done through Fusion and no API developer to do that work so I'm sort of stuck using Fusion. Thank you again!

Todd