Data Sources API - GetJobs with high number of imports

[bjoern__koth]

Hi guys,

I am fairly new to using the Data Sources API and have jusr created a NodeJS importer that queries GSC data and is pushing that data into a data source.

Data import works great so far, and I have imported all 2023 data. 

My client has a fairly big website with large numbers of entries per day (25k+) that requires breaking the import into multiple chunks per day (typically 3) which leaves me at 1000k import jobs so far.

To not accidentally import the same day data twice, I have created a failsafe that is first querying previous job names to see whether the planned day to import has already been processed.

This I do using the GetJobs function

https://api.omniture.com/admin/1.4/rest/?method=DataSources.GetJobs

And here is where the issues start.

This function only returns a maximum of 1000 jobs and my imports run twice because previous imports cannot be detected. Luckily, this is currently on a non-prod report suite.

Does anyone know whether

• there is a pagination in the API call I could use e.g. a rowStart or similar?
• alternatively, a way to filter the job names? There is a "filters" I can set with a "name" array, but it seems like you will have to know the exact name of the import job, that includes a random part that Adobe is adding to the file name that they create during the processing which defeats the purpose of a filter and is absolute nonsense if you ask me. How can you apply a filter if you don't even know what to look for...?!

 

Cheers

Björn

[fhusain]

Hey @bjoern__koth Are you still facing challenges with this issue? 

- On the pagination part, if I check the body of the API it seems we don't any such parameter explicitly.

-  We can check on the filter name part as well and if you haven't log a support ticket yet; can you please create one via Admin Console such that this can be looked at?

[bjoern__koth · Accepted solution]

Hi @fhusain,

support said it's not possible, disabled my checks, filters only work on exact values i.e. no regex or pattern matching.

Don't think it's ever gonna be implemented in a 1.4 API.

Cheers
Björn