Skip to main content
Solved

REST API Pagination Issue: Unable To Retrieve More Than 5000 Records With Offset/Limit

  • January 31, 2026
  • 20 replies
  • 3 views
H
This message originated from Cribl Community Slack.
Click here to view the original link.

Struggling with understanding pagination on a particular REST API: https://docs.dynatrace.com/docs/discover-dynatrace/references/dynatrace-api/environment-api/rum/user-sessions/table#parameters The data that comes in does not explicitly include an indicator for which one of these to use: https://sandbox.cribl.io/coursedocs/rest/docs/7_pagination#offsetlimit https://sandbox.cribl.io/coursedocs/rest/docs/7_pagination#pagesize I've tried both and it either never goes past the 5000 records collected or the job continuously runs without stopping, running the same query over and over again for each page per collector job. Based on the latter, I believe I need to be using Offset/Limit, but the parameters explicitly say pageSize & pageOffset

Best answer by Randy Corelli

For anyone who'd like some more info on the State Tracking, I've done a couple of videos on the topic that have been published to the Cribl U YouTube library: The first video goes through the basics using the _time tracking, while the second provides a deeper review of the state tracking expression and using a different event field. There are some even more advanced scenarios for the state tracking, so if your situation doesn't align to those I've highlighted in the videos, just hit me up with the details and I'm sure we can help you out!

20 replies

Vu Pham 2
  • Vu Pham 2
  • Employee
  • January 31, 2026
from initial look at the doc, the response body does not contain a field with the total record count. So if using offset/limit the collector will keep going until the page limit has been reached. Is it possible to use response body attribute pagination with a last-page expression based on the results data?
H
  • huntmanday147
  • Author
  • Participating Frequently
  • January 31, 2026
The data doesn't return a response attribute as a token either :confused:
Vu Pham 2
  • Vu Pham 2
  • Employee
  • January 31, 2026
any way to have the query output a total number?
H
  • huntmanday147
  • Author
  • Participating Frequently
  • January 31, 2026
Like a count? Yeah that's how we've been validating how many records exist within a timeframe, but unsure how that would be relevant here
Vu Pham 2
  • Vu Pham 2
  • Employee
  • January 31, 2026
If I'm understanding correctly, this API lets you run a custom query. Hypothesis is if you include an output column containing the total number of records, you can use that as your total field to indicate the end of results for limit/offset
H
  • huntmanday147
  • Author
  • Participating Frequently
  • January 31, 2026


Links for this message:
image.png
G
  • gblock712
  • Employee
  • January 31, 2026
Hi @user did you try using the "Stop on empty results" option? That "should" stop the paging if there are no other events coming through the event breaker.
G
  • gblock712
  • Employee
  • January 31, 2026
Also make sure Page Limit under Collect is set to 0.
H
  • huntmanday147
  • Author
  • Participating Frequently
  • January 31, 2026
Hey @user thanks for chiming in. Yes I did try using that and it worked just fine, just ended up being a limitation of the API from the vendor themselves
H
  • huntmanday147
  • Author
  • Participating Frequently
  • January 31, 2026
They cap it at 5000 records, so I just put the Page Limit to 5, record limit at 1000, and implemented proper state tracking to just run the job every 2 minutes until everything is collected
G
  • gblock712
  • Employee
  • January 31, 2026
Great to hear you found a path. If you blog, state tracking is a great topic that many do not understand!
Randy Corelli
  • Randy Corelli
  • Employee
  • Answer
  • January 31, 2026
For anyone who'd like some more info on the State Tracking, I've done a couple of videos on the topic that have been published to the Cribl U YouTube library: The first video goes through the basics using the _time tracking, while the second provides a deeper review of the state tracking expression and using a different event field. There are some even more advanced scenarios for the state tracking, so if your situation doesn't align to those I've highlighted in the videos, just hit me up with the details and I'm sure we can help you out!
G
  • gblock712
  • Employee
  • January 31, 2026
@user but have you blogged? :smiling_imp:
Randy Corelli
lol, not yet @user. Been sticking with the vlogs so far but if I get some spare time I'd blog it up!
H
  • huntmanday147
  • Author
  • Participating Frequently
  • January 31, 2026
@user I feel like I only "understand" state tracking if I'm not required to modify the default state update/state merge expressions. Just morph the incoming data to accurately use whatever field needs to be used for _time, then work backwards to figure out how exactly the REST API needs the pass the correct time value used for state.latestTime
G
  • gblock712
  • Employee
  • January 31, 2026
@user we need this on Cribl's blog or maybe I loop you in with docs and we can turn what you have into doc pages.
H
  • huntmanday147
  • Author
  • Participating Frequently
  • January 31, 2026
I did read the docs over and over but it didn't make the subject much clearer for me
H
  • huntmanday147
  • Author
  • Participating Frequently
  • January 31, 2026
a blog post outlining exact scenarios would be extremely valuable
Randy Corelli
That's a pretty common feeling @user. I usually prefer making videos vs. blogs since I can add my commentary and demonstrate vs. everything just being written, so check out the video below to see if it helps de-mystify the state tracking a bit. https://www.youtube.com/watch?v=gnM-7MUWf-g
Randy Corelli
That still doesn't cover the more advanced scenarios, like how to use an offset/string vs. an incrementing integer, so I'm planning a part 3 at some point but don't have a date or anything on that one
Terms & ConditionsAccessibility statement