I wanted to spawn this discussion off of this post, Running an operation multiple times using a counter which provided suggestions for paginating through available responses from a web service.
We used to take a similar approach of following this article Design Pattern: Handling API Pagination. However, we would often run into memory errors with a large amount of employees. So we implemented a design that created all of the request documents first, based on the available pages and records. This has proven to be more effective, reusable, and easier to debug.
I am hoping this conversation will allow others to share their design suggestions and / or any challenges that they've faced.