What is AWS boto3 Paginator? (with examples)

Today, we're diving deep into a really useful feature of the AWS boto3 Python SDK, the Paginator. If you've ever struggled with dealing with paged API results using NextToken, then this post is for you. We'll use SNS topics as our example to show you how it works. We'll also see how it compares to using NextToken. Let's get started.

What is a Paginator?

Some AWS operations return results that are incomplete and require subsequent requests in order to get the full response. The process of sending subsequent requests to continue where a previous request left off is called pagination.

Paginators are a feature of boto3 that acts as an abstraction over the process of iterating over an entire result set of a truncated API operation.

A Side-by-Side Comparison of NextToken and Paginator with SNS Topics

Let's go straight into the example. I've chosen SNS topics to demonstrate how Paginator and NextToken work. Of course, you can use this approach with any AWS service. As you can see below, I have created 103 SNS topics for this example.

NextToken

First, let's dive into an example using NextToken to list SNS topics in AWS. We're using Python's boto3 library, and here's how it goes.

import boto3

client = boto3.client('sns')
next_token = None
all_topics = []

while True:
    if next_token:
        response = client.list_topics(NextToken=next_token)
    else:
        response = client.list_topics()
    
    all_topics.extend(response['Topics'])
    
    next_token = response.get('NextToken', None)
    if next_token is None:
        break

print(len(all_topics))

# Output
103

In this code, we create an SNS client and then set up a while loop to keep making API calls until there are no more pages left to retrieve. We use a variable called next_token to keep track of where we are. The list_topics() method fetches the topics.

Initially, next_token is None, so the first API call fetches the first 100 topics. We store these topics in the all_topics list using the extend() method. Then, we check for the presence of NextToken in the API response. If it's there, we use it in the next API call to get the remaining topics. The loop continues until NextToken is no longer returned, meaning we've fetched all topics. At the end, we print the total number of topics gathered.

So, we start with the first 100 topics and then grab the remaining 3 in our case in the subsequent API call. All in all, it's a manual process but gets the job done. Next up, let's see how Paginator can make this even easier.

Paginator

Paginators are created via the get_paginator() method of a boto3 client. The get_paginator() method accepts an operation name and returns a reusable Paginator object. You then call the paginate method of the Paginator, passing in any relevant operation parameters to apply to the underlying API operation. The paginate method then returns an iterable PageIterator

import boto3

client = boto3.client('sns')
paginator = client.get_paginator('list_topics')

all_topics = []
for page in paginator.paginate():
    all_topics.extend(page['Topics'])

print(len(all_topics))

# Output
103

In this example, we're using the Paginator feature available in the boto3 Python SDK for AWS. We start by creating an SNS client, just like before. This time, though, we use the get_paginator method to get a Paginator object for the 'list_topics' operation. The Paginator object is responsible for handling the pagination logic for us.

We then initialize an empty list called all_topics to store our results. The Paginator object's paginate() method returns an iterator, so we loop through each page using a for loop. Within the loop, we extend our all_topics list with the topics found in the current page, which are stored in page['Topics']. Finally, we print out the total number of topics collected. The Paginator takes care of dealing with NextToken behind-the-scenes, making the code cleaner and more straightforward.

NextToken vs Paginator

In summary, both NextToken and Paginator serve the same purpose: they help you navigate through paginated API responses. If you like having full control and don't mind managing NextToken yourself, then using NextToken can be a straightforward approach.

However, if you prefer a cleaner, more automated way to handle pagination, then Paginator is the way to go. It abstracts away the complexities of dealing with NextToken and makes your code easier to read and maintain. Both methods have their merits, so the choice between them depends on your specific needs and coding style.

References