How to use 'NextToken' in boto3 AWS API Calls?

Today we're talking about how to use 'NextToken' in AWS API calls, specifically when using the Boto3 Python SDK. If you're dealing with long lists of items from AWS services, this is the guide for you.

Overview

I'm sure many of you are familiar with Boto3. For those who aren't, it's a Python SDK that allows us to interact with AWS services. Think of it as a tool that lets us automate all the stuff we can do manually in the AWS Console, but using Python.

So, why are we talking about this 'NextToken'? Well, when we're dealing with AWS services that have lots of items to list—say, files in an S3 bucket or topics in SNS—AWS usually breaks it down into smaller chunks. The 'NextToken' is essentially our bookmark. It helps us keep track of where we left off so we can loop through everything without missing out.

Before we jump in, this post assumes you have some knowledge of AWS and basic Python skills.

Example - List SNS Topics

Let's move on to the example. I've chosen SNS topics to demonstrate how 'NextToken' functions. Of course, you can use this approach with any AWS service.

In this example, I've created 103 SNS topics. When you call the API to list these topics, it defaults to showing only the first 100. Along with this list, the API response will include a key called 'NextToken', which is a string of data we'll use later.

So you might be wondering about the other three topics that we didn't see. To view them, you'll need to make another API call. This time, include the 'NextToken' from the previous response, and you'll see the remaining three topics.

First Request

# First request
import boto3

client = boto3.client('sns')
response = client.list_topics()

print(len(response['Topics']))
print(response.get('NextToken', 'No more Topics'))

First, we import the Boto3 library and set up an SNS client. Then, we make an API call using client.list_topics() to get a list of SNS topics. I've included a print statement to show the length of the Topics list, which will indicate how many topics we've retrieved. We also print out the value of 'NextToken' if it exists; otherwise, it'll print 'No more Topics'. This 'NextToken' is what we'll use to get the rest of the topics in our next API call.

# Output

100
AaEdHpfgQAoxULcs1t7ZBSsdf5G4erROSy5yp8Lmk1sdf=

When you run the script next_token_ex.py, you see two lines in the output. The first line, which shows '100', tells us that we've successfully pulled 100 SNS topics from the API call.

The second line, that string of characters, is the 'NextToken'. This token is our key to accessing the remaining topics that didn't show up in the first API call. We'll use this 'NextToken' in our next API request to get the rest of the topics.

So, what we've learned is that the API gave us the first 100 topics and a 'NextToken' to continue where we left off. On to the next part to see how to use this token.

Second Request

import boto3

client = boto3.client('sns')
response = client.list_topics(
    NextToken="AaEdHpfgQAoxULcs1t7ZBSsdf5G4erROSy5yp8Lmk1sdf="
)

print(len(response['Topics']))
print(response.get('NextToken', 'No more Topics'))

Let's get into the second request. In this snippet, the code is pretty much the same as the first one, but with a small difference. We're now including the NextToken in our client.list_topics() API call. This token is the one we got from our first request.

3
No more Topics

When you run this updated script, the output shows '3' as the first line. That tells us that we've successfully retrieved the remaining three SNS topics that were not included in the first API call.

The second line in the output says 'No more Topics', indicating that we've reached the end of our list. There's no more 'NextToken', meaning we've successfully looped through all the topics.

This is Just too Hard to Manage

You might be thinking, "Why go through all this trouble with multiple API calls? I might as well just get this information from the AWS Console." And you're right; running these scripts multiple times can feel a bit cumbersome.

But don't worry, there's a more efficient way to handle this. In our next example, we'll use a while loop to automate the process and make it much smoother.

Using a While Loop

Let's dive into this Python code and understand how it fetches all SNS topics using the boto3 library. The output "103" indicates that all 103 topics have been successfully retrieved, but how did we get there? Here's how each part of the code contributes.

import boto3

client = boto3.client('sns')
next_token = None
all_topics = []

while True:
    if next_token:
        response = client.list_topics(NextToken=next_token)
    else:
        response = client.list_topics()
    
    all_topics.extend(response['Topics'])
    
    next_token = response.get('NextToken', None)
    if next_token is None:
        break

print(len(all_topics))

# Output
103

We have two new variables to begin with, next_token, set to None, and an empty list called all_topics. We enter a while True loop to keep fetching topics until there are no more. Inside the loop, we check if we have a next_token. If we do, we use it to get the next set of topics. Otherwise, we get the first set of topics. We add these topics to our all_topics list using extend.

Then, we check for a new NextToken and update next_token accordingly. If there's no new NextToken, we break out of the loop. Finally, we print the length of all_topics to see how many topics we've collected. In this example, we end up with 103, which means we've successfully grabbed all the SNS topics.

A Better Approach - Paginator

Let's dive into a better way of doing things using a Paginator. Paginator is a feature in the boto3 Python SDK, and it's used for fetching multiple pages of data from AWS. Instead of you doing the heavy lifting with NextToken, Paginator does it for you. All you do is set it up once and then loop through the pages. It fetches and compiles all the data for you.

Paginators are created via the get_paginator() method of a boto3 client. The get_paginator() method accepts an operation name and returns a reusable Paginator object. You then call the paginate method of the Paginator, passing in any relevant operation parameters to apply to the underlying API operation. The paginate method then returns an iterable PageIterator

Here is another script that uses a paginator but gives the same results using a While loop.

import boto3

client = boto3.client('sns')
paginator = client.get_paginator('list_topics')

all_topics = []
for page in paginator.paginate():
    all_topics.extend(page['Topics'])

print(len(all_topics))

# Output
103

In this code, we use the boto3 Python SDK to create an SNS client and then create a Paginator object using the get_paginator('list_topics') method. The Paginator object is a specialized class in boto3 designed to handle API pagination automatically. We then call the paginate() method on the Paginator object to iterate through each page of the API results.

Within the loop, the all_topics list is extended with the topics from the current page using the extend method. This allows us to accumulate all topics from multiple API calls into a single list, without having to manually manage NextToken.