Paginating S3 objects using boto3

When using Boto you can only List 1000 objects per request. So to obtain all the objects in  the bucket. You can use s3's paginator. To use paginator you should first have a client instance 

client = boto3.Session.client( service_name = "s3", region_name=<region-name>
                                                aws_access_key_id=<access-id>, aws_secret_access_key=<secret-key>
                                              ) 

This initiates a client object which can be used for Boto3 Operations

Using client object we can start a list_object instance

    paginator = client.get_paginator( "list_objects" )
    page_iterator = paginator.paginate( Bucket = bucket_name, Prefix = prefix )

This will return a paginator Object which we can iterate with for loop and use for Further Operations. For Instance, to create a List of Bucket Object Keys we can do it as 

bucket_object_list = []
for page in page_iterator:
    if "Contents" in page:
        for key in page[ "Contents" ]:
            keyString = key[ "Key" ]
            bucket_object_list.append(keyString)

 

    By Posted On
SENIOR DEVELOPER at MICROPYRAMID

Latest Comments
Related Articles
Configuring and Testing Load Balancer in AWS EC2 Jagadeesh V

When You have an application that is serving Huge Customer Base, so will be your Traffic. Sometimes The Application simply stops responding. We can use ...

Continue Reading...
How to Mount S3 Bucket on Local Disk Dinesh Deshmukh

It all starts with FUSE, FUSE is File System User Space. Operating Systems have Kernel Space and User Space. Kernel Space is where low level ...

Continue Reading...
Easy and Fast way to implement AWS Lambda service Dinesh Deshmukh

We are going to use a simple application called Gordan to prevent creating a lambda function and triggering actions which involves time taking and repetitive ...

Continue Reading...

Subscribe To our news letter

Subscribe to our news letter to receive latest blog posts into your inbox. Please fill your email address in the below form.
*We don't provide your email contact details to any third parties