Paginating S3 objects using boto3

When using Boto you can only List 1000 objects per request. So to obtain all the objects in  the bucket. You can use s3's paginator. To use paginator you should first have a client instance 

client = boto3.Session.client( service_name = "s3", region_name=<region-name>
                                                aws_access_key_id=<access-id>, aws_secret_access_key=<secret-key>
                                              ) 

This initiates a client object which can be used for Boto3 Operations

Using client object we can start a list_object instance

    paginator = client.get_paginator( "list_objects" )
    page_iterator = paginator.paginate( Bucket = bucket_name, Prefix = prefix )

This will return a paginator Object which we can iterate with for loop and use for Further Operations. For Instance, to create a List of Bucket Object Keys we can do it as 

bucket_object_list = []
for page in page_iterator:
    if "Contents" in page:
        for key in page[ "Contents" ]:
            keyString = key[ "Key" ]
            bucket_object_list.append(keyString)

 

    By Posted On
SENIOR DEVELOPER at MICROPYRAMID

Need any Help in your Project?Let's Talk

Latest Comments
Related Articles
Django Hosting on Amazon EC2 with wordpress on same domain Jagadeesh V

Configuring the Wordpress as subdirectory can be tricky. In this tutorial we will Setup a Django Website alongside a wordpress blog.

Continue Reading...
Amazon SES - Handling Bounces and Complaints Vamsi Popuri

In general while sending emails, we will prepare some recipient addresses as our mailing list, which are valid and our recipients want and expect our ...

Continue Reading...
Amazon AWS IAM Roles and Policies Ashwin Kumar

When You want to Provide access to Amazon Web Services Console or if you're planning to provide REST API Keys to your Developers of a ...

Continue Reading...

Subscribe To our news letter

Free news and articles on Django, Python, Machine Learning, Amazon Web Services, DevOps, Salesforce, ReactJS, AngularJS, React Native.
*We don't provide your email contact details to any third parties