Paginating S3 objects using boto3

Reading Time : ~ .

When using Boto you can only List 1000 objects per request. So to obtain all the objects in  the bucket. You can use s3's paginator. To use paginator you should first have a client instance 

client = boto3.Session.client( service_name = "s3", region_name=<region-name>
                                                aws_access_key_id=<access-id>, aws_secret_access_key=<secret-key>
                                              ) 

This initiates a client object which can be used for Boto3 Operations

Using client object we can start a list_object instance

    paginator = client.get_paginator( "list_objects" )
    page_iterator = paginator.paginate( Bucket = bucket_name, Prefix = prefix )

This will return a paginator Object which we can iterate with for loop and use for Further Operations. For Instance, to create a List of Bucket Object Keys we can do it as 

bucket_object_list = []
for page in page_iterator:
    if "Contents" in page:
        for key in page[ "Contents" ]:
            keyString = key[ "Key" ]
            bucket_object_list.append(keyString)

 

    By Posted On
SENIOR DEVELOPER at MICROPYRAMID

Need any Help in your Project?Let's Talk

Latest Comments
Related Articles
Deploying Django project on Elastic Beanstalk Shirisha Gaddi

Here You can learn about how to setup and deploy a Django application to Amazon Web Services (AWS).

Tools/technologies used:
Python v2.7
Django ...

Continue Reading...
Deploy Django using CloudFormation Template Dinesh Deshmukh

CloudFormation helps in Using JSON templates to describe the resources needed from aws. With this approach, we don't have to repeat the same manual configuration ...

Continue Reading...
How to process message queuing system by amazon SQS Nikhila Mergu

Amazon Simple Queue Service (Amazon SQS) is a distributed messaging queue service.

Queued items in SQS are called messages which are variable in size but ...

Continue Reading...

Subscribe To our news letter

Subscribe to our news letter to receive latest blog posts into your inbox. Please fill your email address in the below form.
*We don't provide your email contact details to any third parties