When using Boto you can only List 1000 objects per request. So to obtain all the objects in  the bucket. You can use s3's paginator. To use paginator you should first have a client instance 

client = boto3.Session.client( service_name = "s3", region_name=<region-name>
                                                aws_access_key_id=<access-id>, aws_secret_access_key=<secret-key>
                                              ) 

This initiates a client object which can be used for Boto3 Operations

Using client object we can start a list_object instance

    paginator = client.get_paginator( "list_objects" )
    page_iterator = paginator.paginate( Bucket = bucket_name, Prefix = prefix )

This will return a paginator Object which we can iterate with for loop and use for Further Operations. For Instance, to create a List of Bucket Object Keys we can do it as 

bucket_object_list = []
for page in page_iterator:
    if "Contents" in page:
        for key in page[ "Contents" ]:
            keyString = key[ "Key" ]
            bucket_object_list.append(keyString)

 

Subscribe To our news letter

Subscribe to our news letter to receive latest blog posts into your inbox. Please fill your email address in the below form.
*We don't provide your email contact details to any third parties
Latest Comments
Related Articles
Amazon SES - Handling Bounces and Complaints. Vamsi Popuri

In general while sending emails, we will prepare some recipient addresses as our mailing list, which are valid and our recipients want and expect our ...

Continue Reading...
Creating Elastic Search Cluster (EC2, cloud-aws Plugin) Jagadeesh V

While handling Large amounts of data with elasticsearch, you may run out of server capacity or compute power, Forming a Elasticsearch cluster will reduce the ...

Continue Reading...
how to setup custom domain for amazon cloudfront Ashwin Kumar

We all want our own domain name to be setup for cloud front instead of amazon default cloud front domain name. We need two things ...

Continue Reading...