salesforce logo

If you are looking for salesforce consulting services then

Contact here

Paginating S3 objects using boto3

When using Boto you can only List 1000 objects per request. So to obtain all the objects in the bucket. You can use s3 paginator. To use paginator you should first have a client instance 

client = boto3.Session.client( service_name = "s3", region_name=<region-name>
                                                aws_access_key_id=<access-id>, aws_secret_access_key=<secret-key>
                                              ) 

This initiates a client object which can be used for Boto3 Operations

Using client object we can start a list_object instance

    paginator = client.get_paginator( "list_objects" )
    page_iterator = paginator.paginate( Bucket = bucket_name, Prefix = prefix )

This will return a paginator Object which we can iterate with for loop and use for Further Operations. For Instance, to create a List of Bucket Object Keys we can do it as 

bucket_object_list = []
for page in page_iterator:
    if "Contents" in page:
        for key in page[ "Contents" ]:
            keyString = key[ "Key" ]
            bucket_object_list.append(keyString)

 

About Micropyramid

Micropyramid is a software development and cloud consulting partner for enterprise businesses across the world. We work on python, Django, Salesforce, Angular, Reactjs, React Native, MySQL, PostgreSQL, Docker, Linux, Ansible, git, amazon web services. We are Amazon and salesforce consulting partner with 5 years of cloud architect experience. We develop e-commerce, retail, banking, machine learning, CMS, CRM web and mobile applications.


Posted On 28 September 2015 By MicroPyramid


Need any Help in your Project?Let's Talk

Latest Comments

Subscribe To our news letter

Subscribe and Stay Updated about our Webinars, news and articles on Django, Python, Machine Learning, Amazon Web Services, DevOps, Salesforce, ReactJS, AngularJS, React Native.
* We don't provide your email contact details to any third parties