How to list all (>1000) objects in s3 bucket

help
1

Hello,

This is an echo of the following github issue - https://github.com/rusoto/rusoto/issues/553

(Note: I am using digital ocean spaces)

I tried doing the following, but I can't seem to get the continuation token for getting the next batch of objects after 1000.

(Access keys loaded through environment variables)

let s3 = S3Client::new(Region::Custom {
    name: String::from(""),  // region name in endpoint
    endpoint: String::from("https://sgp1.digitaloceanspaces.com"),
});

let mut request = ListObjectsV2Request {
    bucket: String::from("my_cool_bucket"),
    ..Default::default()
};

let result = s3.list_objects_v2(request.clone()).await?;

assert!(result.next_continuation_token.is_some());  // <-- panics here

I'm actually trying to port some code from python, and boto3's resource API is smart enough to fetch objects all in one go.

s3 = boto3.resource(
    "s3", 
    aws_access_key_id="...", 
    aws_secret_access_key="...", 
    endpoint_url="https://sgp1.digitaloceanspaces.com",
)

s3.Bucket("my_cool_bucket")
print(list(bucket.objects.all()))  # all 21399 fetched in a single call

This is my first day of rust, please forgive me if this is due to some rookie mistake :slight_smile:

2

Digital Ocean doesn't talk about continuation tokens in their API reference /reference/api/spaces-api/

Also:

3

Thanks for all the help.

I wrote a little solution that seems to work.

Any comments are appreciated.

#[async_trait]
trait ListAllExt {
    async fn list_all_objects(
        &self,
        mut request: ListObjectsRequest,
    ) -> Result<Vec<Object>, Box<dyn Error>>;
}

#[async_trait]
impl ListAllExt for S3Client {
    async fn list_all_objects(
        &self,
        mut request: ListObjectsRequest,
    ) -> Result<Vec<Object>, Box<dyn Error>> {
        let mut results = vec![];

        loop {
            let result = self.list_objects(request.clone()).await?;

            match result.contents {
                Some(x) => results.extend(x),
                None => break,
            }

            request.marker = result.next_marker;
            if request.marker.is_none() {
                break;
            }
        }

        Ok(results)
    }
}

Usage -

let s3 = S3Client::new(Region::Custom {
    name: String::from(""),
    endpoint: String::from("https://sgp1.digitaloceanspaces.com"),
});

let request = ListObjectsRequest {
    bucket: "my_cool_bucket",
    ..Default::default()
};

println!("Listed {:?} objects.", s3.list_all_objects(request).await?.len());
1 Like
4

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.