Large File Handling

Handtruck supports both parallel upload and download.

Multipart upload

For uploading large files multipart uploading can be used. It allows you to asynchronously upload multiple parts of a file to S3. S3Client handles retries of part uploads and calculates part hash for integrity checks.

client = S3Client(url="http://your-s3-host", client=httpx.AsyncClient())

await client.put_file_multipart(
    "test/bigfile.csv",
    headers={
        "Content-Type": "text/csv",
    },
    workers_count=8,
)

Parallel download to file

S3 supports GET requests with Range header. It’s possible to download objects in parallel with multiple connections for speedup. S3Client handles retries of partial requests and makes sure that file won’t be changed during download with ETag header.

Parallel disk writing is also handled, either with pwrite() (Linux, Mac) or by just opening the file multiple times.

client = S3Client(url="http://your-s3-host", client=httpx.AsyncClient())

await client.get_file_parallel(
    "dump/bigfile.csv",
    "/home/user/bigfile.csv",
    workers_count=8,
)