One of the array of issues with using Python’s built-in URL support is that it doesn’t really stream uploads. However, there are remedies out there.
A package called poster has functionality for doing so, and simples3 has built-in helpers for using the two together.
Note: this of course requires poster to be installed.
Usage:
>>> from simples3.streaming import StreamingS3Bucket
>>> bucket = StreamingS3Bucket(usual_arguments)
>>> bucket.put_file("huge_cd.iso", "foo/huge_cd.iso", acl="public-read")
Or alternatively, you can pass file-like objects:
>>> with open("foo/huge_image", "rb") as fp:
... bucket.put_file("memdump.bin", fp)
And that’s all there is to it.