Uploading to S3 in 18 lines of Shell

[chrismdp]

https://gist.github.com/chrismdp/6c6b6c825b07f680e710

[estefan]

aws s3 cp src dest...

OK curl is installed by default but if you have to install this script anyway, it's only a `pip install` away...

[skywhopper → estefan]

Some platforms and environments don't have pip handy, or you might not have root access, or you can't install new software for any number of reasons, or the required libraries are not installed. curl and /bin/sh and openssl are more universal, so it's good to be able to do what you need to do without resorting to installing software.

[insertnickname → skywhopper]

You don't need root to install Python packages with pip. You can install them just for your user. See https://pip.readthedocs.org/en/latest/user_guide.html#user-i...

[skywhopper → insertnickname]

... right, which is why I listed all the other reasons you might not be able to install a python package.

[chrismdp → skywhopper]

Exactly my thinking. Nice to have one thing less to install on a new Jenkins node, for example.

[mdellabitta → chrismdp]

Except this won't work with files bigger than 5gb. It might be worth the time to do it right.

[skywhopper → mdellabitta]

I don't think the author intended this script to solve everyone's problems. It's an interesting exercise and can be useful in some situations. If it's not useful for your needs, then as has been pointed out there are many many other options.

[toomuchtodo → mdellabitta]

> It might be worth the time to do it right.

The right way is the most simple way for your use case. For files under 5GB, this is one of the right ways.

[chrismdp → chrismdp]

I'll fix it when my game requires update files of >5GB :)

[falcolas → estefan]

Even if the script itself is not "practical", it's very informative. It makes it quite simple to translate into another language for the times when you don't want to shell out to another process.

[sbochins → estefan]

I found this script a couple weeks ago and was very useful for the minimal docker image I was creating. In some cases you have certain constraints when accessing s3 and this script is helpful in those cases.

[mietek]

If you like this, you may find bashmenot useful. Among other things, it includes GNU bash functions to work with S3.

Documentation:

https://bashmenot.mietek.io/reference

Source:

https://github.com/mietek/bashmenot

[phireal]

Searching GitHub for S3SECRET and S3KEY might get interesting if this script gets used a lot.

[scrollaway → phireal]

I find the wasted opportunity to use the S3CRET environment variable a shame.

[chrismdp → phireal]

Yeah, I know :/

People have crawled github for these in the past and turned up some pretty "interesting" results...

[hijinks → phireal]

github now auto notifies AWS if it sees you publish secrets.. Then you get an email from AWS saying your keys are public and replace them asap

[toomuchtodo → hijinks]

That's actually pretty awesome.

[tonyhburns]

Here's a couple I made as well for upload/download to S3 before Travis CI had Docker containers with support for caching in OSS projects: https://github.com/tabolario/ci-scripts

These use the V4 signature, which I grabbed from an article somewhere on the subject that I can't seem to find anymore :( The cached-bundle script is inspired and based off the one from devise: https://github.com/plataformatec/devise/blob/master/script/c...

[blackace]

There's a 2007 bash implementation of s3 commands here too:

https://code.google.com/p/s3-bash/

[chrismdp → blackace]

Yeah, thanks for linking to this: the reason I didn't use it was mostly because it was longer than 18 lines :) Nice not to have to track and maintain a version of other people's code when you don't have to.

[jaryd]

Important to note that this uses the public-read ACL by default.