On-Demand File Compression in the AWS S3

If you are like me and happen to have an account with Amazon S3, there are certainly times when you want to share those files you have in storage and have been frustrated by your efforts.
I had a program last year that dynamically generated lots of images and needed a way to share those images with third parties later. After looking around for a while, it looked like most web developers faced this same problem and somehow fixed it but the only available information online was not satisfactory to me.

The Old Way

The old way of sharing files in Amazon S3 is to download files from S3 storage, zip them locally, and upload them back to S3. While this method works, it is definitely not good if you have less storage and need to create a large zip file. It is also very resource intensive and time consuming. It also requires cleanup at the end and the entire process consumes almost twice the storage needed for the zipped file.
Another possible option is to compress the files before putting them in S3 storage. This method works well for static files but does not address the case where clients need an assortment of files. It also does not cover dynamically generated files.

The Real Problem

All those solutions are good, but they all fail to address the real world issue where files are generated dynamically. There is no way to know upfront what clients require or which files need sharing. It also does not address the situation where a user or web developer has less RAM and storage memory to squander around.

Introducing S3zipper

S3zipper is written in Go(Golang) , and its main strength is to automate the process of compressing files in Amazon S3 and sharing them. It can do both Zip and Tar compression methods which are obviously the most popular.
All you need to do is make a few API calls, and the rest is taken care of from our end. Only our resources get used from our end, and nothing gets touched from your end. It is also super fast.

Advantages

• S3zipper is good for dynamically generated files and also static files. You can specify a bucket, folder, or link to a file that was just created.
• It uses file like links. It is not complicated.
• It does not use any client server resources .
• It is super fast.
• It is good for file sharing.
• It also works perfectly for small servers that have less storage and memory as none of that get used.

AWS Marketplace

s3zipper is also available in AWS Marketplace as an ec2 instance.
S3zipper

API Documentation

The API is also available below.
S3zipper API