Uploading a file to Amazon S3 using an ASP.NET MVC application directly from the user’s browser

Associated Code

The code associated with this post can be found at the link below (you can also download a .zip file of the code from this link).

https://github.com/floatingfrisbee/amazonfileupload

15th February 2013 Update

There is a limitation in the current implementation in that it does not allow for the FileId to be set to the actual name of the file, and instead uses a pre-generated GUID as the FileId. This was pointed out to me by “DE” in the comments below.

The reason is that the signature is generated before the view is rendered and a file is selected. If the FileId is updated later the upload will fail because the signature will not match the signature that Amazon will compute. This can be remedied (I think) by capturing the onchange event on the file input in the view, and make an ajax call to generate the signature at that time instead of pre-computing it. I’ll work on it if I find time, but anyone is free to fork the Git repo and make that fix!

18th January 2013 Update

A lot of people have asked me if I have the associated code, and I finally was able to find some time to put it up on Git. Here is the link

https://github.com/floatingfrisbee/amazonfileupload

Remember to open web.config and put in your AWS key, secret and bucket id. Besides that it should just run as long as you can run an ASP.NET MVC 4 application.

Original Post

Amazon S3 is a service where you can store large (or small) files. It is organized in terms of “buckets” so before you can store any thing under your account, you have to create a bucket first (or have access to an existing one). Then you can specify the bucket you want to upload the file into, along with other required pieces of information and even your own metadata.

That is a great solution for a lot of web applications, specially those that are already utilizing Amazon’s web services. And being able to upload directly to Amazon S3 from the user’s browser is important because it saves bandwidth and CPU cycles for you as the application owner and time for everyone involved.

Obviously there are a couple of issues here!

1. How do you get a user’s browser to bypass your web application, and upload a file directly to Amazon S3’s storage.

2. How do you ensure that the user’s browser communicates only with Amazon S3 and not an impersonator.

3. How do you ensure that none of the parameters of the upload; things like bucket name, the file’s ACL on S3, and other metadata you might specify, are changed by the user or any intermediary during this process (since the upload is not going though your application).

Fortunately there are solutions for each of the above and really at the end of the day it boils down to the same ol’ trick that most cryptography relies on: symmetric and asymmetric keys. I’ll present the code I used to implement this using an ASP.NET MVC application using C# right after I explain the solution in the far more complicated language of English.

Solutions

1. This one is simple. An HTML form that has a “method” of “post” and an “action” that points to the correct Amazon S3 URI for your bucket will do the trick. Off course the form also needs to have an “input” of type “file” and an “input” of type “submit”. Besides this all the other parameters of the upload; both those that are required by Amazon S3 and others that you might want to store for your purposes, are added as hidden “input” fields to the form.

2. This one is simple too. Just ensure that the form’s action is specified with “https” as the protocol.

Something like this

https://<your bucket name>.s3.amazonaws.com

This will cause the browser to use SSL to communicate with Amazon and the usual handshake, certificate lookup and encryption that goes along with it will ensue.

3. The solution for this is more complicated and judging from the buzz on the web is a source of frustration for many a web developers, and more so for ASP.NET developers because there are not many good code examples available online. The solution boils down this; create a string that has all the parameters of the upload, and then hash that string using your Amazon S3 private key and the HMAC SHA1 algorithm. Upload both that string with the upload parameters and the hash generated using the private key with the HTML form created in step 1, as hidden “input” fields. When Amazon S3 receives the form, it will de-hash the hash (since they have access to your private key), and compare it to the upload parameter string, and to the actual upload parameters. If they match up, they know that the upload parameters were not tampered with, otherwise, they will reject the upload. No one else can generate the correct hash because no one else has your Amazon S3 private key (well, at least that you know about!).

There you have it. Now…

Lets move on to the ASP.NET MVC  implementation

I was using MVC3 and Razor as my view engine. If you’re new to ASP.NET MVC, you probably want to spend a little time understanding the basics.

First the ViewModel

The members of my ViewModel class are basically the things that the view needs to create the form with it hidden and visible fields, and the form action, method and encapsulation type.

public class FileUploadViewModel 
{ 
    public string FormAction { get; set; } 
    public string FormMethod { get; set; } 
    public string FormEnclosureType { get; set; } 
    public string Bucket { get; set; } 
    public string FileId { get; set; } 
    public string AWSAccessKey { get; set; } 
    public string RedirectUrl { get; set; } 
    public string Acl { get; set; } 
    public string Base64Policy { get; set; } 
    public string Signature { get; set; } 
}

Next the code to generate an instance of the ViewModel

You could package this code as a part of another library or as a part of the helpers in your MVC solution but for this example, I’m not going to get into that.

The key part here is off course the generation of the policy string and then generating the signature, which involves taking the hash of the policy string using your Amazon S3 private key and the HMAC SHA1 hashing algorithm. Having a foundational understanding of encodings and string formats is helpful as always. As Joel and many others have said, you should familiarize yourself with base64 and also what a strings and “encoding” mean, specially in .NET.

First lets look at the policy string. The policy string (also called the policy document or the “Access Control” document) must adhere to the rules specified in Amazon S3’s documentation. You should familiarize yourself with the syntax of the policy string described in it. In the end for my case it ended up looking like like this.

{ 
    "expiration": "2011-04-20T11:54:21.032Z", 
    "conditions": [ ["eq", "acl", "private"], ["eq", "bucket": "myas3bucket"], ["eq", "$key", "myfilename.jpg"], ["content-length-range", 0, 20971520], ["eq", "$redirect", "myredirecturl"]] 
}

I used a class marked with the[DataContract] attribute and used the DataContractJsonSerializer to serialize it into the right format. Beats constructing the string in code.

Now lets look at how you would generate the signature. You will probably store the private key in web.config like this

<appSettings>
    <add key="AWSAccessKey" value="MyAmazonS3AccessKey"/>
    <add key="AWSSecretKey" value="MyAmazonS3SecretKey"/>
    <add key="AWSBucket" value="MyAS3Bucket"/>
</appSettings>

and retrieve it like this before passing it to the method to create the signature.

string publicKey = ConfigurationManager.AppSettings["AWSAccessKey"]; 
string secretKey = ConfigurationManager.AppSettings["AWSSecretKey"]; 
string bucketName = ConfigurationManager.AppSettings["AWSBucket"];

Now let’s look at the “Signature”. In Amazon S3’s documentation it is defined as

Signature is the HMAC of the Base64 encoding of the policy

The function below creates the “Signature”. The parameters are “normal” C# strings. Meaning no additional encoding information is being implied. I have made the function more verbose than normal to clarify the steps involved.

private string CreateSignature(string secretKey, string policy) 
{ 
    var encoding = new ASCIIEncoding(); 
    var policyBytes = encoding.GetBytes(policy); 
    var base64Policy = Convert.ToBase64String(policyBytes); 
    var secretKeyBytes = encoding.GetBytes(secretKey); 
    var hmacsha1 = new HMACSHA1(secretKeyBytes); 
    var base64PolicyBytes = encoding.GetBytes(base64Policy); 
    var signatureBytes = hmacsha1.ComputeHash(base64PolicyBytes); 
    return Convert.ToBase64String(signatureBytes); 
}

Let’s walk through the function. First I am converting the policy string into a series of bytes. To do that you must choose an encoding (because different encodings lead to a different series of bytes for the same string). I choose ASCII because that’s what I found was being used in some of the PHP and Ruby samples I came across. Apparently that is correct because this works.

Once you have the series of bytes for the policy string, you need to convert it into a base64 string, which really means reinterpreting the series of bytes in 6 bit chunks to ensure each chunk stays within a certain range. Anyways, .NET provides the handy Convert.ToBase64String function to deal with that.

Once you have the base64 encoded policy string, you need to hash it. To do that you create an instance of the .NET HMAC SHA1 hasher, supplying the bytes of the secret key as a parameter to its constructor. Again we get the bytes of the private key using the ASCII encoder.

Now we use that hasher to hash the policy bytes. The result of that is the set of bytes of the signature which can then be converted into the base64 encoded string and returned.

Base64 encoding has to be used to transmit both the policy string and the generated hash because base64 is a way to guarantee that any kind of characters will be first converted into the ASCII range. That, as I have read, is a fairly common practice.

So in the end the code to generate the view model ends up looking like this

public FileUploadViewModel GenerateViewModel(string publicKey, string secretKey, string bucketName, string fileName, string redirectUrl) 
{ 
     var fileUploadVM = new FileUploadViewMode(); 
     fileUploadVM.FormAction = string.Format(“https://{0}.s3.amazonaws.com/”, bucketName);
     fileUploadVM.FormMethod = “post”; 
     fileUploadVM.FormEnclosureType = “multipart/form-data”; 
     fileUploadVM.Bucket = bucketName; 
     fileUploadVM.FileId = fileName; 
     fileUploadVM.AWSAccessKey = publicKey; 
     fileUploadVM.RedirectUrl = redirectUrl; // one of private, public-read, public-read-write, or authenticated-read 
     fileUploadVM.Acl = “private”; // Do what you have to to create the policy string here 
     var policy = CreatePolicy(); 
     ASCIIEncoding encoding = new ASCIIEncoding(); 
     fileUploadVM.Base64Policy = Convert.ToBase64String(encoding.GetBytes(policy)); 
     fileUploadVM.Signature = CreateSignature(secretKey, policy); 
}

 

Finally the View

The view, like I said is using the Razor engine, and simply lays out a “form” that has all the relevant fields Amazon S3 requires and optionally others you care about.

@model TheApp.ViewModels.FileUploadViewModel
@{ 
    ViewBag.Title = “Upload File”; 
    Layout = “~/Views/Shared/_Layout.cshtml”; 
}

<div id=”fileuploaddiv” class=”fileuploaddivclass”> 
    <form action=”@Model.FormAction” method=”@Model.FormMethod” enctype=”@Model.FormEnclosureType”> 
         <input type=”hidden” name=”key” value=”@Model.FileId” /> 
         <input type=”hidden” name=”AWSAccessKeyId” value=”@Model.AWSAccessKey” /> 
         <input type=”hidden” name=”acl” value=”@Model.Acl” /> 
         <input type=”hidden” name=”policy” value=”@Model.Base64EncodedPolicy” /> 
         <input type=”hidden” name=”signature” value=”@Model.Signature” /> 
         <input type=”hidden” name=”redirect” value=”@Model.RedirectUrl” /> 
         <div> Please specify a file for upload: <input type=”file” name=”file” size=”100″ /> </div> 
         <input type=”submit” value=”Upload” /> 
     </form> 
</div>

Hopefully this helps. Feel free to leave feedback and questions.

About these ads

About floatingfrisbee

A programmer/blogger from New York City
This entry was posted in .net, cloud, integration and tagged , , , , , , , . Bookmark the permalink.

22 Responses to Uploading a file to Amazon S3 using an ASP.NET MVC application directly from the user’s browser

  1. Hairgami_Master says:

    Thanks so much for posting this. I’ve been struggling with this for 2 days now. I wish I could make it work with the AWS .net framework that Amazon publishes. That takes care of the hashing business… I’m just not sure how to connect the dots.

    • Just so I understand, you were able to create the hash using the “CreateSignature” method I have above? If you have a .NET MVC application the rest of the steps will be:

      – Create the policy string
      – Create a view
      – Generate the ViewModel (that has the various properties needed by the hidden fields in the form).

      You should be able to put these pieces together with the examples in my blog post. Shoot me a message or comment if I can help.

      While I was implementing this, I could not find anything that I could use in the Amazon .NET API specifically for this purpose. Also found many RoR and PHP examples but nothing really useful for .NET.

  2. Hi! great article! thanks!
    Do you have any idea how I could limit the size of the file upload with this configuration?
    Rgards,
    Mariano Vicario

  3. Plz don’t answer my question…. I’m reading amazon doc. There it is the answer

    Thanks for your article!

  4. Adam says:

    The problem is the coupling between the client and the storage schema. If you decide to stop using s3 storage in the future, a lot of code will need to be re-written. I’m trying to solve this by creating a separate class library with a FileStorageFactory/Manager wrapper that the client will use.

    • Hi Adam,

      This is kind of a blast from the past but I was just about to upload the code sample because a few different people have asked me for it, and I thought I would reply to your comment. The Factory pattern is exactly how I had written the production version (which unfortunately is no longer in use). The demo version, which I am uploading, doesn’t do that for clarity.

      Thanks,
      Jaspreet

  5. any possibility to mail the code to me?
    thanks

  6. jonny says:

    any possiblity to give code.

  7. jonny says:

    my requirement is i dont have to give a interface just on a click the specified file from a local machine should get uploaded to Amazon Server.

    i am trying the following code:
    PutObjectRequest request = new PutObjectRequest();
    request.WithBucketName(BUCKET_NAME);
    request.WithKey(S3_KEY);//.WithCannedACL(S3CannedACL.PublicRead);
    request.WithCannedACL(S3CannedACL.PublicRead);
    request.WithFilePath(pathToFile);
    client.PutObject(request);

    this code works fine from my visual studio but when hosted on webserver gives me error “Specified File Does Not exists.”.

    if i follow your way. so what will happen to this code do i still require the code i have mentioned ?

    what i feel is it is not able to understand the request.WithFilePath(pathToFile); as it contain my file path.

    Please help me.

  8. Oya says:

    (x2) any possibility to mail the code to me?
    thanks

  9. Adam says:

    I am having problems with generating the policy. Do you have a sample code?

  10. Oya and Adam… I am very busy these days but will try and dig out the code for you. Stay tuned.

  11. I’ve just updated the blog with a link to the the code. It took me a while to get the code uploaded but I hope it is still useful for some of you.

    Thanks
    Jaspreet

  12. DE says:

    How do we get the filename as the ID? Or real question is how do we get the same filename as we uploaded? ?In the sample code the Model.FileId is hardcoded in the key field and is a guid. In the mention in the blog posting filename is there. Thanks, very helpful!

  13. DE,

    My current implementation does not actually allow for the FileId to be set to the name of the file, and as you said, uses a GUID as the FileId. I will update the post with this clarification. The reason that using the real file name is a bit more complex is that in the current implementation the signature needs to be generated before the view is rendered and a file is selected. If the FileId is updated later the upload will fail because the signature will not match the signature that Amazon will compute. An idea to enhance this to support real file names is to capture the onchange event on the file input in the view, and make an ajax call to generate the signature at that time instead of pre-computing it.

    If I find time, I might work on that but of course you are welcome to fork the repo and make that change.

    Thanks
    Jaspreet

  14. Great blog! It addressed all my file uploading questions. Thanks a lot!

  15. barni says:

    Great post! I migrate your code to asp.net (not mvc) and it’s work great. Thanks!
    Do you have what to do with my code?

  16. Joe says:

    Did anyone ever write the AJAX call to set the policy so that we can upload the file name that the user has selected? I am new to AWS/S3 and .NET MVC so it has been a struggle for me.

  17. Rory Kingan says:

    Thanks for this, it’s really helpful to see a fully-worked code example for this.

    Regarding the filename, I don’t think in generaly you need to know the FileId/key before generating your signature, it’s only required because your _current_ policy includes the key. The example on the AWS docs doesn’t specify the key but instead uses starts-with, puts a path ending in “/” in the “key” form field, and the file is therefore uploaded to the given path within the bucket and gets the name from the “file” field.

    http://docs.aws.amazon.com/AmazonS3/latest/dev/HTTPPOSTExamples.html

    Seems like that should work fine for you too?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s