Okay, AWS S3 is actually very good service that offers pretty simplified interface to upload stuffs to it. And it's very fast! Like, real fast! Some of the image assets for the posts in this blog are hosted in my own file server, in which I just used NGINX to serve the files. It was far better than letting Node.js handles the file serving, but somehow still worse than how AWS serves their files through S3. Maybe there is CloudFront behind all of these, but in the end I got a much less waiting time and TTFB (Time to First Byte) response when trying to fetch data from S3. Maybe that's also because of my position (me being in SEA, accessing files in ap-southeast S3 bucket).

But here is the thing, while it's very good to have such a performance, the need to manually upload the files are pretty boring and mendokusai. But of course there are SDK to interact with AWS services, and of course S3 being one of them. It's just that it requires the client secret key in order to be authorized to use these services, which something that we don't want to let our users have their hands on. Of course, we need a security layer, a way to let our users to upload to our S3 buckets, but without giving any unnecessary credentials to use our AWS services.

Enter, presigned URL...

S3 Presigned URL and Server-side Authorization

I might just putting words to that heading because I think it sounds right.

Let's say that you're building an web app that allows user to upload images. Fine example is image sharing website like deviantArt or Pixiv, where user interactions are mostly involve assets that's user-submitted through the website. Of course, you don't want to define a new bucket for each user because not only that's kinda stupid, you'll quickly get into the bucket count limitation of S3. Standard AWS users can get up to 100 buckets and 1000 if your service limit request get approved. But of course, nobody does this. Developers are straight using a single bucket and let the user segmentation goes through the subfolder. You might face some authorization problem with this, but eh, it's a problem for another day.

Now, how do we upload stuff as a user that goes to a single bucket, in which the same bucket are used by other users also? How do we limit user's access before interacting with our bucket? And most importantly, how do we give users access to upload something to the bucket, after everything is verified and authorized? That's where S3 Presigned URL goes in.

S3 Presigned URL

It goes as simple as giving a link to a user where they can use that very URL as a way to upload something to S3 bucket. The term of "presigned" here means that we (developer) is giving an access, and that access is "given approval in advance", so they (users) don't need our manual approval to advance. And this can be done programmatically, so that solves the automation problem.

Here, I'll be using Node.js SDK for AWS, using the v2.0 SDK version. There is newer SDK version, but here I'm using it because I also use another library that still use v2.0 AWS SDK. I definitely can just use both, but eh, probably later on.

import aws from 'aws-sdk'
// setup your credentials
aws.config.update({
  accessKeyId2: [YOUR ACCESS ID HERE],
  secretAccessKey: [YOUR ACCESS KEY HERE],
  region: [YOUR TARGET BUCKET REGION HERE],
  signatureVersion: 'v4',
});
const today = new Date();
today.setTime(today.getTime() + today.getTimezoneOffset() * 60000 + 8 * 3600000);
console.log(today.toISOString());
const keyDirectory = `${today.getFullYear()}/${today.getMonth() + 1}/${today.getDate()}`;
const s3 = new aws.S3();
const presignedUrl = s3.createPresignedPost({
  Bucket: [YOUR BUCKET NAME HERE],
  Fields: {
    key: `${keyDirectory}/${req.body.filename}`,
  },
  Expires: 60,
});

So, what is happening here?

Setting up your AWS credentials. I'm using the standard credentials of access id and access key retrieved from IAM console when creating a user. It's a best practice to create an IAM user with limited permission to prevent nasty things happening in case of credentials leak.
Building up my string path. It's just for convenience so my files will be uploaded into directory structure of `year/month/date/item` structure.
Create S3 API object and create pre-signed URL. It's pretty straightforward, and the result will be a URL that can be used as a target for uploading the file.

Because pre-signed URL is basically granting the user your permission to do stuffs, anyone that can get this URL will be able to do whatever they want with your S3 buckets. So it's always wise to put authentication first when using this.

Server-side Authorization

As I'm using NextJS because it's cool, I can add API routes that can specifically handle the authorization of the said task. As I'll make my users submit a request for a presigned URL, I'll need the basic metadata of the file that's going to be uploaded, and user's data for organizing the directory segmentation. It all goes as a simple POST request to the API route, with all the necessary data needed attached in the body. You can do this in frontend by using Axios or Fetch API. Either way, just make a POST request to the API route, and let the server checks.

I'm authorizing by using cookies, so obviously I'll need to check and verify the cookies. If everything goes well, I'll just let the S3 AWS SDK to handle the generation of presigned URL. The response of the API route will be the metadata of the file and the presigned URL. The next part is to let the user's browser to handle the real file upload through the presigned URL. This way, we've prevented our user to access AWS services by using our credentials, and we avoided unnecessary HTTP traffic into our server just to upload the files.

Creating API routes in NextJS is pretty easy. It's just like when using the standard page routes but here you write it as how you would write an Express.js route handlers.

export default async (req: NextApiRequest, res: NextApiResponse): Promise<any> => {
  // do whatever you want here as how you would do it in Express.js
}

Okay, I know that is very abstracted, but you can read the docs on what stuffs can you do with the request and how you would handle the request. The given request and response object in the function parameters highly resembles Express.js, so yeah, go for it.

In this scenario, make sure you create an API that can return a pre-signed URL when authorized.

React Dropzone Component

Now we're hopping to the UI. You can actually just create a file input and call a handler when file(s) selected. The flow would be like this.

Listen for any file-related events on the input. It's up to you on which event to listen.
Verify the file is within your specification (file size, file format, name conventions, etc...)
Make an API call to the route we made before. This is to request the pre-signed URL.
In case of successful call and we get the pre-signed URL, upload the file using POST request to the pre-signed URL.

For mobile users, you can go straight to creating UI button for handling the file input, but if you're in desktop then you know you can do much better. I personally love the dropzone boxes that various uploaders use, because in most of the time we already have a file explorer window opened up to the location of our files, and it's just drag and drop.

So yeah, we will make dropzone component.

In here, I'll be using a very good library called react-dropzone. It exposes a hook that can be used to handle the pointer events.

import { useDropzone } from 'react-dropzone'

// somewhere in your react component
const onDrop = async (files: File[]) {
  // do the logic described above
}

const { getRootProps, getInputProps, isDragActive } = useDropzone({ onDrop })

return (
  // somewhere in the JSX tags<div {...getRootProps()}><input {...getInputProps()} /></div>
)

And that's it! Create a UI component for dropzone, attach the variables and props needed from the library, and handle the events when there are something dropped. It's up to you on how you would stylize your dropzone. I do it like this.

<div
  {...getRootProps()}
  className={`p-4 ${
    isDragActive ? 'bg-blue-100' : 'bg-blue-200'
  } text-center text-blue-700 text-xl`}
  style={{
    transition: 'background-color 500ms',
  }}
>
  <div className="p-6 border-blue-900 border-dashed border-2">
    <input {...getInputProps()} />
    {isDragActive ? (
      <p>Drop the files here...</p>
    ) : (
      <p>Drag n drop some files here, or click to select files</p>
    )}
  </div>
</div

The result looks like this.

So that's it! Using AWS S3 can be confused at first because there are just too many stuffs to deal with and you probably don't know where to go. But it's all really a part of learning and exploration.

I hope this tutorial helps. Thanks for reading!

See ya.

AWS Shenanigans #2: S3 File Upload using Next JS and React Dropzone