Upload Files To A Minio Object Storage Cloud With Node.js And Multer

The recent Amazon S3 outage that took down much of the internet inspired me to talk about alternatives. Not too long ago I wrote about an open source object storage software called Minio and how I was using it on my Raspberry Pi for backups. The great thing about Minio is it shares the same APIs as AWS S3, but can be deployed to your own hardware, eliminating Amazon as a dependency.

This time around I thought it would be great to share how to use Minio as an object storage for a Node.js application that uses the middleware, Multer, for handling file uploads.

If you’ve been keeping up with the blog, you’ll remember that this isn’t the first time I wrote about file uploads in a Node.js application. Not too long ago I wrote about uploading files via Angular to a Node.js server with Multer. The example we explore here won’t have a front-end, but it shouldn’t stop you from creating your own.

The example we build will use the Minio playground, so be cautious what you choose to upload. Feel free to use your own Minio instance instead.

Create a new Node.js project by executing the following commands:

npm init --y
npm install express minio body-parser multer --save

The above commands will create a new package.json file and install all our project dependencies. This will be an API based application so we’ll be using express and the body-parser package for handling endpoints and requests. The multer package will handle our file uploads and the minio package will allow interactions with a Minio server.

Create a file called app.js which will hold all our application logic. Open this file and include the following code:

var Express = require("express");
var Multer = require("multer");
var Minio = require("minio");
var BodyParser = require("body-parser");
var app = Express();

app.use(BodyParser.json({limit: "4mb"}));

var minioClient = new Minio.Client({
    endPoint: 'play.minio.io',
    port: 9000,
    secure: true,
    accessKey: 'Q3AM3UQ867SPQQA43P2F',
    secretKey: 'zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG'
});

app.post("/upload", Multer({storage: Multer.memoryStorage()}).single("upload"), function(request, response) {
    minioClient.putObject("test", request.file.originalname, request.file.buffer, function(error, etag) {
        if(error) {
            return console.log(error);
        }
        response.send(request.file);
    });
});

app.post("/uploadfile", Multer({dest: "./uploads/"}).single("upload"), function(request, response) {
    minioClient.fPutObject("test", request.file.originalname, request.file.path, "application/octet-stream", function(error, etag) {
        if(error) {
            return console.log(error);
        }
        response.send(request.file);
    });
});

app.get("/download", function(request, response) {
    minioClient.getObject("test", request.query.filename, function(error, stream) {
        if(error) {
            return response.status(500).send(error);
        }
        stream.pipe(response);
    });
});

minioClient.bucketExists("test", function(error) {
    if(error) {
        return console.log(error);
    }
    var server = app.listen(3000, function() {
        console.log("Listening on port %s...", server.address().port);
    });
});

There are several different API endpoints in the above code with a lot going on. We’re going to break it down and figure out what everything means.

The first thing we’re doing is importing all of the packages that were previously downloaded:

var Express = require("express");
var Multer = require("multer");
var Minio = require("minio");
var BodyParser = require("body-parser");
var app = Express();

We are also initializing Express in the above code.

By default, uploads are limited to around 100kb in size. Feel free to leave it as the default, or define your own upload limit. For this application, I’ve set the upload limit to 4mb.

var minioClient = new Minio.Client({
    endPoint: 'play.minio.io',
    port: 9000,
    secure: true,
    accessKey: 'Q3AM3UQ867SPQQA43P2F',
    secretKey: 'zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG'
});

The above code initializes the Minio SDK for Node.js. We define the server node we wish to connect to and provide the relevant credentials to make it possible. As a reminder, in this example we’re using the Minio playground where as in production you should use your own cloud.

Before we worry about the API endpoints, we need to start the Node.js server:

minioClient.bucketExists("test", function(error) {
    if(error) {
        return console.log(error);
    }
    var server = app.listen(3000, function() {
        console.log("Listening on port %s...", server.address().port);
    });
});

Before starting the server we make sure that the Bucket exists. There are commands to create Buckets via Node.js, but for this example we assume it had already been created.

When it comes to uploading files to a Bucket, there are two options. We can upload via a stream, or upload as a file.

app.post("/upload", Multer({storage: Multer.memoryStorage()}).single("upload"), function(request, response) {
    minioClient.putObject("test", request.file.originalname, request.file.buffer, function(error, etag) {
        if(error) {
            return console.log(error);
        }
        response.send(request.file);
    });
});

In the above example we are uploading via a stream. This means that the files will not touch our Node.js filesystem, but instead go straight into our Bucket. If using streams, it is important to understand the limitations of your server. While more convenient, it will be more costly on your server resources.

You also have the option to upload a file that first hits your Node.js filesystem:

app.post("/uploadfile", Multer({dest: "./uploads/"}).single("upload"), function(request, response) {
    minioClient.fPutObject("test", request.file.originalname, request.file.path, "application/octet-stream", function(error, etag) {
        if(error) {
            return console.log(error);
        }
        response.send(request.file);
    });
});

In the above example, the file will first be uploaded to the uploads directory of your project and then saved to the Minio storage. Just remember to clean the files after they’ve been uploaded to save space on your Node.js server.

Finally we have an endpoint for downloading data from Minio:

app.get("/download", function(request, response) {
    minioClient.getObject("test", request.query.filename, function(error, stream) {
        if(error) {
            return response.status(500).send(error);
        }
        stream.pipe(response);
    });
});

In the above example, we download an object stream and pipe it directly to the client that requested it. This prevents us from first having to download it to the Node.js server. However, we have the option to download it first.

Give the application a spin. You can leverage the power of S3, but in an open source alternative solution.

Conclusion

You just saw how to use Node.js and Multer to upload files to a Minio object storage server and download them as well. This is a great alternative to Amazon S3 because it is open source and can be deployed to your own servers. The Minio SDK uses the same APIs that you’d expect in AWS S3.

Want to develop a nice front-end for this Node.js application? Check out the Angular tutorial I wrote for uploading files to Node.js with Angular.

Nic Raboy

Nic Raboy is an advocate of modern web and mobile development technologies. He has experience in C#, JavaScript, Golang and a variety of frameworks such as Angular, NativeScript, and Unity. Nic writes about his development experiences related to making web and mobile development easier to understand.