Create A URL Shortener With Golang And Couchbase NoSQL

Continuing down the road of Golang development I thought it would be a cool learning example to develop a URL shortener application similar to that of TinyURL or Bitly. I think these are great examples because not only does it teach you how to develop a RESTful API that uses a datasource, but it also challenges you to think critically when it comes to the algorithms.

Previously I had written about creating a URL shortener using my other favorite server-side technology, Node.js, but the languages are so different that it makes sense to try the same with the Go programming language. In this example we’re going to create an application that makes use of Golang, Couchbase, and a few other project dependencies.

The Requirements

There are a few requirements that must be satisfied in order to make this project a success. They are the following:

• Go 1.7+
• Couchbase Server 4.1+

This URL shortener application is going to be dependent on what are called N1QL queries, which are SQL queries for Couchbase’s NoSQL database. While those were introduced in Couchbase Server 4.0, you’ll be better off with a newer version.

Preparing the Database and Understanding the Data Model

To keep track of short and long URLs, we’re going to need a database. Based on the information we’re storing, it makes sense to go for a NoSQL solution. For this we’re going to use the open source database, Couchbase.

Download the appropriate version of Couchbase for your operating system and proceed to install it. The database is not a very heavy software, so feel confident when installing it on your local machine that it will not drain all your resources. During the setup phase, you will need to have the query service enabled.

We will need at least one bucket created and configured for our project. This can be the Couchbase default bucket or something else. If you want to create a new bucket, from the administrative dashboard choose Data Buckets and then choose Create New Data Bucket. Give the bucket a name and define a storage capacity.

Because we plan to use N1QL queries, we need at least one index created on the bucket we plan to use. There are several ways to do this. You can use either the Couchbase Query Workbench, or the Couchbase shell called CBQ. The index query would look like the following:

CREATE PRIMARY INDEX ON `bucket-name` USING GSI;

More specific indexes can be created to improve performance in a larger application.

If your version of Couchbase does not have Query Workbench, the shell tool CBQ is more than sufficient for this quick job.

This brings us to our data model. When creating a URL shortener application, the whole idea is that you pass a longer URL and get a shorter URL in return. The URLs in both directions are stored in the database. For example, our data model might look like the following:

{
    "id": "5Qp8oLmWX",
    "longUrl": "https://www.thepolyglotdeveloper.com/2016/08/using-couchbase-server-golang-web-application/",
    "shortUrl": "http://localhost:3000/5Qp8oLmWX"
}

The id is the unique short hash that will reference our document and be used within any URL that we distribute. Nothing super complicated here.

Now we’re ready to start building our Golang application!

Bootstrapping the Golang RESTful Application

We will be creating a RESTful API, but before we start defining the logic of each endpoint, we should worry about getting our server up and running.

Create a new Golang project. I’ll be calling my project main.go and it will be found in my $GOPATH/src/github.com/nraboy/shorturl path. Add the following code to get things started:

package main

import (
    "net/http"
    "github.com/couchbase/gocb"
    "github.com/gorilla/mux"
)

var bucket *gocb.Bucket
var bucketName string

func ExpandEndpoint(w http.ResponseWriter, req *http.Request) { }

func CreateEndpoint(w http.ResponseWriter, req *http.Request) { }

func RootEndpoint(w http.ResponseWriter, req *http.Request) { }

func main() {
    router := mux.NewRouter()
    cluster, _ := gocb.Connect("couchbase://localhost")
    bucketName = "example"
    bucket, _ = cluster.OpenBucket(bucketName, "")
    router.HandleFunc("/{id}", RootEndpoint).Methods("GET")
    router.HandleFunc("/expand/", ExpandEndpoint).Methods("GET")
    router.HandleFunc("/create", CreateEndpoint).Methods("PUT")
    log.Fatal(http.ListenAndServe(":12345", router))
}

Let’s go over what we see in the above Golang code.

You’ll probably notice two imports that are foreign to you as a Go programmer. The imports will allow us to use the Couchbase Go SDK and a Mux tool that makes it easier for us to create a RESTful API. Both packages can be installed through the following:

go get github.com/couchbase/gocb
go get github.com/gorilla/mux

Next we want to create two variables that will be accessible throughout the entire main.go file. We need to keep a public copy of our open bucket and the name of that bucket.

Skipping into the main method we have the initialization of our application. We configure our application router, connect to our locally running Couchbase cluster, and open a bucket of our choosing. In this case I am opening a bucket called example which happened to have existed in my cluster.

Before starting the server, we create three routes which will represent our API endpoints. The /create route will allow us to pass in a long URL and receive a short URL. The /expand endpoint will allow us to pass in a short URL and receive a long URL. Finally, the root endpoint will allow us to pass in a hash and be redirected to the long URL page.

Let’s see how we would accomplish each of the routes.

Adding Logic to the API Endpoints

Before we can worry about the API endpoints we need to define our data model. This will be handled using a Golang data structure.

type MyUrl struct {
    ID       string `json:"id,omitempty"`
    LongUrl  string `json:"longUrl,omitempty"`
    ShortUrl string `json:"shortUrl,omitempty"`
}

The structure MyUrl has three properties, each mapped to a JSON property. The JSON property is what we’ll find the in the database and the structure property is what we’ll use in the code.

This brings us to the /create endpoint which is probably the more complicated of the three:

func CreateEndpoint(w http.ResponseWriter, req *http.Request) {
    var url MyUrl
    _ = json.NewDecoder(req.Body).Decode(&url)
    var n1qlParams []interface{}
    n1qlParams = append(n1qlParams, url.LongUrl)
    query := gocb.NewN1qlQuery("SELECT `" + bucketName + "`.* FROM `" + bucketName + "` WHERE longUrl = $1")
    rows, err := bucket.ExecuteN1qlQuery(query, n1qlParams)
    if err != nil {
        w.WriteHeader(401)
        w.Write([]byte(err.Error()))
        return
    }
    var row MyUrl
    rows.One(&row)
    if row == (MyUrl{}) {
        hd := hashids.NewData()
        h := hashids.NewWithData(hd)
        now := time.Now()
        url.ID, _ = h.Encode([]int{int(now.Unix())})
        url.ShortUrl = "http://localhost:12345/" + url.ID
        bucket.Insert(url.ID, url, 0)
    } else {
        url = row
    }
    json.NewEncoder(w).Encode(url)
}

What the heck is going on in the above code?

The /create endpoint is accessed over a PUT request. In that request we have a JSON body that contains a long URL. For convenience we can store the entire body into a MyUrl object.

The goal behind a URL shortener is to only ever have one copy of the long URL. This means that every short URL needs to be unique. To make this possible, we first need to query the database based on the long URL to see if it exists already:

var n1qlParams []interface{}
n1qlParams = append(n1qlParams, url.LongUrl)
query := gocb.NewN1qlQuery("SELECT `" + bucketName + "`.* FROM `" + bucketName + "` WHERE longUrl = $1")
rows, err := bucket.ExecuteN1qlQuery(query, n1qlParams)

In the above we use a parameterized N1QL query to do our check. A N1QL query is pretty much a SQL query with JSON extras. If there were any errors executing this query, we print out the error in the response and exit out of the method. We do this to prevent an error of trying to call two render commands.

If there was no error, we take the result of the query and see if it is empty. If the result is empty we know that we need to shrink and save the URL.

If we wanted to we could come up with our own short hashing algorithm when shrinking, but I’d rather take the convenience route of using the Hashids package. Before we can use it, we must install it like so:

go get github.com/speps/go-hashids

To make sure we get a unique short URL every time, we are going to hash the current timestamp:

hd := hashids.NewData()
h := hashids.NewWithData(hd)
now := time.Now()
url.ID, _ = h.Encode([]int{int(now.Unix())})

After we receive a unique short hash, we store it as the MyUrl id, along with the new short URL. The long URL is already stored in this object because of the PUT body.

After we save the data, encode whatever is in our object. In the event that the long URL already existed, just encode what was returned in the query.

This brings us to our /expand endpoint. It looks like the following:

func ExpandEndpoint(w http.ResponseWriter, req *http.Request) {
    var n1qlParams []interface{}
    query := gocb.NewN1qlQuery("SELECT `" + bucketName + "`.* FROM `" + bucketName + "` WHERE shortUrl = $1")
    params := req.URL.Query()
    n1qlParams = append(n1qlParams, params.Get("shortUrl"))
    rows, _ := bucket.ExecuteN1qlQuery(query, n1qlParams)
    var row MyUrl
    rows.One(&row)
    json.NewEncoder(w).Encode(row)
}

It is similar to what we saw in the /create endpoint, but this time we have two major differences. Instead of querying with N1QL based on the long URL, we are now querying based on the short URL. Instead of taking in request data from the body of the request, we are now using the query parameters. Whatever is returned from our query will be returned to the user, whether it is an empty object or not.

Finally we have our root endpoint. Because the API and short URLs are hosted on the same server, we can treat all requests to the root endpoint as short URLs. The id to the short URLs will be used in a lookup like the following:

func RootEndpoint(w http.ResponseWriter, req *http.Request) {
    params := mux.Vars(req)
    var url MyUrl
    bucket.Get(params["id"], &url)
    http.Redirect(w, req, url.LongUrl, 301)
}

After doing a lookup based on the id, a redirect will happen to the stored long URL. This can either be a 301 or 302 redirect.

The Full Project Source Code

There wasn’t too much to the URL shortening example. We only had three API endpoints and a structure for holding our data model. The completed project can be seen below:

package main

import (
    "encoding/json"
    "log"
    "net/http"
    "time"

    "github.com/couchbase/gocb"
    "github.com/gorilla/mux"
    "github.com/speps/go-hashids"
)

type MyUrl struct {
    ID       string `json:"id,omitempty"`
    LongUrl  string `json:"longUrl,omitempty"`
    ShortUrl string `json:"shortUrl,omitempty"`
}

var bucket *gocb.Bucket
var bucketName string

func ExpandEndpoint(w http.ResponseWriter, req *http.Request) {
    var n1qlParams []interface{}
    query := gocb.NewN1qlQuery("SELECT `" + bucketName + "`.* FROM `" + bucketName + "` WHERE shortUrl = $1")
    params := req.URL.Query()
    n1qlParams = append(n1qlParams, params.Get("shortUrl"))
    rows, _ := bucket.ExecuteN1qlQuery(query, n1qlParams)
    var row MyUrl
    rows.One(&row)
    json.NewEncoder(w).Encode(row)
}

func CreateEndpoint(w http.ResponseWriter, req *http.Request) {
    var url MyUrl
    _ = json.NewDecoder(req.Body).Decode(&url)
    var n1qlParams []interface{}
    n1qlParams = append(n1qlParams, url.LongUrl)
    query := gocb.NewN1qlQuery("SELECT `" + bucketName + "`.* FROM `" + bucketName + "` WHERE longUrl = $1")
    rows, err := bucket.ExecuteN1qlQuery(query, n1qlParams)
    if err != nil {
        w.WriteHeader(401)
        w.Write([]byte(err.Error()))
        return
    }
    var row MyUrl
    rows.One(&row)
    if row == (MyUrl{}) {
        hd := hashids.NewData()
        h := hashids.NewWithData(hd)
        now := time.Now()
        url.ID, _ = h.Encode([]int{int(now.Unix())})
        url.ShortUrl = "http://localhost:12345/" + url.ID
        bucket.Insert(url.ID, url, 0)
    } else {
        url = row
    }
    json.NewEncoder(w).Encode(url)
}

func RootEndpoint(w http.ResponseWriter, req *http.Request) {
    params := mux.Vars(req)
    var url MyUrl
    bucket.Get(params["id"], &url)
    http.Redirect(w, req, url.LongUrl, 301)
}

func main() {
    router := mux.NewRouter()
    cluster, _ := gocb.Connect("couchbase://localhost")
    bucketName = "example"
    bucket, _ = cluster.OpenBucket(bucketName, "")
    router.HandleFunc("/{id}", RootEndpoint).Methods("GET")
    router.HandleFunc("/expand/", ExpandEndpoint).Methods("GET")
    router.HandleFunc("/create", CreateEndpoint).Methods("PUT")
    log.Fatal(http.ListenAndServe(":12345", router))
}

After running the application, the API can be accessed from http://localhost:12345 in your browser or by some other means.

Conclusion

You just saw how to create a RESTful URL shortening application using the Go programming language. This is to compliment my URL shortener that I created with Node.js previously. The example we saw was very basic, but it can be easily expanded. For example, you could store a counter that increases every time a short URL is visited. You can also store browser agent information amongst other stuff. Essentially any kind of analytic information that might be useful in a URL shortener.

A video version of this article can be seen below.

Nic Raboy

Nic Raboy is an advocate of modern web and mobile development technologies. He has experience in C#, JavaScript, Golang and a variety of frameworks such as Angular, NativeScript, and Unity. Nic writes about his development experiences related to making web and mobile development easier to understand.