API, Golang, Application Architecture, Microservice

Creating a Microservice with Golang and Goa

Microservices are very popular these days, there are lots f interest in breaking apart monoliths into small pieces for easier deployment and smaller, more understandable codebases. Golang is one of the top languages for making microservices, its high performance and readable syntax is making it a no-brainer for many organizations.

It can be daunting to know where to start with microservices. I will walk you through a tutorial on creating an authentication microservice using golang and the Goa framework. All the below code can be found in this repository.

Where to Start?

Getting started can be one of the most significant hurdles to building a microservice. An excellent way to look at it is just a smaller monolith with one purpose. Some examples of possible microservices for a blog would be:

  • Authentication Microservice
  • User Microservice
  • Post Microservice
  • Comment Microservice
  • Analytics Microservice
    This tutorial will be making an authentication microservice, and hopefully, from this, you will have the understanding and confidence moving forward!

Designing the Microservice

The first step we're going to make is designing the microservice. We do this before we write any code because it helps flush out what we want this microservice to do and what endpoints and models we need. Goa makes this very easy with their DSL, I will not dive deep into it in this post, but for more details, please check out their docs.

To start, we make an empty directory called design/ and add a couple empty files in.

api.go
payloads.go
resources.go
responses.go

These files will describe our whole API and allow Goa to generate the scaffolding and tedious code surrounding the API. Now that these empty files are made let's build api.go.

api.go describes our API at a high level, it gives the generated documentation a title and description, adds contact information, and sets a port to listen on, etc... A important thing to note is you can set CORS and the API encoding. I set it to JSON but if you want there are other options such as XML.

package design

import (
    . "github.com/goadesign/goa/design"
    . "github.com/goadesign/goa/design/apidsl"
)

var _ = API("Authentication API", func() {
    Title("The Authentication API")
    Description("An API which an Authentication API")
    Contact(func() {
        Name("Tester Test")
        Email("[email protected]")
    })
    Host("localhost:8080")
    Scheme("http")
    BasePath("/api/")
    Origin("*", func() {
        Headers("Content-Type")
        Methods("GET", "POST", "PATCH", "DELETE", "PUT", "OPTION")
    })
    Consumes("application/json")
    Produces("application/json")
})

Payloads are next, this handles the form inputs for the API. We are going to need to sign users in and let them create accounts, so we want a LoginPayload along with a RegisterPayload. Let's add this to the payloads.go file. These payloads are great because it adds a level of validation to your microservice right out the gate without cluttering up your code. This is one of the key features which made me fall in love with Goa.

package design

import (
    . "github.com/goadesign/goa/design"
    . "github.com/goadesign/goa/design/apidsl"
)

var LoginPayload = Type("LoginPayload", func() {
    Attribute("email", String, func() {
        MinLength(6)
        MaxLength(400)
        Format("email")
        Example("[email protected]")
    })

    Attribute("password", String, func() {
        MinLength(5)
        MaxLength(100)
        Example("abcd1234")
    })
    Required("email", "password")
})

var RegisterPayload = Type("RegisterPayload", func() {
    Attribute("email", String, func() {
        MinLength(6)
        MaxLength(150)
        Format("email")
        Example("[email protected]")
    })

    Attribute("first_name", String, func() {
        MinLength(1)
        MaxLength(200)
        Example("John")
    })

    Attribute("last_name", String, func() {
        MinLength(1)
        MaxLength(200)
        Example("Doe")
    })

    Attribute("password", String, func() {
        MinLength(5)
        MaxLength(100)
        Example("abcd1234")
    })

    Required("email", "password", "first_name", "last_name")
})

Similar to payloads we have responses.go, this contains the responses for each endpoint. We only need the Token response in this tutorial, but I have added the User response as well in case you would like to add on any other endpoints using the user.

package design

import (
    . "github.com/goadesign/goa/design"
    . "github.com/goadesign/goa/design/apidsl"
)

var Token = MediaType("application/vnd.token+json", func() {
    Description("A token")
    Attributes(func() {
        Attribute("token", String, "A JWT token")
    })

    View("default", func() {
        Attribute("token")
    })
})

var User = MediaType("application/vnd.user+json", func() {
    Description("A user")
    Attributes(func() {
        Attribute("id", Integer, "ID of account", func() {
            Example(1)
        })
        Attribute("email", String, "Email of the user", func() {
            Format("email")
            Example("[email protected]")
        })
        Attribute("first_name", String, "First name of the user", func() {
            Example("John")
        })
        Attribute("last_name", String, "Last name of the user", func() {
            Example("Snow")
        })
        Attribute("password", String, "Password of user", func() {
            Example("password")
        })
        Attribute("salt", String, "Salt of the user", func() {
            Example("salt")
        })
    })

    View("default", func() {
        Attribute("id")
        Attribute("email")
        Attribute("first_name")
        Attribute("last_name")
        Attribute("password")
        Attribute("salt")
    })
})

Finally, we have the file which ties this all together! resources.go. This is where all the design code we've been writing gets connected. We create resources which describe all the endpoints to be exposed. We give each endpoint a name, an HTTP method, and the expected input and output. We also describe the errors each endpoint can return.

package design

import (
    . "github.com/goadesign/goa/design"
    . "github.com/goadesign/goa/design/apidsl"
)

var _ = Resource("authentication", func() {
    BasePath("/auth")
    NoSecurity()
    Action("login", func() {
        NoSecurity()
        Routing(
            POST("/login"),
        )
        Description("Sign a user in")
        Payload(LoginPayload)
        Response(OK, Token)
        Response(InternalServerError)
        Response(BadRequest, ErrorMedia)
    })

    Action("register", func() {
        NoSecurity()
        Routing(
            POST("/register"),
        )
        Description("Create a new user")
        Payload(RegisterPayload)
        Response(OK, Token)
        Response(InternalServerError)
        Response(BadRequest, ErrorMedia)
    })

})

var _ = Resource("swagger", func() {
    Files("/swagger.json", "swagger/swagger.json")
})

Once all of this code is written, we should be ready to go to the next step, generating the code!

Generate the Code

Now that the code is ready, you should run the goagen command to generate the scaffolding. You may need to run go get -u github.com/goadesign/goa/... if you haven't before. Once that is complete run the following command. If you are running this in another project change the path, after your go src folder, to your design folder.

goagen bootstrap -d github.com/rymccue/golang-auth-microservice/design

Add the Database

After the code is generated, we need to create a database connection so our microservice can persist data. Create a folder called utils/ and inside that create another folder called database/, inside that folder add a file called database.go and we need to add a function which returns a database instance. It should look something like this.

package database

import (
    "database/sql"
    "fmt"

    _ "github.com/lib/pq"
)

func Connect(user, password, dbname, host, port string) (*sql.DB, error) {
    connStr := fmt.Sprintf("user=%s password=%s dbname=%s host=%s port=%s",
        user, password, dbname, host, port)
    return sql.Open("postgres", connStr)
}

Then go to main.go and add the function in under the service creation.

db, err := database.Connect(os.Getenv("PGUSER"), os.Getenv("PGPASS"), os.Getenv("PGDB"), os.Getenv("PGHOST"), os.Getenv("PGPORT"))
if err != nil {
    service.LogError("startup", "err", err)
}

Once that's added we are going to pass the database to the authentication controller, so add db in as the second parameter in NewAuthenticationController. It will look like this:

c := NewAuthenticationController(service, db)

There should be an error because the function does not take a database instance yet, in one of the next steps we will fix that.

Adding Repositories

An important next step here is creating our queries to the database. A good pattern for creating these is the repository pattern. Let's get started!

package repositories

import (
    "database/sql"

    "github.com/rymccue/golang-auth-microservice/app"
    "github.com/rymccue/golang-auth-microservice/utils/crypto"
)

We need to get a user by an email so lets create a function which takes the database and an email. It should get a user with a given email and then return a user and error message.

// GetUserByEmail gets a user by email
func GetUserByEmail(db *sql.DB, email string) (*app.User, error) {
    const sqlstr = `
    select
        first_name,
        last_name,
        email,
        password,
        salt
    from users
    where email = $1
    `
    var user app.User
    err := db.QueryRow(sqlstr, email).Scan(&user.FirstName, &user.LastName, &user.Email, &user.Password, &user.Salt)
    return &user, err
}

We need a function to create a user. We pass a database, and user data into the function. The function should also create a salt and hash the password so it won't be stored in cleartext.

// AddUserToDatabase creates a new user
func AddUserToDatabase(db *sql.DB, firstName, lastName, email, password string) error {
    const sqlstr = `
    insert into users (
        first_name,
        last_name,
        email,
        password,
        salt
    ) values (
        $1,
        $2,
        $3,
        $4,
        $5
    ) returning id
    `
    salt := crypto.GenerateSalt()
    hashedPassword := crypto.HashPassword(password, salt)
    var id int
    err := db.QueryRow(sqlstr, firstName, lastName, email, hashedPassword, salt).Scan(&id)
    return err
}

The final query we need is one to check if an email exists. You just need to pass a database and email in and it will return a boolean for if the email is in the database.

func CheckEmailExists(db *sql.DB, email string) (bool, error) {
    const sqlstr = "select exists(select 1 from users where email = $1)"

    var exists bool
    err := db.QueryRow(sqlstr, email).Scan(&exists)
    return exists, err
}

Crypto and JWT Utilities

The final utility we need to build before building out the controller is crypto and jwt. You will need to create an RSA private key and set it's path to the JWT_PRIVATE_KEY environment variable.

package jwt

import (
    "context"
    "crypto/rsa"
    "fmt"
    "io/ioutil"
    "net/http"
    "os"
    "path/filepath"
    "time"

    jwtgo "github.com/dgrijalva/jwt-go"
    "github.com/goadesign/goa"
    "github.com/goadesign/goa/middleware/security/jwt"
    uuid "github.com/satori/go.uuid"
)

// LoadJWTPrivateKeys loads PEM encoded RSA private key.
func LoadJWTPrivateKey(keyPath string) (*rsa.PrivateKey, error) {
    keyFile, err := filepath.Glob(keyPath)
    if err != nil {
        return nil, err
    }
    pem, err := ioutil.ReadFile(keyFile[0])
    if err != nil {
        return nil, err
    }
    key, err := jwtgo.ParseRSAPrivateKeyFromPEM([]byte(pem))
    if err != nil {
        return nil, fmt.Errorf("failed to load key %s: %s", keyFile, err)
    }
    return key, nil
}

func CreateJWTToken(email string) (string, error) {
    token := GenerateJWT(email)
    privKey, err := LoadJWTPrivateKey(os.Getenv("JWT_PRIVATE_KEY"))
    if err != nil {
        panic(err)
    }
    return token.SignedString(privKey)
}

func GenerateJWT(email string) *jwtgo.Token {
    token := jwtgo.New(jwtgo.SigningMethodRS512)
    oneMonth := time.Now().Add(time.Duration(24*30) * time.Hour).Unix()
    uuid, _ := uuid.NewV4()
    token.Claims = jwtgo.MapClaims{
        "iss":        "Issuer",
        "aud":        "Audience",
        "exp":        oneMonth,
        "jti":        uuid.String(),
        "iat":        time.Now().Unix(),
        "nbf":        2,
        "sub":        "subject",
        "user.email": email,
    }
    return token
}

We also need some crypto utilities in order to generate a salts and hash passwords.

package crypto

import (
    "crypto/rand"
    "encoding/hex"
    "io"
    "log"

    "golang.org/x/crypto/scrypt"
)

// GenerateSalt generates a random salt
func GenerateSalt() string {
    saltBytes := make([]byte, 16)
    _, err := io.ReadFull(rand.Reader, saltBytes)
    if err != nil {
        log.Fatal(err)
    }
    salt := make([]byte, 32)
    hex.Encode(salt, saltBytes)
    return string(salt)
}

// HashPassword hashes a string
func HashPassword(password, salt string) string {
    hashedPasswordBytes, err := scrypt.Key([]byte(password), []byte(salt), 16384, 8, 1, 32)
    if err != nil {
        log.Fatal("Unable to hash password")
    }
    hashedPassword := make([]byte, 64)
    hex.Encode(hashedPassword, hashedPasswordBytes)
    return string(hashedPassword)
}

Implement the Controllers

We are in the home stretch now, we have all the scaffolding built out for our microservice, now all we need to do is create the two endpoints and it will be complete!

We start off the file by fixing the NewAuthenticationController function which we call in main. Update it to accept an instance of *sql.DB and update the struct to take one as well.

package main

import (
    "database/sql"
    "github.com/goadesign/goa"
    "github.com/rymccue/golang-auth-microservice/app"
    "github.com/rymccue/golang-auth-microservice/repositories"
    "github.com/rymccue/golang-auth-microservice/utils/crypto"
    "github.com/rymccue/golang-auth-microservice/utils/jwt"
)

// AuthenticationController implements the authentication resource.
type AuthenticationController struct {
    *goa.Controller
    *sql.DB
}

// NewAuthenticationController creates a authentication controller.
func NewAuthenticationController(service *goa.Service, db *sql.DB) *AuthenticationController {
    return &AuthenticationController{
        Controller: service.NewController("AuthenticationController"),
        DB:         db,
    }
}

Now for the fun part, let's create our first microservice controller! This is actually the easiest part of creating a microservice, now that all the utilities complete we just need to tie them together with the generated code Goa provides to make a clean controller.

// Login runs the login action.
func (c *AuthenticationController) Login(ctx *app.LoginAuthenticationContext) error {

We first grab the payload from the context and assign it to a variable, then attempt to get a user with the email provided by the client. If there is an error, we check to see if it's a no rows error or another type. If it is a no rows error we return a bad request to the client, if not we return an internal server error.

    payload := ctx.Payload
    u, err := repositories.GetUserByEmail(c.DB, payload.Email)
    if err != nil {
        if err == sql.ErrNoRows {
            return ctx.BadRequest(goa.ErrBadRequest("Invalid email or password"))
        }
        c.Service.LogError("Login User", "err", err)
        return ctx.InternalServerError()
    }

After that we hash the password given by the client and compare it against the hashed password in the database, if it does not match we return a bad request, if not we create a JWT token with the users email.

    hashedPassword := crypto.HashPassword(payload.Password, *u.Salt)
    if *u.Password != hashedPassword {
        return ctx.BadRequest(goa.ErrBadRequest("Invalid email or password"))
    }
    token, err := jwt.CreateJWTToken(*u.Email)
    if err != nil {
        c.Service.LogError("Login User", "err", err)
        return ctx.InternalServerError()
    }

Finally, we add the token to our Token response object and return it successfully to the client!

    res := &app.Token{
        Token: &token,
    }
    return ctx.OK(res)
}

Now lets move on to the register endpoint, it is similar to the login endpoint because we retrieve the payload, and check for the email provided by the client. If it does not exist we can continue with the register process.

// Register runs the register action.
func (c *AuthenticationController) Register(ctx *app.RegisterAuthenticationContext) error {
    payload := ctx.Payload
    exists, err := repositories.CheckEmailExists(c.DB, payload.Email)
    if err != nil {
        c.Service.LogError("Register User", "err", err)
        return ctx.InternalServerError()
    }
    if exists {
        return ctx.BadRequest(goa.ErrBadRequest("Email already exists"))
    }

This is fairly simple, we add the user to the database and then with the provided email generate a token and return it to the user. If there are any errors along the way, we return an internal server error because in theory all errors from this point on is the result of something going wrong in the microservice.

    err = repositories.AddUserToDatabase(c.DB, payload.FirstName, payload.LastName, payload.Email, payload.Password)
    if err != nil {
        c.Service.LogError("Register User", "err", err)
        return ctx.InternalServerError()
    }
    token, err := jwt.CreateJWTToken(payload.Email)
    if err != nil {
        c.Service.LogError("Register User", "err", err)
        return ctx.InternalServerError()
    }
    res := &app.Token{Token: &token}
    return ctx.OK(res)
}

Conclusion

Creating a microservice can be simple, even a complicated microservice like authentication can be easy if you break it down into its individual pieces. Goa is an excellent framework for writing, clean, elegant, and well thought out APIs and microservices in Golang. I believe this is the best framework out there for those reasons and the fact that it can be completely customized to fit your needs.

Author image

About Ryan McCue

Hi, my name is Ryan! I am a Software Developer with experience in many web frameworks and libraries including NodeJS, Django, Golang, and Laravel.