Scala @ Scale, Part 2: Compose Yourself!

Function composition is an extremely useful tool for creating modular, testable programs. One of the most natural applications of functional composition that I’ve found is creating a lightweight, composable HTTP request builder, and with that objective in focus, this post will equip you with the tools you need to simplify your HTTP clients.

In a microservice-based environment, distributed services need to communicate with each other over the network. Many times the protocol of choice is HTTP. What do you do when you need to expose an API for a given service? Create a client of course! Clients are great and should be used, but the issue with clients is that adding new endpoints or changing existing methods can be both tedious and error prone. It’s tough to remember exactly how to accomplish these tasks every time you need to hop into the client to make changes. Ever had to skim through a bunch of code just to remember how to add a new method to the client? Questions that frequently come up are:

  • I forget how to create an HTTP request… how do I do that again???
  • What’s the base set of information required by the API for this specific service?
  • What method should I copy and paste to get exactly what I want?

These issues arise due to your HTTP client not having a composable and expressive DSL (domain specific language) that feels natural to use. That’s the problem that we’re here to solve today. What if I told you that we can build a DSL that feels so natural that you’ll (probably) never again need to copy/paste code every time you add a new endpoint/method to the client? By the end of this post, we will have a set of functions that allows us to express HTTP requests like this:

GET("http://monads4lyfe.net/user/julie") andThen
  addHeader(("x-foo-id" -> "bar")) andThen
  addQueryParam(("sorted" -> "1"))
  addQueryParam(("field" -> "name"))

 

How nice is that?? It is VERY easy to remember because it’s so natural to use. Building a DSL like this requires no external libraries either. Bonus!

Usual Disclaimer: This series assumes knowledge of Scala. Most readers without Scala knowledge should be able to follow a lot of the code, but some nuances may be missed.

Let’s jump right in. Function composition is all about being able to compose functions together to accomplish a task. Building HTTP requests is a great use of this since you are usually working (or “building”) on a single request object. The request object has many fields that allow you to specify how the request should be made and what data needs to be sent in that request.

Let’s first define our HttpRequest model. We’re using a custom, overly simplistic one for purposes of demonstration, but the actual model can come from anything (most likely the library you’re using to make the requests like Akka-Http or http4s):

/**
    * Very simple HttpRequest class. Again this can be whatever class
    * you want it to be (most HTTP libs/frameworks come with their own
    * version of it)
    * @param url The endpoint for the request
    * @param method The HTTP method (GET, POST, etc.)
    * @param body Body for non GET requests
    * @param query Map of URL query parameters
    * @param headers Map of HTTP headers
    */
  case class HttpRequest(url: String = "",
                          method: String = "GET",
                          body: String = "",
                          query: Map[String, String] = Map.empty,
                          headers: Map[String, String] = Map.empty)

 

Now that we have our super-sophisticated HttpRequest class, let’s define the types to be used for request composition:

  // The type for building requests. RequestBuilder is simply
  // a function that takes an HttpRequest and returns a
  // modified HttpRequest. Modifications can be whatever
  // you want as long as they adhere to this definition.
  type RequestBuilder = HttpRequest => HttpRequest

  // The type used to signify lifting of values into RequestBuilder (see usage below)
  type HttpUnit = () => RequestBuilder

 

OK so what’s going on here. First we define a type for RequestBuilder. This defines the operation for our composable functions. Each one will receive an HttpRequest object, modify it however we want, and return a copy of the modified request. Already sounds “chain-able” doesn’t it? Let’s get to the fun stuff. Next we’re going to implement the functions we can use to compose and build our HTTP requests:

  /**
    * The HttpRequestBuilder object contains the basic functions for building HTTP requests
    */
  object HttpRequestBuilder {
    /**
      * Lifts the various arguments into HttpUnit which can ultimately be
      * applied to resolve a RequestBuilder.
      */
    private def lift(url: String, method: String): HttpUnit = () => _ => HttpRequest(url, method)

    /**
      * The Entry point for building requests. You can add any HTTP verbs
      * that you might need here. We define the 4 most basic ones up front.
      *
      * The premise for these is to use the lift() method to ultimately resolve the raw
      * request arguments (url and optional data) into instances of RequestBuilder
      * so that they can be composed and executed at a later point.
      */
    def GET(url: String): RequestBuilder = lift(url, "GET").apply()
    def POST(url: String, data: String = ""): RequestBuilder =
      lift(url, "POST").apply andThen setBody(data)
    def PUT(url: String, data: String = ""): RequestBuilder =
      lift(url, "PUT").apply andThen setBody(data)
    def DELETE(url: String, data: String = ""): RequestBuilder =
      lift(url, "DELETE").apply andThen setBody(data)

    /**
      * These are various functions that we will use to modify/augment the
      * request. Generic functions like addHeader and addQueryParam can be
      * composed to perform more powerful request modifications (we'll
      * see that later).
      * @return
      */
    def addHeaders(headers: Map[String, String]): RequestBuilder = (req) => req.copy(headers = req.headers ++ headers)
    def addHeader(header: (String, String)): RequestBuilder = addHeaders(Map(header._1 -> header._2))
    def addQueryParam(param: (String, String)): RequestBuilder = (req) => req.copy(query = req.query + param)
    def setBody(data: String): RequestBuilder = (req) => req.copy(body = data)
  }

 

Sorry for the big code dump, but it’s a necessary evil to give context to what’s happening. The HttpRequestBuilder object contains various functions for creating/modifying http requests. The entry points for new requests are the HTTP verb methods (GET/POST/PUT/DELETE). We have a private lift method defined which allows us to — you guessed it — “lift” the initial request values into the context of a RequestBuilder. Once we have a RequestBuilder, we can use the various modifier methods to work on the request.

At this point, we can actually generate a function that will resolve to an HttpRequest when applied. Let’s check it out:

val composedReqBuilider =
    POST("http://foo.bar/say/hello") andThen
    setBody("Name: Julie") andThen
    addQueryParam("sort" -> "1") andThen
    addQueryParam("filtered" -> "0")
  val rawRequest = composedReqBuilider(HttpRequest())
  println(s"Raw request ==> $rawRequest")

 

Running the code above ends up printing:

Raw request ==> HttpRequest(http://foo.bar/say/hello,POST,Name: Julie,Map(sort -> 1, filtered -> 0),Map())

 

Nice! So we see that it’s building the request the way we want, but we’re obviously missing an important part to all of this: Actually executing the request! So let’s build a mock HTTP executor that will allow us to “execute” that lovely request we’ve built:

  // Very simple mock Http Executor interface. This can be whatever you want.
  trait HttpExecutor {
    /**
      * The exec method takes an HttpRequest and a mapper function. The
      * mapper function simply takes the response string and can do whatever
      * it wants to it (e.g. returning a case class from the response)
      */
    def exec[A](httpRequest: HttpRequest)(f: String => A): Future[A]
  }

  // Implementation of the executor
  object BasicHttpExecutor extends HttpExecutor {
    /**
      * This exec method takes a fully formed HttpRequest and a transformer function
      * and "executes" it to return a result (as a Future).
      * We built additional logic in for testing. If the HttpRequest has a mock status code
      * set that is not 200, we fail the future with an error.
      *
      * On success we just format the request data in a friendly, human readable format.
      * In a real-world scenario this would be some kind of actual response like
      * JSON or XML that would be passed to a parser.
      */
    override def exec[A](httpRequest: HttpRequest)(f: String => A): Future[A] = httpRequest.headers.get("x-mock-status-code") match {
      case None | Some("200") => Future.successful(f(
        s"""
          Method: ${httpRequest.method}
          Endpoint: ${httpRequest.url}
          Data: ${if(httpRequest.body.isEmpty) "N/A" else httpRequest.body}
          Query Params: ${httpRequest.query}
          Headers: ${httpRequest.headers}
        """.stripMargin))
      case Some(code) => Future.failed(new RuntimeException(s"Received error from API: $code"))
    }

  }

 

Again this is just a mock executor service. In real life, you’d be using an executor from whatever HTTP lib you’ve chosen (Akka, http4s, etc.). It doesn’t matter which one you choose; the overall implementation is the same. In our mock executor, we check a special HTTP header called “x-mock-status-code.” If it’s not defined or set to 200, we consider the request a success. In that case, we just return a successful future with the serialized http request information. If the header is set to a non 200 code, we fail the future and return it with an exception.

Using our example from above, we know how to compose the functions together, and then apply it to receive the HTTP request. So now we could do something like this:

val httpExecutor = BasicHttpExecutor
val fut = httpExecutor.exec(rawRequest)
fut.map( res => … )

 

That will in fact work as you’d expect, but it’s kinda ugly and unintuitive. I’d love a way for us to just call exec() directly on the result of our RequestBuilder. When using basic types (instead of instances of classes), we can augment or “pimp” out our types to add an exec method using implicit classes. Implicit classes are outside the scope of this article, but a quick google search will provide all the info you want. Here’s how we ultimately want to use the request builder:

val fut =
  (GET("http://monads4lyfe.net/user/julie") andThen
    addHeader(("x-foo-id", "bar")) andThen
    addQueryParam(("sorted", "1")) andThen
    addQueryParam(("field", "name"))).exec

 

Oh that’s nice! Instead of having to explicitly call exec on the HttpExecutor, we can just pimp out the RequestBuilder type to be able to handle it for us. Here’s what the implementation looks like for that:

  implicit class PimpedHttp(rb: RequestBuilder) {
    val NOOP: String => String = response => response
    def exec[T](f: String => T)(implicit executor: HttpExecutor): Future[T] =
      executor.exec(rb.apply(HttpRequest()))(f)
    def exec(implicit executor: HttpExecutor): Future[String] = exec(NOOP)
  }

 

We have our mock executor service and a way to implicitly add the exec() function to RequestBuilder types. The original desired API should now fully work!

val fut =
  (GET("http://monads4lyfe.net/user/julie") andThen
    addHeader(("x-foo-id", "bar")) andThen
    addQueryParam(("sorted", "1")) andThen
    addQueryParam(("field", "name"))).exec

fut.foreach(res => println(s"Look at this response!n$res"))

 

This will print out:

Look at this response!

          Method: GET
          Endpoint: http://monads4lyfe.net/user/julie
          Data: N/A
          Query Params: Map(sorted -> 1, field -> name)
          Headers: Map(x-foo-id -> bar)

 

And there we have it! We built our request using RequestBuilder and friends, mocked out an executor service for it, and created an implicit class to execute any instance of RequestBuilder. At this point you have enough knowledge and code to venture on and create your own request builder to make client generation great again! If you’re hungry for more, read on for an actual client implementation using RequestBuilder.

Look at you, still hungry for more client generation goodness! OK. Well we’re going to create a client for our awesome “Foo Service.” We’ll make an aptly named “FooClient” for it so that all our other services can use Foo as much as they want. Let’s first build out some of the case classes we’ll need to handle authentication and responses:

case class FooHost(baseUrl: String)
case class UserCredentials(username: String, userId: String)
case class SayHiResponse(phrase: String)

 

Our Foo service expects credentials on every request to identify who’s using it (set as headers on the HTTP request. We also create a FooHost case class to hold connection information as well as a response class for saying hello to the user. Let’s get into the actual client now:

 case class FooClient(host: FooHost, implicit val httpExecutor: HttpExecutor) {
    private val UserCredsIdHeader = "x-user-id"
    private val UserCredsUsernameHeader = "x-user-name"

    private def addCredentials(creds: UserCredentials): RequestBuilder =
      addHeaders(Map(UserCredsIdHeader -> creds.userId, UserCredsUsernameHeader -> creds.username))

    private def getSayHiPayload(creds: UserCredentials): String =
      s"Hello user ${creds.username}!!"

    def sayHi(creds: UserCredentials): Future[SayHiResponse] = {
      (POST(s"${host.baseUrl}/api/user/${creds.username}/say-hi") andThen
        setBody(getSayHiPayload(creds)) andThen
        addCredentials(creds)).exec(SayHiResponse)
    }
  }

 

Really simple client. Notice the addCredentials method. We are able to reuse the more powerful addHeaders function to extend the base RequestBuilder for the specific use case in FooClient. Even though this is a very simple example, you should already be able to see how extensible this is. Let’s see how this client is used!

  // Reuse the simple executor from before
  val fooClient = FooClient(FooHost("http://monads4lyfe.gov"), BasicHttpExecutor)
  fooClient
    .sayHi(UserCredentials("julie", "xx456yy"))
    .foreach(println(_))

 

This will print out:

SayHiResponse(
          Method: POST
          Endpoint: http://monads4lyfe.gov/api/user/julie/say-hi
          Data: Hello user julie!!
          Query Params: Map()
          Headers: Map(x-user-id -> xx456yy, x-user-name -> julie)
        )

 

Boom! And there we have it. A fully fledged client using the RequestBuilder types. If you got this far, you now have the tools to build a composable, type-safe, expressive DSL for building up HTTP requests regardless of what HTTP library you’re using. Stay tuned for Part 3 for more Scala @ Scale goodness!

For More Scala-Related Articles . . .

If you’re interested in other Scala-related articles based on the experiences of Threat Stack developers, have a look at these: