sttp streaming and the URI interpolator

sttp is a Scala HTTP client library with the goal of providing a simple, no-surprises, immutable API. If you've haven't heard about it, here's an introductory post from last week. The core of the library has no dependencies, and it's very easy to get started both when using SBT or Ammonite - just take a look at the readme.

In this post, I'd like to focus on two non-basic features of sttp: the URI interpolator and streaming.

URI interpolator

Each request definition has one part that is not optional: the URI (there's also another one to be precise, the request method). In sttp, URIs are represented as the Uri case class. While it's possible to construct the desired URI by hand using the various methods on Uri, it's often easier and more readable to convert a string into a Uri instance. That's where the URI interpolator comes in.

Same as s"Hello $world" constructs a string by embedding the value of world into the result, the URI interpolator allows creating instances of the Uri class. For example:

uri"http://example.com/$user/read?filter=$f"

will create an URI by embedding the values of the user and f values in the specified places. As we are constructing a URI, this is done in a URL-safe way: all the parameters are properly escaped. If you want to try, here's an example Ammonite session:

import $ivy.`com.softwaremill.sttp::core:1.0.5`
import com.softwaremill.sttp._

val test = "chrabąszcz majowy"
val testUri = uri"http://httpbin.org/get?bug=$test"

testUri.toString

implicit val backend = HttpURLConnectionSttpBackend()
sttp.get(testUri).send().body

If you are curious, "chrabąszcz majowy" is the May bug in Polish. It's also a good toung-twister ;).

However, there's more! The URI interpolator also supports optional values. If the value of a parameter is None, the parameter will be removed from the query:

val test2: Option[String] = None
val testUri2 = uri"http://httpbin.org/get?bug=$test2"
testUri2.toString
// prints: "http://httpbin.org/get"

Same goes for fragments, sub-domains or path components. But there's even more! You can add a whole key-value map in the parameters section, and it will be unwrapped to key-value pairs in the query:

val test3: Map[String, String] = Map("bug1" -> "chrabąszcz", "bug2" -> "pszczoła")
val testUri3 = uri"http://httpbin.org/get?$test3"
testUri3.toString
// prints: "http://httpbin.org/get?bug1=chrab%C4%85szcz&bug2=pszczo%C5%82a"

Sequences are also supported in the domain and path parts.

The goal of the interpolator is to behave as a developer might expect. If embedding a specific value type into the URI doesn't work or has non-obvious behavior, let us know through the issues!

Streaming

We live in a reactive world, and increasingly often encounter APIs that support streaming of data; sttp is no exception. As far as HTTP requests are concerned, streams make sense in two places: either when sending a request body, or when receiving a response body.

However, as sttp supports multiple backends (at the time of writing, akka-http, async-http-client and OkHttp-based), each of them might support a certain type of streams, while some backends don't support streaming at all.

That's why each backend not only defines if and how the HTTP responses are wrapped (e.g. no wrapper, Future, Task), but also what kind of streams it supports.

Currently there are two backends that support streaming: akka-http and Monix + async-http-client.

In the akka-http case, you can both send and receive bodies of type akka.stream.scaladsl.Source[ByteString, Any], which is the akka-streams type of a stream of bytes. Note that when specifying that a response body should be a stream, it is the responsibility of the caller to ensure that the body is fully consumed (and not discarded).

For monix, the supported stream type is monix.reactive.Observable[ByteBuffer]. Here's an example of sending a monix stream as the request body, and receiving a stream back:

implicit val backend = MonixAsyncHttpClientBackend()
val source: Observable[ByteBuffer] = ...

val response = sttp
  .post(uri"http://httpbin.org/post")
  .streamBody(source)
  .response(asStream[Observable[ByteBuffer]])
  .send()

response: Task[Response[Observable[ByteBuffer]]]

// the body property contains the observable:
def consumer(s: Observable[ByteBuffer]): Unit = ...
response.map(consume(_.body))

Of course it's perfectly fine to send the request body as a stream and read the response as e.g. a String, or send a String as the request body and read the response body as a stream. As in all other cases, you can specify all parts of the request definition (.post, .streamBody, .response, ...) in any order, each time obtaining a new, immutable request definition which can be further specialized or sent.

Summary

sttp aims to be a developer-friendly library. Hopefully the URI interpolator and streaming support help in fulfilling this goal. We are still shaping the API and features, so if you have any suggestions or pull requests, please don't hesitate to ask!