Asynchronous HTTP with spray

Last time, I wrote about the high-level ideas behind the Scala libraries akka and spray, namely the concept of reactive programming in order to meet the demands of modern applications. I finally got the chance to play around a bit with spray in order to get a feel for how the theory of event-driven, asynchronous programming manifests itself in code. I started with a very simple web application that takes a stock ticker symbol and uses the Yahoo Query Language (YQL) to look up basic quote information for the company (using this endpoint). Here's some code for a trait that uses spray-http and spray-client to execute a request against the Yahoo API:

	import scala.concurrent._
	import akka.actor.ActorSystem
	import spray.http._
	import HttpMethods._
	import spray.client.pipelining._
	trait YQL {
	implicit val system: ActorSystem
	import system.dispatcher
	def executeQuery(query: String): Future[HttpResponse] = {
	val pipeline = sendReceive
	pipeline(request(query))
	}
	private def request(query: String): HttpRequest = {
	HttpRequest(GET,
	Uri("http://query.yahooapis.com/v1/public/yql").
	withQuery(
	"q" -> query,
	"format" -> "json",
	"env" -> "store://datatables.org/alltableswithkeys"
	)
	)
	}
	}

view raw YQL.scala hosted with ❤ by GitHub

You'll notice that I included all of the necessary imports in the example (I normally leave them out), and this is because a lot of things look nicer once imported (e.g. GET vs HttpMethods.GET) and spray uses implicits quite extensively to expose a nice domain-specific language (DSL). So knowing what the imports are is actually pretty important for figuring out why the code actually compiles and works properly. Now let's break down the code. The request method is the simplest part, and all it does is take in the YQL query and construct an HTTP request with the appropriate URL and parameters. Spray has some convenient methods for making this easy, but there's nothing too special involved.

The executeQuery method is where all the magic happens. To understand it, we'll need to revisit two concepts: futures and actors. Futures are the Scala way of expressing a value that is computed asynchronously. In this case, executeQuery returns a future because, per the reactive programming principles, we do not want to block the thread while waiting for the request to be sent and the response to be received over the network. This means that when the method returns, it is most likely that the response has not come back, so whoever consumes the result of the future will need to wait for it (e.g. using Await). Actors are an abstraction for handling asynchronous computations using a messaging model; you send messages to an actor, which does some computation based on the contents of the message and potentially returns a result. In this case, we have an implicit ActorSystem that is used by spray-client to run an actor which receives the HttpRequest and returns an HttpResponse. The sendReceive method conveniently encapsulates all of this functionality.

Here is our YQL client in action:

	@RunWith(classOf[JUnitRunner])
	class YQLTest extends Specification with YQL {
	implicit val system = ActorSystem("yql-test")
	"YQL" should {
	"issue request and get response" in {
	val query =
	"""select * from yahoo.finance.quote
	|where symbol in ("YHOO","AAPL","GOOG","MSFT")""".stripMargin
	val future = executeQuery(query)
	val response = Await.result(future, Duration(10, TimeUnit.SECONDS))
	println(response)
	response.status must be(StatusCodes.OK)
	}
	}
	}

view raw YQLTest.scala hosted with ❤ by GitHub

Since this is a test, we define a test ActorSystem to use and then use Await to get the response from the future, but when integrated with spray-can (the HTTP server) it becomes much nicer. In contrast to the Java servlet model where we would have many threads all probably blocked on the network, spray encourages everything to be asynchronous and utilizes a small pool of threads. To that end, the "controllers" you define in spray-can allow futures to be returned and automatically map them to HTTP responses, so any request that requires asynchronous processing will naturally result in a future that is handled entirely by the underlying abstraction. Next time, we'll see how to layer other pieces such as JSON and error handling on top of futures in spray.