Home > F#, HTTP > Web requests in F# now easy! Introducing Http.fs

Web requests in F# now easy! Introducing Http.fs

TL;DR

I’ve made a module which makes HTTP calls (like downloading a web page) easy, available now on GitHub – Http.fs

Introduction

I had a project recently which involved making a lot of HTTP requests and dealing with the responses.  F# being my current language of choice, I was using that.  Unfortunately, .Net’s HttpWebRequest/Response aren’t that nice to use from F# (or C#, frankly).

For example, here’s how you might make an HTTP Post, from F# Snippets:

open System.Text
open System.IO
open System.Net

let url = "http://posttestserver.com/post.php"

let req = HttpWebRequest.Create(url) : ?> HttpWebRequest
req.ProtocolVersion req.Method <- "POST"

let postBytes = Encoding.ASCII.GetBytes("fname=Tomas&lname=Petricek")
req.ContentType <- "application/x-www-form-urlencoded";
req.ContentLength let reqStream = req.GetRequestStream()
reqStream.Write(postBytes, 0, postBytes.Length);
reqStream.Close()

let resp = req.GetResponse()
let stream = resp.GetResponseStream()
let reader = new StreamReader(stream)
let html = reader.ReadToEnd()

There are a few things I don’t really like about doing this:

  • It’s a lot of code!
  • You have to mess around with streams
  • The types used are mutable, so not really idiomatic F#
  • You have to set things (e.g. ‘POST’) as strings, so not typesafe
  • It’s not unit testable
  • You have to cast things (e.g. req) to the correct type

In fact there are many other problems with HttpWebRequest/Response which aren’t demonstrated by this sample, including:

  • Some headers are defined, others aren’t (so you have to set them as strings)
  • You need to mess around with the cookie container to get cookies working
  • If the response code is anything but 200-level, you get an exception (!)
  • Getting headers and cookies from the response isn’t pretty

Since then I’ve discovered HttpClient, which does address some of these issues, but it’s still not great to use from F# (and only available in .Net 4.5).

So I started to write some wrapper functions around this stuff, and it eventually turned into:

Http.fs!

Http.fs is a module which contains a few types and functions for more easily working with Http requests and responses from F#. It uses HttpWebRequest/Response under the hood, although these aren’t exposed directly when you use it.

Downloading a single web page is as simple as:

let page = (createRequest Get "http://www.google.com" |> getResponseBody)

And if you want to do something more in-depth, like the example above, that would look like this:

open HttpClient

let response =
  createRequest Post "http://posttestserver.com/post.php"
  |> withBody "fname=Tomas&lname=Petricek"
  |> withHeader (ContentType "application/x-www-form-urlencoded")
  |> getResponse

Then you could access the response elements like so:

response.StatusCode
response.EntityBody.Value
response.Headers.[Server]

And of course, it has asynchronous functions to let you do things like download multiple pages in parallel:

["http://news.bbc.co.uk"
 "http://www.wikipedia.com"
 "http://www.stackoverflow.com"]
|> List.map (fun url -> createRequest Get url |> getResponseBodyAsync)
|> Async.Parallel
|> Async.RunSynchronously
|> Array.iter (printfn "%s")

There are more details on the GitHub page. The project also contains a sample application which shows how it can be used and tested.

So if you’re using F# and want to make a complex HTTP request – or just download a web page – check out Http.fs!

Update

This is now available on NuGet.  To install:

PM> install-package Http.fs  
Advertisements
Categories: F#, HTTP
  1. Jay
    February 26, 2014 at 5:09 am

    Any ideas on how to loop this in FSharp?
    Thx…

    • Grant Crofton
      May 26, 2014 at 2:44 pm

      Hi Jay, sorry I forgot to answer your question. I guess it’s too late now, but I’ll have a go anyway.

      I’m not exactly sure what it is you want to loop – if you want to repeat the same request a few times, here are a few ways you could do it:

      // imperatively, with a for loop
      for index = 1 to 5 do
      printfn "%i: %s" index (createRequest Get "http://www.google.com" |> getResponseBody)

      // declaratively, with a list/sequence
      [1..5] |> Seq.iter (fun index ->
      printfn "%i: %s" index (createRequest Get "http://www.google.com" |> getResponseBody))

      // recursively
      let rec printPage = function
      | 6 -> ()
      | index ->
      printfn "%i: %s" index (createRequest Get "http://www.google.com" |> getResponseBody)
      printPage (index+1)

      printPage 1

  1. November 17, 2013 at 9:02 pm

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: