I’ve made a module which makes HTTP calls (like downloading a web page) easy, available now on GitHub – Http.fs
I had a project recently which involved making a lot of HTTP requests and dealing with the responses. F# being my current language of choice, I was using that. Unfortunately, .Net’s HttpWebRequest/Response aren’t that nice to use from F# (or C#, frankly).
For example, here’s how you might make an HTTP Post, from F# Snippets:
open System.Text open System.IO open System.Net let url = "http://posttestserver.com/post.php" let req = HttpWebRequest.Create(url) : ?> HttpWebRequest req.ProtocolVersion req.Method <- "POST" let postBytes = Encoding.ASCII.GetBytes("fname=Tomas&lname=Petricek") req.ContentType <- "application/x-www-form-urlencoded"; req.ContentLength let reqStream = req.GetRequestStream() reqStream.Write(postBytes, 0, postBytes.Length); reqStream.Close() let resp = req.GetResponse() let stream = resp.GetResponseStream() let reader = new StreamReader(stream) let html = reader.ReadToEnd()
There are a few things I don’t really like about doing this:
- It’s a lot of code!
- You have to mess around with streams
- The types used are mutable, so not really idiomatic F#
- You have to set things (e.g. ‘POST’) as strings, so not typesafe
- It’s not unit testable
- You have to cast things (e.g. req) to the correct type
In fact there are many other problems with HttpWebRequest/Response which aren’t demonstrated by this sample, including:
- Some headers are defined, others aren’t (so you have to set them as strings)
- You need to mess around with the cookie container to get cookies working
- If the response code is anything but 200-level, you get an exception (!)
- Getting headers and cookies from the response isn’t pretty
Since then I’ve discovered HttpClient, which does address some of these issues, but it’s still not great to use from F# (and only available in .Net 4.5).
So I started to write some wrapper functions around this stuff, and it eventually turned into:
Http.fs is a module which contains a few types and functions for more easily working with Http requests and responses from F#. It uses HttpWebRequest/Response under the hood, although these aren’t exposed directly when you use it.
Downloading a single web page is as simple as:
let page = (createRequest Get "http://www.google.com" |> getResponseBody)
And if you want to do something more in-depth, like the example above, that would look like this:
open HttpClient let response = createRequest Post "http://posttestserver.com/post.php" |> withBody "fname=Tomas&lname=Petricek" |> withHeader (ContentType "application/x-www-form-urlencoded") |> getResponse
Then you could access the response elements like so:
response.StatusCode response.EntityBody.Value response.Headers.[Server]
And of course, it has asynchronous functions to let you do things like download multiple pages in parallel:
["http://news.bbc.co.uk" "http://www.wikipedia.com" "http://www.stackoverflow.com"] |> List.map (fun url -> createRequest Get url |> getResponseBodyAsync) |> Async.Parallel |> Async.RunSynchronously |> Array.iter (printfn "%s")
There are more details on the GitHub page. The project also contains a sample application which shows how it can be used and tested.
So if you’re using F# and want to make a complex HTTP request – or just download a web page – check out Http.fs!
This is now available on NuGet. To install:
PM> install-package Http.fs