Web requests in F# now easy! Introducing Http.fs

TL;DR

I’ve made a module which makes HTTP calls (like downloading a web page) easy, available now on GitHub – Http.fs

Introduction

I had a project recently which involved making a lot of HTTP requests and dealing with the responses.  F# being my current language of choice, I was using that.  Unfortunately, .Net’s HttpWebRequest/Response aren’t that nice to use from F# (or C#, frankly).

For example, here’s how you might make an HTTP Post, from F# Snippets:

open System.Text
open System.IO
open System.Net

let url = "http://posttestserver.com/post.php"

let req = HttpWebRequest.Create(url) : ?> HttpWebRequest
req.ProtocolVersion req.Method <- "POST"

let postBytes = Encoding.ASCII.GetBytes("fname=Tomas&lname=Petricek")
req.ContentType <- "application/x-www-form-urlencoded";
req.ContentLength let reqStream = req.GetRequestStream()
reqStream.Write(postBytes, 0, postBytes.Length);
reqStream.Close()

let resp = req.GetResponse()
let stream = resp.GetResponseStream()
let reader = new StreamReader(stream)
let html = reader.ReadToEnd()

There are a few things I don’t really like about doing this:

• It’s a lot of code!
• You have to mess around with streams
• The types used are mutable, so not really idiomatic F#
• You have to set things (e.g. ‘POST’) as strings, so not typesafe
• It’s not unit testable
• You have to cast things (e.g. req) to the correct type

In fact there are many other problems with HttpWebRequest/Response which aren’t demonstrated by this sample, including:

• Some headers are defined, others aren’t (so you have to set them as strings)
• You need to mess around with the cookie container to get cookies working
• If the response code is anything but 200-level, you get an exception (!)
• Getting headers and cookies from the response isn’t pretty

Since then I’ve discovered HttpClient, which does address some of these issues, but it’s still not great to use from F# (and only available in .Net 4.5).

So I started to write some wrapper functions around this stuff, and it eventually turned into:

Http.fs!

Http.fs is a module which contains a few types and functions for more easily working with Http requests and responses from F#. It uses HttpWebRequest/Response under the hood, although these aren’t exposed directly when you use it.

Downloading a single web page is as simple as:

let page = (createRequest Get "http://www.google.com" |> getResponseBody)

And if you want to do something more in-depth, like the example above, that would look like this:

open HttpClient

let response =
  createRequest Post "http://posttestserver.com/post.php"
  |> withBody "fname=Tomas&lname=Petricek"
  |> withHeader (ContentType "application/x-www-form-urlencoded")
  |> getResponse

Then you could access the response elements like so:

response.StatusCode
response.EntityBody.Value
response.Headers.[Server]

And of course, it has asynchronous functions to let you do things like download multiple pages in parallel:

["http://news.bbc.co.uk"
 "http://www.wikipedia.com"
 "http://www.stackoverflow.com"]
|> List.map (fun url -> createRequest Get url |> getResponseBodyAsync)
|> Async.Parallel
|> Async.RunSynchronously
|> Array.iter (printfn "%s")

There are more details on the GitHub page. The project also contains a sample application which shows how it can be used and tested.

So if you’re using F# and want to make a complex HTTP request – or just download a web page – check out Http.fs!

Update

This is now available on NuGet.  To install:

PM> install-package Http.fs  

Hello Neurons – ENCOG Neural Network XOR example in F#

I’ve been playing with Machine Learning lately, starting with Abhishek Kumar’s Introduction to Machine Learning video on PluralSight.

This video guides you though using the ENCOG library (available on NuGet) to build a simple neural network for the XOR (eXclusive OR) logic table, which is the ‘Hello World’ of Neural Networks.

I’m not going to go into the details of ML or Neural Networks here (I don’t know them, for a start), but one thing I found was that the .Net ENCOG examples were all in C#.  As such, I though I’d post my F# version here. (See the C# version for comparison).

So, without further ado:

open Encog.ML.Data.Basic
open Encog.Engine.Network.Activation
open Encog.Neural.Networks
open Encog.Neural.Networks.Layers
open Encog.Neural.Networks.Training.Propagation.Resilient

let createNetwork() =
    let network = BasicNetwork()
    network.AddLayer( BasicLayer( null, true, 2 ))
    network.AddLayer( BasicLayer( ActivationSigmoid(), true, 2 ))
    network.AddLayer( BasicLayer( ActivationSigmoid(), false, 1 ))
    network.Structure.FinalizeStructure()
    network.Reset()
    network

let train trainingSet (network: BasicNetwork) =
    let trainedNetwork = network.Clone() : ?> BasicNetwork
    let trainer = ResilientPropagation(trainedNetwork, trainingSet)

    let rec trainIteration epoch error =
        match error > 0.001 with
        | false -> ()
        | true -> trainer.Iteration()
                  printfn "Iteration no : %d, Error: %f" epoch error
                  trainIteration (epoch + 1) trainer.Error

    trainIteration 1 1.0
    trainedNetwork

[<EntryPoint>]
let main argv =

    let xor_input =
        [|
            [| 0.0 ; 0.0 |]
            [| 1.0 ; 0.0 |]
            [| 0.0 ; 1.0 |]
            [| 1.0 ; 1.0 |]
        |]

    let xor_ideal =
        [|
            [| 0.0 |]
            [| 1.0 |]
            [| 1.0 |]
            [| 0.0 |]
        |]

    let trainingSet = BasicMLDataSet(xor_input, xor_ideal)
    let network = createNetwork()

    let trainedNetwork = network |> train trainingSet

    trainingSet
    |> Seq.iter (
        fun item ->
            let output = trainedNetwork.Compute(item.Input)
            printfn "Input: %f, %f Ideal: %f Actual: %f"
                item.Input.[0]  item.Input.[1] item.Ideal.[0] output.[0])

    printfn "Press return to exit.."
    System.Console.Read() |> ignore

    0 // return an integer exit code

The main difference over the C# version is that the training iterations are done with recursion instead of looping, and the training returns a new network rather than updating the existing one. Nothing wrong with doing it that way per se, but it gave me a warm feeling inside to make it all ‘functional’.

It may be a while before I create Skynet, but you’ve got to start somewhere..