Archive

Author Archive

Web requests in F# now easy! Introducing Http.fs

November 15, 2013 2 comments

TL;DR

I’ve made a module which makes HTTP calls (like downloading a web page) easy, available now on GitHub – Http.fs

Introduction

I had a project recently which involved making a lot of HTTP requests and dealing with the responses.  F# being my current language of choice, I was using that.  Unfortunately, .Net’s HttpWebRequest/Response aren’t that nice to use from F# (or C#, frankly).

For example, here’s how you might make an HTTP Post, from F# Snippets:

open System.Text
open System.IO
open System.Net

let url = "http://posttestserver.com/post.php"

let req = HttpWebRequest.Create(url) : ?> HttpWebRequest
req.ProtocolVersion req.Method <- "POST"

let postBytes = Encoding.ASCII.GetBytes("fname=Tomas&lname=Petricek")
req.ContentType <- "application/x-www-form-urlencoded";
req.ContentLength let reqStream = req.GetRequestStream()
reqStream.Write(postBytes, 0, postBytes.Length);
reqStream.Close()

let resp = req.GetResponse()
let stream = resp.GetResponseStream()
let reader = new StreamReader(stream)
let html = reader.ReadToEnd()

There are a few things I don’t really like about doing this:

  • It’s a lot of code!
  • You have to mess around with streams
  • The types used are mutable, so not really idiomatic F#
  • You have to set things (e.g. ‘POST’) as strings, so not typesafe
  • It’s not unit testable
  • You have to cast things (e.g. req) to the correct type

In fact there are many other problems with HttpWebRequest/Response which aren’t demonstrated by this sample, including:

  • Some headers are defined, others aren’t (so you have to set them as strings)
  • You need to mess around with the cookie container to get cookies working
  • If the response code is anything but 200-level, you get an exception (!)
  • Getting headers and cookies from the response isn’t pretty

Since then I’ve discovered HttpClient, which does address some of these issues, but it’s still not great to use from F# (and only available in .Net 4.5).

So I started to write some wrapper functions around this stuff, and it eventually turned into:

Http.fs!

Http.fs is a module which contains a few types and functions for more easily working with Http requests and responses from F#. It uses HttpWebRequest/Response under the hood, although these aren’t exposed directly when you use it.

Downloading a single web page is as simple as:

let page = (createRequest Get "http://www.google.com" |> getResponseBody)

And if you want to do something more in-depth, like the example above, that would look like this:

open HttpClient

let response =
  createRequest Post "http://posttestserver.com/post.php"
  |> withBody "fname=Tomas&lname=Petricek"
  |> withHeader (ContentType "application/x-www-form-urlencoded")
  |> getResponse

Then you could access the response elements like so:

response.StatusCode
response.EntityBody.Value
response.Headers.[Server]

And of course, it has asynchronous functions to let you do things like download multiple pages in parallel:

["http://news.bbc.co.uk"
 "http://www.wikipedia.com"
 "http://www.stackoverflow.com"]
|> List.map (fun url -> createRequest Get url |> getResponseBodyAsync)
|> Async.Parallel
|> Async.RunSynchronously
|> Array.iter (printfn "%s")

There are more details on the GitHub page. The project also contains a sample application which shows how it can be used and tested.

So if you’re using F# and want to make a complex HTTP request – or just download a web page – check out Http.fs!

Update

This is now available on NuGet.  To install:

PM> install-package Http.fs  
Categories: F#, HTTP

Hello Neurons – ENCOG Neural Network XOR example in F#

November 14, 2013 1 comment

I’ve been playing with Machine Learning lately, starting with Abhishek Kumar’s Introduction to Machine Learning video on PluralSight.

This video guides you though using the ENCOG library (available on NuGet) to build a simple neural network for the XOR (eXclusive OR) logic table, which is the ‘Hello World’ of Neural Networks.

I’m not going to go into the details of ML or Neural Networks here (I don’t know them, for a start), but one thing I found was that the .Net ENCOG examples were all in C#.  As such, I though I’d post my F# version here. (See the C# version for comparison).

So, without further ado:

open Encog.ML.Data.Basic
open Encog.Engine.Network.Activation
open Encog.Neural.Networks
open Encog.Neural.Networks.Layers
open Encog.Neural.Networks.Training.Propagation.Resilient

let createNetwork() =
    let network = BasicNetwork()
    network.AddLayer( BasicLayer( null, true, 2 ))
    network.AddLayer( BasicLayer( ActivationSigmoid(), true, 2 ))
    network.AddLayer( BasicLayer( ActivationSigmoid(), false, 1 ))
    network.Structure.FinalizeStructure()
    network.Reset()
    network

let train trainingSet (network: BasicNetwork) =
    let trainedNetwork = network.Clone() : ?> BasicNetwork
    let trainer = ResilientPropagation(trainedNetwork, trainingSet)

    let rec trainIteration epoch error =
        match error > 0.001 with
        | false -> ()
        | true -> trainer.Iteration()
                  printfn "Iteration no : %d, Error: %f" epoch error
                  trainIteration (epoch + 1) trainer.Error

    trainIteration 1 1.0
    trainedNetwork

[<EntryPoint>]
let main argv =

    let xor_input =
        [|
            [| 0.0 ; 0.0 |]
            [| 1.0 ; 0.0 |]
            [| 0.0 ; 1.0 |]
            [| 1.0 ; 1.0 |]
        |]

    let xor_ideal =
        [|
            [| 0.0 |]
            [| 1.0 |]
            [| 1.0 |]
            [| 0.0 |]
        |]

    let trainingSet = BasicMLDataSet(xor_input, xor_ideal)
    let network = createNetwork()

    let trainedNetwork = network |> train trainingSet

    trainingSet
    |> Seq.iter (
        fun item ->
            let output = trainedNetwork.Compute(item.Input)
            printfn "Input: %f, %f Ideal: %f Actual: %f"
                item.Input.[0]  item.Input.[1] item.Ideal.[0] output.[0])

    printfn "Press return to exit.."
    System.Console.Read() |> ignore

    0 // return an integer exit code

The main difference over the C# version is that the training iterations are done with recursion instead of looping, and the training returns a new network rather than updating the existing one. Nothing wrong with doing it that way per se, but it gave me a warm feeling inside to make it all ‘functional’.

It may be a while before I create Skynet, but you’ve got to start somewhere..

Categories: F#, Machine Learning

Testing DateTime.Now (and other things) with an Ambient Context

September 30, 2013 Leave a comment

One issue you’re likely to face while writing automated tests is that it can be tricky to check anything related to DateTimes, as they are typically accessed statically using:

var startTime = DateTime.Now;

The approach I’ve typically used is to create some kind of IDateProvider interface, and supply that as a dependency as needed using Dependency Injection.  However, with things used throughout a system like DateTime can be, this becomes quite a chore and make the code more complex with an ever-increasing list of dependencies.

One of the great tips given by Ian Russell at an Agile Yorkshire meetup recently was to use the Ambient Context pattern in this, and other similar situations.

Note: I didn’t write the code below, I found it in various places online, particularly this Stack Overflow question by Mark Seemann, author of Dependency Injection in .Net.  I was having trouble finding the details online, which is why I’ve posted them here.

So, to take full control of DateTime.Now, you need an abstract TimeProvider:

public abstract class TimeProvider {
   private static TimeProvider current =
   DefaultTimeProvider.Instance;

   public static TimeProvider Current {
      get { return TimeProvider.current; }
      set {
         if (value == null) {
            throw new ArgumentNullException("value");
         }
         TimeProvider.current = value;
      }
   }

   public abstract DateTime UtcNow { get; }

   public static void ResetToDefault() {
      TimeProvider.current = DefaultTimeProvider.Instance;
   }
}

A concrete implementation used in your live system:

public class DefaultTimeProvider : TimeProvider {
   private readonly static DefaultTimeProvider instance = 
      new DefaultTimeProvider();

   private DefaultTimeProvider() { }

   public override DateTime UtcNow {
      get { return DateTime.UtcNow; }
   }

   public static DefaultTimeProvider Instance {
      get { return DefaultTimeProvider.instance; }
   }
}

And a mock for testing, for example here set up with Rhino.Mocks:

var fakeTimeProvider = MockRepository.GenerateStub();
fakeTimeProvider.Expect(x => x.UtcNow).Repeat.Once()
   .Return(new DateTime(1999, 12, 31, 11, 59, 59, 999));
fakeTimeProvider.Expect(x => x.UtcNow).Repeat.Once()
   .Return(new DateTime(2000, 1, 1, 0, 0, 0, 0));

For this to work, instead of calling dateTime.Now (or DateTime.UtcNow as we’re using here) from your code, you need to call:

var startTime = TimeProvider.Current.UtcNow;

One last thing – don’t forget to reset the provider after you mock it out, otherwise your mock will remain as the time provider:

[TestCleanup]
public void TearDown() {
   TimeProvider.ResetToDefault();
}

As suggested in the title, this can be used for things other than DateTime, but that’s the only place I’ve tried it so far.

Happy testing!

Categories: Patterns, TDD, Unit Testing

Making a simple State Machine with F# Actors

F# comes with a built-in Actor framework, using the MailboxProcessor class.  The main reason for using actors is to simplify concurrent processing, but they also hit another sweet spot – they’re pretty good for implementing Finite State Machines (FSM).

I won’t go into the detail of how to implement a standard actor, but here’s a simple example which maintains a sum of the numbers posted to it.  It uses an asynchronous workflow which runs it asynchronously in a different thread, and a recursive function to maintain the state (although as an actor’s state is not shared, mutable state would be acceptable):

let countingActor =
  MailboxProcessor.Start(fun inbox ->
    let rec loop num = async {
      do printfn "num = %i" num
      let! msg = inbox.Receive()
      return! loop(num+msg) }
    loop 0)

And here it is being called in FSI:

countingActor.Post 10;;
num = 10
val it : unit = ()

countingActor.Post 5;;
num = 15
val it : unit = ()

Perhaps this is not the most elegant-looking or understandable code – I hear actors in Erlang and Scala are much more succinct – but once you give it a try it starts to make sense. For what we’re about to consider, the important point to note is that the recursive ‘loop’ function is being called infinitely, each time waiting until a message has been received.

So, how to make a state machine with it?  To keep this as simple as possible, let’s consider a hugely inefficient climate control system which has only two states – heating and cooling.  We can define this as a MailboxProcessor:

// define the messages which can be used to change the state,
// using a Discriminated Union
type message =
    | HeatUp
    | CoolDown

// define the actor
let climateControl = MailboxProcessor.Start( fun inbox ->

    // the 'heating' state
    let rec heating() = async {
        printfn "Heating"
        let! msg = inbox.Receive()
        match msg with
        | CoolDown -> return! cooling()
        | _ -> return! heating()}

    // the 'cooling' state
    and cooling() = async {
        printfn "Cooling"
        let! msg = inbox.Receive()
        match msg with
        | HeatUp -> return! heating()
        | _ -> return! cooling()}

    // the initial state
    heating()
    )

And post messages to it to change states:

climateControl.Post HeatUp;;
Heating
val it : unit = ()

climateControl.Post CoolDown;;
Cooling
val it : unit = ()

And that’s it! The main points of interest are the states, each defined by a separate function heating() and cooling() – note they are mutually recursive so have to be declared with ‘and’ – and the ‘match’ expression used to change states.

Of course they can get more complicated, but you have the basic pattern. For a slightly more in-depth look at FSMs with F# actors, check out this post by the monad nomad.

Happy state transitions!

Categories: F# Tags: ,

F# Koans – Stock Example

March 31, 2013 7 comments

At Leeds Sharp the other day, we started doing the F# Koans as a way of of trying out the language, with the aim of expanding our minds with some functional programming.  A bit like coder’s LSD, if you will.

The koans are pretty good at introducing you to things, although initially it’s a bit too easy to get through them without really understanding what’s going on.

However, when you get to the stock example that all changes, as you have to actually use some of what you’ve learned to solve a more involved problem. The purpose of this post is to put up my solution  in the hope of getting into a bit of a discussion with others who’ve done it, at Leeds Sharp or otherwise. I have no doubt that my implementation could be improved, so any advice would be more than welcome!

module ``about the stock example`` =

  let stockData =
    [ "Date,Open,High,Low,Close,Volume,Adj Close";
      "2012-03-30,32.40,32.41,32.04,32.26,31749400,32.26";
      // ...
      "2012-02-29,31.89,32.00,31.61,31.74,59323600,31.74"; ]

  // function to split a comma-separated string into an array
  let splitOnCommas (dataAsString:string) =
    dataAsString.Split([|','|])

  // function to take the desired parts of the array as a tuple,
  // including opening-closing difference
  let createPriceDifferenceTuple (fullList:string[]) =
    ( fullList.[0],
      abs (Double.Parse(fullList.[1]) - Double.Parse(fullList.[4])) )

  let maximumDifferenceTuple =
    // split strings into arrays, ignoring header
    List.map splitOnCommas stockData.Tail
    // map arrays to tuples
    |> List.map createPriceDifferenceTuple
    // get maximum tuple based on second element (price difference)
    |> List.maxBy snd

  [<Koan>]
  let YouGotTheAnswerCorrect() =
    let result = fst maximumDifferenceTuple // get date, first item in tuple
    AssertEquality "2012-03-13" result

Categories: Development, F#
Follow

Get every new post delivered to your Inbox.

Join 116 other followers