Web requests in F# now easy! Introducing Http.fs

November 15, 2013 2 comments

TL;DR

I’ve made a module which makes HTTP calls (like downloading a web page) easy, available now on GitHub – Http.fs

Introduction

I had a project recently which involved making a lot of HTTP requests and dealing with the responses.  F# being my current language of choice, I was using that.  Unfortunately, .Net’s HttpWebRequest/Response aren’t that nice to use from F# (or C#, frankly).

For example, here’s how you might make an HTTP Post, from F# Snippets:

open System.Text
open System.IO
open System.Net

let url = "http://posttestserver.com/post.php"

let req = HttpWebRequest.Create(url) : ?> HttpWebRequest
req.ProtocolVersion req.Method <- "POST"

let postBytes = Encoding.ASCII.GetBytes("fname=Tomas&lname=Petricek")
req.ContentType <- "application/x-www-form-urlencoded";
req.ContentLength let reqStream = req.GetRequestStream()
reqStream.Write(postBytes, 0, postBytes.Length);
reqStream.Close()

let resp = req.GetResponse()
let stream = resp.GetResponseStream()
let reader = new StreamReader(stream)
let html = reader.ReadToEnd()

There are a few things I don’t really like about doing this:

  • It’s a lot of code!
  • You have to mess around with streams
  • The types used are mutable, so not really idiomatic F#
  • You have to set things (e.g. ‘POST’) as strings, so not typesafe
  • It’s not unit testable
  • You have to cast things (e.g. req) to the correct type

In fact there are many other problems with HttpWebRequest/Response which aren’t demonstrated by this sample, including:

  • Some headers are defined, others aren’t (so you have to set them as strings)
  • You need to mess around with the cookie container to get cookies working
  • If the response code is anything but 200-level, you get an exception (!)
  • Getting headers and cookies from the response isn’t pretty

Since then I’ve discovered HttpClient, which does address some of these issues, but it’s still not great to use from F# (and only available in .Net 4.5).

So I started to write some wrapper functions around this stuff, and it eventually turned into:

Http.fs!

Http.fs is a module which contains a few types and functions for more easily working with Http requests and responses from F#. It uses HttpWebRequest/Response under the hood, although these aren’t exposed directly when you use it.

Downloading a single web page is as simple as:

let page = (createRequest Get "http://www.google.com" |> getResponseBody)

And if you want to do something more in-depth, like the example above, that would look like this:

open HttpClient

let response =
  createRequest Post "http://posttestserver.com/post.php"
  |> withBody "fname=Tomas&lname=Petricek"
  |> withHeader (ContentType "application/x-www-form-urlencoded")
  |> getResponse

Then you could access the response elements like so:

response.StatusCode
response.EntityBody.Value
response.Headers.[Server]

And of course, it has asynchronous functions to let you do things like download multiple pages in parallel:

["http://news.bbc.co.uk"
 "http://www.wikipedia.com"
 "http://www.stackoverflow.com"]
|> List.map (fun url -> createRequest Get url |> getResponseBodyAsync)
|> Async.Parallel
|> Async.RunSynchronously
|> Array.iter (printfn "%s")

There are more details on the GitHub page. The project also contains a sample application which shows how it can be used and tested.

So if you’re using F# and want to make a complex HTTP request – or just download a web page – check out Http.fs!

Update

This is now available on NuGet.  To install:

PM> install-package Http.fs  
Categories: F#, HTTP

Hello Neurons – ENCOG Neural Network XOR example in F#

November 14, 2013 1 comment

I’ve been playing with Machine Learning lately, starting with Abhishek Kumar’s Introduction to Machine Learning video on PluralSight.

This video guides you though using the ENCOG library (available on NuGet) to build a simple neural network for the XOR (eXclusive OR) logic table, which is the ‘Hello World’ of Neural Networks.

I’m not going to go into the details of ML or Neural Networks here (I don’t know them, for a start), but one thing I found was that the .Net ENCOG examples were all in C#.  As such, I though I’d post my F# version here. (See the C# version for comparison).

So, without further ado:

open Encog.ML.Data.Basic
open Encog.Engine.Network.Activation
open Encog.Neural.Networks
open Encog.Neural.Networks.Layers
open Encog.Neural.Networks.Training.Propagation.Resilient

let createNetwork() =
    let network = BasicNetwork()
    network.AddLayer( BasicLayer( null, true, 2 ))
    network.AddLayer( BasicLayer( ActivationSigmoid(), true, 2 ))
    network.AddLayer( BasicLayer( ActivationSigmoid(), false, 1 ))
    network.Structure.FinalizeStructure()
    network.Reset()
    network

let train trainingSet (network: BasicNetwork) =
    let trainedNetwork = network.Clone() : ?> BasicNetwork
    let trainer = ResilientPropagation(trainedNetwork, trainingSet)

    let rec trainIteration epoch error =
        match error > 0.001 with
        | false -> ()
        | true -> trainer.Iteration()
                  printfn "Iteration no : %d, Error: %f" epoch error
                  trainIteration (epoch + 1) trainer.Error

    trainIteration 1 1.0
    trainedNetwork

[<EntryPoint>]
let main argv =

    let xor_input =
        [|
            [| 0.0 ; 0.0 |]
            [| 1.0 ; 0.0 |]
            [| 0.0 ; 1.0 |]
            [| 1.0 ; 1.0 |]
        |]

    let xor_ideal =
        [|
            [| 0.0 |]
            [| 1.0 |]
            [| 1.0 |]
            [| 0.0 |]
        |]

    let trainingSet = BasicMLDataSet(xor_input, xor_ideal)
    let network = createNetwork()

    let trainedNetwork = network |> train trainingSet

    trainingSet
    |> Seq.iter (
        fun item ->
            let output = trainedNetwork.Compute(item.Input)
            printfn "Input: %f, %f Ideal: %f Actual: %f"
                item.Input.[0]  item.Input.[1] item.Ideal.[0] output.[0])

    printfn "Press return to exit.."
    System.Console.Read() |> ignore

    0 // return an integer exit code

The main difference over the C# version is that the training iterations are done with recursion instead of looping, and the training returns a new network rather than updating the existing one. Nothing wrong with doing it that way per se, but it gave me a warm feeling inside to make it all ‘functional’.

It may be a while before I create Skynet, but you’ve got to start somewhere..

Categories: F#, Machine Learning

Testing DateTime.Now (and other things) with an Ambient Context

September 30, 2013 Leave a comment

One issue you’re likely to face while writing automated tests is that it can be tricky to check anything related to DateTimes, as they are typically accessed statically using:

var startTime = DateTime.Now;

The approach I’ve typically used is to create some kind of IDateProvider interface, and supply that as a dependency as needed using Dependency Injection.  However, with things used throughout a system like DateTime can be, this becomes quite a chore and make the code more complex with an ever-increasing list of dependencies.

One of the great tips given by Ian Russell at an Agile Yorkshire meetup recently was to use the Ambient Context pattern in this, and other similar situations.

Note: I didn’t write the code below, I found it in various places online, particularly this Stack Overflow question by Mark Seemann, author of Dependency Injection in .Net.  I was having trouble finding the details online, which is why I’ve posted them here.

So, to take full control of DateTime.Now, you need an abstract TimeProvider:

public abstract class TimeProvider {
   private static TimeProvider current =
   DefaultTimeProvider.Instance;

   public static TimeProvider Current {
      get { return TimeProvider.current; }
      set {
         if (value == null) {
            throw new ArgumentNullException("value");
         }
         TimeProvider.current = value;
      }
   }

   public abstract DateTime UtcNow { get; }

   public static void ResetToDefault() {
      TimeProvider.current = DefaultTimeProvider.Instance;
   }
}

A concrete implementation used in your live system:

public class DefaultTimeProvider : TimeProvider {
   private readonly static DefaultTimeProvider instance = 
      new DefaultTimeProvider();

   private DefaultTimeProvider() { }

   public override DateTime UtcNow {
      get { return DateTime.UtcNow; }
   }

   public static DefaultTimeProvider Instance {
      get { return DefaultTimeProvider.instance; }
   }
}

And a mock for testing, for example here set up with Rhino.Mocks:

var fakeTimeProvider = MockRepository.GenerateStub();
fakeTimeProvider.Expect(x => x.UtcNow).Repeat.Once()
   .Return(new DateTime(1999, 12, 31, 11, 59, 59, 999));
fakeTimeProvider.Expect(x => x.UtcNow).Repeat.Once()
   .Return(new DateTime(2000, 1, 1, 0, 0, 0, 0));

For this to work, instead of calling dateTime.Now (or DateTime.UtcNow as we’re using here) from your code, you need to call:

var startTime = TimeProvider.Current.UtcNow;

One last thing – don’t forget to reset the provider after you mock it out, otherwise your mock will remain as the time provider:

[TestCleanup]
public void TearDown() {
   TimeProvider.ResetToDefault();
}

As suggested in the title, this can be used for things other than DateTime, but that’s the only place I’ve tried it so far.

Happy testing!

Categories: Patterns, TDD, Unit Testing

Making a simple State Machine with F# Actors

F# comes with a built-in Actor framework, using the MailboxProcessor class.  The main reason for using actors is to simplify concurrent processing, but they also hit another sweet spot – they’re pretty good for implementing Finite State Machines (FSM).

I won’t go into the detail of how to implement a standard actor, but here’s a simple example which maintains a sum of the numbers posted to it.  It uses an asynchronous workflow which runs it asynchronously in a different thread, and a recursive function to maintain the state (although as an actor’s state is not shared, mutable state would be acceptable):

let countingActor =
  MailboxProcessor.Start(fun inbox ->
    let rec loop num = async {
      do printfn "num = %i" num
      let! msg = inbox.Receive()
      return! loop(num+msg) }
    loop 0)

And here it is being called in FSI:

countingActor.Post 10;;
num = 10
val it : unit = ()

countingActor.Post 5;;
num = 15
val it : unit = ()

Perhaps this is not the most elegant-looking or understandable code – I hear actors in Erlang and Scala are much more succinct – but once you give it a try it starts to make sense. For what we’re about to consider, the important point to note is that the recursive ‘loop’ function is being called infinitely, each time waiting until a message has been received.

So, how to make a state machine with it?  To keep this as simple as possible, let’s consider a hugely inefficient climate control system which has only two states – heating and cooling.  We can define this as a MailboxProcessor:

// define the messages which can be used to change the state,
// using a Discriminated Union
type message =
    | HeatUp
    | CoolDown

// define the actor
let climateControl = MailboxProcessor.Start( fun inbox ->

    // the 'heating' state
    let rec heating() = async {
        printfn "Heating"
        let! msg = inbox.Receive()
        match msg with
        | CoolDown -> return! cooling()
        | _ -> return! heating()}

    // the 'cooling' state
    and cooling() = async {
        printfn "Cooling"
        let! msg = inbox.Receive()
        match msg with
        | HeatUp -> return! heating()
        | _ -> return! cooling()}

    // the initial state
    heating()
    )

And post messages to it to change states:

climateControl.Post HeatUp;;
Heating
val it : unit = ()

climateControl.Post CoolDown;;
Cooling
val it : unit = ()

And that’s it! The main points of interest are the states, each defined by a separate function heating() and cooling() – note they are mutually recursive so have to be declared with ‘and’ – and the ‘match’ expression used to change states.

Of course they can get more complicated, but you have the basic pattern. For a slightly more in-depth look at FSMs with F# actors, check out this post by the monad nomad.

Happy state transitions!

Categories: F# Tags: ,

F# Koans – Stock Example

March 31, 2013 7 comments

At Leeds Sharp the other day, we started doing the F# Koans as a way of of trying out the language, with the aim of expanding our minds with some functional programming.  A bit like coder’s LSD, if you will.

The koans are pretty good at introducing you to things, although initially it’s a bit too easy to get through them without really understanding what’s going on.

However, when you get to the stock example that all changes, as you have to actually use some of what you’ve learned to solve a more involved problem. The purpose of this post is to put up my solution  in the hope of getting into a bit of a discussion with others who’ve done it, at Leeds Sharp or otherwise. I have no doubt that my implementation could be improved, so any advice would be more than welcome!

module ``about the stock example`` =

  let stockData =
    [ "Date,Open,High,Low,Close,Volume,Adj Close";
      "2012-03-30,32.40,32.41,32.04,32.26,31749400,32.26";
      // ...
      "2012-02-29,31.89,32.00,31.61,31.74,59323600,31.74"; ]

  // function to split a comma-separated string into an array
  let splitOnCommas (dataAsString:string) =
    dataAsString.Split([|','|])

  // function to take the desired parts of the array as a tuple,
  // including opening-closing difference
  let createPriceDifferenceTuple (fullList:string[]) =
    ( fullList.[0],
      abs (Double.Parse(fullList.[1]) - Double.Parse(fullList.[4])) )

  let maximumDifferenceTuple =
    // split strings into arrays, ignoring header
    List.map splitOnCommas stockData.Tail
    // map arrays to tuples
    |> List.map createPriceDifferenceTuple
    // get maximum tuple based on second element (price difference)
    |> List.maxBy snd

  [<Koan>]
  let YouGotTheAnswerCorrect() =
    let result = fst maximumDifferenceTuple // get date, first item in tuple
    AssertEquality "2012-03-13" result

Categories: Development, F#

Golden NuGet!

March 29, 2011 1 comment

For anyone using VS2010, one tool you need to try out is NuGet (actually pronounced new-get according to Microsoft, but what do they know?!).

What is it?

NuGet is Microsoft’s answer to the much-lamented gap in the .Net space for a package management tool like Ruby’s Gems or Linux’s RPM.

NuGet simply looks after the download and installation of tools and libraries you use in your projects.  It will get the appropriate version of all needed DLLs, configuration files, and related tools, put them in a folder, and do any configuration if needed.  Simple!  Also useful is the fact that it will download any dependencies – so for example if we want to install Shouldly, which uses Rhino Mocks, NuGet will get the appropriate version of that as well.

And that’s about it.  So it won’t change your world, but it will save you a good few hours each year hunting for, downloading, and installing tools.

History

NuGet started out as NuPack, which kind of took over from the Nu open source project.  Indeed, NuGet is itself open source, continuing Microsoft’s trend of a more transparent and community-involving future, which is nice.

There are other package managers for .Net, including:

However, as usual, things don’t tend to take off in the .Net community until Microsoft makes their version, which then becomes king. I haven’t tried any of those package managers, but I wouldn’t want to bet on them being too successful now NuGet’s out.

Anyway, enough of the history lesson, how do I use this thing?!

Installation

Firstly, you need to get it installed, which is simple as it’s in the Visual Studio Extension Manager (Tools -> Extension Manager).  Search the Online Gallery for ‘NuGet’ and you should find it:

VS2010 Extension Manager Screenshot

VS2010 Extension Manager

A couple of clicks later and you’re ready to go!

Putting it to Use

There are two interfaces.  You can either right-click on a project and select ‘Add Library Reference’, and find what you need with a handy visual interface (the same as the Extension Manager):

NuGet Visual Interface screenshot

NuGet Visual Interface

Or, by selecting Tools -> Library Package Manager -> Package Manager Console,  you can use the Powershell-based console version, like a real developer!

NuGet console screenshot

NuGet Console

(Check out some NuGet Console Commands, if that’s how you want to roll.)

Either way, you’ll end up with a new ‘packages’ folder in your solution, containing the bits you need (as well as having references automatically added to your project, and configuration set):

Packages Folder screenshot

Packages Folder

Et voilá!  You’re ready to log/unit test/mock/whatever to your heart’s content.

You can also delete packages, update packages, and so on – it’s all pretty straightforward, so I’ll leave you to work that out.

So what are you waiting for?  Go and nu-get it now!!  (sorry..)

Categories: Tools Tags: ,

Books Page

January 28, 2011 1 comment

I just put up a page with a load of book reviews.  It’s not finished yet but there are a fair few on there.  Check it out if you’re looking for a good read!

Categories: Books Tags:
Follow

Get every new post delivered to your Inbox.

Join 116 other followers