Functional Flow

Maintaining Backwards Compatibility When Changing APIs in F#

| Comments

Classes

By now, mostly everyone already knows how to change the internal representation of a class without breaking compatibility. In C#:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
// Before
class Rectangle {

    public Point TopLeft { get; private set; }
    public Point BottomRight { get; private set; }

    public Rectangle(Point topLeft, Point bottomRight) {
        TopLeft = topLeft;
        BottomRight = bottomRight;
    }
}

// After
class Rectangle {

    public Point TopLeft { get; private set; }
    public Point Size { get; private set; }

    public Rectangle(Point topLeft, Size size) {
        TopLeft = topLeft;
        Size = size;
    }

    // for backwards compatibility:
    public Rectangle(Point topLeft, Point bottomRight) {
        TopLeft = topLeft;
        Size = new Size(bottomRight.X - topLeft.X, bottomRight.Y - topLeft.Y);
    }

    // for backwards compatibility:
    public Point BottomRight {
        get { return new Point(TopLeft.X + Size.Width, TopLeft.Y + Size.Height); }
    }
}

and in F#:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
// Before
type Rectangle(topLeft:Point, bottomRight:Point) =

    member __.TopLeft = topLeft
    member __.BottomRight = bottomRight

// After
type Rectangle(topLeft:Point, size:Size) =

    member __.TopLeft = topLeft
    member __.Size = size

    // for backwards compatibility:
    new (topLeft:Point, bottomRight:Point) =
        Rectangle(topLeft, Point(bottomRight.X - topLeft.X, bottomRight.Y - topLeft.Y))

    // for backwards compatibility:
    member __.BottomRight =
        Point(topLeft.X + size.Width, topLeft.Y + size.Height)

Records

When using record types, things are a little bit more tricky, though:

1
2
3
4
5
6
7
8
9
10
11
// Before
type Rectangle =
    { TopLeft : Point
      BottomRight : Point }

// After
type Rectangle =
    { TopLeft : Point
      Size : Size }
    member x.BottomRight =
        Point(x.TopLeft.X + x.Size.Width, x.TopLeft.Y + x.Size.Height)

This change will keep working at the usages of Rectangle, but not at creation, as there isn’t a way (at least that I know of) to make something like { TopLeft = Point(2.5, 3.0); BottomRight = Point(4.0, 7.5) } keep working. This can be avoided if we originally had hidden the fact that Rectangle was representation as a record, like this:

1
2
3
4
5
6
7
type Rectangle =
    private { TL : Point
              BR : Point }
    member x.TopLeft = x.TL
    member x.BottomRight = x.BR
    static member Create(topLeft, bottomRight) =
        { TL = topLeft; BR = bottomRight }

But for me this seems to defeat some of the advantages of using a record in the first place. You still get structural equality and pretty printing for free, which classes don’t have, but you lose the nice pattern matching and type inference capabilities that usually come with records.

Single case discriminated unions

The situation for discriminated unions is better, though. For each changed case, you can use one active pattern and a function to maintain backward compatibility:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
// Before
type Rectangle =
    | Rectange of topLeft:Point * bottomRight:Point

// After
type Rectangle =
    | Rectangle2 of topLeft:Point * size:Size

// for backwards compatibility:
let (|Rectangle|) x =
    match x with
    | Rectangle2(topLeft, size) -> topLeft, Point(topLeft.X + size.Width, topLeft.Y + size.Height)

// for backwards compatibility:
let Rectangle(topLeft:Point, bottomRight:Point) =
    Rectangle2(topLeft, Size(bottomRight.X - topLeft.X, bottomRight.Y - topLeft.Y))

The downside is that the new discriminated union case has to have a new name, otherwise the function for backward compatibility will shadow the case constructor. This might not necessarily be a bad thing, though.

We could also make the discriminated union representation private, like we did for the record:

1
2
3
4
5
6
7
8
9
10
type Rectangle =
    private | Rectangle of topLeft:Point * bottomRight:Point
    member x.TopLeft =
        match x with
        | Rectangle(topLeft, _) -> topLeft
    member x.BottomRight =
        match x with
        | Rectangle(_, bottomRight) -> bottomRight
    static member Create(topLeft, bottomRight) =
        Rectangle(topLeft, bottomRight)

When we’re deciding between using a record or a single case discriminated union, these considerations might help decide.

Multiple case discriminated unions

When we have discriminated unions with multiple cases, the same technique also applies, but now we need to use partial active patterns instead ((|Pattern|_|) returning an option instead of (|Pattern|) returning the type directly). Here’s an example:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
// Before
[<RequireQualifiedAccess>]
type JsonValue =
    | String of string
    | Number of decimal
    | Float of float
    | Object of Map<string, JsonValue>
    | Array of JsonValue[]
    | Boolean of bool
    | Null

// After
[<RequireQualifiedAccess>]
type JsonValue =
    | String of string
    | Number of decimal
    | Float of float
    | Record of properties:(string*JsonValue)[]
    | Array of elements:JsonValue[]
    | Boolean of bool
    | Null

[<CompilationRepresentation(CompilationRepresentationFlags.ModuleSuffix)>]
module JsonValue =

    [<Obsolete("Please use JsonValue.Record instead")>]
    let (|Object|_|) x =
        match x with
        | JsonValue.Record properties -> Map.ofArray properties |> Some
        | _ -> None

    [<Obsolete("Please use JsonValue.Record instead")>]
    let Object = Map.toArray >> JsonValue.Record

In this example, because [<RequireQualifiedAccess>] was being used, we have to include the active pattern and the compatibility function in a module, and use [<CompilationRepresentation(CompilationRepresentationFlags.ModuleSuffix)>] to be able to maintain the exact same syntax. We also marked the backward compatibility constructs with the Obsolete attribute because we don’t want new users to be using the old discriminated union case. This attribute will make sure the old case doesn’t show up in IntelliSense, and will also issue a warning to old users.

There’s one extra caveat, though. Consider the following code:

1
2
3
4
5
6
7
8
9
let f x =
    match x with
    | String s -> ...
    | Number n -> ...
    | Float f -> ...
    | Object properties -> ...
    | Array a -> ...
    | Boolean b -> ...
    | Null -> ...

Even though we’re handling all the cases, we get a warning stating this pattern-matching is not exhaustive. This is because the compiler doesn’t know what Object will match on: it’s a custom defined partial active pattern, so the implementation could match on multiple cases or even none at all.

Final thoughts

In all of the above we were only concerned with source compatibility. Binary compatibility has other concerns that would require further exploration.

F# Named Union Fields

| Comments

One of the new features in F# 3.1, and one that I really like, is the possibility to name fields of discriminated unions. From the F# 3.1 announcement blog post:

Discriminated union types are a powerful feature of F# which make programming with shaped data simple, accurate and robust. They also greatly reduce the number of classes needed to represent data in many common circumstances.

In F# 3.1, it is now possible to name union fields within each case of a union type. This is important for large-scale software engineering, particularly when union cases have a large number of fields or multiple fields with the same type.

What wasn’t originally mentioned on that post, I just found out today, is that pattern match by name is not limited to a single element of the case: we can match on multiple elements by separating them by ;.

When we pattern match by position, the several elements are separated by ,, and as that didn’t work with union field names, I assumed it wasn’t possible to do at all. But it turns out it is possible if you use ; instead. Here’s an example taken from F# Data:

1
2
3
4
5
6
7
8
9
10
11
12
13
type InferedType =
  | Primitive of typ:System.Type * unit:option<System.Type> * optional:bool
  | ...

// positional pattern match:
match x with
| Primitive(typ, _, false) -> ...
| ...

// pattern match by name 
match x with
| Primitive(typ = typ; optional = false) -> ...
| ...

FSharp.Data 2.0.0 Released

| Comments

Version 2.0.0 of FSharp.Data was just released. You can get the full version history on the release notes, but here are the highlights:

  • A new type inference algorithm that will generate better and more robust API’s in the presence of missing and heterogeneous types. This means that you’ll get much less errors when using JsonProvider and XmlProvider on real world messy files. The error messages are now also more descriptive when things go wrong.
  • Much improved Freebase support for individuals.
  • The functionality of the Http class was greatly extended. It now supports uploading and downloading binary files, and compression across all supported platforms, among many other tweaks.
  • Improved the overall performance of the type provider itself when running inside the editor, and also of the generated code.
  • The documentation was much improved, there’s now a Japanese translation of it, and comprehensive reference documentation.
  • Mono is now fully supported, as well as F# 3.1, and the new Portable Class Library profile 7.
  • Source link support. This means that you can step trough the source code while debugging if you enable that feature in Visual Studio.

We also took the opportunity of major version bump to do some house cleaning, so things that we weren’t previously very happy with were renamed, dropped or, shuffled around. Unfortunately this means breaking changes, but we think the overall outcome is positive. In this process we dropped support for:

  • Silverlight and Windows Phone separate packages.
  • Apiary provider (which has now moved to a separate project).

Thanks to everyone which contributed directly to this release, Tomas Petricek, Don Syme, Steffen Forkmann, Enrico Sada, Cameron Taggart, Remko Boschker, Vasily Kirichenko, Diego Frata, yukitos, Denis Ok, and James Holwell, and also to everyone who also contributed indirectly through yak-shaving needed in FSharp.Formatting, FSharp.Compiler.Services, and FAKE, including Dave Thomas, Xiang Zhang, Anh-Dung Phan, fbmnds, and Max Malook. And also thanks to everyone who contributed by just placing issues on github, but I couldn’t find a way to generate a list of all of you.

There’s still a lot of room for improvement, and we love pull requests, so please contribute, even if it’s only fixing a typo in the docs. A getting started guide is available here.

For the next versions, we want to add:

  • Html type provider, currently being worked on by Colin Bull, Mathias Brandewinder and me.
  • Write API for Json, currently being worked on by Ross McKinlay.
  • Authentication support, currently being worked on by Veikko Eeva.
  • XSD support, which I hope one of you reading this will pick up :).

Higher Order List Operations Across Languages

| Comments

Did this table out of curiosity, decided to share it:

----------------------------------------------------------------------------------------------
| F#         | C#         | Scala       | Clojure | Python | Ruby     | Haskell   | SQL      |
|------------|------------|-------------|---------|--------|----------|-----------|----------|
| map        | Select     | map         | map     | map    | collect  | map       | Select   |
| filter     | Where      | filter      | filter  | filter | select   | filter    | Where    |
| fold       | Aggregate  | foldLeft    | reduce  | reduce | inject   | foldl     |          |
| foldBack   |            | foldRight   |         |        |          | foldr     |          |
| reduce     | Aggregate  | reduceLeft  | reduce  | reduce | inject   | foldl1    |          |
| reduceBack |            | reduceRight |         |        |          | foldr1    |          |
| collect    | SelectMany | flatMap     | mapcat  |        | flat_map | concatMap | From     |
| exists     | Any        | exists      | some    | any    | any?     | any       | Exists   |
| forall     | All        | forall      | every?  | all    | all?     | all       |          |
| sortBy     | OrderBy    | sortBy      | sort-by | sorted | sort_by  | sortBy    | Order By |
----------------------------------------------------------------------------------------------

PS: please correct me if I got something wrong

F# for Screen Scraping

| Comments

One of the many things F# is great for is screen scraping. Here’s why:

  • Downloading multiple pages asynchronously and in parallel is trivial with F#’s async support
  • Navigating the HTML DOM is a great fit for higher order data processing combined with partial application
  • F# Interactive really shines in iterative processes like this, where you try something out, see it didn’t work quite well, and keep adjusting until you get it right. Doing a full compile-run cycle on each iteration instead of simply evaluating in the REPL would make this task take much more time-consuming

Html Agility Pack is the obvious candidate to use for screen scraping in .NET, but like other LINQ-like libraries that rely heavily on extension methods, its API isn’t ideal for use in F#. A simple wrapper will take care of that problem:

FSharp.Data 1.1.9 Released

| Comments

We’ve just released version 1.1.9 of FSharp.Data, and I realized we’ve not been announcing new releases since version 1.1.1 when Tomas first announced it, so I though I’d rectify that. In addition to bug fixes, here what’s has been added to FSharp.Data since then:

  • The behaviour of CsvProvider can be fine tuned by using the HasHeaders, Schema, IgnoreErrors, SafeMode, and PreferOptionals parameters. See the CsvProvider documentation here.
  • The CSV Parser and Reader can now be used standalone. See the CsvFile documentation here.
  • Guid types are now supported in CsvProvider, JsonProvider, and XmlProvider.
  • Support to disable caching for CSV files too big to fit in memory.
  • Support for cookies, binary files and client certificates in FSharp.Net.Http.
  • Save, Filter, Take, TakeWhile, Skip, SkipWhile, and Truncate operations for CsvProvider and CsvFile.
  • Support for Windows Phone 7.

There’s still a lot of room for improvement, and we love pull requests, so please contribute, even if it’s only fixing a typo in the docs. A getting started guide is available here.

Building Type Providers - Part 1

| Comments

Type Providers are the biggest new feature of F# 3.0. They allow us to leverage the power of the static type system in areas that have typically been the territory of dynamic languages and stringly typing. A picture is worth a thousand words, so here’s 12 pictures that show type providers in action:

All your types are belong to us

There are already several good samples and articles about using type providers, but there still isn’t a lot about creating type providers. Having worked recently on FSharp.Data I decided to share some tips about building type providers.

F#, Windows Phone 7 & Visual Studio 2012

| Comments

F# 2.0 supported Silverlight 4, Silverlight 5, and Windows Phone 7.1, as several versions of FSharp.Core.dll were shipped as part of the standalone F# redistribution. There were even some nice Visual Studio 2010 templates made by Daniel Mohl.

In F# 3.0 there is no standalone redistributable package anymore, and Visual Studio 2012 only ships with two versions of FSharp.Core.dll: the full .NET 4.5 version, and a portable class library version targeting Profile47 (which includes .NET 4.5, Silverlight 5.0, and .NET for Windows Store apps).

I tried to create versions for Windows Phone 7 and for Profile88 (which additionally includes .NET 4.0, Silverlight 4.0, and Windows Phone 7.1) from the source code, and I even got them to compile, but unfortunately I still get a bunch of Invalid Program exceptions at runtime when running on the emulator or on the device for both versions.

So I tried another approach - to reuse the FSharp.Core.dll version from F# 2.0 - and it seems to work just fine. Type providers and other new F# 3.0 features that depend on the library won’t work, but at least we can use Visual Studio 2012 and some of the features like the triple quotes.

F# as a Octave/Matlab Replacement for Machine Learning

| Comments

EDIT: As another version of the ml-class course has started, I've made the repository private

Back when I was in college, I took three different courses that dealt with subjects related to machine learning and data mining. Although I didn’t lose interest on those matters, my work has led me in a totally unrelated direction, so I haven’t exercised any of that knowledge in about eight years or so. A few weeks ago, I stumbled upon Stanford’s online class on Machine Learning and decided to enroll. I want to revive many of the things I have forgotten and try to put them into practice, as nowadays it’s very easy to access large amounts of interesting data from all kinds of online sources.

The programming exercises of this class are supposed to be done in Octave or Matlab, and while I understand the advantages of these tools, my past experience (where all the exercises and projects were done either with SAS or with Matlab) shows me that not using a general purpose programming languages doesn’t help a lot in turning academic exercises into real world programs. As professor Andrew Ng said in the introduction, one of the goals of the class is for us to put machine learning into practice in real world problems we care about, so I decided that I’ll implement all the algorithms and exercises in F#.

Introducing PreSharp

| Comments

Background

Back in 2004, I was doing some code-generation work as part of the OBIWAN project. When I started, CodeDom was being used to do the work, but I really didn’t like it because it made the generator code very hard to read and modify. Realistically, I would not need to support any other language than C#, so I started looking for alternatives. CodeSmith was very popular at the time for generating type-safe collections (.NET 2.0 generics didn’t exist yet), but it was targeted at one-shot generations, and not at creating code generation code. Then I found a very simple tool named CodeGen that appealed to me. I had been playing around with the Boost Preprocessor library recently, so I really liked the idea of using the preprocessor. I did a few tweaks to it and was able to use it for my needs at the time. Later on, around mid-2007, I needed to do code-generation again, so I took this tool and added a good amount of more power to it. At this time, it was very far apart from the original code, so I re-baptized it as PreSharp and published it to CodePlex. I never got around to do any documentation for it, so I’m making this post to try to compensate for that. I also moved the project recently from CodePlex to GitHub.