A while back, I discussed an implementation of the application operator (Haskell’s $) in OCaml. In the closing section of that post, a couple of problems were raised regarding treatment of associativity and composition in OCaml Batteries. These issues have been addressed in Batteries 2.0, released in January 2013; the improvements are outlined here.

Quick recap

Batteries 1.x defines the following operators for composition and application:

• val ( -| ) : ('a -> 'b) -> ('c -> 'a) -> 'c -> 'b
  • “Function composition. f -| g is fun x -> f (g x). Mathematically, this is operator o.”
• val ( **> ) : ('a -> 'b) -> 'a -> 'b
  • “Function application. f **> x is equivalent to f x. This [operator] may be useful for composing sequences of function calls without too many parenthesis.”

The problem pointed out in the comments section (now the closing Update) of the post was that

the precedence you’d expect coming from Haskell is inverted. We’d need to define a new application operator to address this problem as the commenter suggested…

Specifically, the following code sample was shown to exhibit surprising behaviour for anyone familiar with Haskell’s . and $,

# print_endline -| string_of_int **> succ **> sum [1; 2; 3];;
Error: This expression has type string but an expression was expected of type 'a -> string

What’s up, doc?

In Batteries 2.0, both operators have been renamed:

• Composition:
  • (-|) is now (%)
• Application:
  • (**>) is now (@@)

To appreciate the behaviour of the new operators, we can once again consult (a subsection of) the operator associativity table from the language manual:

Construction or operator	Associativity
*... /... %... mod land lor lxor	left
+... -...	left
::	right
@... ^...	right
=... <... >... |... &... $...	left

A couple of things are worth noting:

• The associativity of the application operator has changed.
  • (**>) is left associative, (@@) right.
  • As discussed in the original post, right associativity is an integral feature of the application operator.
• In Batteries 1.x, application had higher precedence than composition.
  • (**>) is covered by the first row of the table, (-|) by the second.
  • As of Batteries 2.0, this precedence has been inverted.

No surprises

Reworking the above code sample for Batteries 2.0 is trivial—simply substitute (%) for (-|) and (@@) for (**>). With these changes in place, the code behaves as follows:

# print_endline % string_of_int @@ succ @@ sum [1; 2; 3];;
7
- : unit = ()

Crucially, the application operator has lower precedence than the composition operator and is right associative.

Returning to the definition of $ referenced at the outset of the original post,

This operator is redundant, since ordinary application (f x) means the same as (f $ x). However, $ has low, right-associative binding precedence, so it sometimes allows parentheses to be omitted.

it should be apparent that these new operators closely conform to Haskell’s treatment of application and composition (in particular, associativity and precedence), allowing for a straightforward translation of the above expression:

Prelude> putStrLn . show $ succ $ sum [1, 2, 3]
7

I’ve migrated this blog from Posterous to Hakyll following the (long anticipated) announcement that Posterous will be shut down in April. Unfortunately, comments for old posts have been lost.

Hopefully folks like the new layout; mobile stylesheet coming soon.

In the previous post I introduced the $ operator to OCaml using two different approaches:

1. Renaming the operator to **$ or @$ in order to achieve the necessary associativity.
2. Leveraging Camlp4 to provide a syntax extension.

In the comments section¹, variants of these operators were provided that mirror Haskell’s relative precedence of application and composition.

As a postscript, I thought it might be interesting to look at the implementation of $ in Standard ML. Here it is in the SML/NJ toplevel,

Standard ML of New Jersey v110.74 [built: Wed Apr 11 13:33:07 2012]
- infixr 0 $;
infixr $
- fun (f $ x) = f x;
val $ = fn : ('a -> 'b) * 'a -> 'b

Of note:

• Standard ML lets us specify the associativity of newly defined operators explicitly (using the infix* fixity directives) whereas OCaml follows an operator naming convention.
• As such, we have no need to fall back on syntax extensions here; $ is a valid name for a right associative operator.

To replicate the target example of the previous post we’ll need to define a few utilities,

- fun succ x = x + 1;
val succ = fn : int -> int
- val sum = foldl op+ 0;
val sum = fn : int list -> int
- fun printLn str = print $ str ^ "\n";
val printLn = fn : string -> unit

Note that printLn is defined using $; the standard approach would be fun printLn str = print (str ^ "\n").

With these definitions in place, we can employ $ to print the desired result:

- printLn $ Int.toString $ succ $ sum [1, 2, 3];
7
val it = () : unit

Finally, since $ was defined with a precedence level of 0, it interacts correctly with SML’s composition operator, o, which has a precedence of 3 (as per the Standard):

- printLn o Int.toString $ succ $ sum [1, 2, 3];
7
val it = () : unit

Haskell provides a convenient function application operator, $, that can improve code readability and concision. This post discusses the problems that arise in trying to implement an identical operator in OCaml.

Why do we need an application operator?

In most situations, we don’t. Having said that, $ can be useful – its right-associativity sometimes obviates the need for parentheses which can result in cleaner code. As the Haskell documentation states:

This operator is redundant, since ordinary application (f x) means the same as (f $ x). However, $ has low, right-associative binding precedence, so it sometimes allows parentheses to be omitted.

It turns out that implementing $ in OCaml is an interesting undertaking in itself.

The shape of things to come

When we’re done, the following code should compile and print “7” upon execution:

let sum = List.fold_left (+) 0
let res xs = string_of_int $ succ $ sum xs
let () = print_endline $ res [1; 2; 3]

By definition of $, res [1; 2; 3] is equivalent to the expression string_of_int (succ (sum [1; 2; 3])).

Naive approach

As a first pass we could try implementing $ as follows:

# let ($) f x = f x;;
val ( $ ) : ('a -> 'b) -> 'a -> 'b = <fun>

Testing on a simple case suggests that this implementation is a suitable analogue of Haskell’s operator:

# let sum xs = List.fold_left (+) 0 xs;;
val sum : int list -> int = <fun>
# succ $ sum [1; 2; 3];;
- : int = 7

However, problems become apparent when we attempt to chain together more than two functions. As an example, let’s try out the body of res:

# string_of_int $ succ $ sum [1; 2; 3];;
Error: This expression has type int -> int
       but an expression was expected of type int

The reported error indicates that string_of_int was applied to a function from int -> int rather than an int as intended: our operator is left-associative. string_of_int is being passed the succ function rather than the result of applying succ to the sum of [1; 2; 3].

Perhaps this is unsurprising given that we didn’t specify associativity anywhere. Here’s the interesting bit: the associativity of our operator is a direct consequence of its name. To clarify, we’ll need to consult the operator associativity table in the language manual. The relevant row is repeated below:

• Construction or operator
  • =... <... >... |... &... $...
• Associativity: left

In other words: any operator identifier beginning with $ is left associative. To get the right-associativity, a right-associative prefix must be used:

• Construction or operator
  • @... ^...
  • **... lsl lsr asr
• Associativity: right

So to fix our earlier implementation we simply need to rename $; here are two possible implementations that behave correctly:

let ( **$ ) f x = f x;;
let (  @$ ) f x = f x;;

Whichever we choose, the toplevel expression now evaluates as anticipated:

# string_of_int **$ succ **$ sum [1; 2; 3];;
- : string = "7"
# string_of_int @$ succ @$ sum [1; 2; 3];;
- : string = "7"

As a final note: there’s nothing special about non-prefix $ when used as part of an operator name – it has been retained for the sake of consistency. If you’ve used OCaml Batteries Included, you might find the first operator familiar – in BatPervasives it’s named **>; the leading ** is necessary for the reasons outlined above.

Enter Camlp4

In the previous section we implemented $ but were forced – due to grammatical constraints – to settle for slightly cryptic operator names. Can we do any better? After all, the goal of this post is to compile the snippet provided in the opening section. So far it can only be approximated as follows:

let sum = List.fold_left (+) 0
let res xs = string_of_int @$ succ @$ sum xs
let () = print_endline @$ res [1; 2; 3]

We need a way of telling OCaml that $ is a right-associative operator in spite of its name. If that sentence makes you wonder whether this is even possible given the grammatical constraints outlined above, you’re on the right track – it’s not. Instead, we’ll need to write a syntax extension with the help of Camlp4 that allows us to (in a sense) “violate” the operator naming rules. More accurately: to extend the grammar such that $ is defined as a right-associative operator.

Camlp4whatNow?

To quote the documentation,

Camlp4 is a Pre-Processor-Pretty-Printer for OCaml. It offers syntactic tools (parsers, extensible grammars), the ability to extend the concrete syntax of OCaml (quotations, syntax extensions), and to redefine it from scratch.

Of relevance here is the ability to extend OCaml’s (concrete) syntax.

Writing the extension

While Camlp4 can be somewhat esoteric (the latest version isn’t officially documented), our requirements are simple enough that a simple 4 line module will do the trick.

open Camlp4.PreCast.Syntax
EXTEND Gram
    expr: BEFORE "apply"
        [ "applOp" RIGHTA [ f = expr; "$"; x = expr -> <:expr<$f$ $x$>> ]];
END

The above code extends the default OCaml grammar Gram by adding a new rule under the expr entry. This rule

• extends the expr (expression) entry by inserting the rule at the supplied precedence level – before application;
• is named applOp (an arbitrarily chosen name);
• is right-associative (RIGHTA);
• rewrites f $ x as f x for all expressions (exprs) f, x.

To make use of this extension it must first be compiled – here with the help of ocamlfind,

$ ocamlfind ocamlc -linkpkg -syntax camlp4o \
    -package camlp4.extend -package camlp4.quotations -c appop.ml

and passed to the -pp switch of ocamlc during batch compilation of dollar.ml,

$ ocamlc -pp "camlp4o ./appop.cmo" dollar.ml -o dollar
$ ./dollar
7

To see what’s going on under the hood, we can pre-process dollar.ml and output the (source) result to stdout,

$ camlp4o ./appop.cmo dollar.ml
let sum = List.fold_left ( + ) 0

let res xs = string_of_int (succ (sum xs))

let () = print_endline (res [ 1; 2; 3 ])

Finally, the extension can also be used interactively from the toplevel,

# #use "topfind";;
- : unit = ()
Findlib has been successfully loaded. Additional directives:
  #require "package";;      to load a package
  #list;;                   to list the available packages
  #camlp4o;;                to load camlp4 (standard syntax)
  #camlp4r;;                to load camlp4 (revised syntax)
  #predicates "p,q,...";;   to set these predicates
  Topfind.reset();;         to force that packages will be reloaded
  #thread;;                 to enable threads

- : unit = ()

# #camlp4o;;
/opt/godi/lib/ocaml/std-lib/dynlink.cma: loaded
/opt/godi/lib/ocaml/std-lib/camlp4: added to search path
/opt/godi/lib/ocaml/std-lib/camlp4/camlp4o.cma: loaded
    Camlp4 Parsing version 3.12.1

# #load "appop.cmo";;

# let sum = List.fold_left (+) 0;;
val sum : int list -> int = <fun>

# string_of_int $ succ $ sum [1; 2; 3];;
- : string = "7"

A health warning

This post is simply meant to introduce the relationship between associativity and operator naming–I don’t meant to suggest that the above syntax extension is the “best” solution. On the contrary, accomodating OCaml’s naming conventions is appealing for two reasons:

1. Simpler implementation;
2. Satisfies the grammatical expectations of its clients.

Having said that, it’s both useful and reassuring to know that should we need to “violate” these conventions, Camlp4 is there to help.

Update

A commenter on the original post (@ManInTheMiddle) pointed out that

Considering the non-Camlp4 approach, it seems to me your solution misses the point that the $ operator should also mix well with function composition (Haskell . operator). This operator is left-associative and should be of higher-precedence than function application. So if you choose the **... prefix for $ then you will not find any such operator for composition.Hence I suggest using only @... or ^... for application. Then you can choose *..., /... or %... for composition.

I agree – the above glosses over the relationship between precedence and operator naming. Interestingly, the Batteries project uses (-|) for composition and (**>) for application (see here); the precedence you’d expect coming from Haskell is inverted. We’d need to define a new application operator to address this problem as the commenter suggested:

let (^^) f x = f x

yielding,

# print_endline -| string_of_int **> succ **> sum [1; 2; 3];;
Error: This expression has type string but an expression was expected of type 'a -> string
# print_endline -| string_of_int ^^ succ ^^ sum [1; 2; 3];;
7

Along similar lines, an improvement to the syntax extension can also be made,

open Camlp4.PreCast.Syntax
EXTEND Gram
    expr: BEFORE "^"
        [ "applOp" RIGHTA [ f = expr; "$"; x = expr -> <:expr><$f$>> ]];
END

thereby licensing:

# print_endline -| string_of_int $ succ $ sum [1; 2; 3];;
7
- : unit = ()

In the previous post we looked at the Functor structure; now we turn to Applicative Functor. One important enrichment is that applicatives allow for the application of functions that are in some context t to values in the same context.

Once more, a couple of definitions to get things rolling,

• Given a function in context, val f: ('a -> 'b) t, Applicative Functor provides an operation, <*>, such that f can be applied to values in like context: val (<*>) : ('a -> 'b) t -> 'a t -> 'b t.
• Given a value 'a, the Applicative Functor function pure will lift 'a into the appropriate (“default” or pure) context: val pure: 'a -> 'a t.

Something that I glossed over last time around are the laws associated with the structures under discussion. A module can implement a signature validly while violating these laws. For Functor, the laws are as follows:

1. Mapping the identity function over a value in context t should have no effect – i.e., fmap id = id.
   • For example, OptionFunctor.fmap (fun x -> x) (Some 5) = Some 5 and EitherStringFunctor.fmap (fun x -> x) (Left "Oops") = Left "Oops".
2. Mapping the composition of functions g and h over a value in context t should be equivalent to mapping function g over this value, then mapping function h over the output (i.e., the composition of mapping g and mapping h).

The salient law for Applicative Functor is addressed below.

Applicative Structure

Let’s begin by providing the signature for an Applicative Functor, defined in terms of Functor:

module type Functor = sig
    type 'a t
    val fmap: ('a -> 'b) -> 'a t -> 'b t
end

module type ApplicativeFunctor = sig
    include Functor

    val pure: 'a -> 'a t
    val (<*>): ('a -> 'b) t -> 'a t -> 'b t
    val (<$>): ('a -> 'b)   -> 'a t -> 'b t
end

In addition to the signatures mentioned in the introduction, we also define a function (operator) named <$> with a signature identical to fmap. Strictly speaking, this should be declared in the Functor signature… since we’ll only be using it with applicatives, though, the above factoring will suffice.

First out of the blocks, option

As before, we’ll start out by implementing something simple – an Applicative Functor module for options. To save some space, only incremental additions will be supplied inline. Here’re our two implementations alongside a simple demo,

module OptionFunctor : Functor
with type 'a t = 'a option = struct
    type 'a t = 'a option
    let fmap f = function
        Some a  -> Some (f a)
        | _     -> None
end

module OptionApplicativeF : ApplicativeFunctor
with type 'a t = 'a option = struct
    include OptionFunctor

    let pure x = Some x

    let (<*>) f a = match (f, a) with
        (Some f, Some a)    -> Some (f a)
        | _                 -> None

    let (<$>) = fmap
end

module Demo1(A: ApplicativeFunctor) = struct
    include A
    let double x = x * 2
    let eg1 x    = double <$> x
    let eg2 x y  = pure (+) <*> x <*> y
    let eg3 x y  = (+) <$> x <*> y
end

module OptionDemo1 = Demo1(OptionApplicativeF)

The inheritance relationship indicated in the module signatures is mirrored in in the implementations – OptionApplicativeF is defined in terms of OptionFunctor: fmap is borrowed; pure, <*> and <$> defined explicitly.

Without further ado, the examples in action:

# OptionDemo1.eg1 (Some 5);;
- : int OptionDemo1.t = Some 10

# OptionDemo1.eg1 None;;
- : int OptionDemo1.t = None

# OptionDemo1.eg2 (Some 5) (Some 10);;
- : int OptionDemo1.t = Some 15

# OptionDemo1.eg2 (Some 5) None;;
- : int OptionDemo1.t = None

# OptionDemo1.eg2 None (Some 10);;
- : int OptionDemo1.t = None

# OptionDemo1.eg2 None None;;
- : int OptionDemo1.t = None

# OptionDemo1.eg3 (Some 5) (Some 10);;
- : int OptionDemo1.t = Some 15

# OptionDemo1.eg3 (Some 5) None;;
- : int OptionDemo1.t = None

# OptionDemo1.eg3 None (Some 10);;
- : int OptionDemo1.t = None

# OptionDemo1.eg3 None None;;
- : int OptionDemo1.t = None

Notice how Demo.eg2 and Demo.eg3 are functionally identical in spite of the variation in their definitions. Where eg2 raises + into context t with pure before applying <*>, eg3 simply fmaps via the convenience operator <$>. This is essentially syntactic sugar, allowing for a pipelined style. Here we’ve hit on the salient law for Applicative Functor:

fmap g x = pure g <*> x

or using <$>:

g <$> x = pure g <*> x

Before we leave OptionApplicativeF, consider the following case in which the function itself is absent (i.e., None) in a <*> chain:

# None <*> (Some 1) <*> (Some 2);;
- : '_a t = None

This follows from definition of <*>: unless both the function and its argument are present (Some _), the result of <*> is None.

Adding either into the mix

To make things more interesting, we’ll re-introduce either and provide an ApplicativeFunctor implementation:

type ('a, 'b) either = Left of 'a | Right of 'b

module type Typed = sig type t end

module EitherFunctor(T: Typed) : Functor
with type 'a t = (T.t, 'a) either = struct
    type 'a t = (T.t, 'a) either
    let fmap f = function
        Right r     -> Right (f r)
        | Left l    -> Left l
end

module EitherApplicativeF(T: Typed) : ApplicativeFunctor
with type 'a t = (T.t, 'a) either = struct
    include EitherFunctor(T)

    let pure x = Right x

    let (<*>) f a = match (f, a) with
        (Right f, Right a)  -> Right (f a)
        | (Left err, _)     -> Left err
        | (_, Left err)     -> Left err

    let (<$>) = fmap
end

(* Demo1 as before *)
module EitherDemo1 = Demo1(EitherApplicativeF(String))

As with OptionFunctorF, the only success condition accounted for by <*> is where both the function and its value are valid – in the option case, Some _ and in the either case, Right _. If the first clause of match can be unified, f is applied to a and wrapped with the Right constructor as an indication of success. In the other two cases, the error (typed T.t) is propagated.

The examples in EitherDemo1 should yield few surprises,

# EitherDemo1.eg1 (Right 5);;
- : int EitherDemo1.t = Right 10

# EitherDemo1.eg3 (Right 5) (Right 10);;
- : int EitherDemo1.t = Right 15

# EitherDemo1.eg3 (Right 5) (Left "Oops");;
- : int EitherDemo1.t = Left "Oops"

# EitherDemo1.eg3 (Left "Um") (Left "Oops");;
- : int EitherDemo1.t = Left "Um"

# EitherDemo1.eg3 (Left "Um") (Right 15);;
- : int EitherDemo1.t = Left "Um"

Recognising abstractions

The demonstration functions defined in Demo1 share a common semantic purpose: generalising a given function f so that it can apply in a context t. This is known as lifting.

By way of example: Consider the double function: this has the signature val double : int -> int; in Demo1.eg1 we wish to lift double into context t and double whatever value is embedded in this context – in the case of OptionDemo1 the context is option, in EitherDemo1, either.

Since we’ve recognised and named this process, all that remains is to implement it. To start with, here’s a module signature that declares three lift operations:

module type Lift = sig
    type 'a t
    val lift:  ('a -> 'b) -> 'a t -> 'b t
    val lift2: ('a -> 'b -> 'c) -> 'a t -> 'b t -> 'c t
    val lift3: ('a -> 'b -> 'c -> 'd) -> 'a t -> 'b t -> 'c t -> 'd t
end

where lift takes a function from 'a -> 'b and a single input in context t, mapping to 'b t. lift2 takes a “two-argument” function (glossing over currying) and performs the same generalisation over a context; similarly for lift3.

We can use an OCaml functor to provide a generic implementation of lifting by leveraging <$> and <*>:

module ApplicativeLift(A: ApplicativeFunctor) : Lift
with type 'a t := 'a A.t = struct
    include A
    let lift  f a       = f <$> a
    let lift2 f a b     = f <$> a <*> b
    let lift3 f a b c   = f <$> a <*> b <*> c
end

Finally, let’s compose the signatures of Lift and ApplicativeFunctor to produce a new signature, Applicative,

(* Composition of ApplicativeFunctor and Lift *)
module type Applicative = sig
    include Lift 
    include ApplicativeFunctor with type 'a t := 'a t
end

Note that this use of destructive substitution (with type 'a t := 'a t) requires OCaml 3.12 – see the documentation for more information.

With the above in place, we can now define a couple of enriched applicative modules (with less unseemly names),

module OptionApplicative : Applicative
with type 'a t = 'a option = struct
    include OptionApplicativeF
    include ApplicativeLift(OptionApplicativeF)
end

module EitherApplicative(T: Typed) : Applicative
with type 'a t = (T.t, 'a) either = struct
    module EA = EitherApplicativeF(T)
    include EA
    include ApplicativeLift(EA)
end

And expand on our earlier set of demos,

module Demo2(A: Applicative) = struct
    include A
    let double x  = x * 2
    let eg1 x     = lift double x
    let eg2 x y   = lift2 (+) x y
    let eg3 x     = double <$> x
    let eg4 x y   = (+) <$> x <*> y
    let eg5 f x y = lift2 f x y
end

module OptionDemo2       = Demo2(OptionApplicative)
module EitherStringDemo2 = Demo2(EitherApplicative(String))
module EitherIntDemo2    = Demo2(EitherApplicative(struct type t = int end))

This composed structure proves to be very useful in practice – as a sketch, here’s an example of lifting a function into either context:

# let foo x y = (x * y) - (x + y);;
val foo : int -> int -> int = <fun>

# EitherStringDemo2.eg5 foo (Left "Err") (Right 10);;
- : int EitherStringDemo2.t = Left "Err"

# EitherStringDemo2.eg5 foo (Right 15) (Right 10);;
- : int EitherStringDemo2.t = Right 125

Further reading

As before, I recommend reading the Haskell Typeclassopedia entry on Applicative Functors. This in turn links to some excellent resources on the subject and addresses the relationship between Applicatives and Monads.

A full code example can be found here.

This post explores the implementation of Functor structures (not to be confused with functor) in OCaml.

Let’s begin with some definitions,

• A Functor is (somewhat informally): a structure that provides a mapping operation that applies to a value in a given context, preserving that context.
• Put differently: given a function val f: 'a -> 'b, a Functor allows us to apply f to values in some context t such that the mapping operation is of type 'a t -> 'b t.
• This mapping operation is typically named fmap and has the type signature val fmap: ('a -> 'b) -> 'a t -> 'b t.

• The signature of fmap may be familiar: when type 'a t = 'a list, the fmap function is simply List.map,

  # List.map;;
  - : ('a -> 'b) -> 'a list -> 'b list = <fun>

Begin the Begin

The option type seems as good a place to start as any. Let’s state the problem clearly: given some function f that maps from values of type 'a -> 'b, how can we apply this f to optional values of type 'a option and mantain the optional context? Here’s a preliminary definition of OptionFunctor and some demo code where f is abs:

module OptionFunctor = struct
    let fmap f = function
        Some a  -> Some (f a)
        | _     -> None
end

module Demo1 = struct
    open OptionFunctor
    let eg1 = fmap abs (Some   5 )
    let eg2 = fmap abs (Some (-5))
    let eg3 = fmap abs None
end

In keeping with the definition of Functor, OptionFunctor provides an fmap function that takes two arguments (we’ll gloss over currying): some function f which maps from 'a -> 'b and an argument of type 'a option. When the second argument is present (in the form Some a), f is applied to a and wrapped with the Some constructor; otherwise (where the second argument is None), the result is the same.

Less verbosely, fmap has type: ('a -> 'b) -> 'a option -> 'b option. Here it is in practice,

# Demo1.eg1;;
- : int option = Some 5

# Demo1.eg2;;
- : int option = Some 5

# Demo1.eg3;;
- : int option = None

To make this example a little more interesting, let’s introduce another type, validation, and implement a Functor instance for it:

type 'a validation = Success of 'a | Failure of string

module OptionFunctor = struct
    let fmap f = function
        Some a  -> Some (f a)
        | _     -> None
end

module ValidationFunctor = struct
    let fmap f = function
        Success   a -> Success (f a)
        | Failure e -> Failure e
end

module Demo1 = struct
    open OptionFunctor
    let eg1 = fmap abs (Some   5 )
    let eg2 = fmap abs (Some (-5))
    let eg3 = fmap abs None
end

module Demo2 = struct
    open ValidationFunctor
    let eg4 = fmap abs (Success   5 )
    let eg5 = fmap abs (Success (-5))
    let eg6 = fmap abs (Failure "Something went wrong")
end

Our validation type has two constructors: Success and Failure – a slight enrichment of the builtin option type through the provision of reason for failure (= absence of value). ValidationFunctor strongly resembles OptionFunctor, the only difference being that a failure string is propagated in the second prong of the function match. The new Demo2 module demonstrates usage; calls to fmap look identical, though validation constructors are employed in Demo2:

# Demo2.eg4;;
- : int validation = Success 5

# Demo2.eg5;;
- : int validation = Success 5

# Demo2.eg6;;
- : int validation = Failure "Something went wrong"

So far, so good – but can we introduce a generic function and use it with both option and validation types? Unsurprisingly, the answer is yes: To do so, let’s introduce a Functor signature and augment our two Functor modules with type specifications.

type 'a validation = Success of 'a | Failure of string

module type Functor = sig
    type 'a t
    val fmap : ('a -> 'b) -> 'a t -> 'b t
end

module OptionFunctor = struct
    type 'a t = 'a option
    let fmap f = function
        Some a  -> Some (f a)
        | _     -> None
end

module ValidationFunctor = struct
    type 'a t = 'a validation
    let fmap f = function
        Success   a -> Success (f a)
        | Failure e -> Failure e
end

module Demo3(F: Functor) = struct
    let eg7 x   = F.fmap abs x
    let eg8 f x = F.fmap   f x
end

module OptionDemo3      = Demo3(OptionFunctor)
module ValidationDemo3  = Demo3(ValidationFunctor)

Demo3 is a functor in the OCaml sense – a module that is parameterised by another module. Since the parameter F is required to implement the Functor signature (here we’re using structural typing to unify with OptionDemo and ValidationDemo), fmap can be used in the module body. The abs examples simply take a value in context,

# OptionDemo3.eg7 (Some 10);;
- : int OptionFunctor.t = Some 10

# OptionDemo3.eg7 None;;
- : int OptionFunctor.t = None

# ValidationDemo3.eg7 (Success 12);;
- : int ValidationFunctor.t = Success 12

# ValidationDemo3.eg7 (Failure "Something went wrong");;
- : int ValidationFunctor.t = Failure "Something went wrong"

while eg8 allows us to provide a function,

# let square x = x * x;;
val square : int -> int = <fun>

# OptionDemo3.eg8 square (Some 4);;
- : int OptionFunctor.t = Some 16

# OptionDemo3.eg8 square None;;
- : int OptionFunctor.t = None

# ValidationDemo3.eg8 square (Success 5);;
- : int ValidationFunctor.t = Success 25

# ValidationDemo3.eg8 square (Failure "Oops");;
- : int ValidationFunctor.t = Failure "Oops"

Finally, here’s how we can define a ListFunctor at the toplevel to make use of Demo3:

# module ListFunctor = struct type 'a t = 'a list let fmap = List.map end;;
module ListFunctor :
  sig type 'a t = 'a list val fmap : ('a -> 'b) -> 'a list -> 'b list end

# module ListDemo3 = Demo3(ListFunctor);;
module ListDemo3 : sig val eg7 : int ListFunctor.t -> int ListFunctor.t end

# ListDemo3.eg7 [1; 2; 3];;
- : int ListFunctor.t = [1; 2; 3]

# ListDemo3.eg7 [1; 2; -3];;
- : int ListFunctor.t = [1; 2; 3]

# ListDemo3.eg7 [];;
- : int ListFunctor.t = []

# ListDemo3.eg8 square [1; 2; 3];;
- : int ListFunctor.t = [1; 4; 9]

# ListDemo3.eg8 square [];;
- : int ListFunctor.t = []

Providing signatures for Functors

Above, we used structural typing to match OptionFunctor and ValidationFunctor against the Functor signature. How about defining these modules with the Functor signature at the outset?

First of all, consider what happens when we attempt to create a new module, OF, by signing OptionFunctor with Functor:

 # module OF = (OptionFunctor : Functor);;
 module OF : Functor

 # OF.fmap abs (Some 5);;
 Error: This expression has type 'a option
        but an expression was expected of type int OF.t

In specifying a signature, the type representation in OptionFunctor has been abstracted over. The same problem is evidenced in the following code,

 # module type A = sig type t val id: t -> t end;;
 module type A = sig type t val id : t -> t end

 # module B = struct type t = int let id x = x end;;
 module B : sig type t = int val id : 'a -> 'a end

 # B.id;;
 - : 'a -> 'a = <fun>

 # B.id 10;;
 - : int = 10

 # module C = (B: A);;
 module C : A

 # C.id;;
 - : C.t -> C.t = <fun>

 # C.id 10;;
 Error: This expression has type int but an expression was expected of type
          C.t

By defining module C as having signature A, type C.t has been rendered abstract. What we meant to express is that C is a kind of A… and that type t = int. To achieve this, a sharing constraint is necessary:

 # module D = (B: A with type t = int);;
 module D : sig type t = int val id : t -> t end

 # D.id 10;;
 - : D.t = 10

Returning to OptionFunctor and its cousin OF, let’s look more closely at each module’s signature (by using the assign-to-peek trick in the toplevel):

 # module T = OptionFunctor;;
 module T :
   sig
     type 'a t = 'a option
     val fmap : ('a -> 'b) -> 'a option -> 'b option
   end

 # module T = OF;;
 module T :
   sig
     type 'a t = 'a OF.t
     val fmap : ('a -> 'b) -> 'a t -> 'b t
   end

As anticipated above, OF.t is abstract; to “fix” fmap we’ll need to add a sharing constraint as follows,

 # module OF2 = (OptionFunctor : Functor with type 'a t = 'a option);;
 module OF2 :
   sig
     type 'a t = 'a option
     val fmap : ('a -> 'b) -> 'a t -> 'b t
   end

Notice that t has a concrete type in OF2 while it remains abstract in OF (as seen through T). The provision of a sharing constraint solves the type incompatibility problem,

 # OF.fmap abs (Some 5);;
 Error: This expression has type 'a option
        but an expression was expected of type int OF.t

 # OF2.fmap abs (Some 5);;
 - : int OF2.t = Some 5

So: is it worth explicitly specifying the signature or not in light of the potential confusion caused by sharing constraints? On the one hand, explicit specification buys encapsulation and compile-time safety – the compiler will alert us to missing definitions. On the other, constraint provision is required to prevent nasty surprises from arising – perhaps we could make do with structural typing.

Since ensuring that implementations satisfy the conditions of formal categories (in this case, Functor) is desirable, explicit signatures will be employed for the remainder of the post. This is by no means required for what follows though.

Back to the code

As a final step, let’s introduce an either type – more general than validation in that its failure type is parametric – and Functor signatures for all of our implementations:

type  'a      validation    = Success of 'a | Failure of string
type ('a, 'b) either        = Left    of 'a | Right   of 'b

module type Functor = sig
    type 'a t
    val fmap : ('a -> 'b) -> 'a t -> 'b t
end

module OptionFunctor : Functor
with type 'a t = 'a option = struct
    type 'a t = 'a option
    let fmap f = function
        Some a  -> Some (f a)
        | _     -> None
end

module ValidationFunctor : Functor
with type 'a t = 'a validation = struct
    type 'a t = 'a validation
    let fmap f = function
        Success   a -> Success (f a)
        | Failure e -> Failure e
end

module ListFunctor : Functor
with type 'a t = 'a list = struct
    type 'a t = 'a list
    let fmap = List.map
end

(* Signature of any module providing its type in `t' *)
module type Typed = sig type t end

module EitherFunctor(T: Typed) : Functor
with type 'a t = (T.t, 'a) either = struct
    type 'a t = (T.t, 'a) either
    let fmap f = function
        Right r     -> Right (f r)
        | Left l    -> Left l
end

module Demo3(F: Functor) = struct
    let eg7 x   = F.fmap abs x
    let eg8 f x = F.fmap   f x
end

module OptionDemo3      = Demo3(OptionFunctor)
module ValidationDemo3  = Demo3(ValidationFunctor)
module ListDemo3        = Demo3(ListFunctor)
module EitherDemo3      = Demo3(EitherFunctor(String))

Here EitherFunctor is an (OCaml) functor (i.e., a module parameterised with another module). The module argument must satisfy the signature Typed by providing a type member t; this is used for the Left (failure) case.

# EitherDemo3.eg7 (Right 12);;
- : int EitherFunctor(String).t = Right 12

# EitherDemo3.eg7 (Left "Something went wrong");;
- : int EitherFunctor(String).t = Left "Something went wrong"

# EitherDemo3.eg8 square (Right 12);;
- : int EitherFunctor(String).t = Right 144

# EitherDemo3.eg8 square (Left "Oh dear");;
- : int EitherFunctor(String).t = Left "Oh dear"

Why do we need to use a functor here? The answer lies with the arity of Functor.t. Recall that either is defined as,

type ('a, 'b) either = Left of 'a | Right of 'b

while Functor expects a type matching

type 'a t

Without the functor, we’d receive a signature mismatch error.

Applicative Functors

Next time we’ll look at Applicative Functor, built on top of the Functor modules described in this post.

For more information on Functor, the Haskell Typeclassopedia is an excellent reference.

This is a simple extension to improve the output of the ‘hg paths’ command. I originally submitted a patch to the mercurial-devel list to achieve the same thing—given that my change alters the format of a builtin command, I don’t expect it to be accepted.

And so, the prettypaths extension: hg-prettypaths.

Configuration is minimal:

First, add the following to your ~/.hgrc:

[extensions]
# ...
prettypaths = /path/to/prettypaths.py

Then (optionally) set --pretty as default:

[defaults]
# ...
paths = --pretty

I’ve recently switched (back) from Git to Mercurial. This post isn’t intended to be a recapitulation of pros and cons; both are great tools. Instead I’d like to mention a problem that came up in my workflow and a small extension written to address it.

Here’s the problem: I use Mercurial for both personal and work projects—and different usernames for each. This situation may well be familiar to some:

1. You save your work (personal) username in the ui section of $HOME/.hgrc
2. For personal (work) repositories you edit the .hg/hgrc and specify a local username
3. You often forget to do (2) and end up committing changesets with the wrong username

The persona extension is designed to make this process slightly less of a headache: It provides an easy way to set ui.username on a repository by repository basis.

At its simplest, the extension lets you specify a username for a repository as follows:

# hg persona -n "Firstname Lastname <firstname@some.domain>"

To save repetition, you can configure individual personas in your ~/.hgrc:

[persona]
home        = Firstname Lastname <firstname@home.domain>
work        = Firstname Lastname <lastname@work.domain>

which allows for quick switching in a given repository:

# hg persona -n home
~ Setting username to 'Firstname Lastname <firstname@home.domain>'

# hg persona -n work
~ Setting username to 'Firstname Lastname <lastname@work.domain>'

Finally, you can specify the --persona option to clone:

# hg clone --persona work src dest

To use this extension, clone http://hg.0branch.com/hg-persona and add the following to your ~/.hgrc:

[extensions]
persona = /path/to/persona.py

Feedback appreciated!

Update: The following hook is also handy for displaying ui.username info on each commit (allowing you to catch repository-username mismatches early):

[hooks]
# ...
pre-commit = echo "(as '`hg showconfig ui.username`')"

Overview

This is the first in a series of posts documenting my experiments in Moose, an OOP framework for Perl 5. Throughout this series, I will be exploring its feature set by building a simple system for writing Interactive Fiction.

The particular focus of this article is Moose’s type system and pitfalls that might crop up on first use.

Prerequisites

If you want to run the examples that follow, you’ll need to install Moose. The easiest way to do so is through CPAN:

# cpan Moose

or

# cpanm Moose

Getting Started

To get started, we’ll need to define classes representing both generic objects in the game world and the player character. For now we won’t worry about the world as a whole (nor its I/O) and will ignore special object behaviour (for example, containment and supporter relationships). Also note that the design decisions taken here are subject to change in later posts — we might, for example, wish to make the Player a special kind of Person.

At its most fundamental, a game object should provide a description to return upon examination and a name to be used by the parser for referring to the object in question. Here’s a first pass,

package Object;
use Moose;

has 'name' => (
  isa       => 'Str',
  is        => 'rw',
  required  => 1,
);

has 'description' => (
  isa       => 'Str',
  is        => 'rw',
  default   => sub {
    "You see nothing special about the " . shift->name;
  },
);

no Moose;
1;

This class introduces us to Moose in general. As with standard Perl OOP, a class is simply a package. Using Moose’s syntactic sugar, we define two attributes for holding an object’s internal name (which will eventually manifest itself as a noun referring to the instance) and a description. Attributes defined with has are similar to C++ member variables or Java fields — though closer to C# properties as Moose provides us with implicit accessor methods.

Both name and description are marked as rw meaning that public accessors are available for setting and getting these attributes. Both are also defined as isa => 'Str' — this is the first sign of the type checking provided out of the box by Moose. Any attempt to set name or description to values that aren’t understood to be strings will result in an exception. Consider the following script,

#!/usr/bin/env perl

use strict;
use warnings;

use Object;

my $foo = Object->new(name => "lamp");
my $bar = Object->new(name => {});

Notice that we attempt to construct $bar by supplying an anonymous hash to name. This is what happens at runtime,

Attribute (name) does not pass the type constraint because: Validation failed for 'Str' with value HASH(0x803eb0) at /Users/marc/perl5/perlbrew/perls/perl-5.12.3/lib/site_perl/5.12.3/darwin-2level/Moose/Meta/Attribute.pm line 883
  Moose::Meta::Attribute::_coerce_and_verify('Moose::Meta::Attribute=HASH(0x9b06d0)', 'HASH(0x803eb0)', 'Object=HASH(0x9199d0)') called at /Users/marc/perl5/perlbrew/perls/perl-5.12.3/lib/site_perl/5.12.3/darwin-2level/Moose/Meta/Attribute.pm line 483
  Moose::Meta::Attribute::initialize_instance_slot('Moose::Meta::Attribute=HASH(0x9b06d0)', 'Moose::Meta::Instance=HASH(0x9b2120)', 'Object=HASH(0x9199d0)', 'HASH(0x803e80)') called at /Users/marc/perl5/perlbrew/perls/perl-5.12.3/lib/site_perl/5.12.3/darwin-2level/Class/MOP/Class.pm line 603
  Class::MOP::Class::_construct_instance('Moose::Meta::Class=HASH(0x97c430)', 'HASH(0x803e80)') called at /Users/marc/perl5/perlbrew/perls/perl-5.12.3/lib/site_perl/5.12.3/darwin-2level/Class/MOP/Class.pm line 576
  Class::MOP::Class::new_object('Moose::Meta::Class=HASH(0x97c430)', 'HASH(0x803e80)') called at /Users/marc/perl5/perlbrew/perls/perl-5.12.3/lib/site_perl/5.12.3/darwin-2level/Moose/Meta/Class.pm line 256
  Moose::Meta::Class::new_object('Moose::Meta::Class=HASH(0x97c430)', 'HASH(0x803e80)') called at /Users/marc/perl5/perlbrew/perls/perl-5.12.3/lib/site_perl/5.12.3/darwin-2level/Moose/Object.pm line 26
  Moose::Object::new('Object', 'name', 'HASH(0x803eb0)') called at ./mismatch.pl line 9

Now for the player character. A trivial implementation that meets the requirements outlined above would be:

package Player;
use Moose;

has 'inventory' => (
  default => sub { [] },
  is      => 'ro',
);

sub list_inventory {
  my $self    = shift;
  my @inv     = @{ $self->inventory };

  return "You are carrying nothing." unless @inv;

  my $output  = "You are currently carrying:\n";
  $output    .= ("* " . $_->name . "\n") foreach(@inv);
  return $output;
}

sub obtain {
  my ($self, $object) = @_;
  push @{ $self->inventory }, $object;
}

no Moose;
1;

Player has a single attribute, inventory, an array reference used to store a list of objects being carried. The list_inventory method returns a string listing the current inventory in a structured way. An obtain method is used to add objects to the player’s inventory by pushing them onto the end of the array referenced by inventory.

Let’s try this out;

#!/usr/bin/env perl

use strict;
use warnings;
use feature 'say';

use Object;
use Player;

my $lamp    = Object->new(name => "lamp");
my $player  = Player->new;

say $player->list_inventory;
$player->obtain($lamp);
say $player->list_inventory;

# but:
$player->obtain($player);

Running entry.pl yields the following:

# ./entry.pl
You are carrying nothing.
You are currently carrying:
* lamp

The code works as expected, but notice that the final line of our script — $player->obtain($player) — is perfectly legal, meaning that a player can pick up instances of Player as well as Object. Is this desirable?

Programmers versed in dynamically typed object systems might well answer yes; we don’t need typing on the inventory to prevent such errors. Rather, we need good API documentation and well written client code. Type mismatches will manifest when, for example, we try to list the inventory after a Player instance has been added:

Can't locate object method "name" via package "Player" at Player.pm line 16.

This run time exception indicates that we’re attempting to call a method on an object of inappropriate type. Good testing practices could iron out such errors, case closed.

If you find this reasoning satisfactory, the rest of this post probably won’t be of interest. My intention is not to debate the relative merits or shortcomings of approaches to typing in general. Rather, I take the position that catching as many errors as early as possible is A Good Thing. While compile-time type checking with Moose isn’t possible, we can prevent instances of the wrong type from being added to our inventory; as such, we don’t have to wait for a method call on an object to tell us that we’ve screwed up somewhere along the way.

Moose and Types

Conceptually, we want to encode the following in Player:

• A Player has an inventory;
• This inventory only contains instances of type Object (or its subtypes)

Notice that I talk of types and subtypes here rather than classes and subclasses. We have opened the door to a headache or two as will become clear below.

Type Parameters

Moose offers a type system for attributes. As noted in the documentation, though,

this is not a “real” type system. Moose does not magically make Perl start associating types with variables. This is just an advanced parameter checking system which allows you to associate a name with a constraint.

With that in mind, let’s add type checking to our inventory:

package Player;
use Moose;

has 'inventory' => (
  handles => { obtain => 'push' },
  isa     => 'ArrayRef[Object]',
  default => sub { [] },
  traits  => ['Array'],
  is      => 'ro',
);

sub list_inventory {
  my $self    = shift;
  my @inv     = @{ $self->inventory };

  return "You are carrying nothing." unless @inv;

  my $output  = "You are currently carrying:\n";
  $output    .= ("* " . $_->name . "\n") foreach(@inv);
  return $output;
}

no Moose;
1;

Notice that we have:

1. Removed the obtain method in favour of a handle key that delegates to push.
2. Changed the type of inventory to ArrayRef[Object]
3. Specified the Array trait.

The array trait,

provides native delegation methods for array references

In particular, it allows us to define obtain with respect to a parameterised type. For more information on the trait, see here.

First Problem

Running our entry script again, we see the same output:

# ./entry.pl
You are carrying nothing.
You are currently carrying:
* lamp

Why hasn’t the type parameter taken effect? The answer is that our class Object clashes with a built in type.

Which Object?

Browsing the Moose documentation, one might be inclined to think that we’re defining a class that already exists — namely, Moose::Object. Consider the following,

[Moose::Object] is the default base class for all Moose-using classes. When you use Moose in this class, your class will inherit from this class.

This description will be familiar to Java and Smalltalk programmers as it suggests a Unified Type System (where the class/type hierarchy has a single root node, Object).

This is not the case here though. Let’s review the isa key used with has:

The isa option uses Moose’s type constraint facilities to set up runtime type checking for this attribute. Moose will perform the checks during class construction, and within any accessors. The $type_name argument must be a string. The string may be either a class name or a type defined using Moose’s type definition features.

We haven’t used any of Moose’s custom type definition features, so let’s take a look at the builtin type hierarchy:

Any
Item
    Bool
    Maybe[`a]
    Undef
    Defined
        Value
            Str
                Num
                    Int
                ClassName
                RoleName
        Ref
            ScalarRef[`a]
            ArrayRef[`a]
            HashRef[`a]
            CodeRef
            RegexpRef
            GlobRef
                FileHandle
            Object

Notice that Object is a built-in type constraint — this constraint shares a name with our Object class and has taken precedence. We are warned of such behaviour in the documentation.

To understand why our Object and Player instances meet the constraint, consider its definition:

subtype 'Object' => as 'Ref' =>
  where { blessed($_) && blessed($_) ne 'Regexp' } =>
  optimize_as \&Moose::Util::TypeConstraints::OptimizedConstraints::Object;

in Moose::Util::TypeConstraints (see here). Any blessed reference will meet this constraint and can therefore be stored in inventory.

Solution

We need to disambiguate our Object — either by placing it in a different package or renaming the class so that it doesn’t clash with a built in type constraint name. For simplicitly, let’s pick a different name, Entity:

package Entity;
use Moose;

has 'name' => (
  isa       => 'Str',
  is        => 'rw',
  required  => 1,
);

has 'description' => (
  isa       => 'Str',
  is        => 'rw',
  default   => sub {
    "You see nothing special about the " . shift->name;
  },
);

no Moose;
1;

meaning that Player needs to be changed to read,

package Player;
use Moose;

has 'inventory' => (
  handles => { obtain => 'push' },
  isa     => 'ArrayRef[Entity]',
  default => sub { [] },
  traits  => ['Array'],
  is      => 'ro',
);

sub list_inventory {
  my $self    = shift;
  my @inv     = @{ $self->inventory };

  return "You are carrying nothing." unless @inv;

  my $output  = "You are currently carrying:\n";
  $output    .= ("* " . $_->name . "\n") foreach(@inv);
  return $output;
}

no Moose;
1;

We also need to update entity.pl:

#!/usr/bin/env perl

use strict;
use warnings;
use feature 'say';

use Entity;
use Player;

my $lamp    = Entity->new(name => "lamp");
my $player  = Player->new;

say $player->list_inventory;
$player->obtain($lamp);
say $player->list_inventory;

# but:
$player->obtain($player);

Running it now gives the following output:

# ./entity.pl
You are carrying nothing.
You are currently carrying:
* lamp
A new member value for 'inventory' does not pass its type constraint because:
Validation failed for 'Entity' with value Player=HASH(0x803f20) (not isa Entity)
  at ./entity.pl line 20

Attempting to add $player to the inventory has thrown an exception as desired.

Finally, we can modify entry.pl to fix the error:

#!/usr/bin/env perl

use strict;
use warnings;
use feature 'say';

use Entity;
use Player;

my $lamp    = Entity->new(name => "lamp");
my $player  = Player->new;

say $player->list_inventory;
$player->obtain($lamp);
say $player->list_inventory;

# this will throw an exception:
# $player->obtain($player);

Summary of pitfalls

Consider what we have learnt so far,

• The existence of a type in Moose is either the result of an explicit constraint or an existing class.
• In the latter case, an implicit type will be inferred on first use.
• While a derived class can be considered a subtype of its base class, we don’t have a homogeneous type system;
• Arbitrary constraints can be written that disregard inheritance relationships entirely.
• As such, there is no clear relationship between subtype and subclass.

Conclusion

First of all, it would be unfair not to repeat the important caveat presented in the Moose documentation:

This is NOT a type system for Perl 5. These are type constraints, and they are not used by Moose unless you tell it to. No type inference is performed, expressions are not typed, etc. etc. etc.

A type constraint is at heart a small “check if a value is valid” function. A constraint can be associated with an attribute. This simplifies parameter validation, and makes your code clearer to read, because you can refer to constraints by name.

Reasoning about types in a Moose system seems quite problematic given the overlap between two rather distinct means of predicating on instances.

Rather than thinking in terms of type systems, it may be more helpful to think of Moose’s “type constraints” as simple predicates that have nothing to do with OOP typing. Rather, Moose provides:

1. A hierarchy of built in predicates that can be used to place constraints on data and,
2. A means for automatically generating transitive predicates on the basis of the class hierarchy.