Can you spot a bug in the following CPP macro? Specifically, when is its behaviour ill-defined?

#define FROBNICATE(__frobptr) do {              \
    struct frobnicable *fptr = (__frobptr);     \
    assert(fptr && "Can't frobnicate NULL!");   \
    frobnicate_init(fptr);                      \
    frobnicate_finish(fptr);                    \
  } while(0)

One interesting thing I discovered about C while debugging this macro is that variable initializations like the following are valid:

{
  int foo = foo;
  /* foo now has an unspecified value. */
}

which means the following piece of code won't do what is expected of it:

{
  struct frobnicable *fptr = malloc(sizeof(struct frobnicable));
  fptr->x = 42;
  FROBNICATE(fptr);
}

The macro will expand to

do {
  struct frobnicable *fptr = (fptr);
  /* ... and so on ...  `fptr` is undefined in the rest of the 
   * macro expansion. */
} while(0);

and the implications won't be pleasant.

I haven't yet discovered an elegant and effective way to deal with this issue. The struct frobnicable *fptr = (__frobptr) is needed to prevent multiple evaluations of __fptr, so the only way I see of preventing this bug is by using an extremely unusual name for the fptr variable (or following a convention of prefixing such variable with some previously defined string). But that is neither elegant nor foolproof.

--

[1] http://spin.atomicobject.com/2012/03/30/preprocessing-with-caution/

We will need to convert ints to types and vice versa (we'll soon see why), so we might as well get that out of the way. One way to convert an int to a type is by exploiting the fact that templates are allowed to take integral values as parameters:

 
template<int N>
struct Integer {
}
 
typedef Integer<0> Zero;
typedef Integer<1> One;
 
// ... and so on

The issue with this is the lack of control over the types we're producing. In this particular case, we want to prevent (at compile time) the creation of types corresponding to negative integers. We do this by following Peano's definition of natural numbers:

template<typename T>
struct S { typedef T Previous;
  enum { ToInt = T::ToInt + 1 };
};
 
struct Z { enum { ToInt = 0 }; };
 
template<int N>
struct ToType {
  typedef S<typename ToType<N - 1>::TypeValue > TypeValue;
};
 
template<>
struct ToType<0> {
  typedef Z TypeValue;
};
 
#define P(n) typename n::Previous
#define TO_INT(n) n::ToInt

In this case, S<S<S<Z > > > represents 3 and it is obvious that using S and Z, it is possible to enumerate only positive integers.

Secondly, we want some mechanism which to use to "lift" normal integers into a corresponding type. This is done by ToType. The recursion is simple, for 0, ToType "returns" Z and for N, it "returns" S< the "return value" of ToType<N - 1> >. P and TO_INT are macros to save typing.

Note that we've not really solved the problem of creation of negative integers -- ToType<-50>::TypeValue will throw a "template depth exceeds maximum of XYZ" error and not a proper compile time error. All this mechanism does is isolate the issue of negative numbers to within ToType so that the rest of the code can work with a cleaner representation. I think this can be improved:

• Create a dummy struct ZeroFlag with one int template paramenter.
• Specialize ZeroFlag on 0 and add a static field DummyF to it.
• Create a struct Abs that computes the absolute value of its int template parameter.
• Add typedef ZeroFlag<Abs<N>::Value >::DummyF to ToType.

This should give a compile time error on instantiating ToType<0>. However, right now, we concentrate elsewhere. Importantly, check the Previous typedef in struct S, which isn't present when we specialize in struct Z -- Previous will be our main weapon in generating compile time errors, as we shall see.

We now create a "linked list" which keeps track of the number of nodes in it at compile time:

// N denotes the size of the list here
template<typename T, typename N>
struct List {
  T data;
  List<T, P(N)> next;
};
 
template<typename T>
struct List<T, Z> {
};

We don't need to have a pointer to a node since the next field always points to a smaller type -- there is no circularity (and hence no possibility of a loop in the linked list). I'm not exactly sure of this, but I think, at least in terms of memory layout, List<100> should be equivalent to an array of 100 ints.

We can actually write up a foldl for this case. First we declare the type of a generic function (we have to use structs everywhere because C++ does not allow partial template specialization on functions):

template<typename A, typename B, typename C>
struct Func2 {
  typedef C functionType(A, B);
};

Now, to fold a type T, we need a function of type A->T->A and an initial value of type A:

template<typename T, typename N>
struct List {
  T data;
  List<T, P(N)> next;
 
  template<typename A>
  A foldl(typename Func2<A, T, A>::functionType f, A a) {
    return next.foldl(f, f(a, data));
  }
};
 
template<typename T>
struct List<T, Z> {
  template<typename A>
  A foldl(typename Func2<A, T, A>::functionType f, A a) {
    return a;
  }
};

I was surprised how close C++ can get to Haskell at this point.

Right now we have a dependently typed (only in a loose sense of the term -- I've not had enough rigour with type systems to formally claim this) linked list in C++. We can now write a "safe" sorting function using this:

(I've used a shitty algorithm here to focus on the things this post is actually supposed to focus on.)

template<typename T, typename N>
struct Sorter {
  static void sort(List<T, N> &list) {
    List<T, P(N)> rest = list.next;
    Sorter<T, P(N)>::sort(rest);
    if (list.data > rest.data) {
      std::swap(rest.data, list.data);
      Sorter<T, P(N)>::sort(rest);
    }
    list.next = rest;
  }
};
 
// List of length 1 is already sorted.
template<typename T>
struct Sorter<T, S<Z> > {
  static void sort(List<T, S<Z> > &list) { };
};

The cute part about this code is, if the specialization Sorter<T, S<Z> > is removed you get a compile time error, since the compiler tries to sort a list of on element using the general template which then tries to sort a list of zero elements. Since the general template tries to access the Previous field (via the P macro), you get a compile time error. That the code compiles is "proof" that your algorithm will not try to walk the past of an empty linked list.

It is easy to create a struct ListCreator which creates a linked list of a specified length and printing is a matter of folding with the function

template<typename T>
std::vector<T> accumulate(std::vector<T> in, T t) {
  in.push_back(t);
  return in;
}

and an empty std::vector and print the resulting vector. The exact code is in [1].

The entire list, the sorting function could all be implemented inside the template system, by the way. However, in that case the elements of the list would all be baked in; in the present situation, we have the option of reading (a compile-time-constant obviously) number of entries from a file or from the user. I think more interesting applications will be seen in "lifting" a partial amount of data and algorithms into the type system, providing a fuzzier boundary between what programmers have to keep in their heads and what the type system proves for them.

For dessert, let's see how you can implement compile-time bound checking on the list. First a template to assert the less-than relation:

template<typename A, typename B>
struct Lt {
  typedef typename Lt<P(A), P(B)>::Flag Flag;
};
 
template<typename B>
struct Lt<Z, S<B> > {
  typedef bool Flag;
};

Essentially A < B iff Lt<A, B>::Flag exists. The non-existence of Flag can be made a compile-time error by typedefing it in the template which requires the relation to be valid. So, in this case, Z is less than all integers which can be written as S<A> for some A and A is less than B iff A - 1 is less than B - 1. Notice that these two axioms are both necessary and sufficient for comparing any two positive integers. Also note how we are inductively asserting in the general case -- since we typedef Lt<P(A), P(B)>::Flag to Flag, Lt<A, B> will have Flag typedefed iff Lt<P(A), P(B)> has it typedefed. Stating a correct indexing operation is now trivial:

template<typename T, typename N>
struct Idx<T, N, Z> {
  typedef typename Lt<Z, N>::Flag Flag;
  static T idx(List<T, N> list) {
    return list.data;
  }
};

--

[1] https://github.com/sanjoy/Snippets/blob/master/DependentTypes.cc

I think the article should, in principle, offend any self-respecting individual due to its tone and any rational person due to the "facts" it presents. From this point on, I tacitly assume that people reading this anti-(anti-drug-rant) rant posses both rationality and self-respect to some extent.

One interesting thing about the article is that after replacing "drug use" with "gay marriage" the reasons still remain "valid", and become strikingly similar to what anti-gay lobbyists routinely come up with. The illegality of two men loving each other, the separation from normal society, "underground" gay groups and "the gateway effect". One of the reasons routinely cited in support of anti-gay laws is that if we allow two women to marry, men will start sleeping with dogs and sisters will engage in incest. Needless to say, such fears are as baseless as the gateway drug theory [2] the authors talk about.

Perhaps the most objectionable aspect of this article the way the authors instruct their reader on the "right" way to lead their lives. Whether someone spends their free time reading Shakespeare or worshipping bananas is entirely for themselves to decide. Advice can only be given conditionally, "If you want to be an astronaut eating zombie then do the following ... ", or in this particular case "If you're giving advice to self-respecting people ...". Unconditional advice speaks poorly of how the author view people around them even more poorly of TSA for publishing such an article.

Getting down to the factual aspects:

It is true that most nations ban drugs, but more and more countries are taking steps towards legalization. London [4] is re-thinking their drug laws and Portugal decriminalized drug use [5] over a decade ago. In our own country we have licensed Bhang Shops [3]. Cocaine and heroin were both legal for several decades -- does it mean it was "okay" to use them back then? The issue is, legality isn't always a good indicator of whether a particular activity is harmful or not. Even practically, the only country I have heard of where such drug-laws are strictly enforced is the US of A, and I don't think the reader will need to be told that their notion of what to "fix" and what not to is a bit skewed. Personally, no one in my social circle ever got into any legal trouble for smoking up once in a while.

The authors mention legality again, and their argument is flawed for the reason already mentioned. Going by this logic, DC++, anything but regular sex, homosexuals (in West Bengal) should all be avoided.

Keeping secrets from some people is not the same as getting disowned from society. Moreover, people choose friends and social circles based on their opinions, not the other way around

You can't vote in this country if you mind sponsoring corruption. :) Plus, at least in West Bengal, marijuana is grown locally by extremely poor people. If anything, by buying pot you're ensuring they have enough to eat for a day.

I agree with the authors on one thing and one thing only -- presenting a distorted picture of drug use, mentioning the good while leaving out the bad and risky is never ethical. In fact, I find this article unethical for doing the exact opposite, which is just as bad.

Fact is, however the authors try to pretend to be omniscient beings who transcend time and space, they're are just like the readers they're trying to advise and I don't think they know any better. People have a right to choose what they do with their lives; giving out unsolicited advice is vulgar and revolting.

Does this mean I condone drug usage? Well, it doesn't matter what I think. Drugs can ruin your life, be the most meaningful experience you take out of this phase of your life or just something you do once in a while. Your life is yours to make and destroy, and I don't get a say in it.

PS: This isn't a personal attack against anyone.

PPS: I don't contribute to any multiple-continent-spanning newspaper.

--

[1] http://www.scholarsavenue.org/2012/03/17/an-open-letter/
[2] http://en.wikipedia.org/wiki/Gateway
[3] http://en.wikipedia.org/wiki/Bhang
[4] http://www.telegraph.co.uk/news/uknews/law-and-order/4581743/Now-Home-Office-drugs-adviser-wants-to-downgrade-LSD-from-A-to-B.html
[5] http://www.huffingtonpost.com/2011/07/03/portugal-drug-laws-decriminalization-_n_889531.html

• attempts to move on a cell already occupied
• attempts to make a move on a game that has already been won

While it does not make sense to compile the program into an executable the game is quite "playable" from the ghci REPL. The code is up on github [1].

In fact, Haskell's type system, with the UndecidableInstances extension is Turing complete, so it isn't a surprise that such a thing is possible. But type level programming has its own set of rules and I had a lot of fun figuring things out in such a foreign domain (Type-Level Instant Insanity [2] was extremely helpful). This post is about some of the things I observed.

Tuples are the type-level lists

This seems obvious now that I say it, since a tuple is the most straightforward way to represent a list of types. However, the direct way to do so is not very useful -- instead of encoding the types A, B, C as the tuple type (A, B, C) it is clearer to represent them using the usual recursive cons idiom: (A, (B, (C, ()))). Here () is the nil element and cons X Y is (X, Y).

Functional dependencies

Haskell normally doesn't allow typeclasses on more than one type. Adding proper functional dependencies can fix this [3]. A multi-parameter typeclass is essentially a n-ary relation between types and a functional dependency constrains the relations that a multi-parameter typeclass can represent. This has interesting consequences when programming with types.

The functional dependency in X a b c | a b -> c constrains the relation it defines by forcing c to be a function of a and b. This can be used to defined type-level functions. Specifically, a function that maps two types to a third type can be represented just like the typeclass X in the example above. The body of the function can then be defined as a bunch of instance declarations. If you're doing something like Peano arithmetic and representing, for instance, 2 by the type S (S ()) (S is a type constructor here), you can implement addition as shown below:

{-# LANGUAGE MultiParamTypeClasses, FunctionalDependencies,
             FlexibleInstances, UndecidableInstances #-}
 
data S x
 
class Add x y z | x y -> z where add :: x -> y -> z
instance Add () x x where add = undefined
instance (Add x y z) => Add (S x) y (S z) where add = undefined

The functional dependency in Add states that the output of Add is a function of the first two types, which is correct. Note how we're building up the induction.

I had a hard time figuring out how predicates like "two things are foo if they are not bar" are to be represented. The reverse was easy, you use a context -- instance (Foo x, Foo y) => Bar x y, but there isn't any way in Haskell to say instance (NOT Foo x, Not Foo y) => Bar x y. If you think about it, there is no reason to have such a feature either -- the only reason for saying "if x is in the typeclass X then it is also in the typeclass Y" is to be able to use X's functions in defining Y's, so having "x is not in the typeclass X" in the context is absolutely useless! The solution (which I read in [2]) is to instead have such predicates "return" a type (which is a value in this weird world we're working in). For better readability, defining two types, data T and data F for this purpose is useful. [1], for instance, is littered with instances like

-- When a is X then it isn't Y and vice versa.  Silly example, but I
-- think it demonstrates the point
 
instance (X a T) => Y a F
instance (X a F) => Y a T

Inverted Induction

Finally, the way you write "function bodies" at the type level is a bit different. Instead of breaking down a function into smaller bits and then recursion on those, you conditionally build smaller structures into larger structures. For instance, to check if an element is in a list, you say "if the object o is in a list l or if x == othen it is in the list cons x l for all x". It isn't any different really, once you get used to it.

--

[1] https://raw.github.com/sanjoy/Snippets/master/TicTacToe.hs
[2] http://www.haskell.org/wikiupload/d/dd/TMR-Issue8.pdf
[3] http://www.haskell.org/haskellwiki/Functional_dependencies

In case you haven't already, you need to first read up (a little bit) on untyped and typed lambda calculus. Try reading to the point where you can comfortably state why $\lambda x \cdot x x$ can not be well-typed. I'll wait right here, I promise. :)

I'll use the symbol $[x \mapsto s] t$ to mean "replace instances of $x$ by $s$ in the term $t$". We'll talk about a typed lambda calculus which has exactly one type $A$ and only one value of that type, $a$. We'll evaluate in normal order.

Firstly, note (informally) that all types will look like $((A \to A) \to (A \to A)) \to (A \to A)$. It is not hard to see (or prove) that the set of types is isomorphic to a set of binary trees with every internal node a $\to$ and every leaf an $A$. This is important since we'll do structural induction on the binary tree corresponding to a type.

We now define a set of what are usually called reducibility candidates. They are a predicate on terms of the lambda calculus with the following conditions

• $R_{A}(t)$ if $t$ halts.
• $R_{T_{1} \to T_{2}}(t)$ $\iff$ $t$ halts and $R_{T_{1}}(s) \Rightarrow R_{T_{2}}(t s)$

Since $R_T$ is a predicate, we can safely say things like $x$ is in $R_A$ etc.

We aim to prove the following

1. Every term in a set $R_{T}$ (for any type $T$) is normalizable. Note that this follows from the definition of the reducibility candidates
2. Every term of type $T$ is in $R_{T}$.

These two, combined, will allow us to draw the conclusion that every well-typed term in our simply typed lambda calculus halts.

Lemma 0: Every term in a reducibility candidate halts

This follows directly from the definition of a reducibility candidate.

Lemma 1: Reduction does not change the reducibility status of a term

Formally, we try to prove $R_T(t) \wedge (t \to t') \iff R_T(t')$. For this, we use the structural induction on binary trees isomorphic to each type we talked about above. There are two cases.

T is the atom type

This is the base case for our induction and also pretty obvious. From $t \to t'$ we conclude $t$ halts iff $t'$ halts. Since the only condition to be in $R_A$ is halting, $R_A(t) \iff R_A(t')$.

T is a function type

Let $T \equiv T_1 \to T_2$. In this case, each $t'$ has to satisfy the additional condition $\forall s \in R_{T_1} \cdot R_{T_2}(t'\;s)$. We begin by taking an arbitrary $s$ in $R_{T_1}$.

Notice how $t \; s \to t' \; s$ [1]? Also notice how $T_2$ is a substructure of $T \equiv (T_1 \to T_2)$? We can invoke structural induction here and say we already know $R_{T_2}(t \; s) \Rightarrow R_{T_2}(t' \; s)$ (statement 0). Also, since we're given $R_T(t)$, we know for all $s$ in $R_{T_1}$, $t \; s$ is in $R_{T_2}$ (property of $R_T$) (statement 1). We can conclude from statement 0 and statement 1 that for any arbitrary $s$ in $R_{T_1}$, we have $R_{T_2} (t' \; s)$. This is the extra condition we needed to show $t'$ is in $R_T$.

Lemma 2: Every well typed term of type T is in $R_T$

This is proved by proving a stronger assertion: if $x_1:T_1, ..., x_n:T_n \vdash t \; : \; T$ and $\emptyset \vdash v_i : T_i$ for all $i$ then $R_T([x_1 \mapsto v_1] ... [x_n \mapsto v_n]t)$. This is a stronger assertion because this also applies to terms with free variables while, at the top level, we'll need to consider only closed terms. The assertion will be invoked on open terms only via induction.

To prove this assertion, we'll invoke induction on the size of a term. So, informally, $t_1 \; t_2$ is greater than either $t_1$ or $t_2$ and $\lambda a \cdot x$ is greater than $x$. We make a case by case analysis on the various forms $t$ can be in

A variable

This is obvious, since $t = x_i$ for some $i$.

An application

This too is easy. Since $t \equiv t_1 \; t_2$, we invoke induction on $t_1$ and $t_2$ separately and then use $([a \mapsto b]c) ([a \mapsto b]d) \equiv ([a \mapsto b] (c \; d))$ to get our result.

An abstraction

We know, from its form that $t$ is a value. The other thing left to prove is that, keeping types compatible, $\forall s \in R_S \cdot R_M(t \; s)$. $t$ is of the form $\lambda x : S \cdot b$. Let $s \in R_S$. We know, from lemma 0 that $s$ will reduce to a value $v$. From lemma 1, we get $R_S(v)$. The condition we're trying to prove is $([x_1 \mapsto v_1]...[x_n \mapsto v_n]t)s \in R_P$. But this beta-reduces to $([x_1 \mapsto v_1]...[x_n \mapsto v_n](t \; s))$ $\to$ $([x_1 \mapsto v_1]...[x_n \mapsto v_n](t \; v))$ $\to$ $([x_1 \mapsto v_1]...[x_n \mapsto v_n]([x \mapsto v]b)$. $b$ is a smaller term satisfying the requirements for this proposition, and we can say, via induction, $R_P([x_1 \mapsto v_1]...[x_n \mapsto v_n]([x \mapsto v]b)$. From lemma 1 we get $R_P(([x_1 \mapsto v_1]...[x_n \mapsto v_n]t)s)$, which is what we wanted to prove.

--

1. In normal order. I haven't tried proving this in other evaluation strategies but I think I can go around by proving that the evaluation strategy does not matter in deciding the behaviour of a program in this lambda calculus.

Contrast a book with a movie, for instance. The information a book contains is certainly more abstract than the information a movie rams into my head. I don't know how Naoko in Norwegian Wood looked like but Martha always conjures up the image of Helena Bontham Carter in a hat, smoking a freshly lit cigarette. Naoko's image is malleable and ethereal but Martha will always be a dark girl with curly hair (this is not to criticize Fight Club, I loved the movie).

I think this makes books more relatable, and a well written text hits me at a very base level. Since the written word is so flexible and abstract, my mind can lift it up to a concrete representation that is very close to my sense of self. A book can drive me to insanity, a movie cannot.

I think the same applies to music too -- I've always found solos by Pink Floyd their most interesting work. When a rapper talks about how he scored last night, there is only so much information my mind can add, there is only so much I can misinterpret. When I hear the crazy slide-guitar in Shine on you crazy diamond, VI-IX, my mind is forced to conjure up something using the only tools it has at its disposal, my imagination and my experience. And it always comes up with something sinister.

The Problem

Let's say A is a runtime function that sometimes gets called by JIT compiled code. When the control is inside A, the stack may look something like this:

[  A  ]
[ Some JIT compiled function (X)  ]
[ Some other JIT compiled function (Y)  ]
[ Some function that launches JIT compiled code (B)  ]
[ Some function that does the JIT compilation (C)  ]
[   .... and so on ....  ]
[  First function for this thread == (D)  ]

Now, for GDB to be able to successfully unwind through the X and Y frames, it needs some extra information. For regular functions (like A, B, C and D) this information is embedded in a section in the ELF file, using the DWARF encoding. However, X and Y don't have any such associated information since they were generated on the fly. On trying to backtrace when the stack is in the state shown above you'll

• See a bunch of ??s instead of X and Y, since GDB does not have the debug symbols.
• A possibly incorrect backtrace. From what I understand, GDB can backtrace by interpreting the prologue, but only up to a point.

The initial solution (still supported by GDB) was this: whenever the JIT compiler compiles a block of code, have it emit the debug information to an ELF file in memory. Then have the JIT compiler notify GDB of these in-memory ELF files using a published interface. This way, GDB knows how to display a pretty backtrace.

This solution (of generating ELF files in memory) is, however, not ideal. ELF files are complex and hard to generate. Generating ELF files bites even more if all that GDB needs to do is display a meaningful stack trace. My job was to find a less complex way to get the debugging information into GDB.

The Solution

It was Tom Tromey's idea that it probably makes more sense to load and use some sort of a plugin provided by the JIT vendors. I too thought this idea was viable and decided to implement it and see how well it worked.

The problem can be divided into two parts -- providing the debug symbols and providing the unwind information. We don't deal with providing type information and other such more sophisticated information, since we're targeting people who just want to see a meaningful stack trace without putting in too much effort. The ELF handling code is still in GDB and anyone wishing to do something more exciting than looking at a backtrace can generate full-fledged ELF files.

For the debug symbols part, we

1. Have the JIT compiler generate the debug symbol information in any format it pleases. GDB does not need to understand this format.
2. Have the plugin parse this debug information (since it understands the format, being written by the JIT vendor) and emit the symbol tables GDB requires.

In the name of a clean interface and better backward compatibility, (2) is done by passing a bunch of callback functions to the plugin, along with the data it is supposed to parse (the data that the JIT compiler passed to GDB). The plugin then constructs the symbol tables using the callbacks it is given. There are callbacks to create a new symbol table (symtab_open), to add a mapping from line numbers to PCs (line_mapping_add) among others. All this is documented in the header jit-reader.h.

jit-reader.h is installed in {includdir}/gdb/, where {includedir} is the system's default include directory. This header is the interface between the plugin and GDB, and needs to be included in the plugin source file. Because of where it is installed by default, usually a #include <gdb/jitreader.h> is sufficient.

For the unwinding part we simply have the plugin provide an unwinder. The unwinder too gets a bunch of callbacks, one to read the value of a register in the current frame, one to tell GDB (write) the value of a register in the previous frame (the one that was called immediately before), one to read some data off the target process' address space, one to generate a frame id (described in jit-reader.h) and so on. The unwinder is especially easy to write if the JIT compiled code stores interesting things at constant offsets from the base register. Pseudo code could be

  READ_REGISTER(cb, AMD64_RBP, FramePtr);
  PreviousFP = READ_MEMORY_AT(FramePtr)
  PreviousPC = READ_MEMORY_AT(FramePtr + 8)

  WRITE_REGISTER(AMD64_RBP, PreviousFP)
  WRITE_REGISTER(AMD64_RA, PreviousPC)

Telling GDB the values of the frame pointer and the program counter is enough for it to display a backtrace.

I've checked in the code, and it is available in the current trunk. This was a very general bird's eye view of the situation, more information can be found in the GDB manual and the jit-reader.h file GDB installs. Thanks to everyone on the GDB mailing list who sent in their valuable reviews.

Comments, suggestions and questions welcome.

Unlike Lisp's macro system, TH is typed -- like everything else in Haskell. You have typed constructors like FunD (function declaration) and InfixP which are used the build the AST of whatever form you wish to splice into your regular code. You can find the current TH user manual at [1].

The AST constructors are in the module Language.Haskell.TH, which looks quite overwhelming at the first glance, and, as such, writing TH expressions from scratch can get quite difficult. Thankfully, we also have a nice runQ function which allows us to write TH expressions without really grokking everything in the module. I'll try to demonstrate this by getting TH to generate a small (read useless / toy) string matcher.

Objective

We'll try to generate a function with the type String->Bool. The function's job will be to check if the String it receives belongs to a list of Strings we provide at compile time. To make things a little more interesting, we'll allow the strings we have at compile time to have a regex-like "?" in them, which can match any character. As an example, if the list of Strings is ["asdf", "a?foo", "P"], the generated function should match "asdf", "P", "amfoo" but not "afoo". We don't address the issue of how to match a literal "?". Heck, out end product boils down to ten lines of code.

Once we're done writing our TH code, we should have a function generateMatcher, which we shall invoke from the top level of a source file, like:

  generateMatcher "generatedF" ["asdf", "a?foo", "P"]

It will then generate the matcher, with the name generatedF and the behaviour we're just described. It can then be called like a regular function. So a source file can look like:

  generateMatcher "generatedF" ["asdf", "a?foo", "P"]

  main = do
    str < - getLine
    putStrLn $ show $ generatedF str

Starting Off

The first thing to think about is how the compiled code should look like. Given the above problem instance, we could generate something like

    generatedF ('a':'s':'d':'f':[]) = True
    generatedF ('a':_:'f':'o':'o':[]) = True
    generatedF ('P':[]) = True
    generatedF _ = False
  

Haskell's pattern matcher is, AFAIK, intelligent enough to convert the cases into a tree structure and quickly jump to the correct clause. Now the second question: how should the AST look like? Here we take the help of the runQ function. First we start GHCi as ghci -XTemplateHaskell (since TH is not enabled by default). Once in GHCi, we import the required module by :m + Language.Haskell.TH. The scene is set! We can now see the AST for generatedF ('a':'s':'d':'f':[]) = True. For that, we evaluate runQ [d|generatedF ('a':'s':'d':'f':[]) = True|]. The [d| ... |] is used to quote a declaration. Detailed information (about this, and generally about TH) is in [1]. We will now see some (rather incomprehensible) output:

[FunD generatedF [Clause [InfixP (LitP (CharL 'a'))
GHC.Types.: (InfixP (LitP (CharL 's')) GHC.Types.:
(InfixP (LitP (CharL 'd')) GHC.Types.:
(InfixP (LitP (CharL 'f')) GHC.Types.:
(ConP GHC.Types.[] []))))] (NormalB
(ConE GHC.Bool.True)) []]]

A little indentation, of course, goes a long way in making things clearer (when does it not? :D):

[FunD
 generatedF [Clause
             [InfixP (LitP (CharL 'a'))
              GHC.Types.:
              (InfixP (LitP (CharL 's'))
               GHC.Types.:
               (InfixP (LitP (CharL 'd'))
                GHC.Types.:
                (InfixP (LitP (CharL 'f'))
                 GHC.Types.:
                 (ConP GHC.Types.[] []))))]
             (NormalB (ConE GHC.Bool.True)) []]]
  

I usually use guesswork from this point, leaving it to Haskell's type system to correct my incompetence. Haskell's AST has the convention of suffixing most data constructors with a letter which hints at what type it belongs to. We can guess (and purely guess, I have not looked up the documentation) that InfixP is an infix pattern, a LitP is a literal pattern, a NormalB is a normal body, a ConE is a constructor expression, a FunD is a function declaration and so forth. The AST for the match-anything clause is simpler:

  [FunD
   generatedF [Clause [WildP]
               (NormalB (ConE GHC.Bool.False)) []]]

You can guess what a WildP is, especially since it derives from a _.

The core idea

The basic idea behind TH is that allowing the user to splice in a bit of custom-generated AST into an otherwise normal Haskell program can lead to all kinds of awesomeness. A splicable AST has to be typed as a Q Exp, a Q Type or a Q [Dec]. The last one stands for a list of declarations, which is the one we need right now. To splice in an expression as an AST, we place it inside a $( ... ). To keep things clean, top level declarations don't need the $( ... ) nesting -- anything of the type Q [Dec] at the top level of a source file is automatically interpreted correctly. Q is a Monad, which is sometimes needed to generate unique identifiers, like Lisp's gensym. We don't need that right now.

Getting back to the point, we need a function that takes the specified input, and outputs the AST of the code we want to generate. Since we need to generate something of the type Q [Dec], and we will be given the name of the function to generate, and a list of Strings, there is at least one line we can write:

  generateMatcher :: String -> [String] -> Q [Dec]

We already have seen the basic structure of the AST. We now write some code to generate it. We will generate only one Dec (i.e. the [Dec] in Q [Dec] will only have one element), corresponding to the one function we want to declare. The Dec will be a FunD (just like in the AST runQ showed us), with multiple Clauses, each representing one pattern (which itself corresponds to one String we were given at compile time).

One thing though -- for some reason I don't really understand, while runQ prints them, things in the GHC module (like GHC.Bool.True, GHC.Types.:) don't really work in a generated AST. A token in TH is essentially a Name, which we can get using mkName :: String -> Name or by using the quote (') operator. So one way to get around the problem is to represent things like GHC.Bool.True as Names -- this seem to work well in practice. The quote operator does not seem to work for symbols -- we replace things like GHC.Bool.True with 'True and things like : with mkName ":". We also need to use mkName for the functions' name, which needs to be a (surprise) Name.

Okay, phew! Time to see some code that actually works. With the little Haskell-foo I have, I wrote the following piece of code:

generateMatcher name strs =
  return [FunD (mkName name) (map thMagic strs ++ baseCase)]
  where
    thMagic str = Clause [transform str] (NormalB (ConE 'True)) []
    baseCase = [Clause [WildP] (NormalB (ConE 'False)) []]
    transform [] = ConP (mkName "[]") []
    transform ('?':rest) =
      let colon = mkName ":" in (InfixP WildP colon $ transform rest)
    transform (x:rest) =
      InfixP (LitP (CharL x)) (mkName ":") $ transform rest

Ten lines, as promised. I used return to lift the [Dec] into the Q monad. Other than, most of it is a direct translation of the input we get into something that looks like the output runQ dumped on me. The code is in my snippets repo, as usual. TH does not, as of now, allow you to declare an expression and splice it in the same file. You need to have, say, a FooTH.hs where you have all the functions for generating the AST and a Foo.hs where you actually use them.

TL;DR

This was a long and meandering post. In short, here are the two basic points I want to make:

• TH is awesome. It can save a lot of time, if used properly.
• Don't beat yourself up trying to write ASTs from scratch, play around with runQ, steal and modify.

[1] http://www.haskell.org/ghc/docs/latest/html/users_guide/template-haskell.html

• Each object has a (header or otherwise) pointer-sized field which stores some information that can be differentiated from a pointer. On a system that is aligns words to two or four byte boundaries, we could set the LSB of non-pointer data stored in the field. We'll call this field info.
• Pointers to objects always point to the beginning of some object. No pointing to a field stored somewhere in the middle.

The algorithm is extremely simple and elegant. All you have to do is this: every time you see a slot (a field, if you like) S point to an object O, change Os info field to point to slot S and set S to whatever value (pointer or non-pointer) info held before it was set to point to S. If you try this out with pen and paper (I'm too lazy to draw a diagram here), you'll see that this simple action, when applied to every slot in the graph, transforms the predicate "Slots S₀, S₁ and S₂ point to object O" to "O's info slot contains a pointer to S₂, S₂ contains a pointer to S₁, S₁ contains a pointer to S₀ and S₀ contains the data O's info field originally held". Since we can always distinguish a data word stored in info from a pointer, we can traverse this list and get a list of the slots. The slots can then be filled with new location of the object.

This is one of the main concepts behind Jonker's compaction algorithm, except that it intelligently divides the compaction into two phases of updating forward pointers and updating backward pointers while moving objects.

I wrote a simple and tiny implementation of threading a graph, which you can find at https://github.com/sanjoy/Snippets/blob/master/Thread.cc. Note how the main threading algorithm only takes thirteen lines of C++.

The Deutsch-Schorr-Waite algorithm does not require adding any extra fields to the nodes. Adding fields to the node structure sort of spoils the fun -- you could, for instance, have a pointer pointing to the parent of every node making stackless traversal a cup of tea. And you haven't really saved any space! Thorelli's approach, however, requires adding a integer to each node that can hold at least the maximum number of outgoing arcs a node can possibly have. But it can traverse graphs with more than two outgoing arcs per node. And you do save space here -- if you have three outgoing arcs (maximum) per node, you could do with just a two bit integer. On x86, that's the last two bits of a pointer.

Note that both the approaches are quite a bit more complicated than a vanilla DFS. Nevertheless, there are applications where such characteristics (of consuming only constant space) becomes important. One of the more visible ones is a GC. A GC has to traverse a (potentially) very large and complicated graph of objects, and it can't use up too much memory (or call-stack space) when collecting -- a time when memory is already a scarce resource.

I like implementing things, and while I did not implement the full algorithm proposed by Thorelli, I did implement one which traverses a tree in constant space. What more, it does not need the extra bits Thorelli's algorithm requires.

The code is here (I believe in making code talk instead of giving long, winded descriptions): https://github.com/sanjoy/Snippets/blob/master/Traversal.cc.

The snippets repo is an idea that I recently got -- I often code small things when I'm bored or want to try something out. Having a public "snippets" repo where I upload all that smelt good.