templates.txt

Templates
=========

A template is a simple form of a macro: It is a simple substitution
mechanism that operates on Nim's abstract syntax trees. It is processed in
the semantic pass of the compiler.

The syntax to *invoke* a template is the same as calling a procedure.

Example:

.. code-block:: nim
  template `!=` (a, b: expr): expr =
    # this definition exists in the System module
    not (a == b)

  assert(5 != 6) # the compiler rewrites that to: assert(not (5 == 6))

The ``!=``, ``>``, ``>=``, ``in``, ``notin``, ``isnot`` operators are in fact 
templates:

| ``a > b`` is transformed into ``b < a``.
| ``a in b`` is transformed into ``contains(b, a)``. 
| ``notin`` and ``isnot`` have the obvious meanings.

The "types" of templates can be the symbols ``expr`` (stands for *expression*), 
``stmt`` (stands for *statement*) or ``typedesc`` (stands for *type 
description*). These are "meta types", they can only be used in certain 
contexts. Real types can be used too; this implies that expressions are 
expected.


Ordinary vs immediate templates
-------------------------------

There are two different kinds of templates: immediate templates and
ordinary templates. Ordinary templates take part in overloading resolution. As
such their arguments need to be type checked before the template is invoked.
So ordinary templates cannot receive undeclared identifiers:

.. code-block:: nim

  template declareInt(x: expr) = 
    var x: int

  declareInt(x) # error: unknown identifier: 'x'

An ``immediate`` template does not participate in overload resolution and so
its arguments are not checked for semantics before invocation. So they can
receive undeclared identifiers:

.. code-block:: nim

  template declareInt(x: expr) {.immediate.} = 
    var x: int

  declareInt(x) # valid


Passing a code block to a template
----------------------------------

If there is a ``stmt`` parameter it should be the last in the template
declaration, because statements are passed to a template via a
special ``:`` syntax:

.. code-block:: nim

  template withFile(f, fn, mode: expr, actions: stmt): stmt {.immediate.} =
    var f: File
    if open(f, fn, mode):
      try:
        actions
      finally:
        close(f)
    else:
      quit("cannot open: " & fn)
      
  withFile(txt, "ttempl3.txt", fmWrite):
    txt.writeln("line 1")
    txt.writeln("line 2")
  
In the example the two ``writeln`` statements are bound to the ``actions``
parameter.


Symbol binding in templates
---------------------------

A template is a `hygienic`:idx: macro and so opens a new scope. Most symbols are
bound from the definition scope of the template:

.. code-block:: nim
  # Module A
  var 
    lastId = 0
  
  template genId*: expr =
    inc(lastId)
    lastId

.. code-block:: nim
  # Module B
  import A
  
  echo genId() # Works as 'lastId' has been bound in 'genId's defining scope

As in generics symbol binding can be influenced via ``mixin`` or ``bind`` 
statements.



Identifier construction
-----------------------

In templates identifiers can be constructed with the backticks notation:

.. code-block:: nim

  template typedef(name: expr, typ: typedesc) {.immediate.} = 
    type
      `T name`* {.inject.} = typ
      `P name`* {.inject.} = ref `T name`
      
  typedef(myint, int)
  var x: PMyInt

In the example ``name`` is instantiated with ``myint``, so \`T name\` becomes
``Tmyint``.


Lookup rules for template parameters
------------------------------------

A parameter ``p`` in a template is even substituted in the expression ``x.p``.
Thus template arguments can be used as field names and a global symbol can be
shadowed by the same argument name even when fully qualified:

.. code-block:: nim
  # module 'm'

  type
    Lev = enum
      levA, levB

  var abclev = levB

  template tstLev(abclev: Lev) =
    echo abclev, " ", m.abclev

  tstLev(levA)
  # produces: 'levA levA'
  
But the global symbol can properly be captured by a ``bind`` statement:

.. code-block:: nim
  # module 'm'

  type
    Lev = enum
      levA, levB

  var abclev = levB

  template tstLev(abclev: Lev) =
    bind m.abclev
    echo abclev, " ", m.abclev

  tstLev(levA)
  # produces: 'levA levB'


Hygiene in templates
--------------------

Per default templates are `hygienic`:idx:\: Local identifiers declared in a
template cannot be accessed in the instantiation context:

.. code-block:: nim
  
  template newException*(exceptn: typedesc, message: string): expr =
    var
      e: ref exceptn  # e is implicitly gensym'ed here
    new(e)
    e.msg = message
    e
    
  # so this works:
  let e = "message"
  raise newException(EIO, e)


Whether a symbol that is declared in a template is exposed to the instantiation
scope is controlled by the `inject`:idx: and `gensym`:idx: pragmas: gensym'ed
symbols are not exposed but inject'ed are.

The default for symbols of entity ``type``, ``var``, ``let`` and ``const``
is ``gensym`` and for ``proc``, ``iterator``, ``converter``, ``template``,
``macro`` is ``inject``. However, if the name of the entity is passed as a 
template parameter, it is an inject'ed symbol:

.. code-block:: nim
  template withFile(f, fn, mode: expr, actions: stmt): stmt {.immediate.} =
    block:
      var f: File  # since 'f' is a template param, it's injected implicitly
      ...
      
  withFile(txt, "ttempl3.txt", fmWrite):
    txt.writeln("line 1")
    txt.writeln("line 2")


The ``inject`` and ``gensym`` pragmas are second class annotations; they have
no semantics outside of a template definition and cannot be abstracted over:

.. code-block:: nim
  {.pragma myInject: inject.}
  
  template t() =
    var x {.myInject.}: int # does NOT work


To get rid of hygiene in templates, one can use the `dirty`:idx: pragma for
a template. ``inject`` and ``gensym`` have no effect in ``dirty`` templates.



Limitations of the method call syntax
-------------------------------------

The expression ``x`` in ``x.f`` needs to be semantically checked (that means
symbol lookup and type checking) before it can be decided that it needs to be
rewritten to ``f(x)``. Therefore the dot syntax has some limiations when it
is used to invoke templates/macros:

.. code-block:: nim
  template declareVar(name: expr): stmt =
    const name {.inject.} = 45

  # Doesn't compile:
  unknownIdentifier.declareVar


Another common example is this:

.. code-block:: nim
  from sequtils import toSeq

  iterator something: string =
    yield "Hello"
    yield "World"

  var info = toSeq(something())

The problem here is that the compiler already decided that ``something()`` as
an iterator is not callable in this context before ``toSeq`` gets its
chance to convert it into a sequence.


Macros
======

A macro is a special kind of low level template. Macros can be used
to implement `domain specific languages`:idx:. Like templates, macros come in
the 2 flavors *immediate* and *ordinary*.

While macros enable advanced compile-time code transformations, they
cannot change Nim's syntax. However, this is no real restriction because
Nim's syntax is flexible enough anyway.

To write macros, one needs to know how the Nim concrete syntax is converted
to an abstract syntax tree.

There are two ways to invoke a macro:
(1) invoking a macro like a procedure call (`expression macros`)
(2) invoking a macro with the special ``macrostmt`` syntax (`statement macros`)


Expression Macros
-----------------

The following example implements a powerful ``debug`` command that accepts a
variable number of arguments:

.. code-block:: nim
  # to work with Nim syntax trees, we need an API that is defined in the
  # ``macros`` module:
  import macros

  macro debug(n: varargs[expr]): stmt =
    # `n` is a Nim AST that contains the whole macro invocation
    # this macro returns a list of statements:
    result = newNimNode(nnkStmtList, n)
    # iterate over any argument that is passed to this macro:
    for i in 0..n.len-1:
      # add a call to the statement list that writes the expression;
      # `toStrLit` converts an AST to its string representation:
      add(result, newCall("write", newIdentNode("stdout"), toStrLit(n[i])))
      # add a call to the statement list that writes ": "
      add(result, newCall("write", newIdentNode("stdout"), newStrLitNode(": ")))
      # add a call to the statement list that writes the expressions value:
      add(result, newCall("writeln", newIdentNode("stdout"), n[i]))

  var
    a: array [0..10, int]
    x = "some string"
  a[0] = 42
  a[1] = 45

  debug(a[0], a[1], x)

The macro call expands to:

.. code-block:: nim
  write(stdout, "a[0]")
  write(stdout, ": ")
  writeln(stdout, a[0])

  write(stdout, "a[1]")
  write(stdout, ": ")
  writeln(stdout, a[1])

  write(stdout, "x")
  write(stdout, ": ")
  writeln(stdout, x)


Arguments that are passed to a ``varargs`` parameter are wrapped in an array
constructor expression. This is why ``debug`` iterates over all of ``n``'s
children.


BindSym
-------

The above ``debug`` macro relies on the fact that ``write``, ``writeln`` and
``stdout`` are declared in the system module and thus visible in the 
instantiating context. There is a way to use bound identifiers
(aka `symbols`:idx:) instead of using unbound identifiers. The ``bindSym`` 
builtin can be used for that:

.. code-block:: nim
  import macros

  macro debug(n: varargs[expr]): stmt =
    result = newNimNode(nnkStmtList, n)
    for i in 0..n.len-1:
      # we can bind symbols in scope via 'bindSym':
      add(result, newCall(bindSym"write", bindSym"stdout", toStrLit(n[i])))
      add(result, newCall(bindSym"write", bindSym"stdout", newStrLitNode(": ")))
      add(result, newCall(bindSym"writeln", bindSym"stdout", n[i]))

  var
    a: array [0..10, int]
    x = "some string"
  a[0] = 42
  a[1] = 45

  debug(a[0], a[1], x)

The macro call expands to:

.. code-block:: nim
  write(stdout, "a[0]")
  write(stdout, ": ")
  writeln(stdout, a[0])

  write(stdout, "a[1]")
  write(stdout, ": ")
  writeln(stdout, a[1])

  write(stdout, "x")
  write(stdout, ": ")
  writeln(stdout, x)

However, the symbols ``write``, ``writeln`` and ``stdout`` are already bound
and are not looked up again. As the example shows, ``bindSym`` does work with
overloaded symbols implicitly.


Statement Macros
----------------

Statement macros are defined just as expression macros. However, they are
invoked by an expression following a colon.

The following example outlines a macro that generates a lexical analyzer from
regular expressions:

.. code-block:: nim
  import macros

  macro case_token(n: stmt): stmt =
    # creates a lexical analyzer from regular expressions
    # ... (implementation is an exercise for the reader :-)
    discard

  case_token: # this colon tells the parser it is a macro statement
  of r"[A-Za-z_]+[A-Za-z_0-9]*":
    return tkIdentifier
  of r"0-9+":
    return tkInteger
  of r"[\+\-\*\?]+":
    return tkOperator
  else:
    return tkUnknown


**Style note**: For code readability, it is the best idea to use the least
powerful programming construct that still suffices. So the "check list" is:

(1) Use an ordinary proc/iterator, if possible.
(2) Else: Use a generic proc/iterator, if possible.
(3) Else: Use a template, if possible.
(4) Else: Use a macro.


Macros as pragmas
-----------------

Whole routines (procs, iterators etc.) can also be passed to a template or 
a macro via the pragma notation: 

.. code-block:: nim
  template m(s: stmt) = discard

  proc p() {.m.} = discard

This is a simple syntactic transformation into:

.. code-block:: nim
  template m(s: stmt) = discard

  m:
    proc p() = discard