Internal Hy Documentation


These bits are mostly useful for folks who hack on Hy itself, but can also be used for those delving deeper in macro programming.

Hy Models

Introduction to Hy Models

Hy models are a very thin layer on top of regular Python objects, representing Hy source code as data. Models only add source position information, and a handful of methods to support clean manipulation of Hy source code, for instance in macros. To achieve that goal, Hy models are mixins of a base Python class and HyObject.


hy.models.HyObject is the base class of Hy models. It only implements one method, replace, which replaces the source position of the current object with the one passed as argument. This allows us to keep track of the original position of expressions that get modified by macros, be that in the compiler or in pure hy macros.

HyObject is not intended to be used directly to instantiate Hy models, but only as a mixin for other classes.

Compound Models

Parenthesized and bracketed lists are parsed as compound models by the Hy parser.

Hy uses pretty-printing reprs for its compound models by default. If this is causing issues, it can be turned off globally by setting hy.models.PRETTY to False, or temporarily by using the hy.models.pretty context manager.

Hy also attempts to color pretty reprs and errors using colorama. These can be turned off globally by setting hy.models.COLORED and hy.errors.COLORED, respectively, to False.


hy.models.HySequence is the abstract base class of “iterable” Hy models, such as HyExpression and HyList.

Adding a HySequence to another iterable object reuses the class of the left-hand-side object, a useful behavior when you want to concatenate Hy objects in a macro, for instance.

HySequences are (mostly) immutable: you can’t add, modify, or remove elements. You can still append to a variable containing a HySequence with += and otherwise construct new HySequences out of old ones.


hy.models.HyList is a HySequence for bracketed [] lists, which, when used as a top-level expression, translate to Python list literals in the compilation phase.


hy.models.HyExpression inherits HySequence for parenthesized () expressions. The compilation result of those expressions depends on the first element of the list: the compiler dispatches expressions between compiler special-forms, user-defined macros, and regular Python function calls.


hy.models.HyDict inherits HySequence for curly-bracketed {} expressions, which compile down to a Python dictionary literal.

Atomic Models

In the input stream, double-quoted strings, respecting the Python notation for strings, are parsed as a single token, which is directly parsed as a HyString.

An uninterrupted string of characters, excluding spaces, brackets, quotes, double-quotes and comments, is parsed as an identifier.

Identifiers are resolved to atomic models during the parsing phase in the following order:


hy.models.HyString represents string literals (including bracket strings), which compile down to unicode string literals (str) in Python.

HyStrings are immutable.

Hy literal strings can span multiple lines, and are considered by the parser as a single unit, respecting the Python escapes for unicode strings.

HyStrings have an attribute brackets that stores the custom delimiter used for a bracket string (e.g., "==" for #[==[hello world]==] and the empty string for #[[hello world]]). HyStrings that are not produced by bracket strings have their brackets set to None.


hy.models.HyBytes is like HyString, but for sequences of bytes. It inherits from bytes.

Numeric Models

hy.models.HyInteger represents integer literals, using the int type.

hy.models.HyFloat represents floating-point literals.

hy.models.HyComplex represents complex literals.

Numeric models are parsed using the corresponding Python routine, and valid numeric python literals will be turned into their Hy counterpart.


hy.models.HySymbol is the model used to represent symbols in the Hy language. Like HyString, it inherits from str (or unicode on Python 2).

Symbols are mangled when they are compiled to Python variable names.


hy.models.HyKeyword represents keywords in Hy. Keywords are symbols starting with a :. See keywords.

Hy Internal Theory


The Hy internals work by acting as a front-end to Python bytecode, so that Hy itself compiles down to Python Bytecode, allowing an unmodified Python runtime to run Hy code, without even noticing it.

The way we do this is by translating Hy into an internal Python AST datastructure, and building that AST down into Python bytecode using modules from the Python standard library, so that we don’t have to duplicate all the work of the Python internals for every single Python release.

Hy works in four stages. The following sections will cover each step of Hy from source to runtime.

Steps 1 and 2: Tokenizing and Parsing

The first stage of compiling Hy is to lex the source into tokens that we can deal with. We use a project called rply, which is a really nice (and fast) parser, written in a subset of Python called rpython.

The lexing code is all defined in hy.lex.lexer. This code is mostly just defining the Hy grammar, and all the actual hard parts are taken care of by rply – we just define “callbacks” for rply in hy.lex.parser, which takes the tokens generated, and returns the Hy models.

You can think of the Hy models as the “AST” for Hy, it’s what Macros operate on (directly), and it’s what the compiler uses when it compiles Hy down.

See also

Section Hy Models for more information on Hy models and what they mean.

Step 3: Hy Compilation to Python AST

This is where most of the magic in Hy happens. This is where we take Hy AST (the models), and compile them into Python AST. A couple of funky things happen here to work past a few problems in AST, and working in the compiler is some of the most important work we do have.

The compiler is a bit complex, so don’t feel bad if you don’t grok it on the first shot, it may take a bit of time to get right.

The main entry-point to the Compiler is HyASTCompiler.compile. This method is invoked, and the only real “public” method on the class (that is to say, we don’t really promise the API beyond that method).

In fact, even internally, we don’t recurse directly hardly ever, we almost always force the Hy tree through compile, and will often do this with sub-elements of an expression that we have. It’s up to the Type-based dispatcher to properly dispatch sub-elements.

All methods that preform a compilation are marked with the @builds() decorator. You can either pass the class of the Hy model that it compiles, or you can use a string for expressions. I’ll clear this up in a second.

First Stage Type-Dispatch

Let’s start in the compile method. The first thing we do is check the Type of the thing we’re building. We look up to see if we have a method that can build the type() that we have, and dispatch to the method that can handle it. If we don’t have any methods that can build that type, we raise an internal Exception.

For instance, if we have a HyString, we have an almost 1-to-1 mapping of Hy AST to Python AST. The compile_string method takes the HyString, and returns an ast.Str() that’s populated with the correct line-numbers and content.


If we get a HyExpression, we’ll attempt to see if this is a known Macro, and push to have it expanded by invoking hy.macros.macroexpand, then push the result back into HyASTCompiler.compile.

Second Stage Expression-Dispatch

The only special case is the HyExpression, since we need to create different AST depending on the special form in question. For instance, when we hit an (if True True False), we need to generate a ast.If, and properly compile the sub-nodes. This is where the @builds() with a String as an argument comes in.

For the compile_expression (which is defined with an @builds(HyExpression)) will dispatch based on the string of the first argument. If, for some reason, the first argument is not a string, it will properly handle that case as well (most likely by raising an Exception).

If the String isn’t known to Hy, it will default to create an ast.Call, which will try to do a runtime call (in Python, something like foo()).

Issues Hit with Python AST

Python AST is great; it’s what’s enabled us to write such a powerful project on top of Python without having to fight Python too hard. Like anything, we’ve had our fair share of issues, and here’s a short list of the common ones you might run into.

Python differentiates between Statements and Expressions.

This might not sound like a big deal – in fact, to most Python programmers, this will shortly become a “Well, yeah” moment.

In Python, doing something like:

print for x in range(10): pass, because print prints expressions, and for isn’t an expression, it’s a control flow statement. Things like 1 + 1 are Expressions, as is lambda x: 1 + x, but other language features, such as if, for, or while are statements.

Since they have no “value” to Python, this makes working in Hy hard, since doing something like (print (if True True False)) is not just common, it’s expected.

As a result, we reconfigure things using a Result object, where we offer up any ast.stmt that need to get run, and a single ast.expr that can be used to get the value of whatever was just run. Hy does this by forcing assignment to things while running.

As example, the Hy:

(print (if True True False))

Will turn into:

if True:
    _temp_name_here = True
    _temp_name_here = False


OK, that was a bit of a lie, since we actually turn that statement into:

print(True if True else False)

By forcing things into an ast.expr if we can, but the general idea holds.

Step 4: Python Bytecode Output and Runtime

After we have a Python AST tree that’s complete, we can try and compile it to Python bytecode by pushing it through eval. From here on out, we’re no longer in control, and Python is taking care of everything. This is why things like Python tracebacks, pdb and django apps work.

Hy Macros

Using gensym for Safer Macros

When writing macros, one must be careful to avoid capturing external variables or using variable names that might conflict with user code.

We will use an example macro nif (see for a more complete description.) nif is an example, something like a numeric if, where based on the expression, one of the 3 forms is called depending on if the expression is positive, zero or negative.

A first pass might be something like:

(defmacro nif [expr pos-form zero-form neg-form]
    (setv obscure-name ~expr)
    (cond [(pos? obscure-name) ~pos-form]
          [(zero? obscure-name) ~zero-form]
          [(neg? obscure-name) ~neg-form])))

where obscure-name is an attempt to pick some variable name as not to conflict with other code. But of course, while well-intentioned, this is no guarantee.

The method gensym is designed to generate a new, unique symbol for just such an occasion. A much better version of nif would be:

(defmacro nif [expr pos-form zero-form neg-form]
  (setv g (gensym))
     (setv ~g ~expr)
     (cond [(pos? ~g) ~pos-form]
           [(zero? ~g) ~zero-form]
           [(neg? ~g) ~neg-form])))

This is an easy case, since there is only one symbol. But if there is a need for several gensym’s there is a second macro with-gensyms that basically expands to a setv form:

(with-gensyms [a b c]

expands to:

  (setv a (gensym)
        b (gensym)
        c (gensym))

so our re-written nif would look like:

(defmacro nif [expr pos-form zero-form neg-form]
  (with-gensyms [g]
       (setv ~g ~expr)
       (cond [(pos? ~g) ~pos-form]
             [(zero? ~g) ~zero-form]
             [(neg? ~g) ~neg-form]))))

Finally, though we can make a new macro that does all this for us. defmacro/g! will take all symbols that begin with g! and automatically call gensym with the remainder of the symbol. So g!a would become (gensym "a").

Our final version of nif, built with defmacro/g! becomes:

(defmacro/g! nif [expr pos-form zero-form neg-form]
     (setv ~g!res ~expr)
     (cond [(pos? ~g!res) ~pos-form]
           [(zero? ~g!res) ~zero-form]
           [(neg? ~g!res) ~neg-form])))

Checking Macro Arguments and Raising Exceptions

Hy Compiler Built-Ins