AST compilation from Python

written by Georg, on Saturday, March 29, 2008 9:55.

Thanks to Thomas Lee, we have now in the Python trunk a built-in way to modify the Abstract Syntax Tree compiled from a piece of Python source, and to compile that to an executable code object.

Quick example:

>>> import _ast
>>> # A dumb source snippet
>>> source = "print 5 + 8"
>>> # Compile using the special PyCF_ONLY_AST flag to get an AST
>>> ast = compile(source, "<string>", "exec", _ast.PyCF_ONLY_AST)
>>> # The toplevel node is always a Module for "exec"
>>> ast
<_ast.Module object at 0xb7b00edc>
>>> # Each AST node has different attributes (see the docs)
>>> ast.body
[<_ast.Print object at 0xb7b020a4>]
>>> # The Print node has a list of values -- expressions to print
>>> ast.body[0].values
[<_ast.BinOp object at 0xb7b02114>]
>>> # This is the addition operator between 5 and 8
>>> ast.body[0].values[0].op
<_ast.Add object at 0xb7c98504>
>>> # Now, replace the operator with subtraction ...
>>> ast.body[0].values[0].op = _ast.Sub()
>>> # ... and compile the resulting AST
>>> code = compile(ast, "<string>", "exec")
>>> # Voila:
>>> exec code
-3

From this, you should be able to work out how to write AST manipulation code that actually makes sense :)

The _ast documentation gives you an idea which classes and attributes are available for individual AST elements.

Comments

My math skills must be rusty. :)

5 + 8 = -3

— John M Camara on Saturday, March 29, 2008 15:01 #
Okay, it was a bit hard to see that I replaced the Add with a Sub node before compiling the AST.

I've now added inline comments to the code, it may be clearer with them.

— Georg on Saturday, March 29, 2008 15:09 #
Awesome! I was playing with compiler.ast just last week and wondering if such a partial compilation scheme would be possible.

In what versions of Python will we see this change? I see the documentation is for 2.6. Will it also be relevant to Py3k?

— David Anderson on Saturday, March 29, 2008 18:57 #
cool! I've been wondering what becomes of the old compiler module.

— kumar on Saturday, March 29, 2008 19:42 #
Thanks for the review love, Georg. :)

— Thomas Lee on Sunday, March 30, 2008 0:30 #
@dave: It is in the trunk, and therefore also will be merged to Py3k.

@kumar: It will also be important in Py3k since the compiler package is gone there.

— Georg on Sunday, March 30, 2008 6:49 #
I was also playing with the compiler module this weekend...

Do you happen to know if there is a nice way to walk the AST, similar to ASTVisitor?

— fitzgeraldsteele on Tuesday, April 8, 2008 5:13 #
There is a module called "ast" that does this and more here.

We'll propose something like this for inclusion in Python 2.6.

— Georg on Wednesday, April 9, 2008 17:51 #

Commenting is no longer possible.