An ast "Hello World": Getting Started with Python's Abstract Syntax Trees

I've been working on a Python library which - for a number of reasons - needs to dynamically alter itself. Essentially, I want it to parse a document and to generate some code based on that parsed file.

ast

It turns out that Python's ast module lets me do exactly what I need. I came across some quite useful supplementary documentation on ast. But, to get started, I needed something simpler that those advanced examples. I therefore wrote a "Hello world!" program using ast. Here it is, in case you were looking for that, too.

Hello world!

Since I've become test infected, I wanted to structure my "Hello world!" ast program using unit tests.

So, to start, I tracked down a suitable "Hello world!" unit test in Python.

Hello world! in ast

Then I rewrote the Greeter.py class to use ast. My version constructs an abstract syntax tree for an Assignment. Specifically, it assigns the string "Hello world!" to the variable "m". The code then fixes the locations, compiles the code and executes it dynamically.

Obviously, the above code is a lot more work than simply assigning the string value to the variable directly. But it meant I now had the world's simplest ast program.

Nothing

Armed with this most basic of unit tests, I was then in a position to work out how to support various other types of code in ast.

For example, here's a snippet of ast code which uses ast to generate an abstract syntax tree to assign an empty type to a variable named "nothing". In other words, equivalent to nothing = ()

Invoking Methods

One of the hardest things for me to figure out was how to invoke a method of a class.

First, I worked out to call a function - one not attached to an instance of a class. But to call a method of a class, I needed to understand a bit more about how Python itself is implemented.

Calling Functions

Here's some Python ast code to call a function _foo() and assign the returned value to a variable called "result", i.e. equivalent to result = foo() And here's a variant where you pass in a value, i.e. equivalent to result = bar("some value")

In Python, Methods are Attributes of Classes

Having figured out how to call functions and pass parameters to them, I reckoned that calling a method on a class would be similar.
And it sort of was - I still needed to use ast.Call to invoke the method. But it took me quite a while to figure out how to tell it which class method to call. For example, if I wanted to call

result = self._baz(theResult)

should I pass in a function name of "self._baz"? (I tried that - it didn't work). Eventually, I worked out that self._baz is an attribute of the instance object referred to as "self". In Python, instance objects have two kinds of valid attribute names, data attributes and methods. Which meant that the code to call one method of an instance from another method looks like this:
I had never thought that profoundly about how Python is really implemented behind the scenes. Although many of the Python design decisions are actually quite well documented.

An ast Short Cut

In the process of working out how to invoke instance object methods, I came up with a general-purpose shortcut. It turns out that - since Python 2.6 - ast has a very handy helper method called "ast.parse()". This - in combination with ast.dump() - will let you very quickly figure out the correct ast pattern to use for a given bit of Python code. For example, here's how to figure out how to invoke an instance object method

005-Syntax by vicdunk
https://flic.kr/p/dUdQkm

Hopefully, that will be enough to get you going on your own Python ast adventures!