|
1 |
|
2 .. _compiler: |
|
3 |
|
4 *********************** |
|
5 Python compiler package |
|
6 *********************** |
|
7 |
|
8 .. deprecated:: 2.6 |
|
9 The :mod:`compiler` package has been removed in Python 3.0. |
|
10 |
|
11 .. sectionauthor:: Jeremy Hylton <jeremy@zope.com> |
|
12 |
|
13 |
|
14 The Python compiler package is a tool for analyzing Python source code and |
|
15 generating Python bytecode. The compiler contains libraries to generate an |
|
16 abstract syntax tree from Python source code and to generate Python |
|
17 :term:`bytecode` from the tree. |
|
18 |
|
19 The :mod:`compiler` package is a Python source to bytecode translator written in |
|
20 Python. It uses the built-in parser and standard :mod:`parser` module to |
|
21 generated a concrete syntax tree. This tree is used to generate an abstract |
|
22 syntax tree (AST) and then Python bytecode. |
|
23 |
|
24 The full functionality of the package duplicates the builtin compiler provided |
|
25 with the Python interpreter. It is intended to match its behavior almost |
|
26 exactly. Why implement another compiler that does the same thing? The package |
|
27 is useful for a variety of purposes. It can be modified more easily than the |
|
28 builtin compiler. The AST it generates is useful for analyzing Python source |
|
29 code. |
|
30 |
|
31 This chapter explains how the various components of the :mod:`compiler` package |
|
32 work. It blends reference material with a tutorial. |
|
33 |
|
34 |
|
35 The basic interface |
|
36 =================== |
|
37 |
|
38 .. module:: compiler |
|
39 :synopsis: Python code compiler written in Python. |
|
40 :deprecated: |
|
41 |
|
42 |
|
43 The top-level of the package defines four functions. If you import |
|
44 :mod:`compiler`, you will get these functions and a collection of modules |
|
45 contained in the package. |
|
46 |
|
47 |
|
48 .. function:: parse(buf) |
|
49 |
|
50 Returns an abstract syntax tree for the Python source code in *buf*. The |
|
51 function raises :exc:`SyntaxError` if there is an error in the source code. The |
|
52 return value is a :class:`compiler.ast.Module` instance that contains the tree. |
|
53 |
|
54 |
|
55 .. function:: parseFile(path) |
|
56 |
|
57 Return an abstract syntax tree for the Python source code in the file specified |
|
58 by *path*. It is equivalent to ``parse(open(path).read())``. |
|
59 |
|
60 |
|
61 .. function:: walk(ast, visitor[, verbose]) |
|
62 |
|
63 Do a pre-order walk over the abstract syntax tree *ast*. Call the appropriate |
|
64 method on the *visitor* instance for each node encountered. |
|
65 |
|
66 |
|
67 .. function:: compile(source, filename, mode, flags=None, dont_inherit=None) |
|
68 |
|
69 Compile the string *source*, a Python module, statement or expression, into a |
|
70 code object that can be executed by the exec statement or :func:`eval`. This |
|
71 function is a replacement for the built-in :func:`compile` function. |
|
72 |
|
73 The *filename* will be used for run-time error messages. |
|
74 |
|
75 The *mode* must be 'exec' to compile a module, 'single' to compile a single |
|
76 (interactive) statement, or 'eval' to compile an expression. |
|
77 |
|
78 The *flags* and *dont_inherit* arguments affect future-related statements, but |
|
79 are not supported yet. |
|
80 |
|
81 |
|
82 .. function:: compileFile(source) |
|
83 |
|
84 Compiles the file *source* and generates a .pyc file. |
|
85 |
|
86 The :mod:`compiler` package contains the following modules: :mod:`ast`, |
|
87 :mod:`consts`, :mod:`future`, :mod:`misc`, :mod:`pyassem`, :mod:`pycodegen`, |
|
88 :mod:`symbols`, :mod:`transformer`, and :mod:`visitor`. |
|
89 |
|
90 |
|
91 Limitations |
|
92 =========== |
|
93 |
|
94 There are some problems with the error checking of the compiler package. The |
|
95 interpreter detects syntax errors in two distinct phases. One set of errors is |
|
96 detected by the interpreter's parser, the other set by the compiler. The |
|
97 compiler package relies on the interpreter's parser, so it get the first phases |
|
98 of error checking for free. It implements the second phase itself, and that |
|
99 implementation is incomplete. For example, the compiler package does not raise |
|
100 an error if a name appears more than once in an argument list: ``def f(x, x): |
|
101 ...`` |
|
102 |
|
103 A future version of the compiler should fix these problems. |
|
104 |
|
105 |
|
106 Python Abstract Syntax |
|
107 ====================== |
|
108 |
|
109 The :mod:`compiler.ast` module defines an abstract syntax for Python. In the |
|
110 abstract syntax tree, each node represents a syntactic construct. The root of |
|
111 the tree is :class:`Module` object. |
|
112 |
|
113 The abstract syntax offers a higher level interface to parsed Python source |
|
114 code. The :mod:`parser` module and the compiler written in C for the Python |
|
115 interpreter use a concrete syntax tree. The concrete syntax is tied closely to |
|
116 the grammar description used for the Python parser. Instead of a single node |
|
117 for a construct, there are often several levels of nested nodes that are |
|
118 introduced by Python's precedence rules. |
|
119 |
|
120 The abstract syntax tree is created by the :mod:`compiler.transformer` module. |
|
121 The transformer relies on the builtin Python parser to generate a concrete |
|
122 syntax tree. It generates an abstract syntax tree from the concrete tree. |
|
123 |
|
124 .. index:: |
|
125 single: Stein, Greg |
|
126 single: Tutt, Bill |
|
127 |
|
128 The :mod:`transformer` module was created by Greg Stein and Bill Tutt for an |
|
129 experimental Python-to-C compiler. The current version contains a number of |
|
130 modifications and improvements, but the basic form of the abstract syntax and of |
|
131 the transformer are due to Stein and Tutt. |
|
132 |
|
133 |
|
134 AST Nodes |
|
135 --------- |
|
136 |
|
137 .. module:: compiler.ast |
|
138 |
|
139 |
|
140 The :mod:`compiler.ast` module is generated from a text file that describes each |
|
141 node type and its elements. Each node type is represented as a class that |
|
142 inherits from the abstract base class :class:`compiler.ast.Node` and defines a |
|
143 set of named attributes for child nodes. |
|
144 |
|
145 |
|
146 .. class:: Node() |
|
147 |
|
148 The :class:`Node` instances are created automatically by the parser generator. |
|
149 The recommended interface for specific :class:`Node` instances is to use the |
|
150 public attributes to access child nodes. A public attribute may be bound to a |
|
151 single node or to a sequence of nodes, depending on the :class:`Node` type. For |
|
152 example, the :attr:`bases` attribute of the :class:`Class` node, is bound to a |
|
153 list of base class nodes, and the :attr:`doc` attribute is bound to a single |
|
154 node. |
|
155 |
|
156 Each :class:`Node` instance has a :attr:`lineno` attribute which may be |
|
157 ``None``. XXX Not sure what the rules are for which nodes will have a useful |
|
158 lineno. |
|
159 |
|
160 All :class:`Node` objects offer the following methods: |
|
161 |
|
162 |
|
163 .. method:: getChildren() |
|
164 |
|
165 Returns a flattened list of the child nodes and objects in the order they |
|
166 occur. Specifically, the order of the nodes is the order in which they |
|
167 appear in the Python grammar. Not all of the children are :class:`Node` |
|
168 instances. The names of functions and classes, for example, are plain |
|
169 strings. |
|
170 |
|
171 |
|
172 .. method:: getChildNodes() |
|
173 |
|
174 Returns a flattened list of the child nodes in the order they occur. This |
|
175 method is like :meth:`getChildren`, except that it only returns those |
|
176 children that are :class:`Node` instances. |
|
177 |
|
178 |
|
179 Two examples illustrate the general structure of :class:`Node` classes. The |
|
180 :keyword:`while` statement is defined by the following grammar production:: |
|
181 |
|
182 while_stmt: "while" expression ":" suite |
|
183 ["else" ":" suite] |
|
184 |
|
185 The :class:`While` node has three attributes: :attr:`test`, :attr:`body`, and |
|
186 :attr:`else_`. (If the natural name for an attribute is also a Python reserved |
|
187 word, it can't be used as an attribute name. An underscore is appended to the |
|
188 word to make it a legal identifier, hence :attr:`else_` instead of |
|
189 :keyword:`else`.) |
|
190 |
|
191 The :keyword:`if` statement is more complicated because it can include several |
|
192 tests. :: |
|
193 |
|
194 if_stmt: 'if' test ':' suite ('elif' test ':' suite)* ['else' ':' suite] |
|
195 |
|
196 The :class:`If` node only defines two attributes: :attr:`tests` and |
|
197 :attr:`else_`. The :attr:`tests` attribute is a sequence of test expression, |
|
198 consequent body pairs. There is one pair for each :keyword:`if`/:keyword:`elif` |
|
199 clause. The first element of the pair is the test expression. The second |
|
200 elements is a :class:`Stmt` node that contains the code to execute if the test |
|
201 is true. |
|
202 |
|
203 The :meth:`getChildren` method of :class:`If` returns a flat list of child |
|
204 nodes. If there are three :keyword:`if`/:keyword:`elif` clauses and no |
|
205 :keyword:`else` clause, then :meth:`getChildren` will return a list of six |
|
206 elements: the first test expression, the first :class:`Stmt`, the second text |
|
207 expression, etc. |
|
208 |
|
209 The following table lists each of the :class:`Node` subclasses defined in |
|
210 :mod:`compiler.ast` and each of the public attributes available on their |
|
211 instances. The values of most of the attributes are themselves :class:`Node` |
|
212 instances or sequences of instances. When the value is something other than an |
|
213 instance, the type is noted in the comment. The attributes are listed in the |
|
214 order in which they are returned by :meth:`getChildren` and |
|
215 :meth:`getChildNodes`. |
|
216 |
|
217 +-----------------------+--------------------+---------------------------------+ |
|
218 | Node type | Attribute | Value | |
|
219 +=======================+====================+=================================+ |
|
220 | :class:`Add` | :attr:`left` | left operand | |
|
221 +-----------------------+--------------------+---------------------------------+ |
|
222 | | :attr:`right` | right operand | |
|
223 +-----------------------+--------------------+---------------------------------+ |
|
224 | :class:`And` | :attr:`nodes` | list of operands | |
|
225 +-----------------------+--------------------+---------------------------------+ |
|
226 | :class:`AssAttr` | | *attribute as target of | |
|
227 | | | assignment* | |
|
228 +-----------------------+--------------------+---------------------------------+ |
|
229 | | :attr:`expr` | expression on the left-hand | |
|
230 | | | side of the dot | |
|
231 +-----------------------+--------------------+---------------------------------+ |
|
232 | | :attr:`attrname` | the attribute name, a string | |
|
233 +-----------------------+--------------------+---------------------------------+ |
|
234 | | :attr:`flags` | XXX | |
|
235 +-----------------------+--------------------+---------------------------------+ |
|
236 | :class:`AssList` | :attr:`nodes` | list of list elements being | |
|
237 | | | assigned to | |
|
238 +-----------------------+--------------------+---------------------------------+ |
|
239 | :class:`AssName` | :attr:`name` | name being assigned to | |
|
240 +-----------------------+--------------------+---------------------------------+ |
|
241 | | :attr:`flags` | XXX | |
|
242 +-----------------------+--------------------+---------------------------------+ |
|
243 | :class:`AssTuple` | :attr:`nodes` | list of tuple elements being | |
|
244 | | | assigned to | |
|
245 +-----------------------+--------------------+---------------------------------+ |
|
246 | :class:`Assert` | :attr:`test` | the expression to be tested | |
|
247 +-----------------------+--------------------+---------------------------------+ |
|
248 | | :attr:`fail` | the value of the | |
|
249 | | | :exc:`AssertionError` | |
|
250 +-----------------------+--------------------+---------------------------------+ |
|
251 | :class:`Assign` | :attr:`nodes` | a list of assignment targets, | |
|
252 | | | one per equal sign | |
|
253 +-----------------------+--------------------+---------------------------------+ |
|
254 | | :attr:`expr` | the value being assigned | |
|
255 +-----------------------+--------------------+---------------------------------+ |
|
256 | :class:`AugAssign` | :attr:`node` | | |
|
257 +-----------------------+--------------------+---------------------------------+ |
|
258 | | :attr:`op` | | |
|
259 +-----------------------+--------------------+---------------------------------+ |
|
260 | | :attr:`expr` | | |
|
261 +-----------------------+--------------------+---------------------------------+ |
|
262 | :class:`Backquote` | :attr:`expr` | | |
|
263 +-----------------------+--------------------+---------------------------------+ |
|
264 | :class:`Bitand` | :attr:`nodes` | | |
|
265 +-----------------------+--------------------+---------------------------------+ |
|
266 | :class:`Bitor` | :attr:`nodes` | | |
|
267 +-----------------------+--------------------+---------------------------------+ |
|
268 | :class:`Bitxor` | :attr:`nodes` | | |
|
269 +-----------------------+--------------------+---------------------------------+ |
|
270 | :class:`Break` | | | |
|
271 +-----------------------+--------------------+---------------------------------+ |
|
272 | :class:`CallFunc` | :attr:`node` | expression for the callee | |
|
273 +-----------------------+--------------------+---------------------------------+ |
|
274 | | :attr:`args` | a list of arguments | |
|
275 +-----------------------+--------------------+---------------------------------+ |
|
276 | | :attr:`star_args` | the extended \*-arg value | |
|
277 +-----------------------+--------------------+---------------------------------+ |
|
278 | | :attr:`dstar_args` | the extended \*\*-arg value | |
|
279 +-----------------------+--------------------+---------------------------------+ |
|
280 | :class:`Class` | :attr:`name` | the name of the class, a string | |
|
281 +-----------------------+--------------------+---------------------------------+ |
|
282 | | :attr:`bases` | a list of base classes | |
|
283 +-----------------------+--------------------+---------------------------------+ |
|
284 | | :attr:`doc` | doc string, a string or | |
|
285 | | | ``None`` | |
|
286 +-----------------------+--------------------+---------------------------------+ |
|
287 | | :attr:`code` | the body of the class statement | |
|
288 +-----------------------+--------------------+---------------------------------+ |
|
289 | :class:`Compare` | :attr:`expr` | | |
|
290 +-----------------------+--------------------+---------------------------------+ |
|
291 | | :attr:`ops` | | |
|
292 +-----------------------+--------------------+---------------------------------+ |
|
293 | :class:`Const` | :attr:`value` | | |
|
294 +-----------------------+--------------------+---------------------------------+ |
|
295 | :class:`Continue` | | | |
|
296 +-----------------------+--------------------+---------------------------------+ |
|
297 | :class:`Decorators` | :attr:`nodes` | List of function decorator | |
|
298 | | | expressions | |
|
299 +-----------------------+--------------------+---------------------------------+ |
|
300 | :class:`Dict` | :attr:`items` | | |
|
301 +-----------------------+--------------------+---------------------------------+ |
|
302 | :class:`Discard` | :attr:`expr` | | |
|
303 +-----------------------+--------------------+---------------------------------+ |
|
304 | :class:`Div` | :attr:`left` | | |
|
305 +-----------------------+--------------------+---------------------------------+ |
|
306 | | :attr:`right` | | |
|
307 +-----------------------+--------------------+---------------------------------+ |
|
308 | :class:`Ellipsis` | | | |
|
309 +-----------------------+--------------------+---------------------------------+ |
|
310 | :class:`Expression` | :attr:`node` | | |
|
311 +-----------------------+--------------------+---------------------------------+ |
|
312 | :class:`Exec` | :attr:`expr` | | |
|
313 +-----------------------+--------------------+---------------------------------+ |
|
314 | | :attr:`locals` | | |
|
315 +-----------------------+--------------------+---------------------------------+ |
|
316 | | :attr:`globals` | | |
|
317 +-----------------------+--------------------+---------------------------------+ |
|
318 | :class:`FloorDiv` | :attr:`left` | | |
|
319 +-----------------------+--------------------+---------------------------------+ |
|
320 | | :attr:`right` | | |
|
321 +-----------------------+--------------------+---------------------------------+ |
|
322 | :class:`For` | :attr:`assign` | | |
|
323 +-----------------------+--------------------+---------------------------------+ |
|
324 | | :attr:`list` | | |
|
325 +-----------------------+--------------------+---------------------------------+ |
|
326 | | :attr:`body` | | |
|
327 +-----------------------+--------------------+---------------------------------+ |
|
328 | | :attr:`else_` | | |
|
329 +-----------------------+--------------------+---------------------------------+ |
|
330 | :class:`From` | :attr:`modname` | | |
|
331 +-----------------------+--------------------+---------------------------------+ |
|
332 | | :attr:`names` | | |
|
333 +-----------------------+--------------------+---------------------------------+ |
|
334 | :class:`Function` | :attr:`decorators` | :class:`Decorators` or ``None`` | |
|
335 +-----------------------+--------------------+---------------------------------+ |
|
336 | | :attr:`name` | name used in def, a string | |
|
337 +-----------------------+--------------------+---------------------------------+ |
|
338 | | :attr:`argnames` | list of argument names, as | |
|
339 | | | strings | |
|
340 +-----------------------+--------------------+---------------------------------+ |
|
341 | | :attr:`defaults` | list of default values | |
|
342 +-----------------------+--------------------+---------------------------------+ |
|
343 | | :attr:`flags` | xxx | |
|
344 +-----------------------+--------------------+---------------------------------+ |
|
345 | | :attr:`doc` | doc string, a string or | |
|
346 | | | ``None`` | |
|
347 +-----------------------+--------------------+---------------------------------+ |
|
348 | | :attr:`code` | the body of the function | |
|
349 +-----------------------+--------------------+---------------------------------+ |
|
350 | :class:`GenExpr` | :attr:`code` | | |
|
351 +-----------------------+--------------------+---------------------------------+ |
|
352 | :class:`GenExprFor` | :attr:`assign` | | |
|
353 +-----------------------+--------------------+---------------------------------+ |
|
354 | | :attr:`iter` | | |
|
355 +-----------------------+--------------------+---------------------------------+ |
|
356 | | :attr:`ifs` | | |
|
357 +-----------------------+--------------------+---------------------------------+ |
|
358 | :class:`GenExprIf` | :attr:`test` | | |
|
359 +-----------------------+--------------------+---------------------------------+ |
|
360 | :class:`GenExprInner` | :attr:`expr` | | |
|
361 +-----------------------+--------------------+---------------------------------+ |
|
362 | | :attr:`quals` | | |
|
363 +-----------------------+--------------------+---------------------------------+ |
|
364 | :class:`Getattr` | :attr:`expr` | | |
|
365 +-----------------------+--------------------+---------------------------------+ |
|
366 | | :attr:`attrname` | | |
|
367 +-----------------------+--------------------+---------------------------------+ |
|
368 | :class:`Global` | :attr:`names` | | |
|
369 +-----------------------+--------------------+---------------------------------+ |
|
370 | :class:`If` | :attr:`tests` | | |
|
371 +-----------------------+--------------------+---------------------------------+ |
|
372 | | :attr:`else_` | | |
|
373 +-----------------------+--------------------+---------------------------------+ |
|
374 | :class:`Import` | :attr:`names` | | |
|
375 +-----------------------+--------------------+---------------------------------+ |
|
376 | :class:`Invert` | :attr:`expr` | | |
|
377 +-----------------------+--------------------+---------------------------------+ |
|
378 | :class:`Keyword` | :attr:`name` | | |
|
379 +-----------------------+--------------------+---------------------------------+ |
|
380 | | :attr:`expr` | | |
|
381 +-----------------------+--------------------+---------------------------------+ |
|
382 | :class:`Lambda` | :attr:`argnames` | | |
|
383 +-----------------------+--------------------+---------------------------------+ |
|
384 | | :attr:`defaults` | | |
|
385 +-----------------------+--------------------+---------------------------------+ |
|
386 | | :attr:`flags` | | |
|
387 +-----------------------+--------------------+---------------------------------+ |
|
388 | | :attr:`code` | | |
|
389 +-----------------------+--------------------+---------------------------------+ |
|
390 | :class:`LeftShift` | :attr:`left` | | |
|
391 +-----------------------+--------------------+---------------------------------+ |
|
392 | | :attr:`right` | | |
|
393 +-----------------------+--------------------+---------------------------------+ |
|
394 | :class:`List` | :attr:`nodes` | | |
|
395 +-----------------------+--------------------+---------------------------------+ |
|
396 | :class:`ListComp` | :attr:`expr` | | |
|
397 +-----------------------+--------------------+---------------------------------+ |
|
398 | | :attr:`quals` | | |
|
399 +-----------------------+--------------------+---------------------------------+ |
|
400 | :class:`ListCompFor` | :attr:`assign` | | |
|
401 +-----------------------+--------------------+---------------------------------+ |
|
402 | | :attr:`list` | | |
|
403 +-----------------------+--------------------+---------------------------------+ |
|
404 | | :attr:`ifs` | | |
|
405 +-----------------------+--------------------+---------------------------------+ |
|
406 | :class:`ListCompIf` | :attr:`test` | | |
|
407 +-----------------------+--------------------+---------------------------------+ |
|
408 | :class:`Mod` | :attr:`left` | | |
|
409 +-----------------------+--------------------+---------------------------------+ |
|
410 | | :attr:`right` | | |
|
411 +-----------------------+--------------------+---------------------------------+ |
|
412 | :class:`Module` | :attr:`doc` | doc string, a string or | |
|
413 | | | ``None`` | |
|
414 +-----------------------+--------------------+---------------------------------+ |
|
415 | | :attr:`node` | body of the module, a | |
|
416 | | | :class:`Stmt` | |
|
417 +-----------------------+--------------------+---------------------------------+ |
|
418 | :class:`Mul` | :attr:`left` | | |
|
419 +-----------------------+--------------------+---------------------------------+ |
|
420 | | :attr:`right` | | |
|
421 +-----------------------+--------------------+---------------------------------+ |
|
422 | :class:`Name` | :attr:`name` | | |
|
423 +-----------------------+--------------------+---------------------------------+ |
|
424 | :class:`Not` | :attr:`expr` | | |
|
425 +-----------------------+--------------------+---------------------------------+ |
|
426 | :class:`Or` | :attr:`nodes` | | |
|
427 +-----------------------+--------------------+---------------------------------+ |
|
428 | :class:`Pass` | | | |
|
429 +-----------------------+--------------------+---------------------------------+ |
|
430 | :class:`Power` | :attr:`left` | | |
|
431 +-----------------------+--------------------+---------------------------------+ |
|
432 | | :attr:`right` | | |
|
433 +-----------------------+--------------------+---------------------------------+ |
|
434 | :class:`Print` | :attr:`nodes` | | |
|
435 +-----------------------+--------------------+---------------------------------+ |
|
436 | | :attr:`dest` | | |
|
437 +-----------------------+--------------------+---------------------------------+ |
|
438 | :class:`Printnl` | :attr:`nodes` | | |
|
439 +-----------------------+--------------------+---------------------------------+ |
|
440 | | :attr:`dest` | | |
|
441 +-----------------------+--------------------+---------------------------------+ |
|
442 | :class:`Raise` | :attr:`expr1` | | |
|
443 +-----------------------+--------------------+---------------------------------+ |
|
444 | | :attr:`expr2` | | |
|
445 +-----------------------+--------------------+---------------------------------+ |
|
446 | | :attr:`expr3` | | |
|
447 +-----------------------+--------------------+---------------------------------+ |
|
448 | :class:`Return` | :attr:`value` | | |
|
449 +-----------------------+--------------------+---------------------------------+ |
|
450 | :class:`RightShift` | :attr:`left` | | |
|
451 +-----------------------+--------------------+---------------------------------+ |
|
452 | | :attr:`right` | | |
|
453 +-----------------------+--------------------+---------------------------------+ |
|
454 | :class:`Slice` | :attr:`expr` | | |
|
455 +-----------------------+--------------------+---------------------------------+ |
|
456 | | :attr:`flags` | | |
|
457 +-----------------------+--------------------+---------------------------------+ |
|
458 | | :attr:`lower` | | |
|
459 +-----------------------+--------------------+---------------------------------+ |
|
460 | | :attr:`upper` | | |
|
461 +-----------------------+--------------------+---------------------------------+ |
|
462 | :class:`Sliceobj` | :attr:`nodes` | list of statements | |
|
463 +-----------------------+--------------------+---------------------------------+ |
|
464 | :class:`Stmt` | :attr:`nodes` | | |
|
465 +-----------------------+--------------------+---------------------------------+ |
|
466 | :class:`Sub` | :attr:`left` | | |
|
467 +-----------------------+--------------------+---------------------------------+ |
|
468 | | :attr:`right` | | |
|
469 +-----------------------+--------------------+---------------------------------+ |
|
470 | :class:`Subscript` | :attr:`expr` | | |
|
471 +-----------------------+--------------------+---------------------------------+ |
|
472 | | :attr:`flags` | | |
|
473 +-----------------------+--------------------+---------------------------------+ |
|
474 | | :attr:`subs` | | |
|
475 +-----------------------+--------------------+---------------------------------+ |
|
476 | :class:`TryExcept` | :attr:`body` | | |
|
477 +-----------------------+--------------------+---------------------------------+ |
|
478 | | :attr:`handlers` | | |
|
479 +-----------------------+--------------------+---------------------------------+ |
|
480 | | :attr:`else_` | | |
|
481 +-----------------------+--------------------+---------------------------------+ |
|
482 | :class:`TryFinally` | :attr:`body` | | |
|
483 +-----------------------+--------------------+---------------------------------+ |
|
484 | | :attr:`final` | | |
|
485 +-----------------------+--------------------+---------------------------------+ |
|
486 | :class:`Tuple` | :attr:`nodes` | | |
|
487 +-----------------------+--------------------+---------------------------------+ |
|
488 | :class:`UnaryAdd` | :attr:`expr` | | |
|
489 +-----------------------+--------------------+---------------------------------+ |
|
490 | :class:`UnarySub` | :attr:`expr` | | |
|
491 +-----------------------+--------------------+---------------------------------+ |
|
492 | :class:`While` | :attr:`test` | | |
|
493 +-----------------------+--------------------+---------------------------------+ |
|
494 | | :attr:`body` | | |
|
495 +-----------------------+--------------------+---------------------------------+ |
|
496 | | :attr:`else_` | | |
|
497 +-----------------------+--------------------+---------------------------------+ |
|
498 | :class:`With` | :attr:`expr` | | |
|
499 +-----------------------+--------------------+---------------------------------+ |
|
500 | | :attr:`vars` | | |
|
501 +-----------------------+--------------------+---------------------------------+ |
|
502 | | :attr:`body` | | |
|
503 +-----------------------+--------------------+---------------------------------+ |
|
504 | :class:`Yield` | :attr:`value` | | |
|
505 +-----------------------+--------------------+---------------------------------+ |
|
506 |
|
507 |
|
508 Assignment nodes |
|
509 ---------------- |
|
510 |
|
511 There is a collection of nodes used to represent assignments. Each assignment |
|
512 statement in the source code becomes a single :class:`Assign` node in the AST. |
|
513 The :attr:`nodes` attribute is a list that contains a node for each assignment |
|
514 target. This is necessary because assignment can be chained, e.g. ``a = b = |
|
515 2``. Each :class:`Node` in the list will be one of the following classes: |
|
516 :class:`AssAttr`, :class:`AssList`, :class:`AssName`, or :class:`AssTuple`. |
|
517 |
|
518 Each target assignment node will describe the kind of object being assigned to: |
|
519 :class:`AssName` for a simple name, e.g. ``a = 1``. :class:`AssAttr` for an |
|
520 attribute assigned, e.g. ``a.x = 1``. :class:`AssList` and :class:`AssTuple` for |
|
521 list and tuple expansion respectively, e.g. ``a, b, c = a_tuple``. |
|
522 |
|
523 The target assignment nodes also have a :attr:`flags` attribute that indicates |
|
524 whether the node is being used for assignment or in a delete statement. The |
|
525 :class:`AssName` is also used to represent a delete statement, e.g. :class:`del |
|
526 x`. |
|
527 |
|
528 When an expression contains several attribute references, an assignment or |
|
529 delete statement will contain only one :class:`AssAttr` node -- for the final |
|
530 attribute reference. The other attribute references will be represented as |
|
531 :class:`Getattr` nodes in the :attr:`expr` attribute of the :class:`AssAttr` |
|
532 instance. |
|
533 |
|
534 |
|
535 Examples |
|
536 -------- |
|
537 |
|
538 This section shows several simple examples of ASTs for Python source code. The |
|
539 examples demonstrate how to use the :func:`parse` function, what the repr of an |
|
540 AST looks like, and how to access attributes of an AST node. |
|
541 |
|
542 The first module defines a single function. Assume it is stored in |
|
543 :file:`/tmp/doublelib.py`. :: |
|
544 |
|
545 """This is an example module. |
|
546 |
|
547 This is the docstring. |
|
548 """ |
|
549 |
|
550 def double(x): |
|
551 "Return twice the argument" |
|
552 return x * 2 |
|
553 |
|
554 In the interactive interpreter session below, I have reformatted the long AST |
|
555 reprs for readability. The AST reprs use unqualified class names. If you want |
|
556 to create an instance from a repr, you must import the class names from the |
|
557 :mod:`compiler.ast` module. :: |
|
558 |
|
559 >>> import compiler |
|
560 >>> mod = compiler.parseFile("/tmp/doublelib.py") |
|
561 >>> mod |
|
562 Module('This is an example module.\n\nThis is the docstring.\n', |
|
563 Stmt([Function(None, 'double', ['x'], [], 0, |
|
564 'Return twice the argument', |
|
565 Stmt([Return(Mul((Name('x'), Const(2))))]))])) |
|
566 >>> from compiler.ast import * |
|
567 >>> Module('This is an example module.\n\nThis is the docstring.\n', |
|
568 ... Stmt([Function(None, 'double', ['x'], [], 0, |
|
569 ... 'Return twice the argument', |
|
570 ... Stmt([Return(Mul((Name('x'), Const(2))))]))])) |
|
571 Module('This is an example module.\n\nThis is the docstring.\n', |
|
572 Stmt([Function(None, 'double', ['x'], [], 0, |
|
573 'Return twice the argument', |
|
574 Stmt([Return(Mul((Name('x'), Const(2))))]))])) |
|
575 >>> mod.doc |
|
576 'This is an example module.\n\nThis is the docstring.\n' |
|
577 >>> for node in mod.node.nodes: |
|
578 ... print node |
|
579 ... |
|
580 Function(None, 'double', ['x'], [], 0, 'Return twice the argument', |
|
581 Stmt([Return(Mul((Name('x'), Const(2))))])) |
|
582 >>> func = mod.node.nodes[0] |
|
583 >>> func.code |
|
584 Stmt([Return(Mul((Name('x'), Const(2))))]) |
|
585 |
|
586 |
|
587 Using Visitors to Walk ASTs |
|
588 =========================== |
|
589 |
|
590 .. module:: compiler.visitor |
|
591 |
|
592 |
|
593 The visitor pattern is ... The :mod:`compiler` package uses a variant on the |
|
594 visitor pattern that takes advantage of Python's introspection features to |
|
595 eliminate the need for much of the visitor's infrastructure. |
|
596 |
|
597 The classes being visited do not need to be programmed to accept visitors. The |
|
598 visitor need only define visit methods for classes it is specifically interested |
|
599 in; a default visit method can handle the rest. |
|
600 |
|
601 XXX The magic :meth:`visit` method for visitors. |
|
602 |
|
603 |
|
604 .. function:: walk(tree, visitor[, verbose]) |
|
605 |
|
606 |
|
607 .. class:: ASTVisitor() |
|
608 |
|
609 The :class:`ASTVisitor` is responsible for walking over the tree in the correct |
|
610 order. A walk begins with a call to :meth:`preorder`. For each node, it checks |
|
611 the *visitor* argument to :meth:`preorder` for a method named 'visitNodeType,' |
|
612 where NodeType is the name of the node's class, e.g. for a :class:`While` node a |
|
613 :meth:`visitWhile` would be called. If the method exists, it is called with the |
|
614 node as its first argument. |
|
615 |
|
616 The visitor method for a particular node type can control how child nodes are |
|
617 visited during the walk. The :class:`ASTVisitor` modifies the visitor argument |
|
618 by adding a visit method to the visitor; this method can be used to visit a |
|
619 particular child node. If no visitor is found for a particular node type, the |
|
620 :meth:`default` method is called. |
|
621 |
|
622 :class:`ASTVisitor` objects have the following methods: |
|
623 |
|
624 XXX describe extra arguments |
|
625 |
|
626 |
|
627 .. method:: default(node[, ...]) |
|
628 |
|
629 |
|
630 .. method:: dispatch(node[, ...]) |
|
631 |
|
632 |
|
633 .. method:: preorder(tree, visitor) |
|
634 |
|
635 |
|
636 Bytecode Generation |
|
637 =================== |
|
638 |
|
639 The code generator is a visitor that emits bytecodes. Each visit method can |
|
640 call the :meth:`emit` method to emit a new bytecode. The basic code generator |
|
641 is specialized for modules, classes, and functions. An assembler converts that |
|
642 emitted instructions to the low-level bytecode format. It handles things like |
|
643 generation of constant lists of code objects and calculation of jump offsets. |
|
644 |