tag

#compiler

10 pages tagged

Deep Reads › CPython Internals › 3. Compilation Pipeline ›

19. Parsing

19. Parsing Parsing is the stage that turns a token stream into syntax structure. The tokenizer recognizes lexical units. The parser recognizes grammatical structure. It decides whether a sequence of tokens is a valid Python program and, when it is valid, builds an abstract syntax tree. For this source: def add(a, b): return a + b the tokenizer produces tokens such as: NAME "def" NAME "add" LPAR "(" NAME "a"...
cpython python internals compiler
Deep Reads › CPython Internals › 3. Compilation Pipeline ›

21. Symbol Tables

21. Symbol Tables After parsing produces an AST, CPython performs scope analysis. This stage builds symbol tables. A symbol table records how names behave inside each scope. It determines whether a name is local, global, free, cell, parameter, imported, annotated, or referenced from nested scopes. For this source: x = 10 def outer(): y = 20 def inner(): return x + y return inner the parser only knows that names...
cpython python internals compiler
Deep Reads › CPython Internals › 3. Compilation Pipeline ›

25. Bytecode Generation

25. Bytecode Generation Bytecode generation is the stage where CPython transforms structured syntax into executable virtual machine instructions. The parser builds an AST. The symbol table determines scope behavior. The compiler then emits bytecode instructions that implement Python semantics. For this source: def add(a, b): return a + b CPython generates bytecode shaped like: LOAD_FAST a LOAD_FAST b BINARY_OP + RETURN_VALUE The exact instruction names and formats vary across Python...
cpython python internals compiler
Deep Reads › CPython Internals › 3. Compilation Pipeline ›

20. The AST

20. The AST The abstract syntax tree, usually called the AST, is the structured representation of Python source code after parsing. The tokenizer produces a flat stream of tokens. The parser recognizes grammar. The AST records the result as a tree of statements and expressions. For this source: x = 1 + 2 the AST is shaped like this: Module Assign targets: Name id="x", ctx=Store value: BinOp left: Constant value=1...
cpython python internals compiler
Deep Reads › CPython Internals › 6. Standard Library Interfaces to the Core ›

54. `dis`

54. dis The dis module exposes CPython bytecode in a readable form. It is the standard tool for seeing what CPython actually executes after source code has passed through the compiler. Python source code is not executed directly. CPython compiles it into code objects. Each code object contains bytecode, constants, names, local variable metadata, exception handling metadata, and source position information. The dis module reads those code objects and formats...
cpython python internals compiler
Deep Reads › CPython Internals › 3. Compilation Pipeline ›

23. Code Objects

23. Code Objects A code object is CPython’s compiled representation of executable Python code. It contains bytecode and metadata. The interpreter can execute it, but the code object itself does not carry runtime state such as globals, default arguments, closure cells, or bound methods. For this source: def add(a, b): return a + b the function object add contains a code object: code = add.__code__ print(code.co_name) print(code.co_varnames) print(code.co_consts) print(code.co_names) The...
cpython python internals compiler
Deep Reads › CPython Internals › 3. Compilation Pipeline ›

24. Constants, Names, and Locals

24. Constants, Names, and Locals A code object does not store Python source code as text. It stores compact tables and bytecode instructions that refer to those tables by index. Three of the most important tables are: co_consts co_names co_varnames They separate repeated data from the instruction stream. For this function: def f(a): b = len(a) return b + 1 CPython stores: constants: None 1 names: len local variables: a...
cpython python internals compiler
Deep Reads › CPython Internals › 3. Compilation Pipeline ›

22. Compiler Passes

22. Compiler Passes Compiler passes turn the AST and symbol table information into an executable code object. Earlier stages answer structural questions. tokenizer: What lexical units are in the source? parser: What syntax tree do these tokens form? symbol table: What scope does each name belong to? The compiler answers execution questions. Which bytecode instructions should be emitted? Which constants belong in co_consts? Which names belong in co_names? Which local...
cpython python internals compiler
Deep Reads › CPython Internals › 3. Compilation Pipeline ›

26. Bytecode Optimization

26. Bytecode Optimization Bytecode optimization is the final cleanup and refinement stage before a code object is ready for execution. Earlier compiler stages decide what the program means and emit instructions. Optimization tries to make those instructions smaller, simpler, or faster while preserving Python semantics. For example: x = 1 + 2 can compile as if written: x = 3 because both operands are constants and integer addition has no...
cpython python internals compiler
Deep Reads › CPython Internals › 3. Compilation Pipeline ›

18. Tokenization

18. Tokenization Tokenization is the first structural stage in CPython’s compilation pipeline. It receives Python source text and produces a stream of tokens. The parser consumes that token stream and builds syntax structure from it. At this stage, CPython does not yet know whether a program is meaningful. It only recognizes lexical units: names, numbers, strings, operators, newlines, indentation, and end-of-file markers. The tokenizer turns this: def add(a, b): return...
cpython python internals compiler