Skip to content

Commit 4721b7d

Browse files
TheOnlyErrorPluriscientyouknowoneDimitrisJimfanninpm
authored
Architecture overview (RustPython#2528)
* Add architecture document --------- Co-authored-by: Eric van der Toorn <[email protected]> Co-authored-by: Pluriscient <[email protected]> Co-authored-by: Jeong YunWon <[email protected]> Co-authored-by: DimitrisJim <[email protected]> Co-authored-by: fanninpm <[email protected]>
1 parent 06c6244 commit 4721b7d

File tree

3 files changed

+152
-0
lines changed

3 files changed

+152
-0
lines changed

‎README.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -205,6 +205,8 @@ cargo doc --no-deps --all # Excluding all dependencies
205205
Documentation HTML files can then be found in the `target/doc` directory or you can append `--open` to the previous commands to
206206
have the documentation open automatically on your default browser.
207207

208+
For a high level overview of the components, see the [architecture](architecture/architecture.md) document.
209+
208210
## Contributing
209211

210212
Contributions are more than welcome, and in many cases we are happy to guide

‎architecture/architecture.md

Lines changed: 150 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,150 @@
1+
# Architecture
2+
3+
This document contains a high-level architectural overview of RustPython, thus it's very well-suited to get to know [the codebase][1].
4+
5+
RustPython is an Open Source (MIT-licensed) Python 3 interpreter written in Rust, available as both a library and a shell environment. Using Rust to implement the Python interpreter enables Python to be used as a programming language for Rust applications. Moreover, it allows Python to be immediately compiled in the browser using WebAssembly, meaning that anyone could easily run their Python code in the browser. For a more detailed introduction to RustPython, have a look at [this blog post][2].
6+
7+
RustPython consists of several components which are described in the section below. Take a look at [this video][3] for a brief walk-through of the components of RustPython. For a more elaborate introduction to one of these components, the parser, see [this blog post][4] for more information.
8+
9+
Have a look at these websites for a demo of RustPython running in the browser using WebAssembly:
10+
11+
- [The demo page.][5]
12+
- [The RustPython Notebook, a toy notebook inspired by the Iodide project.][6]
13+
14+
If, after reading this, you want to contribute to RustPython, take a look at these sources to get to know how and where to start:
15+
16+
- [RustPython Development Guide and Tips][7]
17+
- [How to contribute to RustPython using CPython's unit tests][8]
18+
19+
## Bird's eye view
20+
21+
A high-level overview of the workings of RustPython is visible in the figure below, showing how Python source files are interpreted.
22+
23+
![overview.png](overview.png)
24+
25+
Main architecture of RustPython.
26+
27+
The RustPython interpreter can be decoupled into three distinct components: the parser, compiler and VM.
28+
29+
1. The parser is responsible for converting the source code into tokens, and deriving an Abstract Syntax Tree (AST) from it.
30+
2. The compiler converts the generated AST to bytecode.
31+
3. The VM then executes the bytecode given user supplied input parameters and returns its result.
32+
33+
## Entry points
34+
35+
The main entry point of RustPython is located in `src/main.rs` and simply forwards a call to `run`, located in [`src/lib.rs`][9]. This method will call the compiler, which in turn will call the parser, and pass the compiled bytecode to the VM.
36+
37+
For each of the three components, the entry point is as follows:
38+
39+
- Parser: The Parser is located in a separate project, [RustPython/Parser][10]. See the documentation there for more information.
40+
- Compiler: `compile`, located in [`vm/src/vm/compile.rs`][11], this eventually forwards a call to [`compiler::compile`][12].
41+
- VM: `run_code_obj`, located in [`vm/src/vm/mod.rs`][13]. This creates a new frame in which the bytecode is executed.
42+
43+
## Components
44+
45+
Here we give a brief overview of each component and its function. For more details for the separate crates please take a look at their respective READMEs.
46+
47+
### Compiler
48+
49+
This component, implemented as the `rustpython-compiler/` package, is responsible for translating a Python source file into its equivalent bytecode representation. As an example, the following Python file:
50+
51+
```python
52+
def f(x):
53+
return x + 1
54+
```
55+
56+
Is compiled to the following bytecode:
57+
58+
```python
59+
2 0 LoadFast (0, x)
60+
1 LoadConst (1)
61+
2 BinaryOperation (Add)
62+
3 ReturnValue
63+
```
64+
65+
Note that bytecode is subject to change, and is _not_ a stable interface.
66+
67+
#### Parser
68+
69+
The Parser is the main sub-component of the compiler. All the functionality required for parsing Python sourcecode to an abstract syntax tree (AST) is implemented here:generator.
70+
71+
1. Lexical Analysis
72+
2. Parsing
73+
74+
The functionality for parsing resides in the RustPython/Parser project. See the documentation there for more information.
75+
76+
### VM
77+
78+
The Virtual Machine (VM) is responsible for executing the bytecode generated by the compiler. It is implemented in the `rustpython-vm/` package. The VM is currently implemented as a stack machine, meaning that it uses a stack to store intermediate results. In the `rustpython-vm/` package, additional sub-components are present, for example:
79+
80+
- builtins: the built in objects of Python, such as `int` and `str`.
81+
- stdlib: parts of the standard library that contains built-in modules needed for the VM to function, such as `sys`.
82+
83+
## Additional packages
84+
85+
### common
86+
87+
The `rustpython-common` package contains functionality that is not directly coupled to one of the other RustPython packages. For example, the [`float_ops.rs`][14] file contains operations on floating point numbers
88+
which could be used by other projects if needed.
89+
90+
### derive and derive-impl
91+
92+
Rust language extensions and macros specific to RustPython. Here we can find the definition of `PyModule` and `PyClass` along with useful macros like `py_compile!`.
93+
94+
### jit
95+
96+
This folder contains a _very_ experimental JIT implementation.
97+
98+
### stdlib
99+
100+
Part of the Python standard library that's implemented in Rust. The modules that live here are ones that aren't needed for the VM to function, but are useful for the user. For example, the `random` module is implemented here.
101+
102+
### Lib
103+
104+
Python side of the standard libary, copied over (with care) from CPython sourcecode.
105+
106+
#### Lib/test
107+
108+
CPython test suite, which can be used to compare with CPython in terms of conformance.
109+
Many of these files have been modified to fit with the current state of RustPython (when they were added), in one of three ways:
110+
111+
- The test has been commented out completely if the parser could not create a valid code object. If a file is unable to be parsed
112+
the test suite would not be able to run at all.
113+
- A test has been marked as `unittest.skip("TODO: RustPython <reason>")` if it led to a crash of RustPython. Adding a reason
114+
is useful to know why the test was skipped but not mandatory.
115+
- A test has been marked as `unittest.expectedFailure` with a `TODO: RustPython <reason>` comment left on top of the decorator. This decorator is used if the test can run but the result differs from what is expected.
116+
117+
Note: This is a recommended route to starting with contributing. To get started please take a look [this blog post][15].
118+
119+
### src
120+
121+
The RustPython executable/REPL (Read-Eval-Print-Loop) is implemented here, which is the interface through which users come in contact with library.
122+
Some things to note:
123+
124+
- The CLI is defined in the [`run` function of `src/lib.rs`][16].
125+
- The interface and helper for the REPL are defined in this package, but the actual REPL can be found in `vm/src/readline.rs`
126+
127+
### WASM
128+
129+
Crate for WebAssembly build, which compiles the RustPython package to a format that can be run on any modern browser.
130+
131+
### extra_tests
132+
133+
Integration and snippets that test for additional edge-cases, implementation specific functionality and bug report snippets.
134+
135+
[1]: https://github.com/RustPython/RustPython
136+
[2]: https://2021.desosa.nl/projects/rustpython/posts/vision/
137+
[3]: https://www.youtube.com/watch?v=nJDY9ASuiLc&t=213s
138+
[4]: https://rustpython.github.io/2020/04/02/thing-explainer-parser.html
139+
[5]: https://rustpython.github.io/demo/
140+
[6]: https://rustpython.github.io/demo/notebook/
141+
[7]: https://github.com/RustPython/RustPython/blob/master/DEVELOPMENT.md
142+
[8]: https://rustpython.github.io/guideline/2020/04/04/how-to-contribute-by-cpython-unittest.html
143+
[9]: https://github.com/RustPython/RustPython/blob/0e24cf48c63ae4ca9f829e88142a987cab3a9966/src/lib.rs#L63
144+
[10]: https://github.com/RustPython/Parser
145+
[11]: https://github.com/RustPython/RustPython/blob/0e24cf48c63ae4ca9f829e88142a987cab3a9966/vm/src/vm/compile.rs#LL10C17-L10C17
146+
[12]: https://github.com/RustPython/RustPython/blob/0e24cf48c63ae4ca9f829e88142a987cab3a9966/vm/src/vm/compile.rs#L26
147+
[13]: https://github.com/RustPython/RustPython/blob/0e24cf48c63ae4ca9f829e88142a987cab3a9966/vm/src/vm/mod.rs#L356
148+
[14]: https://github.com/RustPython/RustPython/blob/0e24cf48c63ae4ca9f829e88142a987cab3a9966/common/src/float_ops.rs
149+
[15]: https://rustpython.github.io/guideline/2020/04/04/how-to-contribute-by-cpython-unittest.html
150+
[16]: https://github.com/RustPython/RustPython/blob/0e24cf48c63ae4ca9f829e88142a987cab3a9966/src/lib.rs#L63

‎architecture/overview.png

34.4 KB
Loading

0 commit comments

Comments
 (0)