diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md new file mode 100644 index 000000000000..4aefa8e11d91 --- /dev/null +++ b/ARCHITECTURE.md @@ -0,0 +1,81 @@ +# Rerun architecture +This document describes the technical architecture of Rerun. + +## See also +* [`BUILD.md`](BUILD.md) +* [`CODE_STYLE.md`](CODE_STYLE.md) +* [`CONTRIBUTING.md`](CONTRIBUTING.md) +* [`RELEASES.md`](RELEASES.md) + + +## The major components +### Logging APIs +It all starts with logging. You can log rich data (point clouds, images, etc) with either our Python SDK or our Rust SDK. + +The logging SDK:s encodes the data using Apache Arrow (see more below). + +The logging data can be written to disk as `.rrd` files, or transmitted over TCP to either a Rerun Viewer or a Rerun Server. + +### Rerun viewer +The Rerun Viewer is where log data is visualized. It is usually run as a native app, but can also be compiled to WebAssembly (Wasm) and run in a browser. + +#### Native viewer +The easiest way to launch the viewer is directly from the logging API with `rr.init("my_app", spawn=True)`. However, the standalone viewer can also be run from the command line, for example to view an `.rrd` file: `rerun mydata.rrd`. + +#### Web viewer +You can try running the viewer in a browser using `rr.serve()` in python, or using `rerun --web-viewer mydata.rrd`. + +The web viewer consists of just a few small files - a thin `.html`, a `.wasm` blob, and an auto-generated `.js` bridge for the wasm. These files are served using the [`re_web_server`](https://github.com/rerun-io/rerun/tree/main/crates/re_web_server) crate. + +The web viewer can load `.rrd` files (just drag-drop them into the browser), or read logging data streamed over WebSockets. + +### `.rrd` files +`.rrd` ("**R**e**r**un **D**ata") is just a bunch of log messages appended one after the other to a file. + +NOTE: `.rrd` files do not yet guarantee any backwards or forwards compatibility. One version of Rerun will likely not be able to open an `.rrd` file generated by another Rerun version. + + +## Technologies we use +### Apache Arrow +[Apache Arrow](https://arrow.apache.org/) is a language-independent columnar memory format for arbitrary data. We use it to encode the log data when transmitting it over the network or storing it in an `.rrd` file. We also use it in our in-RAM data store, [`re_arrow_store`](crates/re_arrow_store/README.md). + +In rust, we use the [`arrow2` crate](https://crates.io/crates/arrow2). + +### `wgpu` +The Rerun Viewer uses the [`wgpu`](https://github.com/gfx-rs/wgpu) graphics API. It provides a high-performance abstraction over Vulkan, Metal, D3D12, D3D11, OpenGLES, WebGL and [WebGPU](https://en.wikipedia.org/wiki/WebGPU). This lets us write the same code graphics code for native as for web. + +We use the WebGL backend when compiling for web. Once WebGPU is available in most browsers, we can easily switch to it for a nice performance boost! + +We have written our own high-level rendering crate on top of `wgpu`, called [`re_renderer`](crates/re_renderer/README.md). + +### `egui` +The GUI in the Rerun Viewer is using [`egui`](https://www.egui.rs/), a cross-platform, [immediate mode GUI](https://github.com/emilk/egui#why-immediate-mode). + +We use [`eframe`](https://github.com/emilk/egui/tree/master/crates/eframe), the egui framework, to run `egui` on both native and web. + + +### Wasm +Wasm (short for [WebAssembly](https://webassembly.org/)) is a binary instruction format supported by all major browser. +The Rerun Viewer can be compiled to Wasm and run in a browser. + +Threading support in Wasm is nascent, so care must we taken that we don't spawn any threads when compiling for `wasm32`. + +Wasm has no access to the host system, except via JS calls (something that may change once [WASI](https://wasi.dev/) rolls out), so when compiling for `wasm32` you can NOT use the Rust standard library to: +* Access files +* Read environment variables +* Get the current time (use [`instant`](https://crates.io/crates/instant) instead) +* Use networking (use [`ehttp`](https://github.com/emilk/ehttp), [`reqwest`](https://github.com/seanmonstar/reqwest), or [`ewebsock`](https://github.com/rerun-io/ewebsock) instead) +* etc + + +## Immediate mode +The Rerun Viewer uses an [immediate mode GUI](https://github.com/emilk/egui#why-immediate-mode), [`egui`](https://www.egui.rs/). This means that each frame the entire GUI is being laid out from scratch. + +In fact, the whole of the Rerun Viewer is written in an immediate mode style. Each rendered frame it will query the in-RAM data store, massage the results, and feed it to the renderer. + +The advantage of immediate mode is that is removes all state management. There is no callbacks that are called when some state has already changed, and the state of the blueprint is always in sync with what you see on screen. + +Immediate mode is also a forcing function, forcing us to relentlessly optimize our code. +This leads to a very responsive GUI, where there is no "hickups" when switching data source or doing time scrubbing. + +Of course, this will only take us so far. In the future we plan on caching queries and work submitted to the renderer so that we don't perform unnecessary work each frame. We also plan on doing larger operation in background threads. This will be necessary in order to support viewing large datasets, e.g. several million points. The plan is still to do so within an immediate mode framework, retaining most of the advantages of stateless code. diff --git a/BUILD.md b/BUILD.md new file mode 100644 index 000000000000..a86b15cb36bc --- /dev/null +++ b/BUILD.md @@ -0,0 +1,150 @@ +# Building Rerun +This is a guide to how to build Rerun. + + +## See also +* [`ARCHITECTURE.md`](ARCHITECTURE.md) +* [`CODE_STYLE.md`](CODE_STYLE.md) +* [`CONTRIBUTING.md`](CONTRIBUTING.md) +* [`RELEASES.md`](RELEASES.md) + + +## Getting started with the repository. +* Install the Rust toolchain: +* `git clone git@github.com:rerun-io/rerun.git && cd rerun` +* Run `./scripts/setup_dev.sh`. +* Make sure `cargo --version` prints `1.67.0` once you are done + + +### Apple-silicon Macs + +If you are using an Apple-silicon Mac (M1, M2), make sure `rustc -vV` outputs `host: aarch64-apple-darwin`. If not, this should fix it: + +```sh +rustup set default-host aarch64-apple-darwin && rustup install 1.67 +``` + +## Building the docs + +High-level documentation for rerun can be found at [http://rerun.io/docs](http://rerun.io/docs). It is built from the separate repository [rerun-docs](https://github.com/rerun-io/rerun-docs). + +Python API docs can be found at and are built via `mkdocs` and hosted on GitHub. For details on how more information on the python doc-system see [Writing Docs](https://rerun-io.github.io/rerun/latest/docs). + +Rust documentation is hosted on . You can build them locally with: `cargo doc --all-features --no-deps --open` + +## Build and install the Rerun Python SDK +Rerun is available as a package on PyPi and can be installed with `pip install rerun-sdk` (coming soon!) + +Additionally, prebuilt dev wheels from head of main are available at . + +However, if you want to build from source you can follow the instructions below. + +### Set up virtualenv + +Mac/Linux: + +```sh +python3 -m venv venv # Rerun supports Python version >= 3.7 +source venv/bin/activate +python -m pip install --upgrade pip # We need pip version >=21.3 +``` + +Windows (powershell): + +```ps1 +python -m venv venv +.\venv\Scripts\Activate.ps1 +python -m pip install --upgrade pip +``` + +From here on out, we assume you have this virtualenv activated. + +### Build and install + +You need to setup your build environment once with +```sh +./scripts/setup.sh +``` + +Then install the Rerun SDK with: +``` +pip install ./rerun_py +``` + +> Note: If you are unable to upgrade pip to version `>=21.3`, you need to pass `--use-feature=in-tree-build` to the `pip install` command. + + +## Improving compile times + +As of today, we link everything statically in both debug and release builds, which makes custom linkers and split debuginfo the two most impactful tools we have at our disposal in order to improve compile times. + +These tools can configured through your `Cargo` configuration, available at `$HOME/.cargo/config.toml`. + +### macOS + +On macOS, use the [zld](https://github.com/michaeleisel/zld) linker and keep debuginfo in a single separate file. + +Pre-requisites: +- Install [zld](https://github.com/michaeleisel/zld): `brew install michaeleisel/zld/zld`. + +`config.toml` (x64): +```toml +[target.x86_64-apple-darwin] +rustflags = [ + "-C", + "link-arg=-fuse-ld=/usr/local/bin/zld", + "-C", + "split-debuginfo=packed", +] +``` + +`config.toml` (M1): +```toml +[target.aarch64-apple-darwin] +rustflags = [ + "-C", + "link-arg=-fuse-ld=/opt/homebrew/bin/zld", + "-C", + "split-debuginfo=packed", +] +``` + +### Linux + +On Linux, use the [mold](https://github.com/rui314/mold) linker and keep DWARF debuginfo in separate files. + +Pre-requisites: +- Install [mold](https://github.com/rui314/mold) through your package manager. + +`config.toml`: +```toml +[target.x86_64-unknown-linux-gnu] +linker = "clang" +rustflags = [ + "-C", + "link-arg=-fuse-ld=/usr/bin/mold", + "-C", + "split-debuginfo=unpacked", +] +``` + +### Windows + +On Windows, use LLVM's `lld` linker and keep debuginfo in a single separate file. + +Pre-requisites: +- Install `lld`: +``` +cargo install -f cargo-binutils +rustup component add llvm-tools-preview +``` + +`config.toml`: +```toml +[target.x86_64-pc-windows-msvc] +linker = "rust-lld.exe" +rustflags = [ + "-C", + "split-debuginfo=packed", +] +``` diff --git a/CODE_STYLE.md b/CODE_STYLE.md new file mode 100644 index 000000000000..454809fbcafe --- /dev/null +++ b/CODE_STYLE.md @@ -0,0 +1,128 @@ +# Rerun code style + +## See also +* [`ARCHITECTURE.md`](ARCHITECTURE.md) +* [`BUILD.md`](BUILD.md) +* [`CONTRIBUTING.md`](CONTRIBUTING.md) +* [`RELEASES.md`](RELEASES.md) + +## Rust code + +### Error handling and logging +We log problems using our own `re_log` crate (which is currently a wrapper around [`tracing`](https://crates.io/crates/tracing/)). + +* An error should never happen in silence. +* Validate code invariants using `assert!` or `debug_assert!`. +* Validate user data and return errors using [`thiserror`](https://crates.io/crates/thiserror). +* Attach context to errors as they bubble up the stack using [`anyhow`](https://crates.io/crates/anyhow). +* Log errors using `re_log::error!` or `re_log::error_once!`. +* If a problem is recoverable, use `re_log::warn!` or `re_log::warn_once!`. +* If an event is of interest to the user, log it using `re_log::info!` or `re_log::info_once!`. +* The code should only panic if there is a bug in the code. +* Never ignore an error: either pass it on, or log it. +* Handle each error exactly once. If you log it, don't pass it on. If you pass it on, don't log it. + +### Log levels + +The log is for several distinct users: +* The application user +* The application programmer +* The library user +* The library programmer + +We are all sharing the same log stream, so we must cooperate carefully. + +#### `ERROR` +This is for _unrecoverable_ problems. The application or library couldn't complete an operation. + +Libraries should ideally not log `ERROR`, but instead return `Err` in a `Result`, but there are rare cases where returning a `Result` isn't possible (e.g. then doing an operation in a background task). + +Application can "handle" `Err`ors by logging them as `ERROR` (perhaps in addition to showing a popup, if this is a GUI app). + +#### `WARNING` +This is for _recoverable_ problems. The operation completed, but couldn't do exactly what it was instructed to do. + +Sometimes an `Err` is handled by logging it as `WARNING` and then running some fallback code. + +#### `INFO` +This is the default verbosity level. This should mostly be used _only by application code_ to write interesting and rare things to the application user. For instance, you may perhaps log that a file was saved to specific path, or where the default configuration was read from. These things lets application users understand what the application is doing, and debug their use of the application. + +#### `DEBUG` +This is a level you opt-in to to debug either an application or a library. These are logged when high-level operations are performed (e.g. texture creation). If it is likely going to be logged each frame, move it to `TRACE` instead. + +#### `TRACE` +This is the last-resort log level, and mostly for debugging libraries or the use of libraries. Here any and all spam goes, logging low-level operations. + +The distinction between `DEBUG` and `TRACE` is the least clear. Here we use a rule of thumb: if it generates a lot of continuous logging (e.g. each frame), it should go to `TRACE`. + + +### Libraries +We use [`thiserror`](https://crates.io/crates/thiserror) for errors in our libraries, and [`anyhow`](https://crates.io/crates/anyhow) for type-erased errors in applications. + +For faster hashing, we use [`ahash`](https://crates.io/crates/ahash) (`ahash::HashMap`, …). + +When the hashmap key is high-entropy we use [`nohash-hasher`](https://crates.io/crates/nohash-hasher) (`nohash_hasher::IntMap`). + +### Style +We follow the [Rust API Guidelines](https://rust-lang.github.io/api-guidelines/about.html). + +We use `rust fmt` with default settings. + +We have blank lines before functions, types, `impl` blocks, and docstrings. + +We format comments `// Like this`, and `//not like this`. + +When importing a `trait` to use it's trait methods, do this: `use Trait as _;`. That lets the reader know why you imported it, even though it seems unused. + +When intentionally ignoring a `Result`, prefer `foo().ok();` over `let _ = foo();`. The former shows what is happening, and will fail to compile if `foo`:s return type ever changes. + +### `TODO`:s +When you must remember to do something before merging a PR, write `TODO` or `FIXME` in any file. The CI will not be green until you either remove them or rewrite them as `TODO(yourname)`. + +You can also use the `todo()!` macro during development, but again it won't pass CI until you rewrite it as `todo!("more details")`. Of course, we should try to avoid `todo!` macros in our code. + + +### Misc +Use debug-formatting (`{:?}`) when logging strings in logs and error messages. This will surround the string with quotes and escape newlines, tabs, etc. For instance: `re_log::warn!("Unknown key: {key:?}");`. + +Use `re_error::format(err)` when displaying an error. + +### Naming +When in doubt, be explicit. BAD: `id`. GOOD: `msg_id`. + +Be terse when it doesn't hurt readability. BAD: `message_identifier`. GOOD: `msg_id`. + +Avoid negations in names. A lot of people struggle with double negations, so things like `non_blocking = false` and `if !non_blocking { … }` can become a source of confusion and will slow down most readers. So prefer `connected` over `disconnected`, `initialized` over `uninitialized` etc. + +For UI functions (functions taking an `&mut egui::Ui` argument), we use the name `ui` or `_ui` suffix, e.g. `blueprint_ui(…)` or `blueprint.ui(…)`. + +#### Spaces +Points, vectors, rays etc all live in different _spaces_. Whenever there is room for ambiguity, we explicitly state which space something is in, e.g. with `ray_in_world`. + +Here are some of our standard spaces: + +* `ui`: coordinate system used by `egui`, measured in logical pixels ("points"), with origin in the top left +* `image`: image pixel coordinates, possibly with an added `z=depth` +* `space`: a user-defined space where they log stuff into +* `world`: the common coordinate system of a 3D scene, usually same as `space` +* `view`: X=right, Y=down, Z=back, origin = center of screen + +#### Matrices +We use column vectors, which means matrix multiplication is done as `M * v`, i.e. we read all matrix/vector operations right-to-left. We therefore name all transform matrices as `foo_from_bar`, for instance: + +```rust +let point_in_world = world_from_view * point_in_view; +``` + +This means the name of the space matches up nicely, e.g.: + +```rust +let projection_from_object = projection_from_view * view_from_world * world_from_object; +``` + +See for motivation. + +For consistency, we use the same naming convention for other non-matrix transforms too. For instance, functions: `let screen = screen_from_world(world);`. + +#### Vectors vs points +Vectors are directions with magnitudes. Points are positions. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 5d72df81cd9a..6085c9726ac1 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,7 +1,12 @@ # Contributing to Rerun This is written for anyone who wants to contribute to the Rerun repository. -Rerun is an open core company, and this repository is dual-licensed under MIT and APACHE. However, this repository is NOT YET open source, but IT WILL BE. Therefore we ask you to avoid making public clones of this repository, but in other respects treat it as any other open source GitHub project. + +## See also +* [`ARCHITECTURE.md`](ARCHITECTURE.md) +* [`BUILD.md`](BUILD.md) +* [`CODE_STYLE.md`](CODE_STYLE.md) +* [`RELEASES.md`](RELEASES.md) ## What to contribute * **Examples**: We welcome any examples you would like to add. Follow the pattern of the existing examples in the [`examples/`](examples) folder. @@ -15,11 +20,8 @@ We use [Trunk Based Development](https://trunkbaseddevelopment.com/), which mean All PR:s are merged with [`Squash and Merge`](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/incorporating-changes-from-a-pull-request/about-pull-request-merges#squash-and-merge-your-commits), meaning they all get squashed to just one commit on the `main` branch. This means you don't need to keep a clean commit history on your feature branches. In fact, it is preferable to add new commits to a branch rather than rebasing or squashing. For one, it makes it easier to track progress on a branch, but rebasing and force-pushing also discourages collaboration on a branch. -## Getting started with the repository. -* Install the Rust toolchain: -* `git clone git@github.com:rerun-io/rerun.git && cd rerun` -* Run `./scripts/setup_dev.sh`. -* Make sure `cargo --version` prints `1.67.0` once you are done +Our CI will run benchmarks on each merged PR. The results can be found at . + ## Structure The main crates are found in the [`crates/`](crates) folder, with examples in the [`examples/`](examples) folder. @@ -53,200 +55,3 @@ You can use [bacon](https://github.com/Canop/bacon) to automatically check your ### Other You can view higher log levels with `export RUST_LOG=debug` or `export RUST_LOG=trace`. - -## Rust code - -### Error handling and logging -We log problems using our own `re_log` crate (which is currently a wrapper around [`tracing`](https://crates.io/crates/tracing/)). - -* An error should never happen in silence. -* Validate code invariants using `assert!` or `debug_assert!`. -* Validate user data and return errors using [`thiserror`](https://crates.io/crates/thiserror). -* Attach context to errors as they bubble up the stack using [`anyhow`](https://crates.io/crates/anyhow). -* Log errors using `re_log::error!` or `re_log::error_once!`. -* If a problem is recoverable, use `re_log::warn!` or `re_log::warn_once!`. -* If an event is of interest to the user, log it using `re_log::info!` or `re_log::info_once!`. -* The code should only panic if there is a bug in the code. -* Never ignore an error: either pass it on, or log it. -* Handle each error exactly once. If you log it, don't pass it on. If you pass it on, don't log it. - -### Log levels - -The log is for several distinct users: -* The application user -* The application programmer -* The library user -* The library programmer - -We are all sharing the same log stream, so we must cooperate carefully. - -#### `ERROR` -This is for _unrecoverable_ problems. The application or library couldn't complete an operation. - -Libraries should ideally not log `ERROR`, but instead return `Err` in a `Result`, but there are rare cases where returning a `Result` isn't possible (e.g. then doing an operation in a background task). - -Application can "handle" `Err`ors by logging them as `ERROR` (perhaps in addition to showing a popup, if this is a GUI app). - -#### `WARNING` -This is for _recoverable_ problems. The operation completed, but couldn't do exactly what it was instructed to do. - -Sometimes an `Err` is handled by logging it as `WARNING` and then running some fallback code. - -#### `INFO` -This is the default verbosity level. This should mostly be used _only by application code_ to write interesting and rare things to the application user. For instance, you may perhaps log that a file was saved to specific path, or where the default configuration was read from. These things lets application users understand what the application is doing, and debug their use of the application. - -#### `DEBUG` -This is a level you opt-in to to debug either an application or a library. These are logged when high-level operations are performed (e.g. texture creation). If it is likely going to be logged each frame, move it to `TRACE` instead. - -#### `TRACE` -This is the last-resort log level, and mostly for debugging libraries or the use of libraries. Here any and all spam goes, logging low-level operations. - -The distinction between `DEBUG` and `TRACE` is the least clear. Here we use a rule of thumb: if it generates a lot of continuous logging (e.g. each frame), it should go to `TRACE`. - - -### Libraries -We use [`thiserror`](https://crates.io/crates/thiserror) for errors in our libraries, and [`anyhow`](https://crates.io/crates/anyhow) for type-erased errors in applications. - -For faster hashing, we use [`ahash`](https://crates.io/crates/ahash) (`ahash::HashMap`, …). - -When the hashmap key is high-entropy we use [`nohash-hasher`](https://crates.io/crates/nohash-hasher) (`nohash_hasher::IntMap`). - -### Style -We follow the [Rust API Guidelines](https://rust-lang.github.io/api-guidelines/about.html). - -We use `rust fmt` with default settings. - -We have blank lines before functions, types, `impl` blocks, and docstrings. - -We format comments `// Like this`, and `//not like this`. - -When importing a `trait` to use it's trait methods, do this: `use Trait as _;`. That lets the reader know why you imported it, even though it seems unused. - -When intentionally ignoring a `Result`, prefer `foo().ok();` over `let _ = foo();`. The former shows what is happening, and will fail to compile if `foo`:s return type ever changes. - -### `TODO`:s -When you must remember to do something before merging a PR, write `TODO` or `FIXME` in any file. The CI will not be green until you either remove them or rewrite them as `TODO(yourname)`. - -You can also use the `todo()!` macro during development, but again it won't pass CI until you rewrite it as `todo!("more details")`. Of course, we should try to avoid `todo!` macros in our code. - - -### Misc -Use debug-formatting (`{:?}`) when logging strings in logs and error messages. This will surround the string with quotes and escape newlines, tabs, etc. For instance: `re_log::warn!("Unknown key: {key:?}");`. - -Use `re_error::format(err)` when displaying an error. - -### Naming -When in doubt, be explicit. BAD: `id`. GOOD: `msg_id`. - -Be terse when it doesn't hurt readability. BAD: `message_identifier`. GOOD: `msg_id`. - -Avoid negations in names. A lot of people struggle with double negations, so things like `non_blocking = false` and `if !non_blocking { … }` can become a source of confusion and will slow down most readers. So prefer `connected` over `disconnected`, `initialized` over `uninitialized` etc. - -For UI functions (functions taking an `&mut egui::Ui` argument), we use the name `ui` or `_ui` suffix, e.g. `blueprint_ui(…)` or `blueprint.ui(…)`. - -#### Spaces -Points, vectors, rays etc all live in different _spaces_. Whenever there is room for ambiguity, we explicitly state which space something is in, e.g. with `ray_in_world`. - -Here are some of our standard spaces: - -* `ui`: coordinate system used by `egui`, measured in logical pixels ("points"), with origin in the top left -* `image`: image pixel coordinates, possibly with an added `z=depth` -* `space`: a user-defined space where they log stuff into -* `world`: the common coordinate system of a 3D scene, usually same as `space` -* `view`: X=right, Y=down, Z=back, origin = center of screen - -#### Matrices -We use column vectors, which means matrix multiplication is done as `M * v`, i.e. we read all matrix/vector operations right-to-left. We therefore name all transform matrices as `foo_from_bar`, for instance: - -```rust -let point_in_world = world_from_view * point_in_view; -``` - -This means the name of the space matches up nicely, e.g.: - -```rust -let projection_from_object = projection_from_view * view_from_world * world_from_object; -``` - -See for motivation. - -For consistency, we use the same naming convention for other non-matrix transforms too. For instance, functions: `let screen = screen_from_world(world);`. - -#### Vectors vs points -Vectors are directions with magnitudes. Points are positions. - - -## Improving compile times - -As of today, we link everything statically in both debug and release builds, which makes custom linkers and split debuginfo the two most impactful tools we have at our disposal in order to improve compile times. - -These tools can configured through your `Cargo` configuration, available at `$HOME/.cargo/config.toml`. - -### macOS - -On macOS, use the [zld](https://github.com/michaeleisel/zld) linker and keep debuginfo in a single separate file. - -Pre-requisites: -- Install [zld](https://github.com/michaeleisel/zld): `brew install michaeleisel/zld/zld`. - -`config.toml` (x64): -```toml -[target.x86_64-apple-darwin] -rustflags = [ - "-C", - "link-arg=-fuse-ld=/usr/local/bin/zld", - "-C", - "split-debuginfo=packed", -] -``` - -`config.toml` (M1): -```toml -[target.aarch64-apple-darwin] -rustflags = [ - "-C", - "link-arg=-fuse-ld=/opt/homebrew/bin/zld", - "-C", - "split-debuginfo=packed", -] -``` - -### Linux - -On Linux, use the [mold](https://github.com/rui314/mold) linker and keep DWARF debuginfo in separate files. - -Pre-requisites: -- Install [mold](https://github.com/rui314/mold) through your package manager. - -`config.toml`: -```toml -[target.x86_64-unknown-linux-gnu] -linker = "clang" -rustflags = [ - "-C", - "link-arg=-fuse-ld=/usr/bin/mold", - "-C", - "split-debuginfo=unpacked", -] -``` - -### Windows - -On Windows, use LLVM's `lld` linker and keep debuginfo in a single separate file. - -Pre-requisites: -- Install `lld`: -``` -cargo install -f cargo-binutils -rustup component add llvm-tools-preview -``` - -`config.toml`: -```toml -[target.x86_64-pc-windows-msvc] -linker = "rust-lld.exe" -rustflags = [ - "-C", - "split-debuginfo=packed", -] -``` diff --git a/README.md b/README.md index 5d5dad169ffb..abc9367f06ce 100644 --- a/README.md +++ b/README.md @@ -1,111 +1,51 @@ # Rerun +[![Build Status](https://github.com/emilk/egui/workflows/CI/badge.svg)](https://github.com/emilk/egui/actions?workflow=CI) [![MIT](https://img.shields.io/badge/license-MIT-blue.svg)](https://github.com/rerun-io/rerun/blob/master/LICENSE-MIT) [![Apache](https://img.shields.io/badge/license-Apache-blue.svg)](https://github.com/rerun-io/rerun/blob/master/LICENSE-APACHE) +[![Discord](https://img.shields.io/discord/900275882684477440?label=Rerun%20Community%20Discord)](https://discord.gg/Gcm8BbTaAj) -Rerun is visualization infrastructure for computer vision. - -This repository contains the Rerun SDK and Rerun Viewer. Use the SDK (currently Python only) to log rich data that is streamed to the viewer, where it is visualized live or after the fact. - -# Documentation (Coming soon!) - -## WARNING: The following links don't all work yet -TODO(jleibs): Clean up this section and remove warnign when all links are live - -High-level documentation for rerun can be found at [http://rerun.io/docs](http://rerun.io/docs). - -The documentation is built from a separate [Rerun-Docs Repository](https://github.com/rerun-io/rerun-docs) - -Rust and Python APIs are documented in the code via docstrings. - - The [Rust API docs](https://docs.rs/rerun/) are built via cargo and hosted on docs.rs - - The [Python API docs](https://rerun-io.github.io/rerun) are built via mkdocs and hosted on github: - - For more information on the python doc-system see: [Writing Docs](https://rerun-io.github.io/rerun/latest/docs) - -# For our users - -We don't have any pre-built binaries yet, so you need to build Rerun from source. There is some setup involved, but most of it should be pretty painless. - -## Setup - -- Install the Rust toolchain: -- `git clone git@github.com:rerun-io/rerun.git && cd rerun` -- Run `./scripts/setup.sh`. -- Make sure `cargo --version` prints `1.67.0` once you are done - -### Apple-silicon Macs - -If you are using an Apple-silicon Mac, make sure `rustc -vV` outputs `host: aarch64-apple-darwin`. If not, this should fix it: - -```sh -rustup set default-host aarch64-apple-darwin && rustup install 1.67 -``` - -## Build and install the Rerun Python SDK - -### Set up virtualenv - -Mac/Linux: - -```sh -python3 -m venv venv # Rerun supports Python version >= 3.7 -source venv/bin/activate -python -m pip install --upgrade pip # We need pip version >=21.3 -``` -Windows (powershell): +Rerun is visualization infrastructure for computer vision. -```ps1 -python -m venv venv -.\venv\Scripts\Activate.ps1 -python -m pip install --upgrade pip -``` +You use one of our logging APIs (Python or Rust atm) to log rich data, such as images and point clouds, to the Rerun Viewer, where it is visualized live or after the fact. -From here on out, we assume you have this virtualenv activated. +```py +import rerun as rr -### Build and install +rr.init("my_app", spawn = True) # Spawn a Rerun Viewer and stream log events to it -```sh -./scripts/setup.sh -pip install ./rerun_py +rr.log_image("rgb_image", image) +rr.log_points("points", positions) ``` -> Note: If you are unable to upgrade pip to version `>=21.3`, you need to pass `--use-feature=in-tree-build` to the `pip install` command. + -## Getting started with examples -The easiest way to get started is to run and look at [`examples`](examples). +## Documentation +- [Examples](examples) +- [Python API docs](https://rerun-io.github.io/rerun) +- [Rust getting-started guide](rerun_py/USAGE.md) + -### Buffered or live visualization -By default, the examples run in buffered mode. This means they run through the whole example, and then show the viewer (UI) at the end in the same process by calling blocking function `rerun.show()`. +## Installing the pre-release Python SDK + +1. Download the correct `.whl` from [GitHub Releases](https://github.com/rerun-io/rerun/releases) + (for Mac M1/M2, grab the "universal2" `.whl`) +2. Run `pip install rerun_sdk<...>.whl` (replace `<...>` with the actual filename) +3. Test it: `rerun --version` -If you'd rather see the visualizations live, as data is being logged. Run the examples with the `--connect` flag. The Rerun SDK will then try to connect to a Rerun Viewer running in another process and send the data as it is produced. -To visualize an example live, first in one terminal (with the activated virtualenv) run: +## Installing the Rust SDK +Coming soon + -```sh -python -m rerun # Opens a Rerun Viewer that will wait for data from the Rerun SDK -``` - -Then run the example in a second terminal like: - -```sh -python examples/car/main.py --connect # The Rerun SDK will connect and send data to the separate viewer. -``` - -## Using the Rerun Python SDK - -Most documentation is found in the docstrings of the functions in the Rerun. Either check out the docstrings directly in code or use the built in `help()` function. For example, to see the docstring of the `log_image` function, open a python terminal and run: - -```python -import rerun as rr -help(rr.log_image) -``` - -For a description of how to use the SDK, including some of the key concepts, see [`rerun_py/USAGE.md`](rerun_py/USAGE.md). ## Rerun Viewer without Python - You can also build and install the Rerun Viewer to be used from the terminal without going through Python. To build and install run: @@ -116,13 +56,23 @@ cargo install --path ./crates/rerun/ You should now be able to run `rerun --help` in any terminal. -## Bounded memory use -You can set `--memory-limit=16GB` to tell the Rerun Viewer to purge older log data when memory use goes above that limit. This is useful for using Rerun in _continuous_ mode, i.e. where you keep logging new data to Rerun forever. +## Shortcomings +* Big points clouds (1M+) are slow ([#1136](https://github.com/rerun-io/rerun/issues/1136)) +* The data you want to visualize must fit in RAM. + - See [`rerun_py/USAGE.md`](rerun_py/USAGE.md) for how to bound memory use + - We plan on having a disk-based data store some time in the future -It is still possible to log data faster than the Rerun Viewer can process it, and in those cases you may still run out of memory unless you also set `--drop-at-latency=200ms` or similar. +## Business model +Rerun uses an open core model. Everything in this repository will stay open source and free (as in beer), forever. In the future, Rerun will offer a commercial product that builds on top of the core free project. -# Development +The Rerun open source project targets the needs of individual developers. The commercial product targets the needs specific to teams that build and run computer vision and robotics products. -Take a look at [`CONTRIBUTING.md`](CONTRIBUTING.md). + +# Development +* [`ARCHITECTURE.md`](ARCHITECTURE.md) +* [`BUILD.md`](BUILD.md) +* [`CODE_STYLE.md`](CODE_STYLE.md) +* [`CONTRIBUTING.md`](CONTRIBUTING.md) +* [`RELEASES.md`](RELEASES.md) diff --git a/RELEASES.md b/RELEASES.md index 17cc3edeecf0..ccf9e1b15b1e 100644 --- a/RELEASES.md +++ b/RELEASES.md @@ -2,7 +2,14 @@ This document describes the current release and versioning strategy. This strategy is likely to change as Rerun matures. -## Cadence +## See also +* [`ARCHITECTURE.md`](ARCHITECTURE.md) +* [`BUILD.md`](BUILD.md) +* [`CODE_STYLE.md`](CODE_STYLE.md) +* [`CONTRIBUTING.md`](CONTRIBUTING.md) + + +## Release cCadence New Rerun versions are released every two weeks. Sometimes we do out-of-schedule patch releases. diff --git a/crates/re_analytics/src/cli.rs b/crates/re_analytics/src/cli.rs index 84d383962536..9640388c7248 100644 --- a/crates/re_analytics/src/cli.rs +++ b/crates/re_analytics/src/cli.rs @@ -78,7 +78,7 @@ const DETAILS: &str = " * Anonymous Usage Data Collection in Rerun * Opting out: - - Run `rerun analytics disable` or `python -m rerun analytics disable` to opt out of all usage data collection. + - Run `rerun analytics disable` to opt out of all usage data collection. What data is collected? - The exact set of analytics events and parameters can be found here: diff --git a/crates/re_analytics/src/lib.rs b/crates/re_analytics/src/lib.rs index 1be88e60db5e..995c8f05edc1 100644 --- a/crates/re_analytics/src/lib.rs +++ b/crates/re_analytics/src/lib.rs @@ -102,7 +102,7 @@ const DISCLAIMER: &str = " - We don't log your user name, file paths, or any personal identifiable data. - Usage data we do collect will be sent to and stored on servers within the EU. - For more details and instructions on how to opt out, - run: `rerun analytics details` or `python -m rerun analytics details`. + run: `rerun analytics details` As this is this your first session, _no_ usage data has been sent yet, giving you an opportunity to opt-out first if you wish. diff --git a/crates/re_arrow_store/README.md b/crates/re_arrow_store/README.md new file mode 100644 index 000000000000..8ae98ce4f179 --- /dev/null +++ b/crates/re_arrow_store/README.md @@ -0,0 +1,4 @@ +# Rerun Arrow Store +[Apache Arrow](https://arrow.apache.org/) is a language-independent columnar memory format for arbitrary data. + +The `re_arrow_store` crate is an in-memory time series database for Rerun log data. It is indexed by Entity path, component, timeline, and time. It supports out-of-order insertions, and fast `O(log(N))` queries. diff --git a/crates/re_arrow_store/src/lib.rs b/crates/re_arrow_store/src/lib.rs index 0ee424f9c79e..c32ebf1d473c 100644 --- a/crates/re_arrow_store/src/lib.rs +++ b/crates/re_arrow_store/src/lib.rs @@ -1,6 +1,11 @@ -//! The Rerun Arrow-based datastore. +//! The Rerun datastore, implemented on top of [Apache Arrow](https://arrow.apache.org/) +//! using the [`arrow2`] crate. //! -//! * See [`DataStore`] for an overview of the core datastructures. +//! This crate is an in-memory time series database for Rerun log data. +//! It is indexed by Entity path, component, timeline, and time. +//! It supports out-of-order insertions, and fast `O(log(N))` queries. +//! +//! * See [`DataStore`] for an overview of the core data structures. //! * See [`DataStore::latest_at`] and [`DataStore::range`] for the documentation of the public //! read APIs. //! * See [`DataStore::insert`] for the documentation of the public write APIs. diff --git a/crates/re_renderer/build.rs b/crates/re_renderer/build.rs index 61a8f85d7096..d9c6b90c4b37 100644 --- a/crates/re_renderer/build.rs +++ b/crates/re_renderer/build.rs @@ -76,7 +76,7 @@ impl std::str::FromStr for ImportClause { .map(|path| Self { path }); } - bail!("misformatted import clause: {clause_str:?}") + bail!("malformed import clause: {clause_str:?}") } } @@ -97,7 +97,7 @@ fn check_hermeticity(root_path: impl AsRef, file_path: impl AsRef) { ensure!( clause_path.starts_with(&root_path), "trying to import {:?} which lives outside of the workspace, \ - this is illegal in release and/or WASM builds!", + this is illegal in release and/or Wasm builds!", clause_path ); diff --git a/crates/re_renderer/src/file_server.rs b/crates/re_renderer/src/file_server.rs index 3eaea427182d..0161f520a5cc 100644 --- a/crates/re_renderer/src/file_server.rs +++ b/crates/re_renderer/src/file_server.rs @@ -1,6 +1,6 @@ /// A macro to read the contents of a file on disk, and resolve #import clauses as required. /// -/// - On WASM and/or release builds, this will behave like the standard [`include_str`] +/// - On Wasm and/or release builds, this will behave like the standard [`include_str`] /// macro. /// - On native debug builds, this will actually load the specified path through /// our [`FileServer`], and keep watching for changes in the background (both the root file diff --git a/crates/re_sdk_comms/src/buffered_client.rs b/crates/re_sdk_comms/src/buffered_client.rs index 62db39518f28..b3d9cde4df50 100644 --- a/crates/re_sdk_comms/src/buffered_client.rs +++ b/crates/re_sdk_comms/src/buffered_client.rs @@ -303,7 +303,7 @@ fn send_until_success( // Only produce subsequent warnings once we've saturated the back-off if sleep_ms == MAX_SLEEP_MS && new_err.to_string() != err.to_string() { - re_log::warn!("Still failing to send message: {err}"); + re_log::warn!("Still failing to send message: {err}"); } } else { return None; diff --git a/crates/re_viewer/README.md b/crates/re_viewer/README.md index 302936359f7a..9eff4bd1364c 100644 --- a/crates/re_viewer/README.md +++ b/crates/re_viewer/README.md @@ -2,6 +2,6 @@ The Rerun viewer. This is the main crate with all the GUI. -This can be compiled as a web-app by building for WASM. To run it natively, use the `rerun` binary. +This can be compiled as a web-app by building for Wasm. To run it natively, use the `rerun` binary. Talks to the server over WebSockets (using `re_ws_comms`). diff --git a/crates/re_web_server/Cargo.toml b/crates/re_web_server/Cargo.toml index 01ed183ba9df..5379c5207354 100644 --- a/crates/re_web_server/Cargo.toml +++ b/crates/re_web_server/Cargo.toml @@ -10,7 +10,7 @@ publish = false [features] ## ONLY FOR CI! ## -## When set: will skip building the Web Viewer WASM. +## When set: will skip building the Web Viewer Wasm. ## This makes the CI much faster, but the resulting web server ## will crash if you rey to run it! __ci = [] diff --git a/crates/re_web_server/README.md b/crates/re_web_server/README.md index 0d3cbb2003dd..17caaf5e2870 100644 --- a/crates/re_web_server/README.md +++ b/crates/re_web_server/README.md @@ -1 +1 @@ -Serves the rerun web viewer (`re_viewer` as WASM and HTML) over HTTP. +Serves the rerun web viewer (`re_viewer` as Wasm and HTML) over HTTP. diff --git a/crates/re_web_server/build.rs b/crates/re_web_server/build.rs index 835641fcceee..f9d586b7c905 100644 --- a/crates/re_web_server/build.rs +++ b/crates/re_web_server/build.rs @@ -210,7 +210,7 @@ fn build_web() { // --- fn main() { - // Rebuild the web-viewer WASM, + // Rebuild the web-viewer Wasm, // because the web_server library bundles it with `include_bytes!` let metadata = MetadataCommand::new() diff --git a/crates/re_web_server/src/lib.rs b/crates/re_web_server/src/lib.rs index 33b131b15218..50a7b3313457 100644 --- a/crates/re_web_server/src/lib.rs +++ b/crates/re_web_server/src/lib.rs @@ -73,7 +73,7 @@ impl Service for MakeSvc { // ---------------------------------------------------------------------------- -/// Hosts the Web Viewer WASM+HTML +/// Hosts the Web Viewer Wasm+HTML pub struct WebServer { server: hyper::Server, } diff --git a/crates/rerun_sdk/src/session.rs b/crates/rerun_sdk/src/session.rs index f2aa5e1af991..7455997141db 100644 --- a/crates/rerun_sdk/src/session.rs +++ b/crates/rerun_sdk/src/session.rs @@ -85,7 +85,7 @@ impl Session { .unwrap(); let ws_server_handle = tokio::spawn(ws_server.listen(rerun_rx)); - // This is the server that serves the WASM+HTML: + // This is the server that serves the Wasm+HTML: let web_port = 9090; let web_server = re_web_server::WebServer::new(web_port); let web_server_handle = tokio::spawn(async move { diff --git a/examples/README.md b/examples/README.md index 2be86eed3b7f..a9f2dfd5d6c3 100644 --- a/examples/README.md +++ b/examples/README.md @@ -2,9 +2,11 @@ The simplest example is [`hello_world`](hello_world/main.py). You may want to start there! ## Setup -First follow [the main setup instructions](https://github.com/rerun-io/rerun#setup). +First install the Rerun Python SDK with `pip install rerun-sdk` (coming soon!) -## Running the examples + + +## Dependencies Each example comes with its own set of dependencies listed in a `requirements.txt` file. For example, to install dependencies and run the toy `car` example (which doesn't need to download any data) run: ```sh @@ -18,6 +20,15 @@ You can also install all dependencies needed to run all examples with: pip install -r examples/requirements.txt ``` +## Running the examples +By default, the examples spawn a Rerun Viewer and stream log data to it. + +You can instead save the log data to an `.rrd` file using `examples/car/main.py --save data.rrd`. You can then open that `.rrd` file with `rerun data.rrd`. + +(`rerun` is an alias for `python -m rerun`). + +NOTE: `.rrd` files do not yet guarantee any backwards or forwards compatibility. One version of Rerun will likely not be able to open an `.rrd` file generated by another Rerun version. + ## Contributions welcome Feel free to open a PR to add a new example! diff --git a/justfile b/justfile index eb72d53f710a..aade82a72419 100644 --- a/justfile +++ b/justfile @@ -1,3 +1,7 @@ +# Install just: https://github.com/casey/just +# +# Then run `just --list` to see the available commands + default: @just --list diff --git a/rerun_py/USAGE.md b/rerun_py/USAGE.md index f240803465d4..03722fbe15b7 100644 --- a/rerun_py/USAGE.md +++ b/rerun_py/USAGE.md @@ -1,3 +1,5 @@ + + # Using the `rerun` Python Library Install instructions can be found at . @@ -11,7 +13,7 @@ Rerun assumes you are using `numpy` for any large chunks of data. ```python import rerun as rr -rr.init("my_app", spawn_and_connect = True) # Spawn a Rerun Viewer and stream log events to it +rr.init("my_app", spawn = True) # Spawn a Rerun Viewer and stream log events to it rr.log_image("rgb_image", image) ``` @@ -116,3 +118,17 @@ To do so, don't call `rr.connect()`. Instead, call `rr.show()` at the end of you ## Troubleshooting You can set `RUST_LOG=debug` before running your Python script and/or `rerun` process to get some verbose logging output. + +# Help +Most documentation is found in the docstrings of the functions in the Rerun. Either check out the docstrings directly in code or use the built in `help()` function. For example, to see the docstring of the `log_image` function, open a python terminal and run: + +```python +import rerun as rr +help(rr.log_image) +``` + +## Bounded memory use + +You can set `--memory-limit=16GB` to tell the Rerun Viewer to purge older log data when memory use goes above that limit. This is useful for using Rerun in _continuous_ mode, i.e. where you keep logging new data to Rerun forever. + +It is still possible to log data faster than the Rerun Viewer can process it, and in those cases you may still run out of memory unless you also set `--drop-at-latency=200ms` or similar. diff --git a/scripts/lint.py b/scripts/lint.py index b5e635961947..5ff8c4bdc4e2 100755 --- a/scripts/lint.py +++ b/scripts/lint.py @@ -12,10 +12,9 @@ from typing import Optional todo_pattern = re.compile(r"TODO([^(]|$)") - debug_format_of_err = re.compile(r"\{\:#?\?\}.*, err") - error_match_name = re.compile(r"Err\((\w+)\)") +wasm_caps = re.compile(r"\bWASM\b") def lint_line(line: str) -> Optional[str]: if "NOLINT" in line: @@ -54,6 +53,9 @@ def lint_line(line: str) -> Optional[str]: if name in ("e", "error"): return "Errors should be called 'err', '_err' or '_'" + if wasm_caps.search(line): + return "WASM should be written 'Wasm'" + return None @@ -72,6 +74,8 @@ def test_lint() -> None: 'if let Err(err) = foo', 'if let Err(_err) = foo', 'if let Err(_) = foo', + 'WASM_FOO env var', + 'Wasm', ] should_error = [ @@ -86,6 +90,7 @@ def test_lint() -> None: 'eprintln!("{:?}", err)', 'eprintln!("{:#?}", err)', 'if let Err(error) = foo', + 'We use WASM in Rerun', ] for line in should_pass: diff --git a/taplo.toml b/taplo.toml index 806ef04743fd..0bbd7d5469f9 100644 --- a/taplo.toml +++ b/taplo.toml @@ -1,2 +1,4 @@ +# https://github.com/tamasfe/taplo + [formatting] indent_string = " " diff --git a/web_viewer/index.html b/web_viewer/index.html index 1e8fa4b73a34..f3b951a81206 100644 --- a/web_viewer/index.html +++ b/web_viewer/index.html @@ -96,7 +96,7 @@ - +

@@ -150,7 +150,7 @@ ${error}

- Make sure you use a modern browser with WebGL and WASM enabled. + Make sure you use a modern browser with WebGL and Wasm enabled.

`; }