Improve the dev docs and top-level README.md (rerun-io#1133)

* Create a BUILD.md and CODE_STYLE.md

* Move docs about how to build docs to BUILD.md

* Improve example docs

* Remove `python -m rerun` instructions (`rerun` should work just as well)

* Explain justfile

* Explain taplo.toml

* Create dummy ARCHITECTURE.md and interlink all dev-docs

* Start filling out ARCHITECTURE.md

* Strip down README.md to its minimum

* Document some of the major technologies

* "Wasm" should not be written as WASM

* Add a section about immediate mode

* Add link to benchmarks

* Fix link in TODO

* Add list of shortcomings

* a short section about the native viewer

Co-authored-by: Jeremy Leibs <[email protected]>

* Improve the buisness model section

Co-authored-by: Nikolaus West <[email protected]>

* User feedback

* Unrelated code cleanup

---------

Co-authored-by: Jeremy Leibs <[email protected]>
Co-authored-by: Nikolaus West <[email protected]>

ARCHITECTURE.md

@@ -0,0 +1,81 @@
+# Rerun architecture
+This document describes the technical architecture of Rerun.
+
+## See also
+* [`BUILD.md`](BUILD.md)
+* [`CODE_STYLE.md`](CODE_STYLE.md)
+* [`CONTRIBUTING.md`](CONTRIBUTING.md)
+* [`RELEASES.md`](RELEASES.md)
+
+
+## The major components
+### Logging APIs
+It all starts with logging. You can log rich data (point clouds, images, etc) with either our Python SDK or our Rust SDK.
+
+The logging SDK:s encodes the data using Apache Arrow (see more below).
+
+The logging data can be written to disk as `.rrd` files, or transmitted over TCP to either a Rerun Viewer or a Rerun Server.
+
+### Rerun viewer
+The Rerun Viewer is where log data is visualized. It is usually run as a native app, but can also be compiled to WebAssembly (Wasm) and run in a browser.
+
+#### Native viewer
+The easiest way to launch the viewer is directly from the logging API with `rr.init("my_app", spawn=True)`. However, the standalone viewer can also be run from the command line, for example to view an `.rrd` file: `rerun mydata.rrd`.
+
+#### Web viewer
+You can try running the viewer in a browser using `rr.serve()` in python, or using `rerun --web-viewer mydata.rrd`.
+
+The web viewer consists of just a few small files - a thin `.html`, a `.wasm` blob, and an auto-generated `.js` bridge for the wasm. These files are served using the [`re_web_server`](https://github.com/rerun-io/rerun/tree/main/crates/re_web_server) crate.
+
+The web viewer can load `.rrd` files (just drag-drop them into the browser), or read logging data streamed over WebSockets.
+
+### `.rrd` files
+`.rrd` ("**R**e**r**un **D**ata") is just a bunch of log messages appended one after the other to a file.
+
+NOTE: `.rrd` files do not yet guarantee any backwards or forwards compatibility. One version of Rerun will likely not be able to open an `.rrd` file generated by another Rerun version.
+
+
+## Technologies we use
+### Apache Arrow
+[Apache Arrow](https://arrow.apache.org/) is a language-independent columnar memory format for arbitrary data. We use it to encode the log data when transmitting it over the network or storing it in an `.rrd` file. We also use it in our in-RAM data store, [`re_arrow_store`](crates/re_arrow_store/README.md).
+
+In rust, we use the [`arrow2` crate](https://crates.io/crates/arrow2).
+
+### `wgpu`
+The Rerun Viewer uses the [`wgpu`](https://github.com/gfx-rs/wgpu) graphics API. It provides a high-performance abstraction over Vulkan, Metal, D3D12, D3D11, OpenGLES, WebGL and [WebGPU](https://en.wikipedia.org/wiki/WebGPU). This lets us write the same code graphics code for native as for web.
+
+We use the WebGL backend when compiling for web. Once WebGPU is available in most browsers, we can easily switch to it for a nice performance boost!
+
+We have written our own high-level rendering crate on top of `wgpu`, called [`re_renderer`](crates/re_renderer/README.md).
+
+### `egui`
+The GUI in the Rerun Viewer is using [`egui`](https://www.egui.rs/), a cross-platform, [immediate mode GUI](https://github.com/emilk/egui#why-immediate-mode).
+
+We use [`eframe`](https://github.com/emilk/egui/tree/master/crates/eframe), the egui framework, to run `egui` on both native and web.
+
+
+### Wasm
+Wasm (short for [WebAssembly](https://webassembly.org/)) is a binary instruction format supported by all major browser.
+The Rerun Viewer can be compiled to Wasm and run in a browser.
+
+Threading support in Wasm is nascent, so care must we taken that we don't spawn any threads when compiling for `wasm32`.
+
+Wasm has no access to the host system, except via JS calls (something that may change once [WASI](https://wasi.dev/) rolls out), so when compiling for `wasm32` you can NOT use the Rust standard library to:
+* Access files
+* Read environment variables
+* Get the current time (use [`instant`](https://crates.io/crates/instant) instead)
+* Use networking (use [`ehttp`](https://github.com/emilk/ehttp), [`reqwest`](https://github.com/seanmonstar/reqwest), or [`ewebsock`](https://github.com/rerun-io/ewebsock) instead)
+* etc
+
+
+## Immediate mode
+The Rerun Viewer uses an [immediate mode GUI](https://github.com/emilk/egui#why-immediate-mode), [`egui`](https://www.egui.rs/). This means that each frame the entire GUI is being laid out from scratch.
+
+In fact, the whole of the Rerun Viewer is written in an immediate mode style. Each rendered frame it will query the in-RAM data store, massage the results, and feed it to the renderer.
+
+The advantage of immediate mode is that is removes all state management. There is no callbacks that are called when some state has already changed, and the state of the blueprint is always in sync with what you see on screen.
+
+Immediate mode is also a forcing function, forcing us to relentlessly optimize our code.
+This leads to a very responsive GUI, where there is no "hickups" when switching data source or doing time scrubbing.
+
+Of course, this will only take us so far. In the future we plan on caching queries and work submitted to the renderer so that we don't perform unnecessary work each frame. We also plan on doing larger operation in background threads. This will be necessary in order to support viewing large datasets, e.g. several million points. The plan is still to do so within an immediate mode framework, retaining most of the advantages of stateless code.

BUILD.md

@@ -0,0 +1,150 @@
+# Building Rerun
+This is a guide to how to build Rerun.
+
+
+## See also
+* [`ARCHITECTURE.md`](ARCHITECTURE.md)
+* [`CODE_STYLE.md`](CODE_STYLE.md)
+* [`CONTRIBUTING.md`](CONTRIBUTING.md)
+* [`RELEASES.md`](RELEASES.md)
+
+
+## Getting started with the repository.
+* Install the Rust toolchain: <https://rustup.rs/>
+* `git clone [email protected]:rerun-io/rerun.git && cd rerun`
+* Run `./scripts/setup_dev.sh`.
+* Make sure `cargo --version` prints `1.67.0` once you are done
+
+
+### Apple-silicon Macs
+
+If you are using an Apple-silicon Mac (M1, M2), make sure `rustc -vV` outputs `host: aarch64-apple-darwin`. If not, this should fix it:
+
+```sh
+rustup set default-host aarch64-apple-darwin && rustup install 1.67
+```
+
+## Building the docs
+
+High-level documentation for rerun can be found at [http://rerun.io/docs](http://rerun.io/docs). It is built from the separate repository [rerun-docs](https://github.com/rerun-io/rerun-docs).
+
+Python API docs can be found at <https://rerun-io.github.io/rerun> and are built via `mkdocs` and hosted on GitHub. For details on how more information on the python doc-system see [Writing Docs](https://rerun-io.github.io/rerun/latest/docs).
+
+Rust documentation is hosted on <https://docs.rs/rerun/>. You can build them locally with: `cargo doc --all-features --no-deps --open`
+
+## Build and install the Rerun Python SDK
+Rerun is available as a package on PyPi and can be installed with `pip install rerun-sdk` (coming soon!) <!-- TODO(#1161) -->
+
+Additionally, prebuilt dev wheels from head of main are available at <https://github.com/rerun-io/rerun/releases/tag/latest>.
+
+However, if you want to build from source you can follow the instructions below.
+
+### Set up virtualenv
+
+Mac/Linux:
+
+```sh
+python3 -m venv venv  # Rerun supports Python version >= 3.7
+source venv/bin/activate
+python -m pip install --upgrade pip  # We need pip version >=21.3
+```
+
+Windows (powershell):
+
+```ps1
+python -m venv venv
+.\venv\Scripts\Activate.ps1
+python -m pip install --upgrade pip
+```
+
+From here on out, we assume you have this virtualenv activated.
+
+### Build and install
+
+You need to setup your build environment once with
+```sh
+./scripts/setup.sh
+```
+
+Then install the Rerun SDK with:
+```
+pip install ./rerun_py
+```
+
+> Note: If you are unable to upgrade pip to version `>=21.3`, you need to pass `--use-feature=in-tree-build` to the `pip install` command.
+
+
+## Improving compile times
+
+As of today, we link everything statically in both debug and release builds, which makes custom linkers and split debuginfo the two most impactful tools we have at our disposal in order to improve compile times.
+
+These tools can configured through your `Cargo` configuration, available at `$HOME/.cargo/config.toml`.
+
+### macOS
+
+On macOS, use the [zld](https://github.com/michaeleisel/zld) linker and keep debuginfo in a single separate file.
+
+Pre-requisites:
+- Install [zld](https://github.com/michaeleisel/zld): `brew install michaeleisel/zld/zld`.
+
+`config.toml` (x64):
+```toml
+[target.x86_64-apple-darwin]
+rustflags = [
+    "-C",
+    "link-arg=-fuse-ld=/usr/local/bin/zld",
+    "-C",
+    "split-debuginfo=packed",
+]
+```
+
+`config.toml` (M1):
+```toml
+[target.aarch64-apple-darwin]
+rustflags = [
+    "-C",
+    "link-arg=-fuse-ld=/opt/homebrew/bin/zld",
+    "-C",
+    "split-debuginfo=packed",
+]
+```
+
+### Linux
+
+On Linux, use the [mold](https://github.com/rui314/mold) linker and keep DWARF debuginfo in separate files.
+
+Pre-requisites:
+- Install [mold](https://github.com/rui314/mold) through your package manager.
+
+`config.toml`:
+```toml
+[target.x86_64-unknown-linux-gnu]
+linker = "clang"
+rustflags = [
+    "-C",
+    "link-arg=-fuse-ld=/usr/bin/mold",
+    "-C",
+    "split-debuginfo=unpacked",
+]
+```
+
+### Windows
+
+On Windows, use LLVM's `lld` linker and keep debuginfo in a single separate file.
+
+Pre-requisites:
+- Install `lld`:
+```
+cargo install -f cargo-binutils
+rustup component add llvm-tools-preview
+```
+
+`config.toml`:
+```toml
+[target.x86_64-pc-windows-msvc]
+linker = "rust-lld.exe"
+rustflags = [
+    "-C",
+    "split-debuginfo=packed",
+]
+```

CODE_STYLE.md

@@ -0,0 +1,128 @@
+# Rerun code style
+
+## See also
+* [`ARCHITECTURE.md`](ARCHITECTURE.md)
+* [`BUILD.md`](BUILD.md)
+* [`CONTRIBUTING.md`](CONTRIBUTING.md)
+* [`RELEASES.md`](RELEASES.md)
+
+## Rust code
+
+### Error handling and logging
+We log problems using our own `re_log` crate (which is currently a wrapper around [`tracing`](https://crates.io/crates/tracing/)).
+
+* An error should never happen in silence.
+* Validate code invariants using `assert!` or `debug_assert!`.
+* Validate user data and return errors using [`thiserror`](https://crates.io/crates/thiserror).
+* Attach context to errors as they bubble up the stack using [`anyhow`](https://crates.io/crates/anyhow).
+* Log errors using `re_log::error!` or `re_log::error_once!`.
+* If a problem is recoverable, use `re_log::warn!` or `re_log::warn_once!`.
+* If an event is of interest to the user, log it using `re_log::info!` or `re_log::info_once!`.
+* The code should only panic if there is a bug in the code.
+* Never ignore an error: either pass it on, or log it.
+* Handle each error exactly once. If you log it, don't pass it on. If you pass it on, don't log it.
+
+### Log levels
+
+The log is for several distinct users:
+* The application user
+* The application programmer
+* The library user
+* The library programmer
+
+We are all sharing the same log stream, so we must cooperate carefully.
+
+#### `ERROR`
+This is for _unrecoverable_ problems. The application or library couldn't complete an operation.
+
+Libraries should ideally not log `ERROR`, but instead return `Err` in a `Result`, but there are rare cases where returning a `Result` isn't possible (e.g. then doing an operation in a background task).
+
+Application can "handle" `Err`ors by logging them as `ERROR` (perhaps in addition to showing a popup, if this is a GUI app).
+
+#### `WARNING`
+This is for _recoverable_ problems. The operation completed, but couldn't do exactly what it was instructed to do.
+
+Sometimes an `Err` is handled by logging it as `WARNING` and then running some fallback code.
+
+#### `INFO`
+This is the default verbosity level. This should mostly be used _only by application code_ to write interesting and rare things to the application user. For instance, you may perhaps log that a file was saved to specific path, or where the default configuration was read from. These things lets application users understand what the application is doing, and debug their use of the application.
+
+#### `DEBUG`
+This is a level you opt-in to to debug either an application or a library. These are logged when high-level operations are performed (e.g. texture creation). If it is likely going to be logged each frame, move it to `TRACE` instead.
+
+#### `TRACE`
+This is the last-resort log level, and mostly for debugging libraries or the use of libraries. Here any and all spam goes, logging low-level operations.
+
+The distinction between `DEBUG` and `TRACE` is the least clear. Here we use a rule of thumb: if it generates a lot of continuous logging (e.g. each frame), it should go to `TRACE`.
+
+
+### Libraries
+We use [`thiserror`](https://crates.io/crates/thiserror) for errors in our libraries, and [`anyhow`](https://crates.io/crates/anyhow) for type-erased errors in applications.
+
+For faster hashing, we use [`ahash`](https://crates.io/crates/ahash) (`ahash::HashMap`, …).
+
+When the hashmap key is high-entropy we use [`nohash-hasher`](https://crates.io/crates/nohash-hasher) (`nohash_hasher::IntMap`).
+
+### Style
+We follow the [Rust API Guidelines](https://rust-lang.github.io/api-guidelines/about.html).
+
+We use `rust fmt` with default settings.
+
+We have blank lines before functions, types, `impl` blocks, and docstrings.
+
+We format comments `// Like this`, and `//not like this`.
+
+When importing a `trait` to use it's trait methods, do this: `use Trait as _;`. That lets the reader know why you imported it, even though it seems unused.
+
+When intentionally ignoring a `Result`, prefer `foo().ok();` over `let _ = foo();`. The former shows what is happening, and will fail to compile if `foo`:s return type ever changes.
+
+### `TODO`:s
+When you must remember to do something before merging a PR, write `TODO` or `FIXME` in any file. The CI will not be green until you either remove them or rewrite them as `TODO(yourname)`.
+
+You can also use the `todo()!` macro during development, but again it won't pass CI until you rewrite it as `todo!("more details")`. Of course, we should try to avoid `todo!` macros in our code.
+
+
+### Misc
+Use debug-formatting (`{:?}`) when logging strings in logs and error messages. This will surround the string with quotes and escape newlines, tabs, etc. For instance: `re_log::warn!("Unknown key: {key:?}");`.
+
+Use `re_error::format(err)` when displaying an error.
+
+### Naming
+When in doubt, be explicit. BAD: `id`. GOOD: `msg_id`.
+
+Be terse when it doesn't hurt readability. BAD: `message_identifier`. GOOD: `msg_id`.
+
+Avoid negations in names. A lot of people struggle with double negations, so things like `non_blocking = false` and `if !non_blocking { … }` can become a source of confusion and will slow down most readers. So prefer `connected` over `disconnected`, `initialized` over `uninitialized` etc.
+
+For UI functions (functions taking an `&mut egui::Ui` argument), we use the name `ui` or `_ui` suffix, e.g. `blueprint_ui(…)` or `blueprint.ui(…)`.
+
+#### Spaces
+Points, vectors, rays etc all live in different _spaces_. Whenever there is room for ambiguity, we explicitly state which space something is in, e.g. with `ray_in_world`.
+
+Here are some of our standard spaces:
+
+* `ui`: coordinate system used by `egui`, measured in logical pixels ("points"), with origin in the top left
+* `image`: image pixel coordinates, possibly with an added `z=depth`
+* `space`: a user-defined space where they log stuff into
+* `world`: the common coordinate system of a 3D scene, usually same as `space`
+* `view`: X=right, Y=down, Z=back, origin = center of screen
+
+#### Matrices
+We use column vectors, which means matrix multiplication is done as `M * v`, i.e. we read all matrix/vector operations right-to-left. We therefore name all transform matrices as `foo_from_bar`, for instance:
+
+```rust
+let point_in_world = world_from_view * point_in_view;
+```
+
+This means the name of the space matches up nicely, e.g.:
+
+```rust
+let projection_from_object = projection_from_view * view_from_world * world_from_object;
+```
+
+See <https://www.sebastiansylvan.com/post/matrix_naming_convention/> for motivation.
+
+For consistency, we use the same naming convention for other non-matrix transforms too. For instance, functions: `let screen = screen_from_world(world);`.
+
+#### Vectors vs points
+Vectors are directions with magnitudes. Points are positions.