Starting point for python docs via mkdocs (rerun-io#887)

* Proof-of-concept mkdocs
* Instructions for serving the docs
* Set up manual organization of function lists in parallel with full autogen package
* Make package listing look like folder
* Titles come from literate nav
* Add meta-information on writing the docs themselves

justfile

@@ -58,6 +58,10 @@ py-lint:
 py-test:
     python -m pytest rerun_py/tests/unit/
 
+# Serve the python docs locally
+py-docs-serve:
+    mkdocs serve -f rerun_py/mkdocs.yml -w rerun_py
+
 ### Rust
 
 # Generate and open the documentation for Rerun and all of its Rust dependencies.

rerun_py/README.md

@@ -92,6 +92,25 @@ Now you can install `rerun` in any Python3 environment using:
 pip3 install target/wheels/*.whl
 ```
 
+## Viewing the docs locally
+The rerun python docs are generated using `mkdocs`
+
+Install the doc requirements:
+```
+pip install -r rerun_py/requirements-doc.txt
+```
+
+Serve the docs:
+```
+mkdocs serve -f rerun_py/mkdocs.yml -w rerun_py
+```
+or
+```
+just py-docs-serve
+```
+
+For information on how the docs system works, see: [docs/docs.md](docs/docs.md)
+
 
 ## Troubleshooting
 You can run with `RUST_LOG=debug` to get more output out of the rerun SDK.

rerun_py/docs/SUMMARY.txt

@@ -0,0 +1,3 @@
+* [Common APIs](common/)
+* [Package Listing](package/)
+* [Writing Docs](docs.md)

rerun_py/docs/common/SUMMARY.txt

@@ -0,0 +1,6 @@
+* [index](index.md)
+* [Initialization](initialization.md)
+* [Logging Primitives](primitives.md)
+* [Logging Images](images.md)
+* [Plotting](plotting.md)
+* [Transforms](transforms.md)

rerun_py/docs/common/images.md

@@ -0,0 +1,5 @@
+::: rerun.log_image
+::: rerun.log_image_file
+::: rerun.log_depth_image
+::: rerun.log_segmentation_image
+::: rerun.log_annotation_context

rerun_py/docs/common/index.md

@@ -0,0 +1,4 @@
+# Common APIs
+
+!!! warning
+    TODO(jleibs): Very high level pointers to common API groups

rerun_py/docs/common/initialization.md

@@ -0,0 +1,4 @@
+::: rerun.init
+::: rerun.set_recording_id
+::: rerun.connect
+::: rerun.spawn_and_connect

rerun_py/docs/common/plotting.md

@@ -0,0 +1 @@
+::: rerun.log_scalar

rerun_py/docs/common/primitives.md

@@ -0,0 +1,8 @@
+::: rerun.log_point
+::: rerun.log_points
+::: rerun.log_rect
+::: rerun.log_rects
+::: rerun.log_obb
+::: rerun.log_path
+::: rerun.log_line_segments
+::: rerun.log_arrow

rerun_py/docs/common/transforms.md

@@ -0,0 +1,4 @@
+::: rerun.log_rigid3
+::: rerun.log_pinhole
+::: rerun.log_unknown_transform
+::: rerun.log_view_coordinates

rerun_py/docs/css/mkdocstrings.css

@@ -0,0 +1,10 @@
+/* Indentation. */
+div.doc-contents:not(.first) {
+  padding-left: 25px;
+  border-left: .05rem solid var(--md-typeset-table-color);
+}
+
+/* Avoid breaking parameters name, etc. in table cells. */
+td code {
+  word-break: normal !important;
+}

rerun_py/docs/docs.md

@@ -0,0 +1,23 @@
+# Python Docs
+
+A high-level overview of writing and previewing the Rerun python documentation.
+
+## Getting started with docs
+
+### Dependencies
+All of the dependencies for documentation generation are captured in the requirements file:
+```
+pip install -r rerun_py/requirements-doc.txt
+```
+
+### Serving the docs
+The docs can be previewed locally using `mkdocs`
+
+This will watch the contents of the `rerun_py` folder and refresh documentation live as files are changed.
+```
+mkdocs serve -f rerun_py/mkdocs.yml -w rerun_py
+```
+or
+```
+just serve-py-docs
+```

rerun_py/docs/gen_package_index.py

@@ -0,0 +1,74 @@
+r"""
+Generate the code reference pages and navigation for the source tree.
+
+This helper is used by the [mkdocs-gen-files plugin](https://oprypin.github.io/mkdocs-gen-files)
+
+When building the documentation it will walk our python source-tree and create a collection
+of virtual files, as well as a `SUMMARY.txt` for use with the
+[mkdocs-literature-nav plugin](https://oprypin.github.io/mkdocs-literate-nav)
+
+Example virtual file:
+
+`package/rerun/log/points.md`:
+```
+::: rerun.log.points
+```
+
+`SUMMARY.txt`:
+```
+* [index](index.md)
+* rerun/
+    * [\\__init__.py](rerun/__init__.md)
+    * [color_conversion.py](rerun/color_conversion.md)
+    * components/
+        * [\\__init__.py](rerun/components/__init__.md)
+        * [annotation.py](rerun/components/annotation.md)
+        * [arrow.py](rerun/components/arrow.md)
+        * [box.py](rerun/components/box.md)
+        * [color.py](rerun/components/color.md)
+        * [instance.py](rerun/components/instance.md)
+```
+
+"""
+
+from pathlib import Path
+
+import mkdocs_gen_files
+
+root = Path(__file__).parent.parent.resolve()
+package_dir = Path("package")
+
+nav = mkdocs_gen_files.Nav()
+nav["index"] = "index.md"
+
+for path in sorted(root.joinpath("rerun").rglob("*.py")):
+    rel_path = path.relative_to(root)
+
+    # Build up a python-style dotted identifier for the module
+    # This is how `mkdocstrings-python` will import the module and also how
+    # we refer to it with links internal to the docs
+    module_path = rel_path.with_suffix("")
+    ident_parts = tuple(module_path.parts)
+    if ident_parts[-1] == "__init__":
+        ident_parts = ident_parts[:-1]
+    elif ident_parts[-1] == "__main__":
+        continue
+    ident = ".".join(ident_parts)
+
+    # The doc_path is the md file that we will generate inside the virtual package folder
+    doc_path = rel_path.with_suffix(".md")
+    write_path = package_dir.joinpath(doc_path)
+
+    # Within the nav, we want non-leaf nodes to appear as folders
+    nav_parts = tuple(p + "/" for p in rel_path.parts[:-1]) + (path.parts[-1],)
+
+    # Register the nav-parts index with the generated doc-path
+    nav[nav_parts] = doc_path.as_posix()
+
+    # Write the virtual file
+    with mkdocs_gen_files.open(write_path, "w") as fd:
+        fd.write(f"::: {ident}")
+
+# Generate the SUMMARY.txt file
+with mkdocs_gen_files.open(package_dir.joinpath("SUMMARY.txt"), "w") as nav_file:
+    nav_file.writelines(nav.build_literate_nav())

rerun_py/docs/package/index.md

@@ -0,0 +1,6 @@
+# Full Package
+
+* [rerun][]: Top-level User-facing APIs. See also: [Common APIs](/common)
+    * [rerun.log][]: APIs for logging data
+    * [rerun.color_conversion][]: Conversion utilities related to colors
+    * [rerun.components][]: Helpers for constructing arrow components

rerun_py/docs/templates/python/material/function.html

@@ -0,0 +1,83 @@
+{# Very minimally patched from:
+https://github.com/mkdocstrings/python/blob/master/src/mkdocstrings_handlers/python/templates/material/_base/function.html
+
+See: CHANGE comments below
+#}
+{{ log.debug("Rendering " + function.path) }}
+
+<div class="doc doc-object doc-function">
+{% with html_id = function.path %}
+
+  {% if root %}
+    {% set show_full_path = config.show_root_full_path %}
+    {% set root_members = True %}
+  {% elif root_members %}
+    {% set show_full_path = config.show_root_members_full_path or config.show_object_full_path %}
+    {% set root_members = False %}
+  {% else %}
+    {% set show_full_path = config.show_object_full_path %}
+  {% endif %}
+
+  {% if not root or config.show_root_heading %}
+
+    {% filter heading(heading_level,
+        role="function",
+        id=html_id,
+        class="doc doc-heading",
+        toc_label=function.name ~ "()") %}
+
+      {% if config.separate_signature %}
+        <span class="doc doc-object-name doc-function-name">{% if show_full_path %}{{ function.path }}{% else %}{{ function.name }}{% endif %}</span>
+      {% else %}
+        {% filter highlight(language="python", inline=True) %}
+          {# CHANGE: Insert def before function for better highlighting #}
+          def {% if show_full_path %}{{ function.path }}{% else %}{{ function.name }}{% endif %}
+          {% include "signature.html" with context %}
+        {% endfilter %}
+      {% endif %}
+
+      {% with labels = function.labels %}
+        {% include "labels.html" with context %}
+      {% endwith %}
+
+    {% endfilter %}
+
+    {% if config.separate_signature %}
+      {% filter highlight(language="python", inline=False) %}
+        {# CHANGE: Insert def before function for better highlighting #}
+        def {% filter format_signature(config.line_length) %}
+          {% if show_full_path %}{{ function.path }}{% else %}{{ function.name }}{% endif %}
+          {% include "signature.html" with context %}
+        {% endfilter %}
+      {% endfilter %}
+    {% endif %}
+
+  {% else %}
+    {% if config.show_root_toc_entry %}
+      {% filter heading(heading_level,
+          role="function",
+          id=html_id,
+          toc_label=function.path if config.show_root_full_path else function.name,
+          hidden=True) %}
+      {% endfilter %}
+    {% endif %}
+    {% set heading_level = heading_level - 1 %}
+  {% endif %}
+
+  {# CHANGE: disable gating of first for root docstring #}
+  {#<div class="doc doc-contents {% if root %}first{% endif %}">#}
+  <div class="doc doc-contents {% if root %}{% endif %}">
+    {% with docstring_sections = function.docstring.parsed %}
+      {% include "docstring.html" with context %}
+    {% endwith %}
+
+    {% if config.show_source and function.source %}
+      <details class="quote">
+        <summary>Source code in <code>{{ function.relative_filepath }}</code></summary>
+        {{ function.source|highlight(language="python", linestart=function.lineno, linenums=True) }}
+      </details>
+    {% endif %}
+  </div>
+
+{% endwith %}
+</div>

rerun_py/mkdocs.yml

@@ -0,0 +1,53 @@
+# mkdocs.yml
+# Top-level config for mkdocs
+# See: https://www.mkdocs.org/user-guide/configuration/
+site_name: Rerun Python APIs
+repo_url: https://github.com/rerun-io/rerun/
+
+# Use the material theme
+# Override some options for nav: https://squidfunk.github.io/mkdocs-material/setup/setting-up-navigation/
+theme:
+  name: "material"
+  features:
+    - navigation.tabs
+    - navigation.instant
+    - navigation.indexes
+    - navigation.tracking
+    - navigation.expand
+
+plugins:
+  - search # https://squidfunk.github.io/mkdocs-material/setup/setting-up-site-search/
+  - mkdocstrings: # https://mkdocstrings.github.io/usage/#global-options
+      custom_templates: rerun_py/docs/templates # Override the function template. NOTE: relative to working directory. (https://github.com/mkdocstrings/mkdocstrings/issues/477)
+      handlers:
+        python:
+          paths: ["."] # Lookup python modules relative to this path
+          import: # Cross-references for python and numpy
+            - https://docs.python.org/3/objects.inv
+            - https://numpy.org/doc/stable/objects.inv
+          options: # https://mkdocstrings.github.io/python/usage/#globallocal-options
+            show_source: no
+            docstring_style: numpy
+            members_order: source
+            heading_level: 3
+            show_root_heading: true
+  - gen-files: # https://oprypin.github.io/mkdocs-gen-files
+      scripts:
+        - docs/gen_package_index.py
+  - literate-nav: # https://oprypin.github.io/mkdocs-literate-nav
+      nav_file: SUMMARY.txt
+  - redirects: # https://github.com/mkdocs/mkdocs-redirects
+      redirect_maps:
+        "index.md": "common/index.md"
+
+# https://www.mkdocs.org/user-guide/configuration/#markdown_extensions
+# https://squidfunk.github.io/mkdocs-material/setup/extensions/python-markdown-extensions/
+markdown_extensions:
+  - admonition # https://squidfunk.github.io/mkdocs-material/reference/admonitions/
+  - pymdownx.highlight # https://mkdocstrings.github.io/theming/#syntax-highlighting
+  - toc:
+      toc_depth: 4
+
+# Some extra styling
+extra_css:
+  - css/mkdocstrings.css

rerun_py/requirements-doc.txt

@@ -0,0 +1,8 @@
+mkdocs
+mkdocs-gen-files
+mkdocs-literate-nav
+mkdocs-material
+mkdocs-material-extensions
+mkdocs-redirects
+mkdocstrings
+mkdocstrings-python