[docs] Add contributors docs

CODE_OF_CONDUCT.rst

@@ -0,0 +1,3 @@
+This project adheres to the `Open Code of Conduct`_. By participating, you are expected to honor this code.
+
+.. _Open Code of Conduct: https://github.com/spotify/code-of-conduct/blob/master/code-of-conduct.md

CONTRIBUTING.rst

@@ -0,0 +1,253 @@
+Contributing to the ``klio`` ecosystem
+======================================
+
+Requirements
+------------
+
+* Python 3.5, 3.6, 3.7, and 3.8 (suggestion: use `pyenv <https://github.com/pyenv/pyenv>`_ to install required versions)
+* `tox <https://tox.readthedocs.io/en/latest/>`_ (suggestion: use `pipx <https://pypi.org/project/pipx/>`_ to install ``tox``)
+* `protoc <https://github.com/protocolbuffers/protobuf/>`_
+
+Project Overview
+----------------
+
+Each directory is its own Python package (except for ``docs``, ``examples``, and ``integration``):
+
+.. code-block:: sh
+
+    ├── audio/
+    ├── cli/
+    ├── core/
+    ├── devtools/
+    ├── docs/
+    ├── examples/
+    ├── exec/
+    ├── integration/
+    └── lib/
+
+The ``klio-cli`` (user-facing) commands essentially wrap ``klio-exec`` (non-user-facing) commands.
+``klio-cli $CMD`` sets up the run context needed to correctly run ``klio-exec $CMD`` inside the
+job's Docker container (i.e. mounts the job directory, sets up env vars, copies over credentials).
+In general, changes to ``klio-cli`` require a corresponding change in ``klio-exec``, but there are
+a few ``klio-cli`` commands self-contained and therefore do not need any changes to ``klio-exec`` (i.e. ``klio message publish``).
+
+``klio-core`` is a non-user-facing library of common utilities, including protobuf definitions.
+The ``klio`` library contains the base transform for users to implement, and the main message
+processing logic (i.e. triggering parents, skipping already processed work, etc). ``klio-devtools``
+is just a collection of utilities to help aid the development of klio.
+
+
+Run Tests
+---------
+
+Tests can be invoked in two ways: ``pytest`` and ``tox``.
+
+
+Run tests via ``tox``
+^^^^^^^^^^^^^^^^^^^^^
+
+This is the recommended way to run tests since tox will manage its own virtualenvs and is also
+what will be run as part of the CI checks.
+
+`tox <https://tox.readthedocs.io/en/latest/>`_ is a tool to easily run tests and do other
+development tasks in a Python project, kind of like a Makefile for Python.  It is configured with
+a ``tox.ini`` file in the root folder of a project.  Because Klio is divided into several Python
+projects each one has its own ``tox.ini``.
+
+.. caution::
+
+    Occasionally tox may cache an outdated version of a dependency.
+    If you find yourself getting errors that aren't reproducible by others or by CI, try deleting the ``.tox`` folder in your project's directory.
+
+Before you begin, be sure your shell is **not** using a virtualenv.
+
+First, install ``tox``:
+
+.. code-block:: sh
+
+    $ pipx install tox
+
+Now you can navigate to one of the Klio sub-project directories and use ``tox`` to run unit
+tests.  Currently ``tox`` is configured to run all tests using several different Python versions.
+This can take a while, so it's much faster when testing to just run with one environment:
+
+.. code-block:: sh
+
+    $ tox -e py37
+
+This might take a while the first time as ``tox`` sets up its envs and downloads dependencies, but
+after that it should be pretty fast.
+
+To run a specific test:
+
+.. code-block:: sh
+
+    $ tox -e py37 -- tests/unit/test_message_handler.py::test_preprocess_klio_message_non_klio
+
+
+You can also use
+
+.. code-block:: sh
+
+    $ tox -e format
+
+to run ``black`` and auto-format your code, and
+
+.. code-block:: sh
+
+    $ tox -e lint
+
+to run ``flake8`` which enforces a number of syntactic and formatting conventions.  The CI will
+run both of these (with ``black`` in non-edit mode), so be sure to run these yourself!
+
+
+Run tests via ``pytest`` and ``pyenv`` for Multi-project Dev
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The above testing with ``tox`` will not work if you need to makes changes across several projects
+at once, instead you will need to create your own virtualenv and install your checked-out code as
+a local dependency.  Then you'll bypass ``tox`` and run ``pytest`` directly.
+
+Ensure that you have followed the above instructions and installed `pyenv
+<https://github.com/pyenv/pyenv>`_.  You will also need to install `pyenv-virtualenv
+<https://github.com/pyenv/pyenv-virtualenv>`_.  Now we'll use ``pyenv`` to create a new virtualenv called ``klio-dev`` based on Python 3.7.7:
+
+.. code-block:: sh
+
+    $ pyenv virtualenv 3.7.7 klio-dev
+
+We then have to activate it in order to use it:
+
+.. code-block:: sh
+
+    $ pyenv activate klio-dev
+
+Your console's prompt should now include ``(klio-dev)`` to help you keep track of which venv
+you're currently using.  Now we will install your local ``klio-core`` as a library:
+
+.. code-block:: sh
+
+    $ cd <klio-core-directory>
+    $ pip install -e ".[dev]"
+
+Next, navigate to the ``klio-cli`` dir and do the same thing:
+
+.. code-block:: sh
+
+    $ cd <klio-cli-directory>
+    $ pip install -e ".[dev]"
+
+Now you can run ``pytest`` directly (do _not_ run ``tox``)
+
+.. code-block:: sh
+
+    # in klio-cli directory
+    $ pytest
+
+You can now make changes in either project and they will be picked up immediately, no need to
+re-install each time.
+
+If you want to try running the ``klio-cli`` command directly yourself, you will have to run one
+more command:
+
+.. code-block:: sh
+
+    $ pyenv rehash
+
+This is necessary to make sure your terminal is pointing to the local version.
+
+
+Integration Testing with ``klio-devtools``
+------------------------------------------
+
+The easiest way we've found to do local integration testing of changes to any Klio library *before*
+making PRs is:
+
+1. Identify or create a simple job that you can use to test your changes locally.
+
+2. Temporarily update the package versions locally for ``klio-exec``, ``klio-core``, and ``klio`` libraries (found in their top-level ``__init__.py``). A helpful convention is to bump the patch version and add a suffix of ``.devN``. For instance, if the current version is ``1.2.3``, update it to ``1.2.4.dev1``. This is to ensure the correct versions are installed, and not reusing an already-released version. Once you're satisfied with your changes, undo these changes then run ``bumpversion <part>`` as usual.
+
+3. Make sure the virtualenv for the ``klio-cli`` is activated, and that the ``klio-devtools`` package is installed (usually installed via ``pip install -e ".[dev]"``).
+
+4. In the directory of your simple job, run ``klio-dev develop --klio-path $PATH_TO_REPO``. This will launch you into the job's container with ``klio``, ``klio-core``, and ``klio-exec`` installed as an editable Python package, and will pick up on any changes you make locally.
+
+.. note::
+
+    When using ``klio-dev develop``, the path to the root of the Klio repo can be either
+    relative or absolute.
+
+5. Now that you're inside the job's container, you can run ``klioexec $CMD`` or whatever else is needed to do any manual integration testing.
+
+
+Protobuf
+--------
+
+``klio`` uses protobuf for messages between transforms via Google Pub/Sub. The protobuf definition
+is located in the ``klio-core`` `library <https://github.com/spotify/klio/tree/master/core/src/klio_core/proto>`_.
+
+
+Compile Protobuf
+^^^^^^^^^^^^^^^^
+
+.. admonition:: Attention!
+    :class: caution
+
+    If creating a new protobuf version, be sure to create a new directory and update the command/
+    tox config below.
+
+Manually compile protobuf:
+
+.. code-block:: sh
+
+    # Within top-level of repo
+    $ protoc \
+        --proto_path src/klio/proto/v1beta1 klio.proto \
+        --python_out src/klio/proto/v1beta1
+
+
+Or via ``tox``:
+
+.. code-block:: sh
+
+    # outside a virtualenv
+    $ tox -e protoc
+
+Documentation
+-------------
+
+All documentation related to the Klio ecosystem is in the top-level ``docs/`` directory/
+
+Generate Documentation
+^^^^^^^^^^^^^^^^^^^^^^
+
+Create a new virtual environment for documentation & install dependencies:
+
+.. code-block:: sh
+
+    $ pyenv virtualenv 3.7.7 klio-docs
+    $ pyenv activate klio-docs
+    (klio-docs) $ cd docs
+    (klio-docs) $ pip install -r requirements.txt -r klio-requirements.txt
+
+
+.. note::
+
+    You may see some errors of version conflicts when installing documentation dependencies.
+    However, these should be benign and should not get in the way of generating documentation.
+
+
+Once the environment is setup, documentation can be generated and viewed via ``make``:
+
+.. code-block:: sh
+
+    # in the docs/ directory
+    (klio-docs) $ make clean && make html
+
+To view them locally
+
+.. code-block:: sh
+
+    # in the docs/ directory
+    (klio-docs) $ make clean && make livehtml
+
+Then navigate to ``http://localhost:8888`` in your browser.

docs/src/contributors.rst

@@ -1,6 +1,7 @@
-Contributing
-============
 
-.. todo::
+.. include:: ../../CONTRIBUTING.rst
 
-    add our contributing docs!
+Code of Conduct
+---------------
+
+.. include:: ../../CODE_OF_CONDUCT.rst

docs/src/index.rst

@@ -113,7 +113,7 @@ A collection of all the todos that we should probably address before we publicly
    User Guide <userguide/index>
    Reference <reference/index>
    FAQs <faqs/index>
-   contributors
+   Contributing <contributors>
 
 
 Indices and tables