First draft of the documentation overhaul.

To build the docs, run ./admin/build-doc. To browse them, either get
them on any static website, or just run ./admin/serve-doc to serve
them quickly off of port 8080.

build-doc sets up a virtualenv to avoid needing Sphinx installed
system-wide. serve-doc needs thttpd installed.

Signed-off-by: Tommi Virtanen <[email protected]>

.gitignore

@@ -52,3 +52,4 @@ core
 .settings
 .project
 .cproject
+/build-doc

admin/build-doc

@@ -0,0 +1,23 @@
+#!/bin/sh
+set -e
+cd "$(dirname "$0")"
+cd ..
+
+dia --filter=png-libart --export=doc/overview.png.tmp doc/overview.dia
+mv -- doc/overview.png.tmp doc/overview.png
+
+install -d -m0755 build-doc
+cd build-doc
+
+if [ ! -e virtualenv ]; then
+    virtualenv --no-site-packages virtualenv
+fi
+if [ ! -x virtualenv/bin/sphinx-build ]; then
+    ./virtualenv/bin/pip install sphinx
+fi
+
+install -d -m0755 \
+    output/html \
+    output/man
+./virtualenv/bin/sphinx-build -a -b dirhtml -d doctrees ../doc output/html
+./virtualenv/bin/sphinx-build -a -b man -d doctrees ../doc output/man

admin/serve-doc

@@ -0,0 +1,29 @@
+#!/usr/bin/python
+import SimpleHTTPServer
+import SocketServer
+import os
+import sys
+
+path = os.path.dirname(sys.argv[0])
+os.chdir(path)
+os.chdir('..')
+os.chdir('build-doc/output/html')
+
+class ReusingTCPServer(SimpleHTTPServer.SimpleHTTPRequestHandler):
+    allow_reuse_address = True
+
+    def send_head(self):
+        # horrible kludge because SimpleHTTPServer is buggy wrt
+        # slash-redirecting of requests with query arguments, and will
+        # redirect to /foo?q=bar/ -- wrong slash placement
+        self.path = self.path.split('?', 1)[0]
+        return SimpleHTTPServer.SimpleHTTPRequestHandler.send_head(self)
+
+httpd = SocketServer.TCPServer(
+    ("", 8080),
+    ReusingTCPServer,
+    )
+try:
+    httpd.serve_forever()
+except KeyboardInterrupt:
+    pass

doc/.gitignore

@@ -0,0 +1,2 @@
+*.tmp
+/overview.png

doc/architecture.rst

@@ -0,0 +1,27 @@
+======================
+ Architecture of Ceph
+======================
+
+- Introduction to Ceph Project
+
+  - High-level overview of project benefits for users (few paragraphs, mention each subproject)
+  - Introduction to sub-projects (few paragraphs to a page each)
+
+    - RADOS
+    - RGW
+    - RBD
+    - Ceph
+
+  - Example scenarios Ceph projects are/not suitable for
+  - (Very) High-Level overview of Ceph
+
+    This would include an introduction to basic project terminology,
+    the concept of OSDs, MDSes, and Monitors, and things like
+    that. What they do, some of why they're awesome, but not how they
+    work.
+
+- Discussion of MDS terminology, daemon types (active, standby,
+  standby-replay)
+
+
+.. todo:: write me

doc/conf.py

@@ -0,0 +1,58 @@
+project = u'Ceph'
+copyright = u'2011, New Dream Network'
+version = 'dev'
+release = 'dev'
+
+templates_path = ['_templates']
+source_suffix = '.rst'
+master_doc = 'index'
+exclude_patterns = ['**/.#*', '**/*~']
+pygments_style = 'sphinx'
+
+html_theme = 'nature'
+html_title = "Ceph documentation"
+html_logo = 'logo.jpg'
+html_favicon = 'favicon.ico'
+html_static_path = ['_static']
+html_use_smartypants = True
+html_show_sphinx = False
+
+extensions = ['sphinx.ext.todo']
+todo_include_todos = True
+
+def _get_manpages():
+    import os
+    man_dir = os.path.join(
+        os.path.dirname(__file__),
+        'man',
+        )
+    sections = os.listdir(man_dir)
+    for section in sections:
+        section_dir = os.path.join(man_dir, section)
+        if not os.path.isdir(section_dir):
+            continue
+        for filename in os.listdir(section_dir):
+            base, ext = os.path.splitext(filename)
+            if ext != '.rst':
+                continue
+            if base == 'index':
+                continue
+            with file(os.path.join(section_dir, filename)) as f:
+                one = f.readline()
+                two = f.readline()
+                three = f.readline()
+                assert one == three
+                assert all(c=='=' for c in one.rstrip('\n'))
+                two = two.strip()
+                name, rest = two.split('--', 1)
+                assert name.strip() == base
+                description = rest.strip()
+            yield (
+                os.path.join('man', section, base),
+                base,
+                description,
+                '',
+                section,
+                )
+
+man_pages = list(_get_manpages())

doc/glossary.rst

@@ -0,0 +1,40 @@
+==========
+ Glossary
+==========
+
+.. glossary::
+   :sorted:
+
+   OSD
+     .. todo:: write me
+
+   MDS
+     .. todo:: write me
+
+   Mon
+     .. todo:: write me
+
+   Cap
+     .. todo:: write me
+
+   Object
+     .. todo:: write me
+
+   cosd
+     .. todo:: write me
+
+   cmon
+     .. todo:: write me
+
+   cmds
+     Ceph MDS, the actual daemon blahblah
+
+   rgw
+     .. todo:: write me
+
+   radosgw
+     .. todo:: write me
+
+
+.. todo:: what's missing from glossary
+

doc/index.rst

@@ -0,0 +1,106 @@
+================================
+ Ceph -- petabyte scale storage
+================================
+
+
+Welcome to Ceph
+===============
+
+Ceph is a distributed network storage and file system with distributed
+metadata management and POSIX semantics.
+
+RADOS is a reliable object store, used by Ceph, but also directly
+accessible.
+
+``radosgw`` is an S3-compatible RESTful HTTP service for object
+storage, using RADOS storage.
+
+RBD is a Linux kernel feature that exposes RADOS storage as a block
+device. Qemu/KVM also has a direct RBD client, that avoids the kernel
+overhead.
+
+.. image:: overview.png
+
+
+Getting started
+===============
+
+- :doc:`tutorial`: how to install a cluster for testing
+- `Ceph Blog <http://ceph.newdream.net/news/>`__: news and status info
+
+
+Mailing lists, bug tracker, IRC channel
+=======================================
+
+- The development mailing list is at [email protected], and
+  archived at Gmane_. Send email to subscribe_ or unsubscribe_.
+- `Bug/feature tracker <http://tracker.newdream.net/projects/ceph>`__:
+  for filing bugs and feature requests.
+- IRC channel ``#ceph`` on ``irc.oftc.net``: Many of the core
+  developers are on IRC, especially daytime in the US/Pacific
+  timezone. You are welcome to join and ask questions. You can find
+  logs of the channel `here <http://irclogs.ceph.widodh.nl/>`__.
+- `Commercial support <http://ceph.newdream.net/support/>`__
+
+.. _subscribe: mailto:[email protected]?body=subscribe+ceph-devel
+.. _unsubscribe: mailto:[email protected]?body=unsubscribe+ceph-devel
+.. _Gmane: http://news.gmane.org/gmane.comp.file-systems.ceph.devel
+
+
+Status
+======
+
+The Ceph project is currently focusing on stability. Users are
+welcome, but we do not recommend storing valuable data with it yet
+without proper precautions.
+
+As of this writing, RADOS is the most stable component, and RBD block
+devices are fairly reliable, if not performance tuned yet. The OSD
+component of RADOS relies heavily on the stability and performance of
+the underlying filesystem, and we keep hearing reports of ``btrfs``
+issues; while on the long term we believe in ``btrfs``, in the short
+term you may wish to carefully consider the tradeoffs between ``ext4``
+and ``btrfs``, and make sure you are running the latest Linux kernel.
+
+Radosgw is still going through heavy development, but it will likely
+mature next.
+
+The Ceph filesystem is functionally fairly complete, but has not been
+tested well enough at scale and under load yet. Multi-master MDS is
+still problematic and we recommend running just one active MDS
+(standbys are ok). If you have problems with ``kclient`` or ``cfuse``,
+you may wish to try the other option; in general, ``kclient`` is
+expected to be faster (but be sure to use the latest Linux kernel!)
+while ``cfuse`` provides better stability by not triggering kernel
+crashes.
+
+As individual systems mature enough, we move to improving their
+performance (throughput, latency and jitter). This work is still
+mostly ahead of us.
+
+Ceph is developed on Linux. Other platforms may work, but are not the
+focus of the project. Filesystem access from other operating systems
+can be done via NFS or Samba re-exports.
+
+
+Table of Contents
+=================
+
+.. toctree::
+   :maxdepth: 3
+
+   tutorial
+   architecture
+   ops/index
+   man/index
+   papers
+   glossary
+
+
+Indices and tables
+==================
+
+- :ref:`genindex`
+- :ref:`modindex`
+- :ref:`search`
+

doc/man/8/ceph.rst

@@ -0,0 +1,5 @@
+==========================================
+ ceph -- ceph file system control utility
+==========================================
+
+.. todo:: write me

doc/man/8/index.rst

@@ -0,0 +1,9 @@
+===========================================
+ Section 8, system administration commands
+===========================================
+
+.. toctree::
+   :maxdepth: 1
+   :glob:
+
+   *

doc/man/8/rados.rst

@@ -0,0 +1,5 @@
+=======================================
+ rados -- rados object storage utility
+=======================================
+
+.. todo:: write me