Move from gitbook to docusaurus, build docs in Travis CI (solana-labs…

…#10970)

* fix: ignore unknown fields in more RPC responses

* Remove mdbook infrastructure

* Delete gitattributes and other theme related items

Move all docs to /docs folder to support Docusaurus

* all docs need to be moved to /docs

* can be changed in the future

Add Docusaurus infrastructure

* initialize docusaurus repo

Remove trailing whitespace, add support for eslint

Change Docusaurus configuration to support `src`

* No need to rename the folder! Change a setting and we're all good to
go.

* Fixing rebase items

* Remove unneccessary markdown file, fix type

* Some fonts are hard to read. Others, not so much. Rubik, you've been
sidelined. Roboto, into the limelight!

* As much as we all love tutorials, I think we all can navigate around a
markdown file. Say goodbye, `mdx.md`.

* Setup deployment infrastructure

* Move docs job from buildkite to travic

* Fix travis config

* Add vercel token to travis config

* Only deploy docs after merge

* Docker rust env

* Revert "Docker rust env"

This reverts commit f84bc20.

* Build CLI usage from docker

* Pacify shellcheck

* Run job on PR and new commits for publication

* Update README

* Fix svg image building

* shellcheck

Co-authored-by: Michael Vines <[email protected]>
Co-authored-by: Ryan Shea <[email protected]>
Co-authored-by: publish-docs.sh <[email protected]>

.travis.yml

@@ -106,3 +106,25 @@ jobs:
       script:
         - ../.travis/commitlint.sh
         - source .travis/script.sh
+
+    # docs pull request
+    - name: "docs"
+      if: type IN (push, pull_request)
+      language: node_js
+      node_js:
+        - "node"
+
+      services:
+        - docker
+
+      cache:
+        directories:
+          - ~/.npm
+
+      before_install:
+        - .travis/affects.sh docs/ .travis || travis_terminate 0
+        - cd docs/
+        - source .travis/before_install.sh
+
+      script:
+        - source .travis/script.sh

ci/buildkite-pipeline.sh

@@ -211,12 +211,7 @@ pull_or_push_steps() {
     all_test_steps
   fi
 
-  # doc/ changes:
-  if affects ^docs/; then
-    command_step docs ". ci/rust-version.sh; ci/docker-run.sh \$\$rust_nightly_docker_image docs/build.sh" 5
-  fi
-
-  # web3.js and explorer changes run on Travis...
+  # web3.js, explorer and docs changes run on Travis...
 }
 
 

docs/.eslintrc

@@ -0,0 +1,21 @@
+{
+    "env": {
+      "browser": true,
+      "node": true
+    },
+    "parser": "babel-eslint",
+    "rules": {
+      "strict": 0,
+      "no-unused-vars": ["error", { "argsIgnorePattern": "^_" }],
+      "no-trailing-spaces": ["error", { "skipBlankLines": true }]
+    },
+    "settings": {
+      "react": {
+        "version": "detect", // React version. "detect" automatically picks the version you have installed.
+      }
+    },
+    "extends": [
+      "eslint:recommended",
+      "plugin:react/recommended"
+    ]
+  }

docs/.gitignore

@@ -0,0 +1,22 @@
+# Dependencies
+/node_modules
+
+# Production
+/build
+
+# Generated files
+.docusaurus
+.cache-loader
+.vercel
+/static/img/*.svg
+
+# Misc
+.DS_Store
+.env.local
+.env.development.local
+.env.test.local
+.env.production.local
+
+npm-debug.log*
+yarn-debug.log*
+yarn-error.log*

docs/.travis/before_install.sh

@@ -0,0 +1,9 @@
+# |source| this file
+
+curl -sL https://deb.nodesource.com/setup_12.x | sudo -E bash -
+sudo apt install -y nodejs
+
+npm install --global docusaurus-init
+docusaurus-init
+
+npm install --global vercel

docs/.travis/script.sh

@@ -0,0 +1,4 @@
+# |source| this file
+
+set -ex
+./build.sh

docs/README.md

@@ -1,31 +1,38 @@
-Building the Solana Docs
----
-
-Install dependencies, build, and test the docs:
-
-```bash
-$ brew install coreutils
-$ brew install mscgen
-$ cargo install svgbob_cli
-$ cargo install mdbook-linkcheck
-$ cargo install mdbook
-$ ./build.sh
-```
+# Docs Readme
+
+Solana's Docs are built using [Docusaurus 2](https://v2.docusaurus.io/) with `npm`.
+Static content delivery is handled using `vercel`.
 
-Run any Rust tests in the markdown:
+### Installation
 
-```bash
-$ make test
+```
+$ npm install
 ```
 
-Render markdown as HTML:
+### Local Development
 
-```bash
-$ make build
+```
+$ npm run start
 ```
 
-Render and view the docs:
+This command starts a local development server and open up a browser window. Most changes are reflected live without having to restart the server.
 
-```bash
-$ make open
+### Build
+#### Local Build Testing
+```
+$ npm run build
 ```
+
+This command generates static content into the `build` directory and can be
+served using any static contents hosting service.
+
+#### CI Build Flow
+The docs are built and published in Travis CI with the `docs/build.sh` script.
+On each PR, the docs are built, but not published.
+
+In each post-commit build, docs are built and published using `vercel` to their
+respective domain depending on the build branch.
+
+ - Master branch docs are published to `edge.docs.solana.com`
+ - Beta branch docs are published to `beta.docs.solana.com`
+ - Latest release tag docs are published to `docs.solana.com`

docs/babel.config.js

@@ -0,0 +1,3 @@
+module.exports = {
+  presets: [require.resolve("@docusaurus/core/lib/babel/preset")],
+};

docs/build-cli-usage.sh

@@ -3,6 +3,9 @@ set -e
 
 cd "$(dirname "$0")"
 
+# shellcheck source=ci/rust-version.sh
+source ../ci/rust-version.sh stable
+
 : "${rust_stable:=}" # Pacify shellcheck
 
 usage=$(cargo +"$rust_stable" -q run -p solana-cli -- -C ~/.foo --help | sed -e 's|'"$HOME"'|~|g' -e 's/[[:space:]]\+$//')

docs/build.sh

@@ -1,5 +1,8 @@
 #!/usr/bin/env bash
-set -e
+set -ex
+
+# shellcheck source=ci/env.sh
+source ../ci/env.sh
 
 cd "$(dirname "$0")"
 
@@ -12,6 +15,31 @@ find src -name '*.md' -a \! -name SUMMARY.md |
    fi
  done
 
-mdbook --version
-mdbook-linkcheck --version
-make -j"$(nproc)" test
+: "${rust_stable_docker_image:=}" # Pacify shellcheck
+
+# shellcheck source=ci/rust-version.sh
+source ../ci/rust-version.sh
+../ci/docker-run.sh "$rust_stable_docker_image" docs/build-cli-usage.sh
+../ci/docker-run.sh "$rust_stable_docker_image" docs/convert-ascii-to-svg.sh
+./set-solana-release-tag.sh
+
+# Build from /src into /build
+npm run build
+
+# Deploy the /build content using vercel
+if [[ -d .vercel ]]; then
+  rm -r .vercel
+fi
+./set-vercel-project-name.sh
+
+if [[ -n $CI ]]; then
+  if [[ -z $CI_PULL_REQUEST ]]; then
+    [[ -n $VERCEL_TOKEN ]] || {
+      echo "VERCEL_TOKEN is undefined.  Needed for Vercel authentication."
+      exit 1
+    }
+    vercel deploy . --local-config=vercel.json --confirm --token "$VERCEL_TOKEN" --prod
+  fi
+else
+  vercel deploy . --local-config=vercel.json
+fi

docs/convert-ascii-to-svg.sh

@@ -0,0 +1,21 @@
+#!/usr/bin/env bash
+
+# Convert .bob and .msc files in docs/art to .svg files located where the
+# site build will find them.
+
+set -e
+
+cd "$(dirname "$0")"
+output_dir=static/img
+
+mkdir -p "$output_dir"
+
+while read -r bob_file; do
+  svg_file=$(basename "${bob_file%.*}".svg)
+  svgbob "$bob_file" --output "$output_dir/$svg_file"
+done < <(find art/*.bob)
+
+while read -r msc_file; do
+  svg_file=$(basename "${msc_file%.*}".svg)
+  mscgen -T svg -o "$output_dir/$svg_file" -i "$msc_file"
+done < <(find art/*.msc)

docs/docusaurus.config.js

@@ -0,0 +1,113 @@
+module.exports = {
+  title: "Solana Docs",
+  tagline:
+    "Solana is an open source project implementing a new, high-performance, permissionless blockchain.",
+  url: "https://docs.solana.com",
+  baseUrl: "/",
+  favicon: "img/favicon.ico",
+  organizationName: "solana-labs", // Usually your GitHub org/user name.
+  projectName: "solana", // Usually your repo name.
+  themeConfig: {
+    navbar: {
+      logo: {
+        alt: "Solana Logo",
+        src: "img/logo-horizontal.svg",
+        srcDark: "img/logo-horizontal-dark.svg",
+      },
+      links: [
+        {
+          to: "docs/",
+          activeBasePath: "docs",
+          label: "Docs",
+          position: "left",
+        },
+        {
+          to: "docs/apps/README",
+          activeBasePath: "docs2",
+          label: "Developers",
+          position: "left",
+        },
+        {
+          to: "docs/running-validator/README",
+          activeBasePath: "docs2",
+          label: "Validators",
+          position: "left",
+        },
+        {
+          href: "https://discordapp.com/invite/pquxPsq",
+          label: "Chat",
+          position: "right",
+        },
+
+        {
+          href: "https://github.com/solana-labs/solana",
+          label: "GitHub",
+          position: "right",
+        },
+      ],
+    },
+    footer: {
+      style: "dark",
+      links: [
+        {
+          title: "Docs",
+          items: [
+            {
+              label: "Introduction",
+              to: "docs/introduction",
+            },
+            {
+              label: "Tour de SOL",
+              to: "docs/tour-de-sol/README",
+            },
+          ],
+        },
+        {
+          title: "Community",
+          items: [
+            {
+              label: "Discord",
+              href: "https://discordapp.com/invite/pquxPsq",
+            },
+            {
+              label: "Twitter",
+              href: "https://twitter.com/solana",
+            },
+            {
+              label: "Forums",
+              href: "https://forums.solana.com",
+            },
+          ],
+        },
+        {
+          title: "More",
+          items: [
+            {
+              label: "GitHub",
+              href: "https://github.com/solana-labs/solana",
+            },
+          ],
+        },
+      ],
+      copyright: `Copyright © ${new Date().getFullYear()} Solana Foundation`,
+    },
+  },
+  presets: [
+    [
+      "@docusaurus/preset-classic",
+      {
+        docs: {
+          path: "src",
+          // It is recommended to set document id as docs home page (`docs/` path).
+          homePageId: "introduction",
+          sidebarPath: require.resolve("./sidebars.js"),
+          // Please change this to your repo.
+          editUrl: "https://github.com/solana-labs/solana/edit/master/docs/",
+        },
+        theme: {
+          customCss: require.resolve("./src/css/custom.css"),
+        },
+      },
+    ],
+  ],
+};