replace shell scripts with x.py (apache#725)

Signed-off-by: tison <[email protected]>
Co-authored-by: Twice <[email protected]>

Author: tisonkun

.github/workflows/kvrocks.yaml

@@ -32,7 +32,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v3
-      - uses: apache/skywalking-eyes/header@501a28d2fb4a9b962661987e50cf0219631b32ff
+      - uses: apache/skywalking-eyes/header@v0.4.0
         with:
           config: tools/ci/licenserc.yml
 
@@ -42,19 +42,20 @@ jobs:
     steps:
       - name: Checkout Code Base
         uses: actions/checkout@v3
-
       - name: Install Check Tools
         run: |
-          sudo pip install --upgrade pip
-          sudo pip install --upgrade setuptools
           sudo apt update
           sudo apt install -y cppcheck
-          sudo pip install cpplint==1.5.0
-
+      - name: Setup Python
+        uses: actions/setup-python@v2
+        with:
+          python-version: 3.x
+      - name: Setup dependencies
+        run: pip install cpplint==1.5.0
       - name: Lint and check code
         run: |
-          ./cpplint.sh
-          ./cppcheck.sh
+          ./x.py check cpplint
+          ./x.py check cppcheck
 
   build-and-test:
     name: Build and test
@@ -65,42 +66,50 @@ jobs:
         include:
           - name: Darwin Clang
             os: macos-latest
+            compiler: auto
           - name: Darwin Clang without Jemalloc
             os: macos-latest
+            compiler: auto
             without_jemalloc: -DDISABLE_JEMALLOC=ON
           - name: Ubuntu GCC
             os: ubuntu-18.04
+            compiler: gcc
           - name: Ubuntu Clang
             os: ubuntu-18.04
-            clang: -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++
+            compiler: clang
           - name: Ubuntu GCC ASan
             os: ubuntu-18.04
             with_sanitizer: -DENABLE_ASAN=ON
+            compiler: gcc
           # - name: Ubuntu GCC TSan
           #   os: ubuntu-18.04
           #   with_sanitizer: -DENABLE_TSAN=ON
+          #   compiler: gcc
           - name: Ubuntu GCC without Jemalloc
             os: ubuntu-18.04
             without_jemalloc: -DDISABLE_JEMALLOC=ON
+            compiler: gcc
           - name: Ubuntu Clang ASan
             os: ubuntu-18.04
             with_sanitizer: -DENABLE_ASAN=ON
-            clang: -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++
+            compiler: clang
           # - name: Ubuntu Clang TSan
           #   os: ubuntu-18.04
           #   with_sanitizer: -DENABLE_TSAN=ON
-          #   clang: -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++
+          #   compiler: clang
           - name: Ubuntu Clang without Jemalloc
             os: ubuntu-18.04
             without_jemalloc: -DDISABLE_JEMALLOC=ON
-            clang: -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++
+            compiler: clang
           - name: Ubuntu GCC Ninja
             os: ubuntu-18.04
             with_ninja: --ninja
+            compiler: gcc
           - name: Ubuntu GCC Ninja without Jemalloc
             os: ubuntu-18.04
             with_ninja: --ninja
             without_jemalloc: -DDISABLE_JEMALLOC=ON
+            compiler: gcc
 
     runs-on: ${{ matrix.os }}
     steps:
@@ -117,7 +126,7 @@ jobs:
         run: |
           brew install cmake gcc autoconf automake libtool
           echo "NPROC=$(sysctl -n hw.ncpu)" >> $GITHUB_ENV
-      
+
       - name: Setup Linux
         if: ${{ startsWith(matrix.os, 'ubuntu') }}
         run: |
@@ -136,8 +145,13 @@ jobs:
       - name: Checkout Code Base
         uses: actions/checkout@v3
 
+      - name: Setup Python
+        uses: actions/setup-python@v2
+        with:
+          python-version: 3.x
+
       - name: Build Kvrocks
-        run: ./build.sh build -j$NPROC --unittest ${{ matrix.clang }} ${{ matrix.with_ninja }} ${{ matrix.with_sanitizer }} ${{ matrix.without_jemalloc }}
+        run: ./x.py build -j$NPROC --unittest --compiler ${{ matrix.compiler }} ${{ matrix.with_ninja }} ${{ matrix.with_sanitizer }} ${{ matrix.without_jemalloc }}
 
       - name: Run Unit Test
         run: ./build/unittest

.github/workflows/release.yaml

@@ -47,7 +47,7 @@ jobs:
 
       - name: Build
         run: |
-          ./build.sh build
+          ./x.py build
           cd build
           cp kvrocks release/bin/
           cp kvrocks2redis release/bin/
@@ -113,7 +113,7 @@ jobs:
       - name: Build
         run: |
           mkdir -p build/release
-          ./build.sh build
+          ./x.py build
           cd build
           mkdir -p release/bin release/conf
           cp kvrocks release/bin/

Dockerfile

@@ -22,11 +22,11 @@ ENV TZ=Asia/Shanghai
 RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone
 
 RUN apt update
-RUN apt install -y cmake make git autoconf libtool g++
+RUN apt install -y cmake make git autoconf libtool g++ python3
 WORKDIR /kvrocks
 
 COPY . .
-RUN ./build.sh build
+RUN ./x.py build
 
 FROM ubuntu:focal
 

README.md

@@ -6,18 +6,18 @@
 
 ---
 
-- [Google Group](https://groups.google.com/g/kvrocks)
-- [Slack Channel](https://join.slack.com/t/kvrockscommunity/shared_invite/zt-p5928e3r-OUAK8SUgC8GOceGM6dAz6w)
+* [Google Group](https://groups.google.com/g/kvrocks)
+* [Slack Channel](https://join.slack.com/t/kvrockscommunity/shared_invite/zt-p5928e3r-OUAK8SUgC8GOceGM6dAz6w)
 
 **Apache Kvrocks(Incubating)** is a distributed key value NoSQL database that uses RocksDB as storage engine and is compatible with Redis protocol. Kvrocks intends to decrease the cost of memory and increase the capacity while compared to Redis. The design of replication and storage was inspired by `rocksplicator` and `blackwidow`.
 
 Kvrocks has the following key features:
 
-- Redis protocol, user can use redis client to visit the kvrocks
-- Namespace, similar to redis db but use token per namespace
-- Replication, async replication using binlog like MySQL
-- High Available, supports redis sentinel to failover when master or slave was failed
-- Cluster mode, centralized management but compatible with Redis cluster client access
+* Redis protocol, user can use redis client to visit the kvrocks
+* Namespace, similar to redis db but use token per namespace
+* Replication, async replication using binlog like MySQL
+* High Available, supports redis sentinel to failover when master or slave was failed
+* Cluster mode, centralized management but compatible with Redis cluster client access
 
 > Thanks for designers @[田凌宇](https://github.com/tianlingyu1997) and @范世丽 contribute the kvrocks logo for us.
 
@@ -41,35 +41,34 @@ Kvrocks has the following key features:
 </tr>
 </table>
 
-***Tickets a pull reqeust to let us known that you're using kvrocks and add your logo to README***
+***Tickets a pull request to let us known that you're using kvrocks and add your logo to README***
 
-## Building kvrocks
+## Build and Run kvrocks
 
-#### requirements
-* g++ (required by c++11, version >= 4.8)
-* autoconf automake libtool cmake
-
-#### Build
+### Prerequisite
 
 ```shell
-# Centos/Redhat
-sudo yum install -y epel-release && sudo yum install -y git gcc gcc-c++ make cmake autoconf automake libtool which
+# CentOS / RedHat
+sudo yum install -y epel-release
+sudo yum install -y git gcc gcc-c++ make cmake autoconf automake libtool which
 
-# Ubuntu/Debian
+# Ubuntu / Debian
 sudo apt update
-sudo apt-get install gcc g++ make cmake autoconf automake libtool
+sudo apt install -y gcc g++ make cmake autoconf automake libtool
 
 # macOS
 brew install autoconf automake libtool cmake
 ```
 
+### Build
+
 It is as simple as:
 
 ```shell
 $ git clone https://github.com/apache/incubator-kvrocks.git
 $ cd incubator-kvrocks
-$ ./build.sh build # `./build.sh -h` to check more options;
-                   # especially, `./build.sh build --ghproxy` will fetch dependencies via ghproxy.com.
+$ ./x.py build # `./x.py build -h` to check more options;
+               # especially, `./x.py build --ghproxy` will fetch dependencies via ghproxy.com.
 ```
 
 ### Running kvrocks
@@ -111,7 +110,7 @@ $ ./unittest
 
 ##  Namespace
 
-namespace was used to isolate data between users. unlike all the redis databases can be visited by `requirepass`, we use one token per namespace. `requirepass` was regraded as admin token, only admin token allows to access the namespace command, as well as some commands like `config`, `slaveof`, `bgsave`, etc…
+Namespace is used to isolate data between users. Unlike all the Redis databases can be visited by `requirepass`, we use one token per namespace. `requirepass` is regraded as admin token, and only admin token allows to access the namespace command, as well as some commands like `config`, `slaveof`, `bgsave`, etc..
 
 ```
 # add token
@@ -136,24 +135,25 @@ OK
 
 ## Cluster
 
-Kvrocks implements a proxyless centralized cluster solution but its accessing method is completely compatible with the Redis cluster client. You can use Redis cluster SDKs to access the kvrocks cluster. More details, please see: [Kvrocks Cluster Introduction](https://github.com/apache/incubator-kvrocks/wiki/Kvrocks-Cluster-Introduction)
+Kvrocks implements a proxyless centralized cluster solution but its accessing method is completely compatible with the Redis cluster client. You can use Redis cluster SDKs to access the kvrocks cluster. More details, please see: [Kvrocks Cluster Introduction](https://kvrocks.apache.org/docs/Cluster/kvrocks-cluster-introduction)
 
-## DOCs
+## Documents
 
-* [supported commands](https://github.com/apache/incubator-kvrocks/wiki/Support-Commands)
-* [design complex kv on rocksdb](https://github.com/apache/incubator-kvrocks/blob/master/docs/metadata-design.md)
-* [replication design](https://github.com/apache/incubator-kvrocks/blob/master/docs/replication-design.md)
+Documents are hosted at the [official website](https://kvrocks.apache.org/docs/supported-commands).
 
-For more informations, can see: [Kvrocks Wiki](https://github.com/apache/incubator-kvrocks/wiki)
+* [Supported Commands](https://kvrocks.apache.org/docs/supported-commands)
+* [Design Complex Structure on RocksDB](https://kvrocks.apache.org/docs/Design/design-structure-on-rocksdb)
+* [Replication Design](https://kvrocks.apache.org/docs/Design/replication)
 
 ## Tools
+
 * Export the Kvrocks monitor metrics, please use [kvrocks_exporter](https://github.com/KvrocksLabs/kvrocks_exporter)
-* Migrate from redis to kvrocks, use [redis-migrate-tool](https://github.com/vipshop/redis-migrate-tool) which was developed by vipshop
+* Migrate from redis to kvrocks, use [redis-migrate-tool](https://github.com/vipshop/redis-migrate-tool) which was developed by @vipshop
 * Migrate from kvrocks to redis. use `kvrocks2redis` in build dir
 
 ## Performance
 
-#### Hardware
+### Hardware
 
 * CPU: 48 cores Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz
 * Memory: 32 GiB
@@ -162,34 +162,34 @@ For more informations, can see: [Kvrocks Wiki](https://github.com/apache/incubat
 
 >  Benchmark Client:  multi-thread redis-benchmark(unstable branch)
 
- #### 1. Commands QPS
+### 1. Commands QPS
 
 > kvrocks: workers = 16, benchmark: 8 threads/ 512 conns / 128 payload
 
 latency: 99.9% < 10ms
 
-![image](https://raw.githubusercontent.com/apache/incubator-kvrocks/master/docs/images/chart-commands.png)
+![image](docs/images/chart-commands.png)
 
-#### 2.  QPS on different payloads
+### 2.  QPS on different payloads
 
 > kvrocks: workers = 16, benchmark: 8 threads/ 512 conns
 
 latency: 99.9% < 10ms
 
-![image](https://raw.githubusercontent.com/apache/incubator-kvrocks/master/docs/images/chart-values.png)
+![image](docs/images/chart-values.png)
 
 #### 3. QPS on different workers
 
 > kvrocks: workers = 16, benchmark: 8 threads/ 512 conns / 128 payload
 
 latency: 99.9% < 10ms
 
-![image](https://raw.githubusercontent.com/apache/incubator-kvrocks/master/docs/images/chart-threads.png)
+![image](docs/images/chart-threads.png)
 
 ## License
 
-Kvrocks is under the Apache License Version 2.0. See the LICENSE file for details.
+Kvrocks is under the Apache License Version 2.0. See the [LICENSE](LICENSE) file for details.
 
 ## WeChat Official Account (微信公众号)
 
-<img src="docs/images/wechat_account.jpg" alt="WeChat offical account" />
+![WeChat official account](docs/images/wechat_account.jpg)