updated readme and provided brief description of the run-hermesKV.sh …

…args

README.md

@@ -1,3 +1,61 @@
-######################################################################################
-##### WARNING: Do not compile through through cmake use the Makefile in /exec/ to compile instead!!!!
-######################################################################################
+# Hermes Reliable Replication Protocol
+---------------------------------------
+This is the publicly available artifact repository supporting the ASPLOS'20 paper
+_"Hermes: Fast and Reliable Data Replication with Linearizability"_.
+The repository contains both code to experimentally evaluate Hermes(KV) 
+and complete Hermes TLA+ specifications which can be used to verify Hermes 
+correctness via model-checking.
+
+---
+
+## Hardware dependencies
+A homogeneous cluster of x86_64 nodes interconnected via RDMA network cards and switched 
+(tested on "Mellanox ConnectX-4" Infiniband infrastructure).
+
+---
+
+## Software requirements
+Linux OS (tested on Ubuntu 18.04 4.15.0-55-generic) with root access.
+
+The software is tested using the following version of Mellanox OFED RDMA drivers
+`MLNX_OFED_LINUX-4.4-2.0.7.0`.
+
+Third-party libraries that you will require to run the experiments include:
+1. _parallel_ (Cluster management scripts only)
+1. _libmemcached-dev_ (used to exchange QP informations for the setup of RDMA connections)
+1. _libnuma-dev_	(for mbind)
+---
+## Setup
+On every node:
+1. Install Mellanox OFED ibverbs drivers
+1. `./hermes/bin/setup.sh`
+
+On manager (just pick on node in the cluster):
+1. Fill variables in `/hermes/exec/hosts.sh`
+1. Configure setup and default parameters in `/hermes/include/hermes/config.h`
+1. From `/hermes/exec/` compile _hermesKV_ through make
+1. scp  _hermesKV_ and the configured hosts.sh in the `/hermes/exec/` directory of all other nodes in the cluster. 
+---
+##Compilation 
+`cd hermes/exec; make`
+
+_Warning_: Do not compile through through cmake; instead use the Makefile in exec/ directory.
+
+---
+## Run
+Run first on manager:
+`./run-hermesKV.sh <experiment_parameters>`
+
+Then run on all other member nodes 
+`./run-hermesKV.sh <experiment_parameters>`
+
+> Note that some members will eagerly terminate if experiment 
+  uses smaller number of nodes than specified in hosts.sh
+
+An experiment example for three nodes 12 worker threads and 35% write ratio would be as follows:
+`./run-hermesKV.sh -W 12 -w 350 -M 3`
+Supported command-line arguments for the experiments are detailed in the run-hermesKV.sh script.
+
+---
+## Contact
+ Antonios Katsarakis: `[email protected]`

exec/run-hermesKV.sh

@@ -19,28 +19,34 @@ LAT_WORKER="-1"
 while getopts ":W:w:l:R:C:c:b:M:h" opt; do
   case $opt in
      W)
-       NUM_WORKERS=$OPTARG
+       NUM_WORKERS=$OPTARG # Number of threads: this must be smaller than MAX_WORKERS_PER_MACHINE of config.h
        ;;
      w)
-       WRITE_RATIO=$OPTARG
+       WRITE_RATIO=$OPTARG # given number is divided by 10 to give write rate % (i.e., 55 means 5.5 % writes)
        ;;
      R)
-       RMW_RATIO=$OPTARG
+       RMW_RATIO=$OPTARG # percentage of writes to be rmws (i.e., -w 500 -R 500 means 25 % of RMWs and 25% of writes)
+                         # RMW is disabled by default (no usage through the artifact) can be enabled through config.h)
        ;;
      C)
-       MAX_COALESCE=$OPTARG
+       MAX_COALESCE=$OPTARG # maximum number of readily-available messages to be "batched" in a network packet
+                            # must be smaller than MTU and it is capped by MAX_REQ_COALESCE in config.h
        ;;
      c)
-       CREDITS=$OPTARG
+       CREDITS=$OPTARG      # maximum number of credits per node per thread; credits correspond to messages and not packets
+                            # it is capped by MAX_CREDITS_PER_REMOTE_WORKER in config.h
        ;;
      b)
-       MAX_BATCH_SIZE=$OPTARG
+       MAX_BATCH_SIZE=$OPTARG   # amount of requests and protocol messages that can be batched to the KVS
+                                # it is capped by MAX_BATCH_KVS_OPS_SIZE in config.h
        ;;
      M)
-       NUM_MACHINES=$OPTARG
+       NUM_MACHINES=$OPTARG # it is capped by MAX_MACHINE_NUM in config.h and the number of IPS as indicated in hosts.sh
        ;;
      l)
-       LAT_WORKER=$OPTARG
+       LAT_WORKER=$OPTARG # An id of the worker who is measuring the latency
+                          # if -1 Latency is disabled
+                          # otherwise it is capped by running worker threads (NUM_WORKERS-1)
        ;;
      h)
       echo "Usage: -W <# workers> -w <write ratio>  (x1000 --> 10 for 1%)"

include/hermes/config.h

@@ -21,14 +21,29 @@
 #define TOTAL_CORES_PER_SOCKET  10
 #define TOTAL_NUMBER_OF_SOCKETS  2
 
+/*-------------------------------------------------
+-------------------------------------------------
+-------------------------------------------------
+-------- No need to change beyond this point ----
+-------------------------------------------------
+-------------------------------------------------
+--------------------------------------------------*/
+
 // Default workload writes / updates accesses (the rest are reads)
-#define DEFAULT_UPDATE_RATIO  1000  // both writes and RMWs (RMW_RATIO inderectly provides WRITE_RATIO)
-#define ENABLE_RMWs              0  // if RMWs is not enabled then all UPDATE_RATIO == WRITE_RATIO
-#define DEFAULT_RMW_RATIO        0  // percentage of UPDATE_RATIO to be RMWs
+#define DEFAULT_UPDATE_RATIO  1000  // is divided by 10 (i.e., 25 --> 2.5 %)
+// both writes and RMWs (RMW_RATIO inderectly provides WRITE_RATIO)
+
 
+#define ENABLE_RMWs              0  // if RMWs is not enabled then all UPDATE_RATIO == WRITE_RATIO
+#define DEFAULT_RMW_RATIO        0  // is divided by 10 (i.e., 25 --> 2.5 %)
+// percentage of UPDATE_RATIO to be RMWs
 
 // Max operations per-thread to batches to the KVS (either received packets or read/write/RMW requests)
-#define MAX_BATCH_KVS_OPS_SIZE 250 // up to 254
+#define MAX_BATCH_KVS_OPS_SIZE 250
+static_assert(MAX_WORKERS_PER_MACHINE <= 254, "");
+static_assert(MAX_WORKERS_PER_MACHINE <= TOTAL_NUMBER_OF_SOCKETS *
+                                         TOTAL_THREADS_PER_CORE * TOTAL_CORES_PER_SOCKET, "");
+static_assert(DEFAULT_UPDATE_RATIO <= 1000  && DEFAULT_RMW_RATIO >= 0, "");
 
 
 /*-------------------------------------------------