Evaluation

This folder contains the necessary code to reproduce the evaluation results from the paper

Benchmark

The files in this folder can be used to reproduce (the runtime component of) Figure 5 ("Runtime and Memory consumption [...] for different vector sizes") from the HECO paper.

In order to compile the programs from the high-level intermediate representation (HIR) form given here to *.cpp, run the following (assuming your current working directory is the repositoy root):

./build/bin/heco --full-pass < evaluation/benchmark/heco_input/hammingdistance_4.mlir > evaluation/heco_output/hammingdist_4.mlir
./build/bin/emitc-translate --mlir-to-cpp  < evaluation/benchmark/heco_output/hammingdistance_4.mlir > evaluation/heco_output/hammingdist_4.cpp

In order to avoid needing to compile each HECO input individually, you can instead use a helper script that does this for all files in the heco_input folder (assuming your current working directory is the repositoy root):

./evaluation/benchmark/heco_helper.sh

You can then compile and build the benchmark target (assuming your current working directory is the repositoy root):

cmake --build build --target benchmark

Finally, you can run the benchmarking (assuming your current working directory is the repositoy root):

./build/bin/benchmark

This creates a set of *.csv files of the format <workload>_HECO_<size>.csv (e.g., HammingDistance_HECO_4.csv) in evaluation/plotting/data/benchmark. These contain one iteration on each line, reporting keygen time, encryption time, evaluation time, and decryption time (in this order) in microseconds.

Comparison

The files in this folder can be used to reproduce Figure 6 ("Runtime [...] comparing a naive non-batched baseline, the solution generated by our system (HECO), and an optimally-batched solution synthesized by the Porcupine tool") from the HECO paper.

In order to compile the programs from the high-level intermediate representation (HIR) form given here to *.cpp, run the following (assuming your current working directory is the repositoy root):

./build/bin/heco --full-pass < evaluation/comparison/heco_input/hammingdistance_4.mlir > evaluation/heco_output/hammingdist_4.mlir
./build/bin/emitc-translate --mlir-to-cpp < evaluation/comparison/heco_output/hammingdistance_4.mlir > evaluation/heco_output/hammingdist_4.cpp

In order to avoid needing to compile each HECO input individually, you can instead use a helper script that does this for all files in the heco_input folder (assuming your current working directory is the repositoy root):

./evaluation/comparison/heco_helper.sh

You can then compile and build the comparison target (assuming your current working directory is the repositoy root):

cmake --build build --target comparison

Finally, you can run the comparison (assuming your current working directory is the repositoy root):

./build/bin/comparison

This creates a set of *.csv files of the format <workload>_<version>_<size>.csv (e.g., HammingDistance_HECO_4.csv) in evaluation/plotting/data/comparison. These contain one iteration on each line, reporting keygen time, encryption time, evaluation time, and decryption time (in this order) in microseconds.

Plotting Runtimes

A (rough) plotting script is provided in the form of evaluation/plotting/plot_all.py. A pipfile is provided for the necessary dependencies. However, note that the plotting requires several LaTeX packages, which might need to be installed via the distro package manager rather than pip or similar.

The script assumes that data for batch benchmark and comparison has been created as described above. The resulting figures are stored in evaluation/plotting/out.

Compile Time Evaluation

The files in this folder can be used to reproduce Table 1 ("Compile time") from the HECO paper.

MLIR includes utilities for detailed compile time reports, which can be used by passing -mlir-timing -mlir-timing-display=list to the compiler. Specifically, to evaluate, e.g., hammingdist_4.mlir, you should run the following (assuming your current working directory is the repositoy root):

./build/bin/heco --full-pass -mlir-timing -mlir-timing-display=list < test/evaluation/hammingdistance_4.mlir

In order to avoid needing to compile each HECO input individually, you can instead use a helper script that does this for all files in the heco_input folder (assuming your current working directory is the repositoy root):

./evaluation/compile_time/heco_helper.sh

This will produce a report that will look something like this:

===-------------------------------------------------------------------------===
                         ... Execution time report ...
===-------------------------------------------------------------------------===
  Total Execution Time: 0.1311 seconds

  ----Wall Time----  ----Name----
    0.1311 (100.0%)  root
    0.0702 ( 53.5%)  Canonicalizer
    0.0435 ( 33.2%)  NaryPass
    0.0055 (  4.2%)  CSE
    0.0041 (  3.2%)  BatchingPass
    0.0031 (  2.4%)  UnrollLoopsPass
    0.0030 (  2.3%)  Tensor2BatchedSecretPass
    0.0005 (  0.4%)  Output
    0.0004 (  0.3%)  InternalOperandBatchingPass
    0.0003 (  0.2%)  Parser
    0.0001 (  0.1%)  CombineSimplifyPass
    0.0000 (  0.0%)  (A) DominanceInfo
    0.0004 (  0.3%)  Rest
    0.1311 (100.0%)  Total

Note that because a significant part of the compile time for HECO is spent in the "Canonicalizer" pass, which is a built-in pass from MLIR, the compile time might vary significantly as MLIR updates.

Note There is a significant performance gap between debug mode and release mode when using MLIR! Please ensure that both MLIR and HECO have been built in Release configuration when evaluating compile time!