Skip to content

Commit

Permalink
eval results update
Browse files Browse the repository at this point in the history
  • Loading branch information
@imtypist
imtypist committed Apr 28, 2023
1 parent c383a81 commit 6fae5ea
Show file tree
Hide file tree
Showing 23 changed files with 527 additions and 511 deletions.
14 changes: 14 additions & 0 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -33,3 +33,17 @@ EOF
```

- console v3.0.0 has minor issues on calling precompiled contracts, the fixed method refers to [https://github.com/FISCO-BCOS/console/issues/601](https://github.com/FISCO-BCOS/console/issues/601).

## Evaluation

- estimated gas cost of model inference on-chain
![](dist/figures/evm-gas-cost.png)

- time cost of TEE register, distributed attestation, and secret provisioning
![](dist/figures/das-and-secretprov.png)

- time cost comparison in sgx and non-sgx environments
![](dist/figures/sgx-inference-comp.png)

- end-to-end latency of SMART framework
![](dist/figures/e2e-latency-multi.png)
Binary file modified dist/figures/das-and-secretprov.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified dist/figures/e2e-latency-multi.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
12 changes: 6 additions & 6 deletions dist/figures/e2e-latency-multi.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -86,17 +86,17 @@
ax.bar(_pos+width/2, private_off_chain_pre_check, width, bottom=private_send_contract_call, label='enclave initialization', color='lightgray', hatch='.', edgecolor='black')
ax.bar(_pos+width/2, private_key_provision, width, bottom=private_send_contract_call+private_off_chain_pre_check, label='secret key provisioning (private model only)', color='orange', hatch='\\', edgecolor='black')
ax.bar(_pos+width/2, private_model_inference, width, bottom=private_send_contract_call+private_off_chain_pre_check+private_key_provision, label='model inference', color='lightblue', hatch='x', edgecolor='black')
ax.bar(_pos+width/2, private_generate_quote, width, bottom=private_send_contract_call+private_off_chain_pre_check+private_key_provision+private_model_inference, label='quote generation', color='lightpink', hatch='o')
ax.bar(_pos+width/2, private_verify_quote, width, bottom=private_send_contract_call+private_off_chain_pre_check+private_key_provision+private_model_inference+private_generate_quote, label='quote verification', color='lightseagreen', hatch='+')
ax.bar(_pos+width/2, private_execution_return, width, bottom=private_send_contract_call+private_off_chain_pre_check+private_key_provision+private_model_inference+private_generate_quote+private_verify_quote, label='contract context recovery (back to on-chain)', color='red', hatch='*')
ax.bar(_pos+width/2, private_generate_quote, width, bottom=private_send_contract_call+private_off_chain_pre_check+private_key_provision+private_model_inference, label='quote generation', color='lightpink', hatch='o', edgecolor='black')
ax.bar(_pos+width/2, private_verify_quote, width, bottom=private_send_contract_call+private_off_chain_pre_check+private_key_provision+private_model_inference+private_generate_quote, label='quote verification', color='lightseagreen', hatch='+', edgecolor='black')
ax.bar(_pos+width/2, private_execution_return, width, bottom=private_send_contract_call+private_off_chain_pre_check+private_key_provision+private_model_inference+private_generate_quote+private_verify_quote, label='contract context recovery (back to on-chain)', color='red', hatch='*', edgecolor='black')


ax.bar(_pos-width/2, send_contract_call, width, color='lightyellow', hatch='///', edgecolor='black')
ax.bar(_pos-width/2, off_chain_pre_check, width, bottom=send_contract_call, color='lightgray', hatch='.', edgecolor='black')
ax.bar(_pos-width/2, model_inference, width, bottom=off_chain_pre_check+send_contract_call, color='lightblue', hatch='x', edgecolor='black')
ax.bar(_pos-width/2, generate_quote, width, bottom=off_chain_pre_check+send_contract_call+model_inference, color='lightpink', hatch='o')
ax.bar(_pos-width/2, verify_quote, width, bottom=off_chain_pre_check+send_contract_call+model_inference+generate_quote, color='lightseagreen', hatch='+')
ax.bar(_pos-width/2, execution_return, width, bottom=off_chain_pre_check+send_contract_call+model_inference+generate_quote+verify_quote, color='red', hatch='*')
ax.bar(_pos-width/2, generate_quote, width, bottom=off_chain_pre_check+send_contract_call+model_inference, color='lightpink', hatch='o', edgecolor='black')
ax.bar(_pos-width/2, verify_quote, width, bottom=off_chain_pre_check+send_contract_call+model_inference+generate_quote, color='lightseagreen', hatch='+', edgecolor='black')
ax.bar(_pos-width/2, execution_return, width, bottom=off_chain_pre_check+send_contract_call+model_inference+generate_quote+verify_quote, color='red', hatch='*', edgecolor='black')

# Add some text for labels, title and custom x-axis tick labels, etc.
ax.set_ylabel('time cost (seconds)')
Expand Down
Binary file modified dist/figures/evm-gas-cost.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified dist/figures/sgx-inference-comp.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
2 changes: 1 addition & 1 deletion dist/figures/sgx-inference-comp.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@
width = 0.35 # the width of the bars

fig = plt.figure()
bax = brokenaxes(ylims=((0, .6), (2.5, 7), (14, 18), (29, 31)), hspace=.05)
bax = brokenaxes(ylims=((0, .6), (4, 9), (13, 16), (29, 31)), hspace=.05)

rects1 = bax.bar(x - width/2, _nonsgx, width, label='non-SGX', color='lightyellow', edgecolor='black', hatch='/')
rects2 = bax.bar(x + width/2, _sgx, width, label='SGX', color='lightgrey', edgecolor='black')
Expand Down
96 changes: 48 additions & 48 deletions eval_logs/alexnet.log
View file
Original file line number Diff line number Diff line change
@@ -1,48 +1,48 @@
 [SMART][1682659288655] stop existing blockchain nodes 
 [SMART][1682659292840] remove existing blockchain data 
 [SMART][1682659292905] remove existing console log 
 [SMART][1682659292910] remove existing local pck database 
 [SMART][1682659292971] restart pck cache server 
 [SMART][1682659292991] generate 4 new blockchain nodes 
 [SMART][1682659294955] start 4 new blockchain nodes 
 [SMART][1682659296505] sync blockchain nodes config to console 
 [SMART][1682659296509] deploy TEE provider management contract 
 [SMART][1682659299380] system initialization is completed 
 [SMART][1682659299392] obtain TPM contract address 
 [SMART][1682659299401] retrieve enc_ppid from PCE and request pck cache update 
 [SMART][1682659299438] obtain enc_ppid 
 [SMART][1682659299443] register TEE provider in the TPM contract 
 [SMART][1682659301892] query the latest registered TEE provider 
 [SMART][1682659304123] query an exsited enc_ppid in the pck cache database 
 [SMART][1682659304177] TEE provider registration is completed 
 [SMART][1682659304182] deploy smart contract 
 [SMART][1682659306828] call inference function (non-sgx-gramine, public alexnet model) 
 [SMART][1682659310615] call inference function (sgx-gramine, public alexnet model) 
 [SMART][1682659392029] client or TEE provider starts secret key provisioning server 
 [SMART][1682659392034] call inference function (sgx-gramine, private alexnet model) 
 [SMART][1682659492934] close server_dcap 
 [SMART][1682659492946] verify execution result (sgx-gramine, public alexnet model) 
 [SMART][1682659493109] verify execution result (sgx-gramine, private alexnet model) 
 [SMART][1682659493274] on-chain and off-chain execution model test is completed 
[SMART][CPP][1682659308695] switch to off-chain execution
[SMART][CPP][1682659308695] outsource to TEE
[SMART][PYTHON][1682659309886] start to execute model inference
[SMART][PYTHON][1682659310119] Done. The result was written to `result.txt`.
[SMART][CPP][1682659310262] receive results from TEE
[SMART][CPP][1682659310262] leave off-chain execution
[SMART][CPP][1682659312596] switch to off-chain execution
[SMART][CPP][1682659312596] outsource to TEE
[SMART][PYTHON][1682659375822] start to execute model inference
[SMART][PYTHON][1682659390559] start to attestation (generate quote)
[SMART][PYTHON][1682659390575] Done. The result was written to `result.quote`.
[SMART][CPP][1682659391683] receive results from TEE
[SMART][CPP][1682659391683] leave off-chain execution
[SMART][CPP][1682659393911] switch to off-chain execution
[SMART][CPP][1682659393911] outsource to TEE
[SMART][PYTHON][1682659479624] start to execute model inference
[SMART][PYTHON][1682659491232] start to attestation (generate quote)
[SMART][PYTHON][1682659491247] Done. The result was written to `result.quote`.
[SMART][CPP][1682659492585] receive results from TEE
[SMART][CPP][1682659492585] leave off-chain execution
[SMART][SERVERDCAP][1682659407428] secret provisioning server starts connection
[SMART][SERVERDCAP][1682659430308] secret provisioning is completed
 [SMART][1682698929541] stop existing blockchain nodes 
 [SMART][1682698929614] remove existing blockchain data 
 [SMART][1682698929691] remove existing console log 
 [SMART][1682698929695] remove existing local pck database 
 [SMART][1682698929707] restart pck cache server 
 [SMART][1682698929724] generate 4 new blockchain nodes 
 [SMART][1682698930667] start 4 new blockchain nodes 
 [SMART][1682698932226] sync blockchain nodes config to console 
 [SMART][1682698932230] deploy TEE provider management contract 
 [SMART][1682698934971] system initialization is completed 
 [SMART][1682698934982] obtain TPM contract address 
 [SMART][1682698934992] retrieve enc_ppid from PCE and request pck cache update 
 [SMART][1682698935041] obtain enc_ppid 
 [SMART][1682698935047] register TEE provider in the TPM contract 
 [SMART][1682698937532] query the latest registered TEE provider 
 [SMART][1682698939744] query an exsited enc_ppid in the pck cache database 
 [SMART][1682698939799] TEE provider registration is completed 
 [SMART][1682698939805] deploy smart contract 
 [SMART][1682698942347] call inference function (non-sgx-gramine, public alexnet model) 
 [SMART][1682698946154] call inference function (sgx-gramine, public alexnet model) 
 [SMART][1682699026694] client or TEE provider starts secret key provisioning server 
 [SMART][1682699026700] call inference function (sgx-gramine, private alexnet model) 
 [SMART][1682699112801] close server_dcap 
 [SMART][1682699112810] verify execution result (sgx-gramine, public alexnet model) 
 [SMART][1682699112986] verify execution result (sgx-gramine, private alexnet model) 
 [SMART][1682699113141] on-chain and off-chain execution model test is completed 
[SMART][CPP][1682698948019] switch to off-chain execution
[SMART][CPP][1682698948019] outsource to TEE
[SMART][PYTHON][1682699011245] start to execute model inference
[SMART][PYTHON][1682699024932] start to attestation (generate quote)
[SMART][PYTHON][1682699024951] Done. The result was written to `result.quote`.
[SMART][CPP][1682699026349] receive results from TEE
[SMART][CPP][1682699026349] leave off-chain execution
[SMART][CPP][1682699028558] switch to off-chain execution
[SMART][CPP][1682699028558] outsource to TEE
[SMART][PYTHON][1682699099956] start to execute model inference
[SMART][PYTHON][1682699111305] start to attestation (generate quote)
[SMART][PYTHON][1682699111319] Done. The result was written to `result.quote`.
[SMART][CPP][1682699112456] receive results from TEE
[SMART][CPP][1682699112456] leave off-chain execution
[SMART][CPP][1682698944191] switch to off-chain execution
[SMART][CPP][1682698944191] outsource to TEE
[SMART][PYTHON][1682698945421] start to execute model inference
[SMART][PYTHON][1682698945680] Done. The result was written to `result.txt`.
[SMART][CPP][1682698945800] receive results from TEE
[SMART][CPP][1682698945800] leave off-chain execution
[SMART][SERVERDCAP][1682699042073] secret provisioning server starts connection
[SMART][SERVERDCAP][1682699047874] secret provisioning is completed
96 changes: 48 additions & 48 deletions eval_logs/alexnet.log.sorted
View file
Original file line number Diff line number Diff line change
@@ -1,48 +1,48 @@
 [SMART][1682659288655] stop existing blockchain nodes 
 [SMART][1682659292840] remove existing blockchain data 
 [SMART][1682659292905] remove existing console log 
 [SMART][1682659292910] remove existing local pck database 
 [SMART][1682659292971] restart pck cache server 
 [SMART][1682659292991] generate 4 new blockchain nodes 
 [SMART][1682659294955] start 4 new blockchain nodes 
 [SMART][1682659296505] sync blockchain nodes config to console 
 [SMART][1682659296509] deploy TEE provider management contract 
 [SMART][1682659299380] system initialization is completed 
 [SMART][1682659299392] obtain TPM contract address 
 [SMART][1682659299401] retrieve enc_ppid from PCE and request pck cache update 
 [SMART][1682659299438] obtain enc_ppid 
 [SMART][1682659299443] register TEE provider in the TPM contract 
 [SMART][1682659301892] query the latest registered TEE provider 
 [SMART][1682659304123] query an exsited enc_ppid in the pck cache database 
 [SMART][1682659304177] TEE provider registration is completed 
 [SMART][1682659304182] deploy smart contract 
 [SMART][1682659306828] call inference function (non-sgx-gramine, public alexnet model) 
[SMART][CPP][1682659308695] switch to off-chain execution
[SMART][CPP][1682659308695] outsource to TEE
[SMART][PYTHON][1682659309886] start to execute model inference
[SMART][PYTHON][1682659310119] Done. The result was written to `result.txt`.
[SMART][CPP][1682659310262] receive results from TEE
[SMART][CPP][1682659310262] leave off-chain execution
 [SMART][1682659310615] call inference function (sgx-gramine, public alexnet model) 
[SMART][CPP][1682659312596] switch to off-chain execution
[SMART][CPP][1682659312596] outsource to TEE
[SMART][PYTHON][1682659375822] start to execute model inference
[SMART][PYTHON][1682659390559] start to attestation (generate quote)
[SMART][PYTHON][1682659390575] Done. The result was written to `result.quote`.
[SMART][CPP][1682659391683] receive results from TEE
[SMART][CPP][1682659391683] leave off-chain execution
 [SMART][1682659392029] client or TEE provider starts secret key provisioning server 
 [SMART][1682659392034] call inference function (sgx-gramine, private alexnet model) 
[SMART][CPP][1682659393911] switch to off-chain execution
[SMART][CPP][1682659393911] outsource to TEE
[SMART][SERVERDCAP][1682659407428] secret provisioning server starts connection
[SMART][SERVERDCAP][1682659430308] secret provisioning is completed
[SMART][PYTHON][1682659479624] start to execute model inference
[SMART][PYTHON][1682659491232] start to attestation (generate quote)
[SMART][PYTHON][1682659491247] Done. The result was written to `result.quote`.
[SMART][CPP][1682659492585] receive results from TEE
[SMART][CPP][1682659492585] leave off-chain execution
 [SMART][1682659492934] close server_dcap 
 [SMART][1682659492946] verify execution result (sgx-gramine, public alexnet model) 
 [SMART][1682659493109] verify execution result (sgx-gramine, private alexnet model) 
 [SMART][1682659493274] on-chain and off-chain execution model test is completed 
 [SMART][1682698929541] stop existing blockchain nodes 
 [SMART][1682698929614] remove existing blockchain data 
 [SMART][1682698929691] remove existing console log 
 [SMART][1682698929695] remove existing local pck database 
 [SMART][1682698929707] restart pck cache server 
 [SMART][1682698929724] generate 4 new blockchain nodes 
 [SMART][1682698930667] start 4 new blockchain nodes 
 [SMART][1682698932226] sync blockchain nodes config to console 
 [SMART][1682698932230] deploy TEE provider management contract 
 [SMART][1682698934971] system initialization is completed 
 [SMART][1682698934982] obtain TPM contract address 
 [SMART][1682698934992] retrieve enc_ppid from PCE and request pck cache update 
 [SMART][1682698935041] obtain enc_ppid 
 [SMART][1682698935047] register TEE provider in the TPM contract 
 [SMART][1682698937532] query the latest registered TEE provider 
 [SMART][1682698939744] query an exsited enc_ppid in the pck cache database 
 [SMART][1682698939799] TEE provider registration is completed 
 [SMART][1682698939805] deploy smart contract 
 [SMART][1682698942347] call inference function (non-sgx-gramine, public alexnet model) 
[SMART][CPP][1682698944191] switch to off-chain execution
[SMART][CPP][1682698944191] outsource to TEE
[SMART][PYTHON][1682698945421] start to execute model inference
[SMART][PYTHON][1682698945680] Done. The result was written to `result.txt`.
[SMART][CPP][1682698945800] receive results from TEE
[SMART][CPP][1682698945800] leave off-chain execution
 [SMART][1682698946154] call inference function (sgx-gramine, public alexnet model) 
[SMART][CPP][1682698948019] switch to off-chain execution
[SMART][CPP][1682698948019] outsource to TEE
[SMART][PYTHON][1682699011245] start to execute model inference
[SMART][PYTHON][1682699024932] start to attestation (generate quote)
[SMART][PYTHON][1682699024951] Done. The result was written to `result.quote`.
[SMART][CPP][1682699026349] receive results from TEE
[SMART][CPP][1682699026349] leave off-chain execution
 [SMART][1682699026694] client or TEE provider starts secret key provisioning server 
 [SMART][1682699026700] call inference function (sgx-gramine, private alexnet model) 
[SMART][CPP][1682699028558] switch to off-chain execution
[SMART][CPP][1682699028558] outsource to TEE
[SMART][SERVERDCAP][1682699042073] secret provisioning server starts connection
[SMART][SERVERDCAP][1682699047874] secret provisioning is completed
[SMART][PYTHON][1682699099956] start to execute model inference
[SMART][PYTHON][1682699111305] start to attestation (generate quote)
[SMART][PYTHON][1682699111319] Done. The result was written to `result.quote`.
[SMART][CPP][1682699112456] receive results from TEE
[SMART][CPP][1682699112456] leave off-chain execution
 [SMART][1682699112801] close server_dcap 
 [SMART][1682699112810] verify execution result (sgx-gramine, public alexnet model) 
 [SMART][1682699112986] verify execution result (sgx-gramine, private alexnet model) 
 [SMART][1682699113141] on-chain and off-chain execution model test is completed 
Loading

0 comments on commit 6fae5ea

Please sign in to comment.