Use less SSH connections / UX improvements for interactive SSH login flows

[Mynacol]

Hi, I'm just starting with deploy-rs. I added a minimal deployment config trying to push my working NixOS config to my server.

While running deploy, I get asked multiple times for my PIN to unlock the SSH key in my TPM, just like for usual SSH connections.
But upon arriving at the Waiting for confirmation event... step I get no such prompt and after the timeout deploy-rs tries to revert to a previous config, which also fails (and it panicked).

Even after deploy-rs exits the shell is all messed up, showing no user input and seemingly still trying to connect to the server in the background.

If deploy-rs exits, all background ssh connections should be stopped/killed. I'd recommend reusing existing SSH connections to limit the amount of potentially interactive SSH logins necessary. I understand that the confirmation phase should use a separate SSH connection, but two connections should suffice.

Full shell session

[nix-shell:~/projects/nixos]$ deploy
🚀 ℹ️ [deploy] [INFO] Running checks for flake in .
warning: Git tree '/home/user/projects/nixos' is dirty
warning: unknown flake output 'deploy'
🚀 ℹ️ [deploy] [INFO] Evaluating flake in .
warning: Git tree '/home/user/projects/nixos' is dirty
🚀 ℹ️ [deploy] [INFO] The following profiles are going to be deployed:
[server.system]
user = "root"
ssh_user = "root"
path = "/nix/store/l296igrc9r4gpix9x014fhpdznkj3miz-activatable-nixos-system-server-23.05.20230701.0de8605"
hostname = "server"
ssh_opts = []

🚀 ℹ️ [deploy] [INFO] Building profile `system` for node `server` on remote host
WARNING: Listing FAPI token objects failed: "fapi:Provisioning was not executed."
Please see https://github.com/tpm2-software/tpm2-pkcs11/blob/1.9.0/docs/FAPI.md for more details
WARNING: Getting tokens from fapi backend failed.
Enter PIN for 'server':
WARNING: Listing FAPI token objects failed: "fapi:Provisioning was not executed."
Please see https://github.com/tpm2-software/tpm2-pkcs11/blob/1.9.0/docs/FAPI.md for more details
WARNING: Getting tokens from fapi backend failed.
Enter PIN for 'server':
🚀 ℹ️ [deploy] [INFO] Activating profile `system` for node `server`
🚀 ℹ️ [deploy] [INFO] Creating activation waiter
WARNING: Listing FAPI token objects failed: "fapi:Provisioning was not executed."
Please see https://github.com/tpm2-software/tpm2-pkcs11/blob/1.9.0/docs/FAPI.md for more details
WARNING: Getting tokens from fapi backend failed.
WARNING: Listing FAPI token objects failed: "fapi:Provisioning was not executed."
Please see https://github.com/tpm2-software/tpm2-pkcs11/blob/1.9.0/docs/FAPI.md for more details
WARNING: Getting tokens from fapi backend failed.
Enter PIN for 'server':
⭐ ℹ️ [activate] [INFO] Activating profile
updating GRUB 2 menu...
removing obsolete file /efi1/kernels/yqbmracgwis884jwivz5dqn06wxsrv94-nixos-system-server-23.05.872.948dcbc16aa-secrets
updating GRUB 2 menu...
removing obsolete file /efi2/kernels/yqbmracgwis884jwivz5dqn06wxsrv94-nixos-system-server-23.05.872.948dcbc16aa-secrets
activating the configuration...
[agenix] creating new generation in /run/agenix.d/12
[agenix] decrypting secrets...
decrypting '/nix/store/4d2ay3rygpdy3wx2pjk9768kqmmhy7nj-mail-mynacol_mynacol.xyz.age' to '/run/agenix.d/12/privkey'...
[agenix] symlinking new secrets to /run/agenix (generation 12)...
[agenix] removing old secrets (generation 11)...
[agenix] chowning...
setting up /etc...
reloading user units for root...
reloading user units for user...
setting up tmpfiles
⭐ ℹ️ [activate] [INFO] Activation succeeded!
⭐ ℹ️ [activate] [INFO] Magic rollback is enabled, setting up confirmation hook...
⭐ ℹ️ [activate] [INFO] Waiting for confirmation event...


WARNING:esys:src/tss2-esys/api/Esys_Unseal.c:295:Esys_Unseal_Finish() Received TPM Error
ERROR:esys:src/tss2-esys/api/Esys_Unseal.c:98:Esys_Unseal() Esys Finish ErrorCode (0x0000098e)
ERROR: Esys_Unseal: tpm:session(1):the authorization HMAC check failed and DA counter incremented
ERROR: Error unsealing wrapping key
PKCS#11 login failed: PIN incorrect
login failed
pkcs11_get_key failed
sign_and_send_pubkey: signing failed for ECDSA "": error in libcrypto
no such identity: /home/user/.ssh/server: No such file or directory
root@server's password: ⭐ ❌ [activate] [ERROR] Error waiting for confirmation event: Timeout elapsed for confirmation
⭐ ⚠️ [activate] [WARN] De-activating due to error
switching profile from version 816 to 815
⭐ ⚠️ [activate] [WARN] Removing generation by ID 816
removing profile version 816
⭐ ℹ️ [activate] [INFO] Attempting to re-activate the last generation
⭐ ❌ [activate] [ERROR] Error de-activating due to another error waiting for confirmation, oh no...: Failed to run command for re-activating the last generation: No such file or directory (os error 2)
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: RecvError(())', /build/source/src/deploy.rs:404:30
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

[nix-shell:~/projects/nixos]$
Permission denied, please try again.
root@server's password:
Permission denied, please try again.
root@server's password:
root@server: Permission denied (publickey,password).


[nix-shell:~/projects/nixos]$

For reference

A usual SSH login for me is the following:

$ ssh server
WARNING: Listing FAPI token objects failed: "fapi:Provisioning was not executed."
Please see https://github.com/tpm2-software/tpm2-pkcs11/blob/1.9.0/docs/FAPI.md for more details
WARNING: Getting tokens from fapi backend failed.
Enter PIN for 'server':
[root@server:~]# 

A failed SSH login due to a wrong PIN gives the following output:

$ ssh server
WARNING: Listing FAPI token objects failed: "fapi:Provisioning was not executed."
Please see https://github.com/tpm2-software/tpm2-pkcs11/blob/1.9.0/docs/FAPI.md for more details
WARNING: Getting tokens from fapi backend failed.
Enter PIN for 'server':
WARNING:esys:src/tss2-esys/api/Esys_Unseal.c:295:Esys_Unseal_Finish() Received TPM Error
ERROR:esys:src/tss2-esys/api/Esys_Unseal.c:98:Esys_Unseal() Esys Finish ErrorCode (0x0000098e)
ERROR: Esys_Unseal: tpm:session(1):the authorization HMAC check failed and DA counter incremented
ERROR: Error unsealing wrapping key
PKCS#11 login failed: PIN incorrect
login failed
pkcs11_get_key failed
sign_and_send_pubkey: signing failed for ECDSA "": error in libcrypto
no such identity: /home/user/.ssh/server: No such file or directory
root@server's password:
Permission denied, please try again.
root@server's password:
Permission denied, please try again.
root@server's password:
root@server: Permission denied (publickey,password).