Thread saturation metrics 📈 (hashicorp#489)

Adds metrics suggested in hashicorp#488, to record the percentage of time the
main and FSM goroutines are busy with work vs available to accept new
work, to give operators an idea of how close they are to hitting
capacity limits.

add back deprecated field `Leader` to the `LeaderObservation` struct (h…

…ashicorp#500)

Add HeartbeatTimeout and ElectionTimeout to reloadable config. (hashi…

…corp#496)

a NonVoter node should never try to bootstrap (hashicorp#492)

This is a follow-up to hashicorp#483

Snapshot restore progress (hashicorp#490)

When restoring a snapshot (on startup, installed from the leader, or during recovery) the logs are extremely terse. There are typically bookend messages indicating that a restore is going to happen, and that it is complete, but there's a big dead space in the middle.

For small snapshots this is probably fine, but for larger multi-GB snapshots this can stretch out and can be unnerving as an operator to know if it's stuck or still making progress.

This PR adjusts the logging to indicate a simple progress log message every 10s about overall completion in bytes-consumed.

Leadership transfer gaps (hashicorp#487)

* setLeadershipTransferInProgress before starting the transfer go routine to avoid a race.

* do not transition to follower state in a middle of a leadership transfer

a NonVoter node should never be able to transition to a Candidate sta…

…te (hashicorp#483)

* only voters can transition to a `Candidate` state

* clarify log message and remove not used func `inConfig`

* add tests to make sure leader transition happen as expected when the `HeartbeatTimeout` is reached

* fix race

add missing import

add retraction to version 1.1.3