Skip to content

Commit

Permalink
KAFKA-16101: Fix KRaft migration documentation (apache#15193)
Browse files Browse the repository at this point in the history 
This PR fixes some bugs in the KRaft migration documentation and reorganizes it to be easier to read. (Specifically, there were some steps that were previously out of order.)

In order to keep it all straight, the revert documentation is now in the form of a table which maps the latest migration state to the actions which the system administrator should perform.

Reviewers: Luke Chen <[email protected]>, David Arthur <[email protected]>, Liu Zeyu <[email protected]>, Paolo Patierno <[email protected]>
  • Loading branch information
@cmccabe
cmccabe authored Jan 29, 2024
1 parent 16ed735 commit 5b42261
Showing 1 changed file with 139 additions and 21 deletions.
160 changes: 139 additions & 21 deletions docs/ops.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3874,7 +3874,7 @@ <h3>Provisioning the KRaft controller quorum</h3>
<p><em>Note: The KRaft cluster <code>node.id</code> values must be different from any existing ZK broker <code>broker.id</code>.
In KRaft-mode, the brokers and controllers share the same Node ID namespace.</em></p>

<h3>Enabling the migration on the brokers</h3>
<h3>Enter Migration Mode on the Brokers</h3>
<p>
Once the KRaft controller quorum has been started, the brokers will need to be reconfigured and restarted. Brokers
may be restarted in a rolling fashion to avoid impacting cluster availability. Each broker requires the
Expand Down Expand Up @@ -3919,9 +3919,10 @@ <h3>Enabling the migration on the brokers</h3>

<h3>Migrating brokers to KRaft</h3>
<p>
Once the KRaft controller completes the metadata migration, the brokers will still be running in ZK mode. While the
KRaft controller is in migration mode, it will continue sending controller RPCs to the ZK mode brokers. This includes
RPCs like UpdateMetadata and LeaderAndIsr.
Once the KRaft controller completes the metadata migration, the brokers will still be running
in ZooKeeper mode. While the KRaft controller is in migration mode, it will continue sending
controller RPCs to the ZooKeeper mode brokers. This includes RPCs like UpdateMetadata and
LeaderAndIsr.
</p>

<p>
Expand Down Expand Up @@ -3956,29 +3957,16 @@ <h3>Migrating brokers to KRaft</h3>
Each broker is restarted with a KRaft configuration until the entire cluster is running in KRaft mode.
</p>

<h3>Reverting to ZooKeeper mode During the Migration</h3>
While the cluster is still in migration mode, it is possible to revert to ZK mode. In order to do this:
<ol>
<li>
For each KRaft broker:
<ul>
<li>Stop the broker.</li>
<li>Remove the __cluster_metadata directory on the broker.</li>
<li>Remove the <code>zookeeper.metadata.migration.enable</code> configuration and the KRaft controllers related configurations like <code>controller.quorum.voters</code>
and <code>controller.listener.names</code> from the broker configuration properties file.</li>
<li>Restart the broker in ZooKeeper mode.</li>
</ul>
</li>
<li>Take down the KRaft quorum.</li>
<li>Using ZooKeeper shell, delete the controller node using <code>rmr /controller</code>, so that a ZooKeeper-based broker can become the next controller.</li>
</ol>

<h3>Finalizing the migration</h3>
<p>
Once all brokers have been restarted in KRaft mode, the last step to finalize the migration is to take the
KRaft controllers out of migration mode. This is done by removing the "zookeeper.metadata.migration.enable"
property from each of their configs and restarting them one at a time.
</p>
<p>
Once the migration has been finalized, you can safely deprovision your ZooKeeper cluster, assuming you are
not using it for anything else. After this point, it is no longer possible to revert to ZooKeeper mode.
</p>

<pre>
# Sample KRaft cluster controller.properties listening on 9093
Expand All @@ -3996,6 +3984,136 @@ <h3>Finalizing the migration</h3>

# Other configs ...</pre>

<h3>Reverting to ZooKeeper mode During the Migration</h3>
<p>
While the cluster is still in migration mode, it is possible to revert to ZooKeeper mode. The process
to follow depends on how far the migration has progressed. In order to find out how to revert,
select the <b>final</b> migration step that you have <b>completed</b> in this table.
</p>
<p>
Note that the directions given here assume that each step was fully completed, and they were
done in order. So, for example, we assume that if "Enter Migration Mode on the Brokers" was
completed, "Provisioning the KRaft controller quorum" was also fully completed previously.
</p>
<p>
If you did not fully complete any step, back out whatever you have done and then follow revert
directions for the last fully completed step.
</p>

<table class="data-table">
<tbody>
<tr>
<th>Final Migration Section Completed</th>
<th>Directions for Reverting</th>
<th>Notes</th>
</tr>
<tr>
<td>Preparing for migration</td>
<td>
The preparation section does not involve leaving ZooKeeper mode. So there is nothing to do in the
case of a revert.
</td>
<td>
</td>
</tr>
<tr>
<td>Provisioning the KRaft controller quorum</td>
<td>
<ul>
<li>
Deprovision the KRaft controller quorum.
</li>
<li>
Then you are done.
</li>
</ul>
</td>
<td>
</td>
</tr>
<tr>
<td>Enter Migration Mode on the brokers</td>
<td>
<ul>
<li>
Deprovision the KRaft controller quorum.
</li>
<li>
Using <code>zookeeper-shell.sh</code>, run <code>rmr /controller</code> so that one
of the brokers can become the new old-style controller.
</li>
<li>
On each broker, remove the <code>zookeeper.metadata.migration.enable</code>,
<code>controller.listener.names</code>, and <code>controller.quorum.voters</code>
configurations, and replace <code>node.id</code> with <code>broker.id</code>.
Then perform a rolling restart of all brokers.
</li>
<li>
Then you are done.
</li>
</ul>
</td>
<td>
It is important to perform the <code>zookeeper-shell.sh</code> step quickly, to minimize the amount of
time that the cluster lacks a controller.
</td>
</tr>
<tr>
<td>Migrating brokers to KRaft</td>
<td>
<ul>
<li>
On each broker, remove the <code>process.roles</code> configuration, and
restore the <code>zookeeper.connect</code> configuration to its previous value.
If your cluster requires other ZooKeeper configurations for brokers, such as
<code>zookeeper.ssl.protocol</code>, re-add those configurations as well.
Then perform a rolling restart of all brokers.
</li>
<li>
Deprovision the KRaft controller quorum.
</li>
<li>
Using <code>zookeeper-shell.sh</code>, run <code>rmr /controller</code> so that one
of the brokers can become the new old-style controller.
</li>
<li>
On each broker, remove the <code>zookeeper.metadata.migration.enable</code>,
<code>controller.listener.names</code>, and <code>controller.quorum.voters</code>
configurations. Replace <code>node.id</code> with <code>broker.id</code>.
Then perform a second rolling restart of all brokers.
</li>
<li>
Then you are done.
</li>
</ul>
</td>
<td>
<ul>
<li>
It is important to perform the <code>zookeeper-shell.sh</code> step <b>quickly</b>, to minimize the amount of
time that the cluster lacks a controller.
</li>
<li>
Make sure that on the first cluster roll, <code>zookeeper.metadata.migration.enable</code> remains set to
</code>true</code>. <b>Do not set it to false until the second cluster roll.</b>
</li>
</ul>
</td>
</tr>
<tr>
<td>Finalizing the migration</td>
<td>
If you have finalized the ZK migration, then you cannot revert.
</td>
<td>
Some users prefer to wait for a week or two before finalizing the migration. While this
requires you to keep the ZooKeeper cluster running for a while longer, it may be helpful
in validating KRaft mode in your cluster.
</td>
</tr>
</tbody>
</table>


<h3 class="anchor-heading"><a id="tiered_storage" class="anchor-link"></a><a href="#kraft">6.11 Tiered Storage</a></h3>

Expand Down

0 comments on commit 5b42261

Please sign in to comment.