Skip to content

Commit

Permalink
Align nni.experiment tuner behavior with nnictl (microsoft#3419)
Browse files Browse the repository at this point in the history
  • Loading branch information
@liuzhe-lz
liuzhe-lz authored Mar 8, 2021
1 parent f897829 commit bc55eec
Show file tree
Hide file tree
Showing 18 changed files with 316 additions and 362 deletions.
53 changes: 22 additions & 31 deletions docs/en_US/Tutorial/HowToLaunchFromPython.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -13,59 +13,50 @@ Since ``nni v2.0``, we provide a new way to launch experiments. Before that, you

Run a New Experiment
--------------------
After successfully installing ``nni``, you can start the experiment with a python script in the following 3 steps.
After successfully installing ``nni``, you can start the experiment with a python script in the following 2 steps.

..
Step 1 - Initialize a tuner you want to use

Step 1 - Initialize an experiment instance and configure it

.. code-block:: python
from nni.algorithms.hpo.hyperopt_tuner import HyperoptTuner
tuner = HyperoptTuner('tpe')
Very simple, you have successfully initialized a ``HyperoptTuner`` instance called ``tuner``.

See all real `builtin tuners <../builtin_tuner.rst>`__ supported in NNI.

..
Step 2 - Initialize an experiment instance and configure it

.. code-block:: python
experiment = Experiment(tuner=tuner, training_service='local')
from nni.experiment import Experiment
experiment = Experiment('local')
Now, you have a ``Experiment`` instance with ``tuner`` you have initialized in the previous step, and this experiment will launch trials on your local machine due to ``training_service='local'``.
Now, you have a ``Experiment`` instance, and this experiment will launch trials on your local machine due to ``training_service='local'``.

See all `training services <../training_services.rst>`__ supported in NNI.

.. code-block:: python
experiment.config.experiment_name = 'test'
experiment.config.experiment_name = 'MNIST example'
experiment.config.trial_concurrency = 2
experiment.config.max_trial_number = 5
experiment.config.max_trial_number = 10
experiment.config.search_space = search_space
experiment.config.trial_command = 'python3 mnist.py'
experiment.config.trial_code_directory = Path(__file__).parent
experiment.config.tuner.name = 'TPE'
experiment.config.tuner.class_args['optimize_mode'] = 'maximize'
experiment.config.training_service.use_active_gpu = True
Use the form like ``experiment.config.foo = 'bar'`` to configure your experiment.

See all real `builtin tuners <../builtin_tuner.rst>`__ supported in NNI.

See `parameter configuration <../reference/experiment_config.rst>`__ required by different training services.

..
Step 3 - Just run
Step 2 - Just run

.. code-block:: python
experiment.run(port=8081)
experiment.run(port=8080)
Now, you have successfully launched an NNI experiment. And you can type ``localhost:8081`` in your browser to observe your experiment in real time.
Now, you have successfully launched an NNI experiment. And you can type ``localhost:8080`` in your browser to observe your experiment in real time.

.. Note:: In this way, experiment will run in the foreground and will automatically exit when the experiment finished. If you want to run an experiment in an interactive way, use ``start()`` in Step 3.
.. Note:: In this way, experiment will run in the foreground and will automatically exit when the experiment finished. If you want to run an experiment in an interactive way, use ``start()`` in Step 2.

Example
^^^^^^^
Expand All @@ -74,10 +65,8 @@ Below is an example for this new launching approach. You can also find this code
.. code-block:: python
from pathlib import Path
from nni.experiment import Experiment
from nni.algorithms.hpo.hyperopt_tuner import HyperoptTuner
tuner = HyperoptTuner('tpe')
from nni.experiment import Experiment
search_space = {
"dropout_rate": { "_type": "uniform", "_value": [0.5, 0.9] },
Expand All @@ -87,16 +76,18 @@ Below is an example for this new launching approach. You can also find this code
"learning_rate": { "_type": "choice", "_value": [0.0001, 0.001, 0.01, 0.1] }
}
experiment = Experiment(tuner, 'local')
experiment.config.experiment_name = 'test'
experiment = Experiment('local')
experiment.config.experiment_name = 'MNIST example'
experiment.config.trial_concurrency = 2
experiment.config.max_trial_number = 5
experiment.config.max_trial_number = 10
experiment.config.search_space = search_space
experiment.config.trial_command = 'python3 mnist.py'
experiment.config.trial_code_directory = Path(__file__).parent
experiment.config.tuner.name = 'TPE'
experiment.config.tuner.class_args['optimize_mode'] = 'maximize'
experiment.config.training_service.use_active_gpu = True
experiment.run(8081)
experiment.run(8080)
Start and Manage a New Experiment
---------------------------------
Expand Down
51 changes: 29 additions & 22 deletions docs/en_US/Tutorial/python_api_connect.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -26,8 +26,7 @@
"name": "stdout",
"output_type": "stream",
"text": [
"[2021-02-25 07:50:38] Tuner not set, wait for connect...\n",
"[2021-02-25 07:50:38] Connect to port 8080 success, experiment id is IF0JnfLE, status is RUNNING.\n"
"[2021-03-05 12:18:28] Connect to port 8080 success, experiment id is DH8pVfXc, status is RUNNING.\n"
]
}
],
Expand All @@ -53,27 +52,27 @@
{
"data": {
"text/plain": [
"{'id': 'IF0JnfLE',\n",
" 'revision': 6,\n",
" 'execDuration': 28,\n",
" 'logDir': '/home/ningshang/nni-experiments/IF0JnfLE',\n",
" 'nextSequenceId': 2,\n",
"{'id': 'DH8pVfXc',\n",
" 'revision': 4,\n",
" 'execDuration': 10,\n",
" 'logDir': '/home/ningshang/nni-experiments/DH8pVfXc',\n",
" 'nextSequenceId': 1,\n",
" 'params': {'authorName': 'default',\n",
" 'experimentName': 'example_sklearn-classification',\n",
" 'trialConcurrency': 1,\n",
" 'maxExecDuration': 3600,\n",
" 'maxTrialNum': 5,\n",
" 'maxTrialNum': 100,\n",
" 'searchSpace': '{\"C\": {\"_type\": \"uniform\", \"_value\": [0.1, 1]}, \"kernel\": {\"_type\": \"choice\", \"_value\": [\"linear\", \"rbf\", \"poly\", \"sigmoid\"]}, \"degree\": {\"_type\": \"choice\", \"_value\": [1, 2, 3, 4]}, \"gamma\": {\"_type\": \"uniform\", \"_value\": [0.01, 0.1]}, \"coef0\": {\"_type\": \"uniform\", \"_value\": [0.01, 0.1]}}',\n",
" 'trainingServicePlatform': 'local',\n",
" 'tuner': {'builtinTunerName': 'TPE',\n",
" 'classArgs': {'optimize_mode': 'maximize'},\n",
" 'checkpointDir': '/home/ningshang/nni-experiments/IF0JnfLE/checkpoint'},\n",
" 'checkpointDir': '/home/ningshang/nni-experiments/DH8pVfXc/checkpoint'},\n",
" 'versionCheck': True,\n",
" 'clusterMetaData': [{'key': 'trial_config',\n",
" 'value': {'command': 'python3 main.py',\n",
" 'codeDir': '/home/ningshang/nni/examples/trials/sklearn/classification/.',\n",
" 'gpuNum': 0}}]},\n",
" 'startTime': 1614239412494}"
" 'startTime': 1614946699989}"
]
},
"execution_count": 2,
Expand All @@ -90,9 +89,17 @@
"execution_count": 3,
"id": "printable-bookmark",
"metadata": {},
"outputs": [],
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[2021-03-05 12:18:32] (root) Successfully update maxTrialNum.\n"
]
}
],
"source": [
"experiment.update_max_trial_number(10)"
"experiment.update_max_trial_number(200)"
]
},
{
Expand All @@ -104,27 +111,27 @@
{
"data": {
"text/plain": [
"{'id': 'IF0JnfLE',\n",
" 'revision': 8,\n",
" 'execDuration': 32,\n",
" 'logDir': '/home/ningshang/nni-experiments/IF0JnfLE',\n",
" 'nextSequenceId': 2,\n",
"{'id': 'DH8pVfXc',\n",
" 'revision': 5,\n",
" 'execDuration': 14,\n",
" 'logDir': '/home/ningshang/nni-experiments/DH8pVfXc',\n",
" 'nextSequenceId': 1,\n",
" 'params': {'authorName': 'default',\n",
" 'experimentName': 'example_sklearn-classification',\n",
" 'trialConcurrency': 1,\n",
" 'maxExecDuration': 3600,\n",
" 'maxTrialNum': 10,\n",
" 'maxTrialNum': 200,\n",
" 'searchSpace': '{\"C\": {\"_type\": \"uniform\", \"_value\": [0.1, 1]}, \"kernel\": {\"_type\": \"choice\", \"_value\": [\"linear\", \"rbf\", \"poly\", \"sigmoid\"]}, \"degree\": {\"_type\": \"choice\", \"_value\": [1, 2, 3, 4]}, \"gamma\": {\"_type\": \"uniform\", \"_value\": [0.01, 0.1]}, \"coef0\": {\"_type\": \"uniform\", \"_value\": [0.01, 0.1]}}',\n",
" 'trainingServicePlatform': 'local',\n",
" 'tuner': {'builtinTunerName': 'TPE',\n",
" 'classArgs': {'optimize_mode': 'maximize'},\n",
" 'checkpointDir': '/home/ningshang/nni-experiments/IF0JnfLE/checkpoint'},\n",
" 'checkpointDir': '/home/ningshang/nni-experiments/DH8pVfXc/checkpoint'},\n",
" 'versionCheck': True,\n",
" 'clusterMetaData': [{'key': 'trial_config',\n",
" 'value': {'command': 'python3 main.py',\n",
" 'codeDir': '/home/ningshang/nni/examples/trials/sklearn/classification/.',\n",
" 'gpuNum': 0}}]},\n",
" 'startTime': 1614239412494}"
" 'startTime': 1614946699989}"
]
},
"execution_count": 4,
Expand Down Expand Up @@ -154,8 +161,8 @@
"name": "stdout",
"output_type": "stream",
"text": [
"[2021-02-25 07:50:49] Stopping experiment, please wait...\n",
"[2021-02-25 07:50:49] Experiment stopped\n"
"[2021-03-05 12:18:36] Stopping experiment, please wait...\n",
"[2021-03-05 12:18:38] Experiment stopped\n"
]
}
],
Expand Down
Loading

0 comments on commit bc55eec

Please sign in to comment.