Revereted optim.py, added distributed example and guide

.gitignore

@@ -60,3 +60,8 @@ target/
 
 #Ipython Notebook
 .ipynb_checkpoints
+
+# IDE settings
+.idea/
+
+.pytest_cache

.travis.yml

@@ -1,35 +1,26 @@
-sudo: required
 dist: trusty
 language: python
-python:
-  - "2.7"
-  # - "3.4" # Note that hyperopt currently seems to have issues with 3.4
+matrix:
+   include:
+      - python: 2.7
+        env:
+          - IPYTHON="ipython[all]==5.4.1"
+      - python: 3.5
+        env:
+          - IPYTHON=ipython[all]
 install:
-  # code below is taken from http://conda.pydata.org/docs/travis.html
-  # We do this conditionally because it saves us some downloading if the
-  # version is the same.
-  - if [[ "$TRAVIS_PYTHON_VERSION" == "2.7" ]]; then
-      wget https://repo.continuum.io/miniconda/Miniconda-latest-Linux-x86_64.sh -O miniconda.sh;
-    else
-      wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda.sh;
-    fi
-  - bash miniconda.sh -b -p $HOME/miniconda
-  - export PATH="$HOME/miniconda/bin:$PATH"
-  - hash -r
-  - conda config --set always_yes yes --set changeps1 no
-  - conda update -q conda
-  - conda info -a
-  - conda create -q -n test-environment python=$TRAVIS_PYTHON_VERSION numpy scipy matplotlib pandas pytest h5py flask scikit-learn networkx
-  - source activate test-environment
-  - pip install pytest-cov python-coveralls
-  - pip install pymongo==2.7.2
-  - pip install hyperopt
-  - pip install git+git://github.com/Theano/Theano.git
-  - pip install keras
+  - sudo apt-get remove ipython
+  - pip install --upgrade pip
+  - pip install --upgrade setuptools
+  - pip install --upgrade wheel
+  - pip install $IPYTHON
+  - pip install entrypoints
+  - pip install tensorflow
+  - pip install --only-binary=numpy,scipy numpy scipy
+  - pip install networkx==1.11
   - python setup.py install
-
-# Just run an example for now
+  - pip install pytest pytest-cov pep8 pytest-pep8
 script:
-  - python $PWD/examples/simple.py
+  - python -m pytest
 after_success:
   - coveralls

ISSUE_TEMPLATE.md

@@ -0,0 +1,13 @@
+Before filing an issue, please make sure to tick the following boxes.
+
+- [ ] Make sure your issue hasn't been filed already. Use GitHub search or manually check the [existing issues](https://github.com/maxpumperla/hyperas/issues), also the closed ones. Also, make sure to check the FAQ section of our [readme](https://github.com/maxpumperla/hyperas/blob/master/README.md).
+
+- [ ] Install latest hyperas from GitHub:
+pip install git+git://github.com/maxpumperla/hyperas.git
+
+- [ ] Install latest hyperopt from GitHub:
+pip install git+git://github.com/hyperopt/hyperopt.git
+
+- [ ] We have continuous integration running with Travis and make sure the build stays "green". If, after installing test utilities with `pip install pytest pytest-cov pep8 pytest-pep8`, you can't successfully run `python -m pytest` there's very likely a problem on your side that should be addressed before creating an issue.
+
+- [ ] Create a gist containing your complete script, or a minimal version of it, that can be used to reproduce your issue. Also, add your _full stack trace_ to that gist. In many cases your error message is enough to at least give some guidance.

MANIFEST.in

@@ -0,0 +1,5 @@
+include LICENSE
+include README.md
+include CONTRIBUTING.md
+graft examples
+graft tests

README.md

@@ -1,4 +1,4 @@
-# Hyperas [![Build Status](https://travis-ci.org/maxpumperla/hyperas.svg?branch=master)](https://travis-ci.org/maxpumperla/hyperas)  [![PyPI version](https://badge.fury.io/py/hyperas.svg)](https://badge.fury.io/py/hyperas) 
+# Hyperas [![Build Status](https://travis-ci.org/maxpumperla/hyperas.svg?branch=master)](https://travis-ci.org/maxpumperla/hyperas)  [![PyPI version](https://badge.fury.io/py/hyperas.svg)](https://badge.fury.io/py/hyperas)
 A very simple convenience wrapper around hyperopt for fast prototyping with keras models. Hyperas lets you use the power of hyperopt without having to learn the syntax of it. Instead, just define your keras model as you are used to, but use a simple template notation to define hyper-parameter ranges to tune.
 
 ## Installation
@@ -7,54 +7,90 @@ pip install hyperas
 ```
 
 ## Quick start
-Assume you have an existing keras model like the following.
+
+Assume you have data generated as such
+
 ```{python}
-model = Sequential()
-model.add(Dense(512, input_shape=(784,)))
-model.add(Activation('relu'))
-model.add(Dropout(0.2))
-model.add(Dense(512))
-model.add(Activation('relu'))
-model.add(Dropout(0.2)
-model.add(Dense(10))
-model.add(Activation('softmax'))
+def data():
+    x_train = np.zeros(100)
+    x_test = np.zeros(100)
+    y_train = np.zeros(100)
+    y_test = np.zeros(100)
+    return x_train, y_train, x_test, y_test
 ```
-To do hyper-parameter optimization on this model, just wrap the parameters you want to optimize into double curly brackets and choose a distribution over which to run the algorithm. In the above example, let's say we want to optimize for the best dropout probability in both dropout layers. Choosing a uniform distribution over the interval ```[0,1]```, this translates into the following definition.
+
+and an existing keras model like the following
 
 ```{python}
-from hyperas.distributions import uniform
+def create_model(x_train, y_train, x_test, y_test):
+    model = Sequential()
+    model.add(Dense(512, input_shape=(784,)))
+    model.add(Activation('relu'))
+    model.add(Dropout(0.2))
+    model.add(Dense(512))
+    model.add(Activation('relu'))
+    model.add(Dropout(0.2)
+    model.add(Dense(10))
+    model.add(Activation('softmax'))
 
-model = Sequential()
-model.add(Dense(512, input_shape=(784,)))
-model.add(Activation('relu'))
-model.add(Dropout({{uniform(0, 1)}}))
-model.add(Dense(512))
-model.add(Activation('relu'))
-model.add(Dropout({{uniform(0, 1)}}))
-model.add(Dense(10))
-model.add(Activation('softmax'))
+    # ... model fitting
+
+    return model
 ```
 
-After having trained the model, to optimize, we also have to define which evaluation metric of the model is important to us. For example, if we wish to optimize for accuracy, the following example does the trick:
+
+To do hyper-parameter optimization on this model,
+just wrap the parameters you want to optimize into double curly brackets
+and choose a distribution over which to run the algorithm.
+
+In the above example, let's say we want to optimize
+for the best dropout probability in both dropout layers.
+Choosing a uniform distribution over the interval ```[0,1]```,
+this translates into the following definition.
+Note that before returning the model, to optimize,
+we also have to define which evaluation metric of the model is important to us.
+For example, in the following, we optimize for accuracy.
+
+**Note**: In the following code we use `'loss': -accuracy`, i.e. the negative of accuracy. That's because under the hood `hyperopt` will always minimize whatever metric you provide. If instead you want to actually want to minimize a metric, say MSE or another loss function, you keep a positive sign (e.g. `'loss': mse`).
+
 
 ```{python}
-score = model.evaluate(X_test, Y_test, verbose=0)
-accuracy = score[1]
-return {'loss': -accuracy, 'status': STATUS_OK}
+from hyperas.distributions import uniform
+
+def create_model(x_train, y_train, x_test, y_test):
+    model = Sequential()
+    model.add(Dense(512, input_shape=(784,)))
+    model.add(Activation('relu'))
+    model.add(Dropout({{uniform(0, 1)}}))
+    model.add(Dense(512))
+    model.add(Activation('relu'))
+    model.add(Dropout({{uniform(0, 1)}}))
+    model.add(Dense(10))
+    model.add(Activation('softmax'))
+
+    # ... model fitting
+
+    score = model.evaluate(x_test, y_test, verbose=0)
+    accuracy = score[1]
+    return {'loss': -accuracy, 'status': STATUS_OK, 'model': model}
 ```
+
 The last step is to actually run the optimization, which is done as follows:
 
 ```{python}
-best_run = optim.minimize(model=model,
+best_run = optim.minimize(model=create_model,
                           data=data,
                           algo=tpe.suggest,
                           max_evals=10,
                           trials=Trials())
 ```
 In this example we use at most 10 evaluation runs and the TPE algorithm from hyperopt for optimization.
 
+Check the "complete example" below for more details.
+
+
 ## Complete example
-**Note:** It is important to wrap your data and model into functions as shown below, and then pass them as parameters to the minimizer. ```data()``` returns the data the ```model()``` needs. An extended version of the above example in one script reads as follows. This example shows many potential use cases of hyperas, including:
+**Note:** It is important to wrap your data and model into functions as shown below, and then pass them as parameters to the minimizer. ```data()``` returns the data the ```create_model()``` needs. An extended version of the above example in one script reads as follows. This example shows many potential use cases of hyperas, including:
 - Varying dropout probabilities, sampling from a uniform distribution
 - Different layer output sizes
 - Different optimization algorithms to use
@@ -65,36 +101,39 @@ In this example we use at most 10 evaluation runs and the TPE algorithm from hyp
 
 ```{python}
 from __future__ import print_function
+
 from hyperopt import Trials, STATUS_OK, tpe
-from hyperas import optim
-from hyperas.distributions import choice, uniform, conditional
 from keras.datasets import mnist
-from keras.utils import np_utils
-from keras.models import Sequential
 from keras.layers.core import Dense, Dropout, Activation
+from keras.models import Sequential
+from keras.utils import np_utils
+
+from hyperas import optim
+from hyperas.distributions import choice, uniform
+
 
 def data():
-    '''
+    """
     Data providing function:
 
-    This function is separated from model() so that hyperopt
+    This function is separated from create_model() so that hyperopt
     won't reload data for each evaluation run.
-    '''
-    (X_train, y_train), (X_test, y_test) = mnist.load_data()
-    X_train = X_train.reshape(60000, 784)
-    X_test = X_test.reshape(10000, 784)
-    X_train = X_train.astype('float32')
-    X_test = X_test.astype('float32')
-    X_train /= 255
-    X_test /= 255
+    """
+    (x_train, y_train), (x_test, y_test) = mnist.load_data()
+    x_train = x_train.reshape(60000, 784)
+    x_test = x_test.reshape(10000, 784)
+    x_train = x_train.astype('float32')
+    x_test = x_test.astype('float32')
+    x_train /= 255
+    x_test /= 255
     nb_classes = 10
-    Y_train = np_utils.to_categorical(y_train, nb_classes)
-    Y_test = np_utils.to_categorical(y_test, nb_classes)
-    return X_train, Y_train, X_test, Y_test
+    y_train = np_utils.to_categorical(y_train, nb_classes)
+    y_test = np_utils.to_categorical(y_test, nb_classes)
+    return x_train, y_train, x_test, y_test
 
 
-def model(X_train, Y_train, X_test, Y_test):
-    '''
+def create_model(x_train, y_train, x_test, y_test):
+    """
     Model providing function:
 
     Create Keras model with double curly brackets dropped-in as needed.
@@ -103,7 +142,7 @@ def model(X_train, Y_train, X_test, Y_test):
         - status: Just use STATUS_OK and see hyperopt documentation if not feasible
     The last one is optional, though recommended, namely:
         - model: specify the model just created so that we can later use it again.
-    '''
+    """
     model = Sequential()
     model.add(Dense(512, input_shape=(784,)))
     model.add(Activation('relu'))
@@ -113,9 +152,11 @@ def model(X_train, Y_train, X_test, Y_test):
     model.add(Dropout({{uniform(0, 1)}}))
 
     # If we choose 'four', add an additional fourth layer
-    if conditional({{choice(['three', 'four'])}}) == 'four':
+    if {{choice(['three', 'four'])}} == 'four':
         model.add(Dense(100))
+
         # We can also choose between complete sets of layers
+
         model.add({{choice([Dropout(0.5), Activation('linear')])}})
         model.add(Activation('relu'))
 
@@ -125,23 +166,85 @@ def model(X_train, Y_train, X_test, Y_test):
     model.compile(loss='categorical_crossentropy', metrics=['accuracy'],
                   optimizer={{choice(['rmsprop', 'adam', 'sgd'])}})
 
-    model.fit(X_train, Y_train,
+    model.fit(x_train, y_train,
               batch_size={{choice([64, 128])}},
-              nb_epoch=1,
-              show_accuracy=True,
+              epochs=1,
               verbose=2,
-              validation_data=(X_test, Y_test))
-    score, acc = model.evaluate(X_test, Y_test, verbose=0)
+              validation_data=(x_test, y_test))
+    score, acc = model.evaluate(x_test, y_test, verbose=0)
     print('Test accuracy:', acc)
     return {'loss': -acc, 'status': STATUS_OK, 'model': model}
 
+
 if __name__ == '__main__':
-    best_run, best_model = optim.minimize(model=model,
+    best_run, best_model = optim.minimize(model=create_model,
                                           data=data,
                                           algo=tpe.suggest,
                                           max_evals=5,
                                           trials=Trials())
     X_train, Y_train, X_test, Y_test = data()
     print("Evalutation of best performing model:")
     print(best_model.evaluate(X_test, Y_test))
+    print("Best performing model chosen hyper-parameters:")
+    print(best_run)
+```
+
+## FAQ
+
+Here is a list of a few popular errors
+
+### `TypeError: require string label`
+
+You're probably trying to execute the model creation code, with the templates, directly in python.
+That fails simply because python cannot run the templating in the braces, e.g. `{{uniform..}}`.
+The `def create_model(...)` function is in fact not a valid python function anymore.
+
+You need to wrap your code in a `def create_model(...): ...` function,
+and then call it from `optim.minimize(model=create_model,...` like in the example.
+
+The reason for this is that hyperas works by doing template replacement
+of everything in the `{{...}}` into a separate temporary file,
+and then running the model with the replaced braces (think jinja templating).
+
+This is the basis of how hyperas simplifies usage of hyperopt by being a "very simple wrapper".
+
+
+### `TypeError: 'generator' object is not subscriptable`
+
+This is currently a [known issue](https://github.com/maxpumperla/hyperas/issues/125).
+
+Just `pip install networkx==1.11`
+
+
+### `NameError: global name 'X_train' is not defined`
+
+Maybe you forgot to return the `x_train` argument in the `def create_model(x_train...)` call
+from the `def data(): ...` function.
+
+You are not restricted to the same list of arguments as in the example.
+Any arguments you return from `data()` will be passed to `create_model()`
+
+### notebook adjustment
+
+If you find error like ["No such file or directory"](https://github.com/maxpumperla/hyperas/issues/83) or [OSError, Err22](https://github.com/maxpumperla/hyperas/issues/149), you may need add `notebook_name='simple_notebook'`(assume your current notebook name is `simple_notebook`) in `optim.minimize` function like this:
+
+```python
+best_run, best_model = optim.minimize(model=model,
+                                      data=data,
+                                      algo=tpe.suggest,
+                                      max_evals=5,
+                                      trials=Trials(),
+                                      notebook_name='simple_notebook')
 ```
+
+### How does hyperas work?
+
+All we do is parse the `data` and `model` templates and translate them into proper `hyperopt` by reconstructing the `space` object that's then passed to `fmin`. Most of the relevant code is found in [optim.py](https://github.com/maxpumperla/hyperas/blob/master/hyperas/optim.py) and [utils.py](https://github.com/maxpumperla/hyperas/blob/master/hyperas/utils.py).
+
+### How to read the output of a hyperas model?
+
+Hyperas translates your script into `hyperopt` compliant code, see [here](https://github.com/maxpumperla/hyperas/issues/140) for some guidance on how to interpret the result.
+
+### What if I need more flexibility loading data and adapting my model?
+
+Hyperas is a convenience wrapper around Hyperopt that has some limitations. If it's not _convenient_ to use in your situation, simply don't use it -- and choose Hyperopt instead. All you can do with Hyperas you can also do with Hyperopt, it's just a different way of defining your model. If you want to squeeze some flexibility out of Hyperas anyway, take a look [here](https://github.com/maxpumperla/hyperas/issues/141).

examples/cifar_generator_cnn.py

@@ -60,7 +60,7 @@ def model(datagen, X_train, Y_train, X_test, Y_test):
     model = Sequential()
 
     model.add(Convolution2D(32, 3, 3, border_mode='same',
-                            input_shape=(img_channels, img_rows, img_cols)))
+                            input_shape=X_train.shape[1:]))
     model.add(Activation('relu'))
     model.add(Convolution2D(32, 3, 3))
     model.add(Activation('relu'))

examples/cnn_lstm.py

@@ -17,7 +17,7 @@ def data():
     max_features = 20000
     maxlen = 100
 
-    (X_train, y_train), (X_test, y_test) = imdb.load_data(nb_words=max_features, test_split=0.2)
+    (X_train, y_train), (X_test, y_test) = imdb.load_data(nb_words=max_features)
     X_train = sequence.pad_sequences(X_train, maxlen=maxlen)
     X_test = sequence.pad_sequences(X_test, maxlen=maxlen)