mlregressor, add web buttons for mlregressor, add some suggestions

docs/develop.md

@@ -221,6 +221,11 @@ For those who wish to mount/sync the local `data` folder with the data folder fr
 docker run ... -v $(pwd)/data/:/app/data ...
 ```
 
+You can also mount data (ex .csv)  files separately
+```bash
+docker run... -v $(pwd)/data/heating_prediction.csv:/app/data/ ...
+```
+
 #### Issue with TARGETARCH
 If your docker build fails with an error related to `TARGETARCH`. It may be best to add your devices architecture manually:
 
@@ -301,7 +306,7 @@ git checkout $branch
 ```bash
 #testing addon (build and run)
 docker build -t emhass/docker --build-arg build_version=addon-local .
-docker run --rm -it -p 5000:5000 --name emhass-container -v $(pwd)/options.json:/app/options.json -e LAT="45.83" -e LON="6.86" -e ALT="4807.8" -e TIME_ZONE="Europe/Paris" emhass/docker --url $HAURL --key $HAKEY
+docker run --rm -it -p 5000:5000 --name emhass-container -v $(pwd)/data/heating_prediction.csv:/app/data/heating_prediction.csv -v $(pwd)/options.json:/app/options.json -e LAT="45.83" -e LON="6.86" -e ALT="4807.8" -e TIME_ZONE="Europe/Paris" emhass/docker --url $HAURL --key $HAKEY
 ```
 ```bash
 #run actions on a separate terminal
@@ -311,6 +316,8 @@ curl -i -H 'Content-Type:application/json' -X POST -d {} http://localhost:5000/a
 curl -i -H 'Content-Type:application/json' -X POST -d {} http://localhost:5000/action/forecast-model-fit
 curl -i -H 'Content-Type:application/json' -X POST -d {} http://localhost:5000/action/forecast-model-predict
 curl -i -H 'Content-Type:application/json' -X POST -d {} http://localhost:5000/action/forecast-model-tune
+curl -i -H "Content-Type:application/json" -X POST -d  '{"csv_file": "heating_prediction.csv", "features": ["degreeday", "solar"], "target": "hour", "regression_model": "RandomForestRegression", "model_type": "heating_hours_degreeday", "timestamp": "timestamp", "date_features": ["month", "day_of_week"], "new_values": [12.79, 4.766, 1, 2] }' http://localhost:5000/action/regressor-model-fit
+curl -i -H "Content-Type:application/json" -X POST -d  '{"mlr_predict_entity_id": "sensor.mlr_predict", "mlr_predict_unit_of_measurement": "h", "mlr_predict_friendly_name": "mlr predictor", "new_values": [8.2, 7.23, 2, 6], "model_type": "heating_hours_degreeday" }' http://localhost:5000/action/regressor-model-predict
 curl -i -H 'Content-Type:application/json' -X POST -d {} http://localhost:5000/action/publish-data
 ```
 
@@ -326,7 +333,7 @@ lat: 45.83
 lon: 6.86
 alt: 4807.8
 EOT
-docker run --rm -it -p 5000:5000 --name emhass-container -v $(pwd)/config_emhass.yaml:/app/config_emhass.yaml -v $(pwd)/secrets_emhass.yaml:/app/secrets_emhass.yaml emhass/docker 
+docker run --rm -it -p 5000:5000 --name emhass-container -v $(pwd)/data/heating_prediction.csv:/app/data/heating_prediction.csv -v $(pwd)/config_emhass.yaml:/app/config_emhass.yaml -v $(pwd)/secrets_emhass.yaml:/app/secrets_emhass.yaml emhass/docker 
 ```
 ```bash
 #run actions on a separate terminal
@@ -336,10 +343,12 @@ curl -i -H 'Content-Type:application/json' -X POST -d {} http://localhost:5000/a
 curl -i -H 'Content-Type:application/json' -X POST -d {} http://localhost:5000/action/forecast-model-fit
 curl -i -H 'Content-Type:application/json' -X POST -d {} http://localhost:5000/action/forecast-model-predict
 curl -i -H 'Content-Type:application/json' -X POST -d {} http://localhost:5000/action/forecast-model-tune
+curl -i -H "Content-Type:application/json" -X POST -d  '{"csv_file": "heating_prediction.csv", "features": ["degreeday", "solar"], "target": "hour", "regression_model": "RandomForestRegression", "model_type": "heating_hours_degreeday", "timestamp": "timestamp", "date_features": ["month", "day_of_week"], "new_values": [12.79, 4.766, 1, 2] }' http://localhost:5000/action/regressor-model-fit
+curl -i -H "Content-Type:application/json" -X POST -d  '{"mlr_predict_entity_id": "sensor.mlr_predict", "mlr_predict_unit_of_measurement": "h", "mlr_predict_friendly_name": "mlr predictor", "new_values": [8.2, 7.23, 2, 6], "model_type": "heating_hours_degreeday" }' http://localhost:5000/action/regressor-model-predict
 curl -i -H 'Content-Type:application/json' -X POST -d {} http://localhost:5000/action/publish-data
 ```
 
-User may wish to re-test with tweaked parameters such as `lp_solver` and `weather_forecast_method`, in `config_emhass.yaml` *(standalone)* or `options.json` *(addon)*, to broaden the testing scope. 
+User may wish to re-test with tweaked parameters such as `lp_solver`, `weather_forecast_method` and `load_forecast_method`, in `config_emhass.yaml` *(standalone)* or `options.json` *(addon)*, to broaden the testing scope. 
 *see [EMHASS & EMHASS-Add-on differences](https://emhass.readthedocs.io/en/latest/differences.html) for more information on how these config_emhass & options files differ*
 
 *Note: may need to set `--build-arg TARGETARCH=YOUR-ARCH` in docker build*

docs/mlregressor.md

@@ -4,10 +4,9 @@ Starting with v0.9.0, a new framework is proposed within EMHASS. It provides a m
 
 This API provides two main methods:
 
-- fit: To train a model with the passed data. This method is exposed with the `regressor-model-fit` end point.
-
-- predict: To obtain a prediction from a pre-trained model. This method is exposed with the `regressor-model-predict` end point.
+- **fit**: To train a model with the passed data. This method is exposed with the `regressor-model-fit` end point.
 
+- **predict**: To obtain a prediction from a pre-trained model. This method is exposed with the `regressor-model-predict` end point.
 
 ## A basic model fit
 
@@ -29,10 +28,11 @@ Some paramters can be optionally defined at runtime:
 
 - `date_features`: A list of 'date_features' to take into account when fitting the model. Possibilities are `year`, `month`, `day_of_week` (monday=0, sunday=6), `day_of_year`, `day`(day_of_month) and `hour`
 
-```
+### Examples: 
+```yaml
 runtimeparams = {
     "csv_file": "heating_prediction.csv",
-    "features":["degreeday", "solar"],
+    "features": ["degreeday", "solar"],
     "target": "heating_hours",
     "regression_model": "RandomForestRegression",
     "model_type": "heating_hours_degreeday",
@@ -43,20 +43,25 @@ runtimeparams = {
 
 A correct `curl` call to launch a model fit can look like this:
 
+```bash
+curl -i -H "Content-Type:application/json" -X POST -d '{"csv_file": "heating_prediction.csv", "features": ["degreeday", "solar"], "target": "heating_hours"}' http://localhost:5000/action/regressor-model-fit
 ```
-curl -i -H "Content-Type:application/json" -X POST -d '{}' http://localhost:5000/action/regressor-model-fit
+or 
+```bash
+curl -i -H "Content-Type:application/json" -X POST -d  '{"csv_file": "heating_prediction.csv", "features": ["degreeday", "solar"], "target": "hour", "regression_model": "RandomForestRegression", "model_type": "heating_hours_degreeday", "timestamp": "timestamp", "date_features": ["month", "day_of_week"], "new_values": [12.79, 4.766, 1, 2] }' http://localhost:5000/action/regressor-model-fit
 ```
+
 A Home Assistant `rest_command` can look like this:
 
-```
+```yaml
 fit_heating_hours:
   url: http://127.0.0.1:5000/action/regressor-model-fit
   method: POST
   content_type: "application/json"
   payload: >-
     {
     "csv_file": "heating_prediction.csv",
-    "features":["degreeday", "solar"],
+    "features": ["degreeday", "solar"],
     "target": "hours",
     "regression_model": "RandomForestRegression",
     "model_type": "heating_hours_degreeday",
@@ -91,7 +96,8 @@ The list of parameters needed to set the data publish task is:
 
 - `model_type`: The model type that has to be predicted
 
-```
+### Examples: 
+```yaml
 runtimeparams = {
     "mlr_predict_entity_id": "sensor.mlr_predict",
     "mlr_predict_unit_of_measurement": None,
@@ -103,13 +109,17 @@ runtimeparams = {
 
 Pass the correct `model_type` like this:
 
-```
+```bash
 curl -i -H "Content-Type:application/json" -X POST -d '{"model_type": "heating_hours_degreeday"}' http://localhost:5000/action/regressor-model-predict
 ```
+or
+```bash
+curl -i -H "Content-Type:application/json" -X POST -d  '{"mlr_predict_entity_id": "sensor.mlr_predict", "mlr_predict_unit_of_measurement": "h", "mlr_predict_friendly_name": "mlr predictor", "new_values": [8.2, 7.23, 2, 6], "model_type": "heating_hours_degreeday" }' http://localhost:5000/action/regressor-model-predict
+```
 
 A Home Assistant `rest_command` can look like this:
 
-```
+```yaml
 predict_heating_hours:
   url: http://localhost:5001/action/regressor-model-predict
   method: POST
@@ -136,17 +146,38 @@ After predicting the model the following information is logged by EMHASS:
 The predict method will publish the result to a Home Assistant sensor.
 
 
-## How to store data in a csv file from Home Assistant
-Notify to a file
+## Storing CSV files  
+
+### Standalone container - how to mount a .csv files in data_path folder
+If running EMHASS as Standalone container, you will need to volume mount a folder to be the `data_path`, or mount a single .csv file inside `data_path`
+
+Example of mounting a folder as data_path *(.csv files stored inside)*
+```bash
+docker run -it --restart always -p 5000:5000 -e LOCAL_COSTFUN="profit" -v $(pwd)/data:/app/data -v $(pwd)/config_emhass.yaml:/app/config_emhass.yaml -v $(pwd)/secrets_emhass.yaml:/app/secrets_emhass.yaml --name DockerEMHASS <REPOSITORY:TAG>
+```
+Example of mounting a single csv file
+```bash
+docker run -it --restart always -p 5000:5000 -e LOCAL_COSTFUN="profit" -v $(pwd)/data/heating_prediction.csv:/app/data/heating_prediction.csv -v $(pwd)/config_emhass.yaml:/app/config_emhass.yaml -v $(pwd)/secrets_emhass.yaml:/app/secrets_emhass.yaml --name DockerEMHASS <REPOSITORY:TAG>
 ```
+
+### Add-on - How to store data in a csv file from Home Assistant
+
+#### Change data_path
+If running EMHASS-Add-On, you will likley need to change the `data_path` to a folder your Home Assistant can access. 
+To do this, set the `data_path` to `/share/` in the addon *Configuration* page. 
+
+#### Store sensor data to csv
+
+Notify to a file
+```yaml
 notify:
   - platform: file
     name: heating_hours_prediction
     timestamp: false
     filename: /share/heating_prediction.csv
 ```
 Then you need an automation to notify to this file
-```
+```yaml
 alias: "Heating csv"
 id: 157b1d57-73d9-4f39-82c6-13ce0cf42
 trigger:

src/emhass/command_line.py

@@ -225,8 +225,9 @@ def set_input_data_dict(emhass_conf: dict, costfun: str,
                 df_input_data = pd.read_csv(filename_path, parse_dates=True)
 
             else:
-                logger.error("The cvs file was not found.")
-                raise ValueError("The CSV file " + csv_file + " was not found.")
+                logger.error("The CSV file " + csv_file + " was not found in path: " + str(emhass_conf["data_path"]))
+                return False
+                #raise ValueError("The CSV file " + csv_file + " was not found.")
             required_columns = []
             required_columns.extend(features)
             required_columns.append(target)
@@ -236,9 +237,11 @@ def set_input_data_dict(emhass_conf: dict, costfun: str,
             if not set(required_columns).issubset(df_input_data.columns):
                 logger.error("The cvs file does not contain the required columns.")
                 msg = f"CSV file should contain the following columns: {', '.join(required_columns)}"
-                raise ValueError(
-                    msg,
-                )
+                logger.error(msg)
+                return False
+                #raise ValueError(
+                #    msg,
+                #)
 
     elif set_type == "publish-data":
         df_input_data, df_input_data_dayahead = None, None
@@ -615,12 +618,36 @@ def regressor_model_fit(
     :type debug: Optional[bool], optional
     """
     data = copy.deepcopy(input_data_dict["df_input_data"])
-    model_type = input_data_dict["params"]["passed_data"]["model_type"]
-    regression_model = input_data_dict["params"]["passed_data"]["regression_model"]
-    features = input_data_dict["params"]["passed_data"]["features"]
-    target = input_data_dict["params"]["passed_data"]["target"]
-    timestamp = input_data_dict["params"]["passed_data"]["timestamp"]
-    date_features = input_data_dict["params"]["passed_data"]["date_features"]
+    if "model_type" in input_data_dict["params"]["passed_data"]:
+        model_type = input_data_dict["params"]["passed_data"]["model_type"]
+    else:
+        logger.error("parameter: 'model_type' not passed")
+        return False
+    if "regression_model" in input_data_dict["params"]["passed_data"]:
+        regression_model = input_data_dict["params"]["passed_data"]["regression_model"]
+    else:
+        logger.error("parameter: 'regression_model' not passed")
+        return False
+    if "features" in input_data_dict["params"]["passed_data"]:
+        features = input_data_dict["params"]["passed_data"]["features"]
+    else:
+        logger.error("parameter: 'features' not passed")
+        return False    
+    if "target" in input_data_dict["params"]["passed_data"]:
+        target = input_data_dict["params"]["passed_data"]["target"]
+    else:
+        logger.error("parameter: 'target' not passed")
+        return False        
+    if "timestamp" in input_data_dict["params"]["passed_data"]:
+        timestamp = input_data_dict["params"]["passed_data"]["timestamp"]
+    else:
+        logger.error("parameter: 'timestamp' not passed")
+        return False       
+    if "date_features" in input_data_dict["params"]["passed_data"]:
+        date_features = input_data_dict["params"]["passed_data"]["date_features"]
+    else:
+        logger.error("parameter: 'date_features' not passed")
+        return False           
 
     # The MLRegressor object
     mlr = MLRegressor(
@@ -658,7 +685,11 @@ def regressor_model_predict(
     :param debug: True to debug, useful for unit testing, defaults to False
     :type debug: Optional[bool], optional
     """
-    model_type = input_data_dict["params"]["passed_data"]["model_type"]
+    if "model_type" in input_data_dict["params"]["passed_data"]:
+        model_type = input_data_dict["params"]["passed_data"]["model_type"]
+    else:
+        logger.error("parameter: 'model_type' not passed")
+        return False   
     filename = model_type + "_mlr.pkl"
     filename_path = input_data_dict["emhass_conf"]["data_path"] / filename
     if not debug:
@@ -669,20 +700,18 @@ def regressor_model_predict(
             logger.error(
                 "The ML forecaster file was not found, please run a model fit method before this predict method",
             )
-            return
-    new_values = input_data_dict["params"]["passed_data"]["new_values"]
+            return False
+    if "new_values" in input_data_dict["params"]["passed_data"]:    
+        new_values = input_data_dict["params"]["passed_data"]["new_values"]
+    else:
+        logger.error("parameter: 'new_values' not passed")
+        return False         
     # Predict from csv file
     prediction = mlr.predict(new_values)
-
-    mlr_predict_entity_id = input_data_dict["params"]["passed_data"][
-        "mlr_predict_entity_id"
-    ]
-    mlr_predict_unit_of_measurement = input_data_dict["params"]["passed_data"][
-        "mlr_predict_unit_of_measurement"
-    ]
-    mlr_predict_friendly_name = input_data_dict["params"]["passed_data"][
-        "mlr_predict_friendly_name"
-    ]
+
+    mlr_predict_entity_id = input_data_dict["params"]["passed_data"].get("mlr_predict_entity_id","sensor.mlr_predict")
+    mlr_predict_unit_of_measurement = input_data_dict["params"]["passed_data"].get("mlr_predict_unit_of_measurement","h")
+    mlr_predict_friendly_name = input_data_dict["params"]["passed_data"].get("mlr_predict_friendly_name","mlr predictor")
     # Publish prediction
     idx = 0
     if not debug:

src/emhass/static/advanced.html

@@ -14,6 +14,9 @@ <h4>Use the buttons below to fit, predict and tune a machine learning model for
 <button type="button" id="forecast-model-predict" class="button button2">ML forecast model
     predict</button>
 <button type="button" id="forecast-model-tune" class="button button3">ML forecast model tune</button>
+</br></br>
+<button type="button" id="regressor-model-fit" class="button button1">ML regressor model fit</button>
+<button type="button" id="regressor-model-predict" class="button button2">ML regressor model predict</button>
 <!-- -->
 <!--dynamic input elements section -->
 <h4>Input Runtime Parameters</h4>

src/emhass/static/script.js

@@ -16,6 +16,8 @@ function loadButtons(page) {
                 "forecast-model-fit",
                 "forecast-model-predict",
                 "forecast-model-tune",
+                "regressor-model-fit",
+                "regressor-model-predict",
                 "perfect-optim",
                 "publish-data",
                 "naive-mpc-optim"

src/emhass/utils.py

@@ -216,13 +216,16 @@ def treat_runtimeparams(
         freq = int(retrieve_hass_conf["freq"].seconds / 60.0)
         delta_forecast = int(optim_conf["delta_forecast"].days)
         forecast_dates = get_forecast_dates(freq, delta_forecast)
-        if set_type == "regressor-model-fit":
-            csv_file = runtimeparams["csv_file"]
-            features = runtimeparams["features"]
-            target = runtimeparams["target"]
-            params["passed_data"]["csv_file"] = csv_file
-            params["passed_data"]["features"] = features
-            params["passed_data"]["target"] = target
+        if set_type == "regressor-model-fit":            
+            if "csv_file" in runtimeparams:
+                csv_file = runtimeparams["csv_file"]
+                params["passed_data"]["csv_file"] = csv_file
+            if "features" in runtimeparams:
+                features = runtimeparams["features"]
+                params["passed_data"]["features"] = features
+            if "target" in runtimeparams:
+                target = runtimeparams["target"]
+                params["passed_data"]["target"] = target
             if "timestamp" not in runtimeparams:
                 params["passed_data"]["timestamp"] = None
             else:
@@ -233,10 +236,10 @@ def treat_runtimeparams(
             else:
                 date_features = runtimeparams["date_features"]
                 params["passed_data"]["date_features"] = date_features
-
         if set_type == "regressor-model-predict":
-            new_values = runtimeparams["new_values"]
-            params["passed_data"]["new_values"] = new_values
+            if "new_values" in runtimeparams:
+                new_values = runtimeparams["new_values"]
+                params["passed_data"]["new_values"] = new_values
             if "csv_file" in runtimeparams:
                 csv_file = runtimeparams["csv_file"]
                 params["passed_data"]["csv_file"] = csv_file

src/emhass/web_server.py

@@ -195,15 +195,21 @@ def action_call(action_name):
             return make_response(msg, 201)
         return make_response(grabLog(ActionStr), 400)
     elif action_name == 'regressor-model-fit':
-        app.logger.info(" >> Performing a machine learning regressor fit...")
+        ActionStr = " >> Performing a machine learning regressor fit..."
+        app.logger.info(ActionStr)
         regressor_model_fit(input_data_dict, app.logger)
         msg = f'EMHASS >> Action regressor-model-fit executed... \n'
-        return make_response(msg, 201)
+        if not checkFileLog(ActionStr):
+            return make_response(msg, 201)
+        return make_response(grabLog(ActionStr), 400)
     elif action_name == 'regressor-model-predict':
-        app.logger.info(" >> Performing a machine learning regressor predict...")
+        ActionStr = " >> Performing a machine learning regressor predict..."
+        app.logger.info(ActionStr)
         regressor_model_predict(input_data_dict, app.logger)
         msg = f'EMHASS >> Action regressor-model-predict executed... \n'
-        return make_response(msg, 201)
+        if not checkFileLog(ActionStr):
+            return make_response(msg, 201)
+        return make_response(grabLog(ActionStr), 400)
     else:
         app.logger.error("ERROR: passed action is not valid")
         msg = f'EMHASS >> ERROR: Passed action is not valid... \n'