Merge pull request letscontrolit#3640 from TD-er/feature/fetchConfig

[Provisioning] Allow to fetch config via commands from HTTP server

docs/source/Plugin/P000_commands.repl

@@ -300,6 +300,36 @@
 
     ``Password,<new password>``"
     "
+    ProvisionConfig","
+    :red:`Internal`","
+    Fetch ``config.dat`` as configured in the Provisioning configuration (see Settings Archive)
+
+    ``ProvisionConfig``"
+    "
+    ProvisionSecurity","
+    :red:`Internal`","
+    Fetch ``security.dat`` as configured in the Provisioning configuration (see Settings Archive)
+
+    ``ProvisionSecurity``"
+    "
+    ProvisionNotification","
+    :red:`Internal`","
+    Fetch ``notification.dat`` as configured in the Provisioning configuration (see Settings Archive)
+
+    ``ProvisionNotification``"
+    "
+    ProvisionProvision","
+    :red:`Internal`","
+    Fetch ``provisioning.dat`` as configured in the Provisioning configuration (see Settings Archive)
+
+    ``ProvisionProvision``"
+    "
+    ProvisionRules","
+    :red:`Internal`","
+    Fetch ``rulesN.txt`` as configured in the Provisioning configuration (see Settings Archive)
+
+    ``ProvisionRules,1`` to fetch ``rules1.txt`` "
+    "
     Publish","
     :green:`Rules`","
     Send command using MQTT broker service

docs/source/Tools/Tools.rst

@@ -833,14 +833,44 @@ Especially if the node is hard to reach for a proper clean setup.
 .. image:: images/SettingsArchive_download2.png
 
 After downloading the files, a summary is given.
+
 A returned error can be something like 404 (file not available) or 401 (not authorized).
 These are the standard HTTP error codes.
+The error will be ``-1`` if the host is unreachable.
+
+If a file already exists, the new file is downloaded with ``_tmp`` appended to the filename.
+If successful, the original file will be renamed to one with ``_bak`` appended to the filename and then the ``_tmp`` version is renamed to the original filename.
+However, if the ``_bak`` is present, it may fail to rename the original one, so the operation fails.
+The presence of the ``_bak`` file is also some protection to not being able to fetch a new version, unless the "Delete First" option is checked.
 
 If ''config.dat'' or ''security.dat'' was downloaded, it is very important to do a reboot and not try to change (and save) anything on the ESPeasy node.
 The old settings are still active in memory and if something will be saved, only the changed part may be saved.
 This would corrupt the settings file.
 
 
+With only ``USE_SETTINGS_ARCHIVE`` defined during build, the URL and credentials cannot be stored.
+For this the build must be made with ``USE_CUSTOM_PROVISIONING`` defined.
+
+N.B. ``USE_CUSTOM_PROVISIONING`` is added on 2022/05/13.
+
+
+URL with Settings
+^^^^^^^^^^^^^^^^^
+
+This holds the full URL without file name where the files must be fetched from.
+
+Since builds made after 2022/05/13, the URL may also contain system variables.
+This allows for an URL like: ``http://192.168.10.127/%mac%``
+
+System variables will be converted into an URL encoded form, which may end up like this:
+
+* ``http://192.168.10.127/A0%3a20%3aA6%3a14%3a84%3a81/rules4.txt`` MAC address: ``A0:20:A6:14:84:81``
+
+The URL will not be stored, unless the build is made with ``USE_CUSTOM_PROVISIONING`` defined and the option is checked to save the URL. (option only present when ``USE_CUSTOM_PROVISIONING`` defined)
+
+Using system variables may allow for multi stage setup of a node, as you could for example fetch a rule which may set a variable to a new value and thus new files may be fetched from a different URL.
+
+
 Side Effects on cloning
 -----------------------
 
@@ -854,3 +884,47 @@ If the original node is configured to use static IP, the clone will use the same
 This can render both inaccessible.
 
 
+Provisioning
+============
+
+Added: 2022/05/13
+
+When the build is made with ``USE_CUSTOM_PROVISIONING`` defined, this Settings Archive screen does allow for more settings helping deployment and remote administration of ESPEasy nodes.
+
+All Settings on the Settings Archive page can be stored in a file named ``provisioning.dat``.
+This file also can store the factory default settings like the device model to ease deployment of a large number of nodes.
+
+N.B. The ``USE_SETTINGS_ARCHIVE`` define is needed to allow to edit the ``provisioning.dat`` file, but it is not needed to use the provisioning feature.
+
+
+.. image:: images/SettingsArchive_provisioning.png
+
+As can be seen, the URL and credentials can be stored.
+This will be stored in a file named ``provisioning.dat`` 
+Such a file may also be fetched from a server.
+
+The ``provisioning.dat`` file can also be automatically generated when performing a factory reset.
+For this the (custom) build must be prepared via a number of defined defaults.
+See the ``Custom-sample.h`` file for some examples.
+
+
+Allow Fetch by Command
+^^^^^^^^^^^^^^^^^^^^^^
+
+This checkbox allows provisioning via commands.
+These commands are not restricted, so they can also be given via HTTP or MQTT.
+
+However, they can only be executed when:
+
+* Allow Fetch by Command is enabled
+* the file to download is checked
+* URL (+ optional credentials) is stored
+
+The commands are:
+
+
+* ``ProvisionConfig`` Fetch ``config.dat``
+* ``ProvisionSecurity`` Fetch ``security.dat``
+* ``ProvisionNotification`` Fetch ``notification.dat``
+* ``ProvisionProvision`` Fetch ``provisioning.dat``
+* ``ProvisionRules,1`` Fetch ``rules1.txt``

src/Custom-sample.h

@@ -169,6 +169,12 @@
 
 #define CUSTOM_EMERGENCY_FALLBACK_ALLOW_MINUTES_UPTIME 10
 
+// Allow for remote provisioning of a node.
+// This is only allowed for custom builds.
+// To setup the configuration of the provisioning file, one must also define USE_SETTINGS_ARCHIVE
+// Default setting is to not allow to configure a node remotely, unless explicitly enabled.
+// #define USE_CUSTOM_PROVISIONING
+
 #define USES_SSDP
 
 #define USE_EXT_RTC                // Support for external RTC clock modules like PCF8563/PCF8523/DS3231/DS1307 
@@ -181,6 +187,35 @@
 // #define ADAGFX_ARGUMENT_VALIDATION  0 // Disable argument validation in AdafruitGFX_helper
 // #define ADAGFX_SUPPORT_7COLOR  0 // Disable the support of 7-color eInk displays by AdafruitGFX_helper
 
+
+#ifdef USE_CUSTOM_PROVISIONING
+// For device models, see src/src/DataTypes/DeviceModel.h
+// #ifdef ESP32
+//  #define DEFAULT_FACTORY_DEFAULT_DEVICE_MODEL  0 // DeviceModel_default
+// #endif
+// #ifdef ESP8266
+//  #define DEFAULT_FACTORY_DEFAULT_DEVICE_MODEL  0 // DeviceModel_default
+// #endif
+//  #define DEFAULT_PROVISIONING_FETCH_RULES1      false
+//  #define DEFAULT_PROVISIONING_FETCH_RULES2      false
+//  #define DEFAULT_PROVISIONING_FETCH_RULES3      false
+//  #define DEFAULT_PROVISIONING_FETCH_RULES4      false
+//  #define DEFAULT_PROVISIONING_FETCH_NOTIFICATIONS false
+//  #define DEFAULT_PROVISIONING_FETCH_SECURITY     false
+//  #define DEFAULT_PROVISIONING_FETCH_CONFIG       false
+//  #define DEFAULT_PROVISIONING_FETCH_PROVISIONING false
+//  #define DEFAULT_PROVISIONING_SAVE_URL           false
+//  #define DEFAULT_PROVISIONING_SAVE_CREDENTIALS   false
+//  #define DEFAULT_PROVISIONING_ALLOW_FETCH_COMMAND false
+//  #define DEFAULT_PROVISIONING_URL                ""
+//  #define DEFAULT_PROVISIONING_USER               ""
+//  #define DEFAULT_PROVISIONING_PASS               ""
+#endif
+
+
+
+#define USES_SSDP
+
 /*
  #######################################################################################################
    Defining web interface

src/ESPEasy_common.h

@@ -79,10 +79,6 @@ namespace std
 
   #include <core_version.h>
   #define NODE_TYPE_ID      NODE_TYPE_ID_ESP_EASYM_STD
-  #define FILE_CONFIG       "config.dat"
-  #define FILE_SECURITY     "security.dat"
-  #define FILE_NOTIFICATION "notification.dat"
-  #define FILE_RULES        "rules1.txt"
   #include <lwip/init.h>
   #ifndef LWIP_VERSION_MAJOR
     #error
@@ -124,10 +120,6 @@ namespace std
   #if ESP_IDF_VERSION_MAJOR < 3
     #define ICACHE_RAM_ATTR IRAM_ATTR
   #endif
-  #define FILE_CONFIG       "/config.dat"
-  #define FILE_SECURITY     "/security.dat"
-  #define FILE_NOTIFICATION "/notification.dat"
-  #define FILE_RULES        "/rules1.txt"
   #include <WiFi.h>
 //  #include  "esp32_ping.h"
 

src/src/Commands/InternalCommands.cpp

@@ -23,6 +23,7 @@
 
 #include "../Commands/Networks.h"
 #include "../Commands/Notifications.h"
+#include "../Commands/Provisioning.h"
 #include "../Commands/RTC.h"
 #include "../Commands/Rules.h"
 #include "../Commands/SDCARD.h"
@@ -384,6 +385,13 @@ bool executeInternalCommand(command_case_data & data)
         COMMAND_CASE_A(       "pcfpulse", Command_GPIO_Pulse,           3); // GPIO.h
       }
       COMMAND_CASE_R("password", Command_Settings_Password, 1); // Settings.h
+#ifdef USE_CUSTOM_PROVISIONING
+      COMMAND_CASE_A(       "provisionconfig", Command_Provisioning_Config,       0); // Provisioning.h
+      COMMAND_CASE_A(     "provisionsecurity", Command_Provisioning_Security,     0); // Provisioning.h
+      COMMAND_CASE_A( "provisionnotification", Command_Provisioning_Notification, 0); // Provisioning.h
+      COMMAND_CASE_A(    "provisionprovision", Command_Provisioning_Provision,    0); // Provisioning.h
+      COMMAND_CASE_A(        "provisionrules", Command_Provisioning_Rules,        1); // Provisioning.h
+#endif
       COMMAND_CASE_A(   "pulse", Command_GPIO_Pulse,        3); // GPIO.h
 #ifdef USES_MQTT
       COMMAND_CASE_A( "publish", Command_MQTT_Publish,      2); // MQTT.h

src/src/Commands/Provisioning.cpp

@@ -0,0 +1,42 @@
+#include "../Commands/Provisioning.h"
+
+
+#ifdef USE_CUSTOM_PROVISIONING
+
+# include "../Commands/Common.h"
+
+# include "../../ESPEasy_common.h"
+# include "../DataTypes/ESPEasyFileType.h"
+# include "../DataStructs/ESPEasy_EventStruct.h"
+# include "../Helpers/ESPEasy_Storage.h"
+
+
+String Command_Provisioning_Config(struct EventStruct *event, const char *Line)
+{
+  return downloadFileType(FileType::CONFIG_DAT);
+}
+
+String Command_Provisioning_Security(struct EventStruct *event, const char *Line)
+{
+  return downloadFileType(FileType::SECURITY_DAT);
+}
+
+String Command_Provisioning_Notification(struct EventStruct *event, const char *Line)
+{
+  return downloadFileType(FileType::NOTIFICATION_DAT);
+}
+
+String Command_Provisioning_Provision(struct EventStruct *event, const char *Line)
+{
+  return downloadFileType(FileType::PROVISIONING_DAT);
+}
+
+String Command_Provisioning_Rules(struct EventStruct *event, const char *Line)
+{
+  if ((event->Par1 <= 0) || (event->Par1 > 4)) {
+    return F("ProvisionRules: rules index out of range");
+  }
+  return downloadFileType(FileType::RULES_TXT, event->Par1 - 1);
+}
+
+#endif // ifdef USE_CUSTOM_PROVISIONING

src/src/Commands/Provisioning.h

@@ -0,0 +1,24 @@
+#ifndef COMMANDS_PROVISIONING_H
+#define COMMANDS_PROVISIONING_H
+
+#include "../../ESPEasy_common.h"
+
+#ifdef USE_CUSTOM_PROVISIONING
+
+class String;
+
+String Command_Provisioning_Config(struct EventStruct *event,
+                                   const char         *Line);
+String Command_Provisioning_Security(struct EventStruct *event,
+                                     const char         *Line);
+String Command_Provisioning_Notification(struct EventStruct *event,
+                                         const char         *Line);
+String Command_Provisioning_Provision(struct EventStruct *event,
+                                      const char         *Line);
+String Command_Provisioning_Rules(struct EventStruct *event,
+                                  const char         *Line);
+
+
+#endif // ifdef USE_CUSTOM_PROVISIONING
+
+#endif // ifndef COMMANDS_PROVISIONING_H

src/src/CustomBuild/ESPEasyDefaults.h

@@ -312,6 +312,55 @@
 #define DEFAULT_SYNC_UDP_PORT                   0                       // Used for ESPEasy p2p. (IANA registered port: 8266)
 #endif
 
+// --- Defaults to be used for custom automatic provisioning builds ------------------------------------
+#ifdef USE_CUSTOM_PROVISIONING
+  #ifndef DEFAULT_FACTORY_DEFAULT_DEVICE_MODEL
+    #define DEFAULT_FACTORY_DEFAULT_DEVICE_MODEL  0 // DeviceModel_default
+  #endif
+  #ifndef DEFAULT_PROVISIONING_FETCH_RULES1
+    #define DEFAULT_PROVISIONING_FETCH_RULES1      false
+  #endif
+  #ifndef DEFAULT_PROVISIONING_FETCH_RULES2
+    #define DEFAULT_PROVISIONING_FETCH_RULES2      false
+  #endif
+  #ifndef DEFAULT_PROVISIONING_FETCH_RULES3
+    #define DEFAULT_PROVISIONING_FETCH_RULES3      false
+  #endif
+  #ifndef DEFAULT_PROVISIONING_FETCH_RULES4
+    #define DEFAULT_PROVISIONING_FETCH_RULES4      false
+  #endif
+  #ifndef DEFAULT_PROVISIONING_FETCH_NOTIFICATIONS
+    #define DEFAULT_PROVISIONING_FETCH_NOTIFICATIONS false
+  #endif
+  #ifndef DEFAULT_PROVISIONING_FETCH_SECURITY
+    #define DEFAULT_PROVISIONING_FETCH_SECURITY     false
+  #endif
+  #ifndef DEFAULT_PROVISIONING_FETCH_CONFIG
+    #define DEFAULT_PROVISIONING_FETCH_CONFIG       false
+  #endif
+  #ifndef DEFAULT_PROVISIONING_FETCH_PROVISIONING
+    #define DEFAULT_PROVISIONING_FETCH_PROVISIONING false
+  #endif
+  #ifndef DEFAULT_PROVISIONING_SAVE_URL
+    #define DEFAULT_PROVISIONING_SAVE_URL           false
+  #endif
+  #ifndef DEFAULT_PROVISIONING_SAVE_CREDENTIALS
+    #define DEFAULT_PROVISIONING_SAVE_CREDENTIALS   false
+  #endif
+  #ifndef DEFAULT_PROVISIONING_ALLOW_FETCH_COMMAND
+    #define DEFAULT_PROVISIONING_ALLOW_FETCH_COMMAND false
+  #endif
+  #ifndef DEFAULT_PROVISIONING_URL
+    #define DEFAULT_PROVISIONING_URL                ""
+  #endif
+  #ifndef DEFAULT_PROVISIONING_USER
+    #define DEFAULT_PROVISIONING_USER               ""
+  #endif
+  #ifndef DEFAULT_PROVISIONING_PASS
+    #define DEFAULT_PROVISIONING_PASS               ""
+  #endif
+#endif
+
 #ifndef BUILD_IN_WEBHEADER
 #define BUILD_IN_WEBHEADER                      false
 #endif

src/src/CustomBuild/define_plugin_sets.h

@@ -1855,6 +1855,12 @@ To create/register a plugin, you have to :
   #endif
 #endif
 
+#if defined(USE_SETTINGS_ARCHIVE) || defined(USE_CUSTOM_PROVISIONING)
+  #ifndef USE_DOWNLOAD
+    #define USE_DOWNLOAD
+  #endif
+#endif
+
 // Here we can re-enable specific features in the TESTING sets as we have created some space there by splitting them up
 #if defined(TESTING_USE_RTTTL) && (defined(PLUGIN_SET_TESTING_A) || defined(PLUGIN_SET_TESTING_B) || defined(PLUGIN_SET_TESTING_C) || defined(PLUGIN_SET_TESTING_D) || defined(PLUGIN_SET_TESTING_E))
   #ifndef USE_RTTTL

src/src/DataStructs/FactoryDefaultPref.cpp

@@ -94,6 +94,39 @@ void ResetFactoryDefaultPreference_struct::deleteFirst(bool checked) {
   bitWrite(_preference, 21, checked);
 }
 
+bool ResetFactoryDefaultPreference_struct::saveURL() const {
+  return bitRead(_preference, 22);
+}
+
+void ResetFactoryDefaultPreference_struct::saveURL(bool checked) {
+  bitWrite(_preference, 22, checked);
+}
+
+bool ResetFactoryDefaultPreference_struct::allowFetchByCommand() const {
+  return bitRead(_preference, 23);
+}
+
+void ResetFactoryDefaultPreference_struct::allowFetchByCommand(bool checked) {
+  bitWrite(_preference, 23, checked);
+}
+
+bool ResetFactoryDefaultPreference_struct::storeCredentials() const {
+  return bitRead(_preference, 24);
+}
+
+void ResetFactoryDefaultPreference_struct::storeCredentials(bool checked) {
+  bitWrite(_preference, 24, checked);
+}
+
+bool ResetFactoryDefaultPreference_struct::fetchProvisioningDat() const {
+  return bitRead(_preference, 25);
+}
+
+void ResetFactoryDefaultPreference_struct::fetchProvisioningDat(bool checked) {
+  bitWrite(_preference, 25, checked);
+}
+
+
 uint32_t ResetFactoryDefaultPreference_struct::getPreference() {
   return _preference;
 }