Merge remote-tracking branch 'origin/topic/dnthayer/doc-fixes-for-2.6'

* origin/topic/dnthayer/doc-fixes-for-2.6:
  Fix some typos and improve formatting in NEWS
  Update the operators documentation
  Replace references to libgeoip in the documentation
  Update install instructions for python-ipaddress
  Update documentation of "option" and "redef" declarations
  Improvements to the config framework documentation
  Rearrange some lines on the "Log Files" documentation page
  Improve install/setup instructions for libmaxminddb
  Update NEWS for config framework clusterization changes
  Update config framework doc for clusterization changes
  Fix typos and formatting issues in config framework docs

CHANGES

@@ -1,4 +1,8 @@
 
+2.5-870 | 2018-08-17 17:07:57 -0500
+
+  * Documentation improvements (Daniel Thayer)
+
 2.5-855 | 2018-08-17 16:34:51 -0500
 
   * Add script to support the old DHCP events (Vlad Grigorescu)

NEWS

@@ -114,7 +114,11 @@ New Functionality
   - Option variables: The new "option" keyword allows variables to be
     declared as runtime options. Such variables cannot be changed
     using normal assignments. Instead, they can be changed using the
-    new function Option::set.
+    new function Config::set_value.  This function will automatically
+    apply the change to all nodes in a cluster.  Note that options can also
+    be changed using the new function Option::set, but this function will
+    not send the change to any other nodes, so Config::set_value should
+    typically be used instead of Option::set.
 
     It is possible to "subscribe" to an option through
     Option::set_change_handler, which will trigger a handler callback
@@ -162,7 +166,7 @@ New Functionality
         }
 
     The specified file will now be monitored continuously for changes, so
-    that writing "testbool T" into /path/to/config.dat will
+    that writing "TestConfig::testbool T" into /path/to/config.dat will
     automatically update the option's value accordingly.
 
     The configuration framework creates a config.log that shows all
@@ -220,7 +224,6 @@ New Functionality
   ssl_server_curve event, ssl_server_curve is now marked as deprecated.
 
 - Functions for retrieving files by their ID have been added:
-
   Files::file_exists, Files::lookup_File
 
 - New functions in the logging API: Log::get_filter_names, Log::enable_stream
@@ -237,13 +240,13 @@ New Functionality
     mount_proc_umnt, mount_proc_umnt_all, mount_proc_not_implemented,
     mount_reply_status.
 
-- Added new NFS events: nfs_proc_symlink, nfs_proc_link, nfs_proc_sattr
+- Added new NFS events: nfs_proc_symlink, nfs_proc_link, nfs_proc_sattr.
 
 - The SMB scripts in policy/protocols/smb are now moved into base/protocols/smb
   and loaded/enabled by default.
 
 - Added new SMB events: smb1_transaction_secondary_request,
-  smb1_transaction2_secondary_request, smb1_transaction_response
+  smb1_transaction2_secondary_request, smb1_transaction_response.
 
 - Bro can now decrypt Kerberos tickets, and retrieve the authentication from
   them, given a suitable keytab file.
@@ -260,20 +263,18 @@ New Functionality
 
 - You can now specify that a pattern matches in a case-insensitive
   fashion by adding 'i' to the end of its specification.  So for example
-  /fOO/i == "Foo" yields T, as does /fOO/i in "xFoObar".  Characters
-  enclosed in quotes however keep their casing, so /"fOO"/i in "xFoObar"
-  yields F, though it yields T for "xfOObar".
+  /fOO/i == "Foo" yields T, as does /fOO/i in "xFoObar".
 
   You can achieve the same functionality for a subpattern enclosed in
   parentheses by adding "?i:" to the open parenthesis.  So for example
-  "/foo|(?i:bar)/" will match "BaR", but not "FoO".
+  /foo|(?i:bar)/ will match "BaR", but not "FoO".
 
   For both ways of specifying case-insensitivity, characters enclosed in
-  double quotes maintain their case-sensitivity.  So for example /"foo"/i
-  will not match "Foo", but it will match "foo".
+  double quotes remain case-sensitive.  So for example /"foo"/i will not
+  match "Foo", but it will match "foo".
 
 - "make install" now installs Bro's include headers (and more) into
-  --prefix so that compiling plugins does no longer need access to a
+  --prefix so that compiling plugins no longer needs access to a
   source/build tree. For OS distributions, this also facilitates
   creating "bro-devel" packages providing all files necessary to build
   plugins.
@@ -313,7 +314,7 @@ New Functionality
   of non-equality, proper superset, and superset-or-equal.
 
 - An expression of the form "v += e" will append the value of the expression
-  "e" to the end of the vector "v" (of course assuming type-compatbility).
+  "e" to the end of the vector "v" (of course assuming type-compatibility).
   "redef v += { a, b, c }" will similarly extend a vector previously declared
   with &redef by appending the result of expressions "a", "b", and "c" to
   the vector at initialization-time.
@@ -336,9 +337,9 @@ Changed Functionality
     script since it's generally less relevant now with the updated log.
 
   - Removed the base/protocols/dhcp/utils.bro script and thus the
-    'reverse_ip' function.
+    "reverse_ip" function.
 
-  - Replaced all DHCP events with the single 'dhcp_message' event.
+  - Replaced all DHCP events with the single "dhcp_message" event.
     The list of removed events includes:
 
     - dhcp_discover
@@ -381,9 +382,9 @@ Changed Functionality
   redef SOCKS::default_capture_password = T;
 
 - The DNS base scripts no longer generate some noisy and annoying
-  weirds (dns_unmatched_msg, dns_unmatched_msg_quantity, dns_unmatched_reply)
+  weirds (dns_unmatched_msg, dns_unmatched_msg_quantity, dns_unmatched_reply).
 
-- The 'tunnel_parents' field of conn.log is now marked &optional, so, for
+- The "tunnel_parents" field of conn.log is now marked &optional, so, for
   the default configuration of logs, this field will show "-" instead of
   "(empty)" for connections that lack any tunneling.
 
@@ -403,7 +404,7 @@ Changed Functionality
   - event ssl_server_signature now has an additional argument
     "signature_and_hashalgorithm".
 
-- The "dnp3_header_block" event no longer has the "start" parameter
+- The "dnp3_header_block" event no longer has the "start" parameter.
 
 - The string_to_pattern() built-in (and the now-deprecated merge_pattern()
   built-in) is no longer restricted to only be called at initialization time.
@@ -426,11 +427,11 @@ Removed Functionality
   https://github.com/bro/packages for a list of Bro packages currently
   available.
 
-- BroControl: The option 'IPv6Comm' and 'ZoneID' options are no longer
-  available (though Broker should be able to handle IPv6 automatically).
-
 - The "ocsp_request" event no longer has "requestorName" parameter.
 
+- BroControl: The "IPv6Comm" and "ZoneID" options are no longer
+  available (though Broker should be able to handle IPv6 automatically).
+
 Deprecated Functionality
 ------------------------
 
@@ -445,10 +446,6 @@ Deprecated Functionality
   as BiFs like send_id(). Use Broker data stores and the new
   configuration framework instead.
 
-- BroControl: The 'update' command is deprecated and scheduled for
-  removal with the next Bro release. Bro's new configuration framework
-  is taking its place.
-
 - Mixing of scalars and vectors, such as "v + e" yielding a vector
   corresponding to the vector v with the scalar e added to each of
   its elements, has been deprecated.
@@ -459,6 +456,10 @@ Deprecated Functionality
 - The undocumented feature of using "&&" and "||" operators for patterns
   has been deprecated.
 
+- BroControl: The "update" command is deprecated and scheduled for
+  removal with the next Bro release. Bro's new configuration framework
+  is taking its place.
+
 Bro 2.5.1
 =========
 

VERSION

@@ -1 +1 @@
-2.5-855
+2.5-870

doc/frameworks/configuration.rst

@@ -11,7 +11,7 @@ Bro includes a "configuration framework" that allows
 updating script options dynamically at runtime. This functionality
 consists of several components: an "option" declaration, the
 ability to specify input files to enable changing the value of options at
-runtime, a couple of built-in functions, and a log file "config.log"
+runtime, a couple of functions, and a log file "config.log"
 which contains information about every change to option values.
 
 
@@ -50,14 +50,28 @@ The "option" keyword allows variables to be declared as configuration options.
         option my_networks: set[subnet] = {};
         option enable_feature = F;
         option hostname = "testsystem";
+        option timeout = 1min;
+        option my_ports: vector of port = {};
     }
 
 The rules regarding options can be thought of as being in between global
 variables and constants.  Like global variables, options cannot be declared
 inside a function, hook, or event handler.  Like constants, options must be
-initialized when declared.  The value of an option can change at runtime,
+initialized when declared (the type can often be inferred from the initializer
+but may need to be specified).  The value of an option can change at runtime,
 but options cannot be assigned a new value using normal assignments.
 
+The initial value of an option can be redefined with a :bro:keyword:`redef`
+declaration just like for global variables and constants.  The only difference
+being that there is no need to specify the :bro:attr:`&redef` attribute in
+the declaration of an option.  For example, given the above option
+declarations, here are some possible redefs:
+
+.. code:: bro
+
+    redef TestModule::enable_feature = T;
+    redef TestModule::my_networks += { 10.1.0.0/16, 10.2.0.0/16 };
+
 
 Changing options
 ----------------
@@ -70,21 +84,32 @@ The format for these files looks like this:
 
     [option name][tab/spaces][new value]
 
-Configuration files can be specified by adding them to Config::config_files.
+Configuration files can be specified by adding them
+to :bro:id:`Config::config_files`.  Note that in a cluster configuration,
+only the manager node attempts to read the specified configuration files.
+
 For example, simply add something like this to local.bro:
 
 .. code:: bro
 
     redef Config::config_files += { "/path/to/config.dat" };
 
-The specified configuration file will then be monitored continuously for changes,
-so that writing ``TestModule::enable_feature T`` into that file will
-automatically update the option's value accordingly.  Here is an example
-configuration file::
+The specified configuration file will then be monitored continuously for
+changes, so that writing ``TestModule::enable_feature T`` into that file will
+automatically update the option's value accordingly (in a cluster
+configuration, the change will be sent from the manager to all other nodes in
+the cluster).  Here is an example configuration file::
 
     TestModule::my_networks 10.0.12.0/24,192.168.17.0/24
     TestModule::enable_feature  T
     TestModule::hostname  host-1
+    TestModule::timeout 50.5
+    TestModule::my_ports 80/tcp,53/udp
+
+Note that as seen in the above example, for options of
+type :bro:type:`interval`, the numeric value in the config file
+is interpreted as seconds and there cannot be any time units
+(such as sec, min, etc.).
 
 Internally, the configuration framework uses the Bro input framework
 with a type of input reader specifically for reading config files. Users
@@ -94,13 +119,34 @@ for configuration files: the files need no header lines and either
 tabs or spaces are accepted as separators.
 
 If you inspect the configuration framework scripts, you will notice that the
-scripts simply catch events from the input framework and then a built-in
-function :bro:see:`Option::set` is called to set an option to the new value.
-If you want to change an option yourself during runtime, you can
-call Option::set directly from a script.
+scripts simply catch events from the input framework and then a
+function :bro:see:`Config::set_value` is called to set an option to the new
+value.  If you want to change an option yourself during runtime, you can
+call Config::set_value directly from a script (in a cluster configuration,
+this only needs to happen on the manager, as the change will be automatically
+sent to all other nodes in the cluster).
+
+Note that some data types (such as pattern, table, and record) are not
+supported by the config input reader.  In that case you would need to use
+the Config::set_value function to change the value of such an option as
+shown in the following example.
+
+.. code:: bro
+
+    module TestModule;
 
-The log file "config.log" contains information about each configuration
-change that occurs during runtime.
+    export {
+        option host_port: table[addr] of port = {};
+    }
+
+    event bro_init() {
+        local t: table[addr] of port = { [10.0.0.2] = 123/tcp };
+        Config::set_value("TestModule::host_port", t);
+    }
+
+Regardless of whether an option change is triggered by a config file or by
+the Config::set_value function, the change is always logged to the
+log file "config.log".
 
 
 Change handlers
@@ -114,7 +160,11 @@ accordingly):
 
 .. code:: bro
 
-    option testaddr = 127.0.0.1;
+    module TestModule;
+
+    export {
+        option testaddr = 127.0.0.1;
+    }
 
     # Note: the data type of 2nd parameter and return type must match
     function change_addr(ID: string, new_value: addr): addr
@@ -125,24 +175,26 @@ accordingly):
 
     event bro_init()
         {
-        Option::set_change_handler("testaddr", change_addr);
+        Option::set_change_handler("TestModule::testaddr", change_addr);
         }
 
-Each time the specified option value is changed, the change handler
-function will be called before the change is performed.  The value returned
-by the change handler is the value finally assigned to the option. This
-allows, for example, checking of values to reject invalid input (the original
-value can be returned to reject the change).
-
-A change handler can optionally have a third argument, which is the location
-string (this is normally the pathname of the configuration file that triggered
+Immediately before the specified option value is changed, the change handler
+function will be called.  The value returned by the change handler is the
+value finally assigned to the option. This allows, for example, checking of
+values to reject invalid input (the original value can be returned to reject
 the change).
 
-It is also possible to chain together multiple change handlers.  In this
-case, the value returned by the first change handler is the "new value" seen
-by the next change handler, and so on.  The built-in function
-:bro:see:`Option::set_change_handler` takes an optional third argument
-that can specify a priority for the handlers.
+It is possible define multiple change handlers for a single option.  In
+this case, the change handlers are chained together: the value returned by the
+first change handler is the "new value" seen by the next change handler, and
+so on.  The built-in function :bro:see:`Option::set_change_handler` takes an
+optional third argument that can specify a priority for the handlers.
+
+A change handler function can optionally have a third argument of type
+string.  When a config file triggers a change, then the third argument is
+the pathname of the config file.  When the Config::set_value function triggers
+a change, then the third argument of the change handler is the value passed
+to the optional third argument of the Config::set_value function.
 
 Note that change handlers are also used internally by the
 configuration framework. If you look at the script level source code of