mgr/cephadm: adding dynamic prometheus config based on http_sd

Signed-off-by: Redouane Kachach <[email protected]>

doc/cephadm/services/monitoring.rst

@@ -71,10 +71,10 @@ steps below:
 
      ceph orch apply prometheus
 
-   or 
+   or
 
    .. prompt:: bash #
-     
+
      ceph orch apply prometheus --placement 'count:2'
 
 #. Deploy grafana:
@@ -193,12 +193,26 @@ configuration files for monitoring services.
 
 Internally, cephadm already uses `Jinja2
 <https://jinja.palletsprojects.com/en/2.11.x/>`_ templates to generate the
-configuration files for all monitoring components. To be able to customize the
-configuration of Prometheus, Grafana or the Alertmanager it is possible to store
-a Jinja2 template for each service that will be used for configuration
-generation instead. This template will be evaluated every time a service of that
-kind is deployed or reconfigured. That way, the custom configuration is
-preserved and automatically applied on future deployments of these services.
+configuration files for all monitoring components. Starting from version x.x.x,
+cephadm uses Prometheus http service discovery support `http_sd_config
+<https://prometheus.io/docs/prometheus/2.28/configuration/configuration/#http_sd_config>`
+in order to get the currently configured targets from Ceph. Internally, `ceph-mgr`
+provides a service discovery endpoint at `<https://<mgr-ip>:8765/sd/` (port is
+configurable through the variable `service_discovery_port`) which is used by
+Prometheus to get the needed targets.
+
+Customers with external monitoring stack can use `ceph-mgr` service discovery endpoint
+to get scraping configuration. Root certificate of the server can be obtained by the
+following command:
+
+   .. prompt:: bash #
+
+     ceph orch sd dump cert
+
+The configuration of Prometheus, Grafana, or Alertmanager may be customized by storing
+a Jinja2 template for each service. This template will be evaluated every time a service
+of that kind is deployed or reconfigured. That way, the custom configuration is preserved
+and automatically applied on future deployments of these services.
 
 .. note::
 
@@ -282,6 +296,12 @@ cluster.
   By default, ceph-mgr presents prometheus metrics on port 9283 on each host
   running a ceph-mgr daemon.  Configure prometheus to scrape these.
 
+To make this integration easier, Ceph provides by means of `ceph-mgr` a service
+discovery endpoint at `<https://<mgr-ip>:8765/sd/` which can be used by an external
+Prometheus to retrieve targets information. Information reported by this EP used
+the format specified by `http_sd_config
+<https://prometheus.io/docs/prometheus/2.28/configuration/configuration/#http_sd_config>`
+
 * To enable the dashboard's prometheus-based alerting, see :ref:`dashboard-alerting`.
 
 * To enable dashboard integration with Grafana, see :ref:`dashboard-grafana`.

src/cephadm/cephadm

@@ -4403,7 +4403,7 @@ WantedBy=ceph-{fsid}.target
                                'port': self.listener_port})
             data = data.encode('ascii')
 
-            url = f'https://{self.target_ip}:{self.target_port}/data'
+            url = f'https://{self.target_ip}:{self.target_port}/data/'
             try:
                 req = Request(url, data, {'Content-Type': 'application/json'})
                 send_time = time.monotonic()
@@ -5177,7 +5177,7 @@ def create_mgr(
     # Note:the default port used by the Prometheus node exporter is opened in fw
     ctx.meta_json = json.dumps({'service_name': 'mgr'})
     deploy_daemon(ctx, fsid, 'mgr', mgr_id, mgr_c, uid, gid,
-                  config=config, keyring=mgr_keyring, ports=[9283])
+                  config=config, keyring=mgr_keyring, ports=[9283, 8765])
 
     # wait for the service to become available
     logger.info('Waiting for mgr to start...')