Add datadog.cluster_agent.leader_election.is_leader

[keisku]

What does this PR do?

Datadog Cluster Agent exposes leader_election_is_leader.

https://github.com/DataDog/datadog-agent/blob/c7f4f4e228f546dfb60fdceb6cb6ae92a14bca88/pkg/util/kubernetes/apiserver/leaderelection/metrics/metrics.go#L24-L32

Motivation

This metric is useful for troubleshooting the issue regarding leader election.

Review checklist (to be filled by reviewers)

• Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
• Add the qa/skip-qa label if the PR doesn't need to be tested during QA.
• If you need to backport this PR to another branch, you can add the backport/<branch-name> label to the PR and it will automatically open a backport PR once this one is merged

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 87.63%. Comparing base (642b2f9) to head (c2c5bcd).

Additional details and impacted files

Flag	Coverage Δ
activemq	?
cassandra	?
datadog_cluster_agent	90.19% <ø> (ø)
hive	?
hivemq	?
hudi	?
ignite	?
jboss_wildfly	?
kafka	?
presto	?
solr	?

Flags with carried forward coverage won't be shown. Click here to find out more.

[clamoriniere]

I added a comment to suggest another approach to have a "is leader" metric easier to use.

Also it would be nice to add the metric in the OOTB Dashboard too.

[clamoriniere]

this metric wasn't designed to be exposed directly.
Instead it was added to be use as "label join", see:

integrations-core/datadog_cluster_agent/datadog_checks/datadog_cluster_agent/check.py

Lines 100 to 105 in c2c5bcd

	'label_joins': {
	'leader_election_is_leader': {
	'labels_to_match': ['*'],
	'labels_to_get': ['is_leader'],
	}
	},

IMO, if we want to make a metric that exposes the leader status we should use the metric value (1 if leader, 0 if follower) instead of the using a label.
Maybe it can be done in this check thanks to a metric transformer function that will convert the is_leader label's value to a metric value.

you can find some transformer example in other openmetrics base check like

integrations-core/kube_controller_manager/datadog_checks/kube_controller_manager/kube_controller_manager.py

Line 231 in 642b2f9

def workqueue_transformer(self, metric, scraper_config):

[keisku]

IMO, if we want to make a metric that exposes the leader status we should use the metric value (1 if leader, 0 if follower) instead of the using a label.

What do you think if I achieve that in Cluster Agent side? DataDog/datadog-agent#32511

We could keep datadog cluster check(python) simple with approach.

[keisku]

Also it would be nice to add the metric in the OOTB Dashboard too.

Will do!