Keyboard shortcuts

Press or to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Monitoring

Node Readiness Controller exposes Prometheus-compatible metrics. This page describes the Prometheus metrics exposed by Node Readiness Controller for monitoring rule evaluation, taint operations, failures, and bootstrap progress.

Metrics Endpoint

The controller serves metrics on /metrics only when metrics are explicitly enabled. Depending on the installation, the endpoint is served either over HTTP or over HTTPS. See Installation for deployment details.

Supported Metrics

node_readiness_rules_total

Number of NodeReadinessRule objects tracked by the controller.

PropertyValue
Typegauge
Labelsnone
Recorded whenThe controller refreshes or removes a tracked rule

node_readiness_taint_operations_total

Total number of taint operations performed by the controller.

PropertyValue
Typecounter
Labelsrule, operation
Recorded whenThe controller successfully adds or removes a taint

Labels

LabelDescriptionValues
ruleNodeReadinessRule nameAny rule name
operationTaint operation performed by the controlleradd, remove

node_readiness_evaluation_duration_seconds

Duration of rule evaluations per rule.

PropertyValue
Typehistogram
Labelsrule
BucketsPrometheus default histogram buckets
Recorded whenThe controller evaluates a rule against a node

Labels

LabelDescriptionValues
ruleNodeReadinessRule nameAny rule name

node_readiness_failures_total

Total number of failure events recorded by the controller.

PropertyValue
Typecounter
Labelsrule, reason
Recorded whenThe controller records an evaluation failure or taint add/remove failure

Labels

LabelDescriptionValues
ruleNodeReadinessRule nameAny rule name
reasonFailure label recorded by the controllerEvaluationError, AddTaintError, RemoveTaintError

node_readiness_build_info

Available starting from the v0.6.0 release.

Build information for the node-readiness-controller binary.

PropertyValue
Typegauge
Labelsversion
Recorded whenThe controller starts up

Labels

LabelDescriptionValues
versionBuild version of the running binaryAny version string, or unknown if not set at build time

node_readiness_rule_nodes

Available starting from the v0.6.0 release.

Number of nodes currently held or released by each NodeReadinessRule, collected at scrape time from the controller cache.

PropertyValue
Typegauge
Labelsrule, state
Recorded whenComputed on each Prometheus scrape from the cached node list

Labels

LabelDescriptionValues
ruleNodeReadinessRule nameAny non-dry-run rule name with a valid selector
stateWhether matching nodes are still tainted by the rule or have had the taint removedheld, released

node_readiness_bootstrap_completed_total

Total number of nodes that have completed bootstrap.

PropertyValue
Typecounter
Labelsrule
Recorded whenThe controller marks bootstrap as completed for a node under a bootstrap-only rule

Labels

LabelDescriptionValues
ruleNodeReadinessRule nameAny rule name

node_readiness_bootstrap_duration_seconds

Time from node creation to bootstrap completion (taint removal) for bootstrap-only rules.

PropertyValue
Typehistogram
Labelsrule
Buckets1, 5, 10, 30, 60, 120, 300, 600, 1200 seconds
Recorded whenThe controller removes the bootstrap taint from a node under a bootstrap-only rule

Labels

LabelDescriptionValues
ruleNodeReadinessRule nameAny rule name

node_readiness_reconciliation_latency_seconds

End-to-end latency from a node condition change to the taint operation that responds to it.

PropertyValue
Typehistogram
Labelsrule, operation
Buckets0.01, 0.05, 0.1, 0.5, 1, 2, 5, 10, 30, 60, 120, 300 seconds
Recorded whenThe controller adds or removes a taint after a node condition change

Labels

LabelDescriptionValues
ruleNodeReadinessRule nameAny rule name
operationTaint operation that closed the reconciliationadd_taint, remove_taint

node_readiness_nodes_by_state

Number of nodes in each readiness state per rule.

PropertyValue
Typegauge
Labelsrule, state
Recorded whenThe controller syncs node state, including on each rule reconciliation

Labels

LabelDescriptionValues
ruleNodeReadinessRule nameAny rule name
stateReadiness state of the counted nodesready, not_ready, bootstrapping

node_readiness_condition_failures_total

Total number of failed condition evaluations, broken down by the condition that failed. Use this to find which check within a rule is blocking node readiness.

PropertyValue
Typecounter
Labelsrule, condition
Recorded whenThe controller evaluates a rule condition against a node and the condition is not met

Labels

LabelDescriptionValues
ruleNodeReadinessRule nameAny rule name
conditionType of the condition that failedAny condition type defined on the rule

node_readiness_rule_last_reconciliation_timestamp_seconds

Unix timestamp of the last reconciliation for each rule. Use this to detect rules that have stopped reconciling.

PropertyValue
Typegauge
Labelsrule
Recorded whenThe controller reconciles the rule

Labels

LabelDescriptionValues
ruleNodeReadinessRule nameAny rule name

Reporter Metrics

The readiness-condition-reporter serves its own Prometheus metrics on /metrics, on the address configured by METRICS_BIND_ADDRESS. See Reporter Configuration for deployment details.

node_readiness_reporter_build_info

Reporter binary version to track fleet version skew.

PropertyValue
Typegauge
Labelsversion
Recorded whenThe reporter process starts

Labels

LabelDescriptionValues
versionReporter binary versionAny version string

node_readiness_reporter_check_duration_seconds

Duration of health probe checks.

PropertyValue
Typehistogram
Labelsnone
Buckets0.005, 0.1, 0.25, 0.5, 1, 2.5, 5, 10 seconds
Recorded whenThe reporter completes a health check request to CHECK_ENDPOINT

node_readiness_reporter_checks_total

Total probe check results over time.

PropertyValue
Typecounter
Labelsresult
Recorded whenThe reporter completes a health check and classifies the result

Labels

LabelDescriptionValues
resultOutcome of the health checkhealthy, unhealthy, error

node_readiness_reporter_condition_writes_total

Total node condition update outcomes, including writes skipped when state was unchanged.

PropertyValue
Typecounter
Labelsresult
Recorded whenThe reporter evaluates whether to write the Node condition after a health check, including when the write is skipped

Labels

LabelDescriptionValues
resultOutcome of the condition write attemptsuccess, error, skipped