Documentation for the Korrel8r Project icon

Overview

Observing a cluster involves different types of signal; for examples logs, metrics, traces, cluster events, cluster alerts, network events and so on. This usually means using several Observability tools, each with their own data model, storage technology, and query language. These differences make it difficult to recognize relationships between different signals and resources in a cluster.

Korrel8r is a correlation engine for observable signals and Kubernetes resources. Each distinct data set, with its own labelling, storage, and schema, is known as a domain. Examples of domains include:

  • Kubernetes Resources, managed by the Kubernetes API server.

  • Metrics stored in Prometheus.

  • Logs stored in Loki in a known format.

New domains can be added to deal with new signal types, for example Network Observability Events.

Korrel8r uses an extendable set of rules to define relationships between data items. Rules can define relationships within a domain, or between different domains. For example, rules can define

  • How to find logs or metrics related to Kubernetes resources and vice-versa.

  • How to find Kubernetes resources related to network events.

  • How to find logs, metrics or resources related to Prometheus Alerts

Given a start object (resource or signal), Korrel8r searches its rule graph to find a network of related objects. There are two types of search:

Goal search

given a goal data type (log, metric, Pod etc.), find all paths from the start to objects of the goal type.

Neighbourhood search

given a numeric depth find all objects reachable from start by following depth rules or less.

Architecture

About domains

A Korrel8r domain represents a family of related objects with a common vocabulary and related storage and query technologies. Domains are descrievd in the domain reference documentation.

Each domain defines its own class, object, query, and store. These abstractions allow Korrel8r to treat different domains in the same way.

Class

A subset of objects in a domain with a common schema for serialization. Some domains have many classes. Domain k8s has a class for each resource kind. For example k8s:Pod and k8s:DaemonSet. Other domains have only a single class, for example Domain alert

Store

A source of stored objects from a single domain.

For example:

Object

The data associated with an individual signal or resource instance. For example a log record, or a serialized Kubernetes resource.

Query

A query selects a set of objects from a store. The structure of a query depends on the domain.

About rules

Rules express relationships between classes, possibly in different domains.

A Rule applies to an object of a start class, and generates a query for a goal class. The start and goal can be in different domains (e.g. k8s:Pod → log:application)

A rule definition contains a template that uses the vocabulary of the start domain and generates a query in the vocabulary of the goal domain. Rules are bridge between domains different vocabularies, schema, labels and query languages.

Getting Started

The korrel8r command

See korrel8r help for more details.

The korrel8r command implements the correlation engine. It loads correlation rules and connects to data stores based on a configuration file

Install the korrel8r engine
go install github.com/korrel8r/korrel8r/cmd/korrel8r@latest

Authentication and Authorization

Korrel8r queries observability signal stores with sensitive data. It requires a Bearer Token to ensure that only authorized data is returned.

On the command line, korrel8r uses the token associated with your kubectl login. When run as a REST server, it uses the bearer token from the incoming request to impersonate the client.

This means you need to be logged on as an admin cluster user to run korrel8r locally, and you need to use an appropriate bearer token with clients like curl or korrel8rcli.

If you are logged in to an Red Hat OpenShift Container Platform cluster, you can get your bearer token with

oc whoami -t

Configuration

There are some example configuration files to get started. You can download them locally or use them directly by URL.

openshift-route.yaml

Run korrel8r outside the cluster, connect to stores via routes:

openshift-svc.yaml

Used to run korrel8r as a clutser service, connect to stores via service URLs. See Korrel8r User Guide for more about running as a cluster service.

Command-line Examples

List the known korrel8r domains.
korrel8r --config https://raw.githubusercontent.com/korrel8r/korrel8r/main/etc/korrel8r/openshift-route.yaml list
List korrel8r classes in the k8s domain.
korrel8r --config https://raw.githubusercontent.com/korrel8r/korrel8r/main/etc/korrel8r/openshift-route.yaml list k8s

Run as a REST service

Run the korrel8r engine outside the cluster, with REST API at localhost:8080.
korrel8r --config https://raw.githubusercontent.com/korrel8r/korrel8r/main/etc/korrel8r/openshift-route.yaml web --http=localhost:8080

This runs the engine as a REST service that can be queried by other services. For example, the Red Hat OpenShift Container Platform web console uses Korrel8r to implement "related data" links and a troubleshooting panel.

You can also use the korrel8rcli command to query a korrel8r engine from the command line.

See Authentication and Authorization for details about access control.

Korrel8r can also run as a cluster service

Deploying in a cluster

Deploy resources from github

You can deploy korrel8r deployment and service in namespace korrel8r like this:

Deploy the latest from github.
oc apply -k github.com/korrel8r/korrel8r/config/?version=main

Replace version=main with version=vX.Y.Z to deploy a specific version.

On an Red Hat OpenShift Container Platform cluster you can create a route to the korrel8r service like this:

Deploy a route to the korrel8r service
oc apply -k github.com/korrel8r/korrel8r/config/route?version=main

You can use the deployment files as a guide to create your own deployment resources.

See Authentication and Authorization for details about access control.

OperatorHub community operator

There is a community operator for Korrel8r on OperatorHub. It has been tested with Red Hat OpenShift Container Platform, it may or may not work with other k8s clusters.

Prerequisites

  • You have administrator permissions on an Red Hat OpenShift Container Platform cluster version 4.12 or newer.

  • You have installed the Red Hat OpenShift Logging Operator version 5.8.1 or newer provided by Red Hat.

  • You have installed the Loki Operator version 5.8.1 or newer provided by Red Hat, and have created a LokiStack custom resource (CR).

  • You have created a ClusterLogging CR that configures LokiStack as the log store.

Procedure

  1. In the Red Hat OpenShift Container Platform web console go to Home > Console > Operators > OperatorHub

  2. Type "korrel8r" in the filter box and select the Korrel8r Community tile
    Korrel8r Community Tile

  3. Click Install, Install again.

  4. You will be asked to create a Korrel8r instance - accept the default settings.

  5. Korrel8r is now running in your cluster!

Using Korrel8r via the Openshift Console

The Red Hat OpenShift Container Platform web console has links that are powered by korrel8r.

  1. Navigate to Observe → Logs. Click the Metrics link to see correlated metrics of each log record.

  2. Navigate to Observe → Alerting, and click an alert. Alerts that have related logs have a see related logs link.

Not all alerts have a see related logs link. Only alerts that are related to container workloads can have related logs. For example, KubePodCrashLooping is related to a pod and so can have related logs. See the There is no "see related logs" link documentation for more information.

The korrel8rcli command

See korrel8cli help for more details.

korrel8rcli is a client that can make requests on a korrel8r server and visualize the results. It is intended for development and experimentation. In production use korrel8r normally functions as a back-end for a console or other visualization or analysis tool.

Install the korrel8rcli client
go install github.com/korrel8r/korrel8r/client/cmd/korrel8rcli@latest

Examples

Every korrel8rcli command requires a URL for the remote Korrel8r engine.

If you run the engine on your local host as described here, the URL is http://localhost:8080.

If you run the engine as a cluster service, use the external URL for the service. For example if you deploy korrel8r with a route as described here then the URL would be:

https://$(oc get -n korrel8r route/korrel8r -o template='{{.spec.host}}')
Domains
Get the list of known domains with store status for each one.
korrel8rcli domains {korrel8rcli-url}
Example output
- name: alert
  stores:
  - alertmanager: https://alertmanager-main.openshift-monitoring.svc:9094
    certificateAuthority: ./run/secrets/kubernetes.io/serviceaccount/service-ca.crt
    domain: alert
    metrics: https://thanos-querier.openshift-monitoring.svc:9091
- name: k8s
  stores:
  - domain: k8s
- name: log
  stores:
  - certificateAuthority: ./run/secrets/kubernetes.io/serviceaccount/service-ca.crt
    domain: log
    lokiStack: https://logging-loki-gateway-http.openshift-logging.svc:8080
- name: metric
  stores:
  - certificateAuthority: ./run/secrets/kubernetes.io/serviceaccount/service-ca.crt
    domain: metric
    metric: https://thanos-querier.openshift-monitoring.svc:9091
- name: mock
  stores: null
- name: netflow
  stores:
  - certificateAuthority: ./run/secrets/kubernetes.io/serviceaccount/service-ca.crt
    domain: netflow
    lokiStack: https://loki-gateway-http.netobserv.svc:8080
Neighbours
Get the neighbours graph of the korreler deployment
korrel8rcli -o json neighbours {korrel8rcli-url} --query 'k8s:Deployment:{namespace: korrel8r}'
Output is a JSON-encoded graph, not intended for human consumption.
{"edges":[{"goal":"k8s:Pod.v1.","start":"k8s:Deployment.v1.apps"},...
 "nodes":[{"class":"log:application","count":15,"queries":[...
Web server
This tool is experimental and may change dramatically or be dropped in future.

korrel8rcli can run as a web server that shows interactive query results as a graph in a browser.

Run a web server to vizualize results.
korrel8rcli web http://korrel8r.example --addr :9090

Open localhost:9090 in a web browser. You can type a korrel8r query in the Start box, and a numeric depth (for neighbours) or korrel8r class name (for goal search). You should see a graph of the correlations, like this.

Screenshot
Figure 1. Example graph.

Configuration

Korrel8r loads configuration from the file specified by the --config flag:

$ korrel8r --config <path_to_config_file>

The default configuration can be loaded from the internet:

korrel8r -c https://raw.githubusercontent.com/korrel8r/korrel8r/main/etc/korrel8r/korrel8r.yaml get

You can examine the default configuration files at the Korrel8r project provides some example configuration files

The configuration file is a YAML file with the following sections:

Include paths to other configuratoin files to include in this one.
include:
  - "path_to_config_file"
Stores defines how to connect to data stores. See [_stores].
stores:
  - domain: "domain_name" (1)
    # Domain-specific fields (2)
1 Domain name of the store (required).
2 Domain-specific fields for connection parameters. See Domain Reference.
Rules defines rules to relate different classes of data. See [_rules].
rules:
  - name: "rule_name" (1)
    start: (2)
      domain: "domain_name"
      classes:
        - "class_name"
    goal: (3)
      domain: "domain_name"
      classes:
        - "class_name"
    result:
      query: "query_template" (4)
1 Name identifies the rule in graphs and for debugging.
2 Start objects for this rule must belong to one of the classes in the domain.
3 Goal queries generated by this rule may must retrieve one of the classes in the domain.
4 Result queries are generated by executing the query template with the start object as context.
Aliases is a list of short-hand alias names for groups of rules.
aliases:
  - name: "alias_name" (1)
    domain: "domain_name" (2)
    classes: (3)
      - "class_name"
1 Alias name can be used as a class name in rule definitions.
2 Domain for classes in this alias.
3 Classes belonging to this alias.

Connecting to Stores

Every entry in the stores section has a domain field to identify the domain. Other fields depend on the domain, see Domain Reference. Store fields can contain plain URL strings or templates that expand to URLs.

Example: configuring a store URL from an Openshift Route resource.
stores:
  - domain: log
    lokiStack: <-
      {{$r := get "k8s:Route:{namespace: openshift-logging, name: logging-loki}" -}} (1)
      https://{{ (first $r).Spec.Host -}} (2)
1 Get a list of routes in "openshift-logging" named "logging-loki".
2 Use the .Spec.Host field of the first route as the host for the store URL.

Writing Rules

Korrel8r comes with a good set of default rules, but you can modify them or add your own.

A rule has the following key elements:

  • A set of start classes. The rule can apply to objects belonging to one of these classes.

  • A set of goal classes. The rule can generate queries for any of these classes.

  • A Go template to generate a goal query from a start object. See the section on templates below for more about writing rule templates.

The query template should generate a string of the form:

<domain-name>:<class-name>:<query-details>

The query-details part depends on the domain, see Domain Reference

Example of rules and aliases.
TODO: Examples

Templates

Some fields in the stores and rules section are Go templates. Go template syntax is also used by the Kubernetes kubectl tool when run with the --output=template option. Korrel8r provides additional template functions to simplify writing rule and configuration templates.

Additional template functions

  • The sprig library of general purpose template functions is always available.

  • Some domains provide additional functions, see Domain Reference.

The following functions are always available:

query

Executes its argument as a korrel8r query, returns []any. May return an error.

Reference

Domains

Reference details for the for the classes, objects, queries and stores of each available domain.

Domain alert

Domain alert provides Prometheus alerts, queries and access to Thanos and AlertManager stores.

Class

There is a single class alert:alert.

Object

An alert object is represented by this Go type. Rules starting from an alert should use the capitalized Go field names rather than the lowercase JSON names.

type Object struct {
	// Common fields.
	Labels		map[string]string	`json:"labels"`
	Annotations	map[string]string	`json:"annotations"`
	Status		string			`json:"status"`	// inactive|pending|firing|suppressed
	StartsAt	time.Time		`json:"startsAt"`

	// Prometheus fields.
	Value		string	`json:"value"`
	Expression	string	`json:"expression"`
	Fingerprint	string	`json:"fingerprint"`

	// Alertmanager fields.
	EndsAt		time.Time	`json:"endsAt"`
	UpdatedAt	time.Time	`json:"updatedAt"`
	Receivers	[]Receiver	`json:"receivers"`
	InhibitedBy	[]string	`json:"inhibitedBy"`
	SilencedBy	[]string	`json:"silencedBy"`
	GeneratorURL	string		`json:"generatorURL"`
}

Query

A JSON map of string names to string values, matched against alert labels, for example:

alert:alert:{"alertname":"KubeStatefulSetReplicasMismatch","container":"kube-rbac-proxy-main","namespace":"openshift-logging"}

Store

A client of Prometheus and/or AlertManager. Store configuration:

domain: alert
metrics: PROMETHEUS_URL
alertmanager: ALERTMANAGER_URL

Either or both of metrics or alertmanager may be present.

Domain k8s

Domain k8s implements Kubernetes resources stored in a Kube API server.

Class

A k8s class corresponds to a kind of Kubernetes resource, the class name is KIND.VERSION.GROUP VERSION and/or GROUP can be omitted if there is no ambiguity. Example class names: k8s:Pod.v1, ks8:Pod, k8s:Deployment.v1.apps, k8s:Deployment.apps, k8s:Deployment

Object

Objects are represented by the standard Go types used by k8s.io/client-go/api, and by Kube-generated CRD struct types. Rules starting from the k8s domain should use the capitalized Go field names rather than the lowercase JSON field names.

Query

Queries are the JSON-serialized form of this struct:

type Query struct {
	// Namespace restricts the search to a namespace.
	Namespace	string	`json:"namespace,omitempty"`
	Name		string	`json:"name,omitempty"`
	// Labels restricts the search to objects with matching label values (optional)
	Labels	client.MatchingLabels	`json:"labels,omitempty"`
	// Fields restricts the search to objects with matching field values (optional)
	Fields	client.MatchingFields	`json:"fields,omitempty"`
	// contains filtered or unexported fields
}

For example:

k8s:Pod.v1.:{"namespace":"openshift-cluster-version","name":"cluster-version-operator-8d86bcb65-btlgn"}

Store

k8s stores connects to the current logged-in Kubernetes cluster, no other configuration is needed than:

domain: k8s

Copyright: This file is part of korrel8r, released under https://github.com/korrel8r/korrel8r/blob/main/LICENSE

Template Functions

k8sClass
    Takes string arguments (apiVersion, kind).
    Returns the korrel8r.Class implied by the arguments, or an error.

Domain log

Domain log is a domain for openshift-logging ViaQ logs stored in Loki or LokiStack.

Class

There are 3 classes corresponding to the 3 openshift logging log types:

log:application
log:infrastructure
log:audit

Object

A log object is a JSON map\[string]any in ViaQ format.

Query

A query is a LogQL query string, prefixed by the logging class, for example:

log:infrastructure:{ kubernetes_namespace_name="openshift-cluster-version", kubernetes_pod_name=~".*-operator-.*" }

Store

To connect to a lokiStack store use this configuration:

domain: log
lokistack: URL_OF_LOKISTACK_PROXY

To connect to plain loki store use:

domain: log
loki: URL_OF_LOKI

Copyright: This file is part of korrel8r, released under https://github.com/korrel8r/korrel8r/blob/main/LICENSE

Template Functions

logTypeForNamespace
    Takes a namespace string argument.
    Returns the log type ("application" or "infrastructure") of a container in the namespace.

logSafeLabel
    Convert the string argument into a  safe label containing only alphanumerics '_' and ':'.

Domain metric

Domain metric represents Prometheus metric time-series as objects.

Class

There is only one class: metric:metric

Object

A Metric is a time series identified by a label set. Korrel8r does not consider load the sample data for a time series, or use it in rules. If a korrel8r search time constraints, then metrics that have no values that meet the constraint are ignored.

Query

Query data is a PromQL instant vector selector, for example:

metric:metric:http_requests_total{environment=~"staging|testing",method!="GET"}

Store

Prometheus is the store, store configuration:

domain: metric
metric: URL_OF_PROMETHEUS

Domain netflow

Domain netflow is a domain for network observability flow events stored in Loki or LokiStack.

Class

There is a single class netflow:network

Object

A log object is a JSON map\[string]any in NetFlow format.

Query

A query is a LogQL query string, prefixed by netflow:network:, for example:

netflow:network:{SrcK8S_Type="Pod", SrcK8S_Namespace="myNamespace"}

Store

To connect to a netflow lokiStack store use this configuration:

domain: netflow
lokistack: URL_OF_LOKISTACK_PROXY

To connect to plain loki store use:

domain: netflow
loki: URL_OF_LOKI

Domain trace

Domain trace is a domain for network observability flow events stored in Tempo or TempoStack.

Class

There is a single class trace:trace

Object

A trace object is a JSON map\[string]any in \[NetFlow] format.

Query

A query is a TraceQL query string, prefixed by trace:trace:, for example:

trace:trace:{resource.k8s.namespace.name="tracing-app-k6"}

Store

To connect to a tempoStack store use this configuration:

domain: trace
tempostack: URL_OF_TEMPOSTACK_PROXY

To connect to plain tempo store use:

domain: trace
tempo: URL_OF_TEMPO

REST API

REST API for the Korrel8r correlation engine.

Version

v1alpha1

License

Apache 2.0

Contact

Project Korrel8r https://github.com/korrel8r/korrel8r

Content negotiation

URI Schemes

  • http

  • https

Consumes

  • application/json

Produces

  • application/json

Endpoints by group

operations

Method URI Name Summary

GET

/api/v1alpha1/domains

get domains

Get name, configuration and status for each domain.

GET

/api/v1alpha1/domains/{domain}/classes

get domains domain classes

Get class names and descriptions for a domain.

GET

/api/v1alpha1/objects

get objects

Execute a query, returns a list of JSON objects.

POST

/api/v1alpha1/graphs/goals

post graphs goals

Create a correlation graph from start objects to goal queries.

POST

/api/v1alpha1/graphs/neighbours

post graphs neighbours

Create a neighbourhood graph around a start object to a given depth.

POST

/api/v1alpha1/lists/goals

post lists goals

Create a list of goal nodes related to a starting point.

Paths

Get name, configuration and status for each domain.

GET /api/v1alpha1/domains
All responses
Code Status Description Has headers Schema

200

OK

OK

schema

default

schema
Responses
200 - OK

Status: OK

Schema

[]Domain

Default Response
Schema

any

Get class names and descriptions for a domain.

GET /api/v1alpha1/domains/{domain}/classes
Parameters
Name Source Type Go type Separator Required Default Description

domain

path

string

string

required

Domain name
All responses
Code Status Description Has headers Schema

200

OK

OK

schema

default

schema
Responses
200 - OK

Status: OK

Schema

Classes

Default Response
Schema

any

Execute a query, returns a list of JSON objects.

GET /api/v1alpha1/objects
Parameters
Name Source Type Go type Separator Required Default Description

query

query

string

string

required

query string
All responses
Code Status Description Has headers Schema

200

OK

OK

schema

default

schema
Responses
200 - OK

Status: OK

Schema

[]interface{}

Default Response
Schema

any

Create a correlation graph from start objects to goal queries.

POST /api/v1alpha1/graphs/goals
Parameters
Name Source Type Go type Separator Required Default Description

rules

query

boolean

bool

optional

include rules in graph edges

request

body

Goals

models.Goals

✓

search from start to goal classes
All responses
Code Status Description Has headers Schema

200

OK

OK

schema

default

schema
Responses
200 - OK

Status: OK

Schema

Graph

Default Response
Schema

any

Create a neighbourhood graph around a start object to a given depth.

POST /api/v1alpha1/graphs/neighbours
Parameters
Name Source Type Go type Separator Required Default Description

rules

query

boolean

bool

optional

include rules in graph edges

request

body

Neighbours

models.Neighbours

✓

search from neighbours
All responses
Code Status Description Has headers Schema

200

OK

OK

schema

default

schema
Responses
200 - OK

Status: OK

Schema

Graph

Default Response
Schema

any

Create a list of goal nodes related to a starting point.

POST /api/v1alpha1/lists/goals
Parameters
Name Source Type Go type Separator Required Default Description

request

body

Goals

models.Goals

✓

search from start to goal classes
All responses
Code Status Description Has headers Schema

200

OK

OK

schema

default

schema
Responses
200 - OK

Status: OK

Schema

[]Node

Default Response
Schema

any

Models

Classes

Classes is a map from class names to a short description.

Classes

Constraint

Constraint constrains the objects that will be included in search results.

Properties

Name Type Go type Required Default Description Example

end

date-time (formatted string)

strfmt.DateTime

End of time interval, quoted RFC 3339 format.

limit

integer

int64

Limit number of objects returned per query, ⇐0 means no limit.

start

date-time (formatted string)

strfmt.DateTime

Start of time interval, quoted RFC 3339 format.

timeout

string

string

Timeout per request, h/m/s/ms/ns format

Domain

Domain configuration information.

Properties

Name Type Go type Required Default Description Example

name

string

string

Name of the domain.

stores

[]Store

[]Store

Stores configured for the domain.

Edge

Directed edge in the result graph, from Start to Goal classes.

Properties

Name Type Go type Required Default Description Example

goal

string

string

Goal is the class name of the goal node.

domain:class

rules

[]Rule

[]*Rule

Rules is the set of rules followed along this edge.

start

string

string

Start is the class name of the start node.

Goals

Starting point for a goals search.

Properties

Name Type Go type Required Default Description Example

goals

[]string

[]string

Goal classes for correlation.

["domain:class"]

start

Start

Start

Graph

Graph resulting from a correlation search.

Properties

Name Type Go type Required Default Description Example

edges

[]Edge

[]*Edge

nodes

[]Node

[]*Node

Neighbours

Starting point for a neighbours search.

Properties

Name Type Go type Required Default Description Example

depth

integer

int64

Max depth of neighbours graph.

start

Start

Start

Node

Node in the result graph, contains results for a single class.

Properties

Name Type Go type Required Default Description Example

class

string

string

Class is the full class name in "DOMAIN:CLASS" form.

domain:class

count

integer

int64

Count of results found for this class, after de-duplication.

queries

[]QueryCount

[]*QueryCount

Queries yielding results for this class.

QueryCount

Query run during a correlation with a count of results found.

Properties

Name Type Go type Required Default Description Example

count

integer

int64

Count of results or -1 if the query was not executed.

query

string

string

Query for correlation data.

Rule

Rule is a correlation rule with a list of queries and results counts found during navigation.

Properties

Name Type Go type Required Default Description Example

name

string

string

Name is an optional descriptive name.

queries

[]QueryCount

[]*QueryCount

Queries generated while following this rule.

Start

Start identifies a set of starting objects for correlation.

Properties

Name Type Go type Required Default Description Example

class

string

string

Class for objects

constraint

Constraint

Constraint

objects

interface{}

interface{}

Objects of class serialized as JSON

queries

[]string

[]string

Queries for starting objects

Store

Store is a map of name:value attributes used to connect to a store.

Store

Troubleshooting

Installation errors on Red Hat OpenShift Container Platform 4.15

Red Hat OpenShift Container Platform 4.15 enforces additional security restrictions that did were not included in Red Hat OpenShift Container Platform 4.14 and earlier versions. Installing the Operator bundle on Red Hat OpenShift Container Platform 4.15 causes security policy errors.

You can use the following workarounds:

  • Deploy Korrel8r in the default namespace instead of creating a new korrel8r namespace, by running the following command:

    $ operator-sdk -n default run bundle quay.io/korrel8r/operator-bundle:latest

  • Apply labels to the korrel8r namespace before installing, by running the following commands:

    $ kubectl label ns/korrel8r pod-security.kubernetes.io/enforce=privileged --overwrite
    $ kubectl label ns/korrel8r pod-security.kubernetes.io/warn=privileged --overwrite

See also https://issues.redhat.com/browse/OU-304.

The see related logs link does not appear unless the following criteria is met:

  1. The alert is related to a container workload.

  2. The workload has generated logs.

  3. The logs have been collected by the logging subsystem for Red Hat OpenShift.

For example, the UpdateAvailable alert indicates an update is available for the entire cluster, it is not related to any specific workload.

You can force the creation of an alert with see related logs by using the following procedure.

Procedure

  1. Run the following command to create a broken deployment in a system namespace:

    kubectl apply -f - << EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: bad-deployment
  namespace: default (1)
spec:
  selector:
    matchLabels:
      app: bad-deployment
  template:
    metadata:
      labels:
        app: bad-deployment
    spec:
      containers: (2)
      - name: bad-deployment
      	image: quay.io/openshift-logging/vector:5.8
    1 The deployment must be in a system namespace (such as default) to cause the desired alerts.
    2 This container deliberately tries to start a vector server with no configuration file. The server will log a few messages, and then exit with an error. Any container could be used for this.

  2. View the alerts:

    1. Go to Observe → Alerting and click clear all filters. View the Pending alerts.

      Alerts first appear in the Pending state. They do not start Firing until the container has been crashing for some time. By showing Pending alerts you can see them much more quickly.

    2. Look for KubeContainerWaiting alerts, KubePodCrashLooping alerts, or KubePodNotReady alerts. These alerts have a show related logs link.

There are no logs for the logging subsystem for Red Hat OpenShift

The log collector does not collect its own logs. Doing so might create a circular condition where collecting its own logs causes the collector to log something, which it then collects, which causes it to log something, and so on in an endless cycle.

To avoid this risk, the log collector does not collect any logs from pods that are part of the logging system.

You can still see view logging subsystem for Red Hat OpenShift logs by using the kubectl logs command directly.