AWS Embedded Metrics Format (EMF) Collection ¶

If you are running on AWS, you might want to publish the collected metrics to AWS CloudWatch. To achieve that, the kafka connect watcher uses AWS EMF which will publish logs captured by AWS CloudWatch, and transformed into metrics automatically.

There are two places where you can configure EMF metrics settings:

aws_emf at the root of the configuration file
cluster.metrics.aws_emf , at the connector level

aws_emf ¶

           log_group_name: str
service_name: str
service_type: str
watcher_config:
  enabled: bool
  namespace: str
  dimensions:
    str: str

          

log_group_name ¶

Importance: HIGH

The log group to which the EMF metrics will be published. We recommend to have a 1 day retention on the log group, as metrics will be persisted after ingestion, therefore you don’t need the additional costs incurred by logs.

service_name ¶

Importance: LOW Optional setting - used in the metadata of EMF Metric log

service_type ¶

Importance: LOW Optional setting - used in the metadata of EMF Metric log

watcher_config ¶

Configuration that drives the general behaviour of aws_emf

enabled ¶

Importance: HIGH Default: false

When true, EMF metrics for the watcher (clusters count etc.) are collected and published

namespace ¶

Importance: HIGH Default: KafkaConnect/Watcher

Allows you to define the Namespace to which the custom metrics are published.

dimensions ¶

Importance: MEDIUM Optional setting

Arbitrary key/value set (unique keys) that will used as dimensions of the published metrics. This settings helps with distinguishing metrics if you have multiple instances running.

cluster.metrics.aws_emf ¶

Settings that drive the EMF metrics collection for a cluster connectors (healthy, failed, unassigned, ignored).

The parameters namespace , dimensions and enabled are identical to the ones for the watcher settings.

Attention

The top level aws_emf settings do not override the cluster level settings.