Skip to content

Kafka Connect Watcher Kafka Connect Watcher

johnpreston/kafka-connect-watcher

Kafka Connect Watcher ¶

BUILD

Actively monitor your Kafka connectors ¶

Service that will actively probe and monitor your Kafka connect clusters using the Connect API. It can report metrics to AWS CloudWatch (Prometheus coming) using AWS EMF to allow creating alerts and alarms.

Features ¶

Scan multiple clusters at once
Implement different remediation rules
Include/Exclude lists for connectors to evaluate/ignore

Run / Execute ¶

With docker ¶

             docker run --rm -it --network host -v ${PWD}/config.yaml:/config.yaml public.ecr.aws/compose-x/kafka-connect-watcher -c /config.yaml

            

With python ¶

             python -m venv watcher
source watcher/bin/activate
pip install pip -U; pip install kafka-connect-watcher
kafka-connect-watcher -c config.yaml

            

How does it work? ¶

Provided a configuration file, the watcher will use the Apache Kafka Connect API to query the connectors and evaluate their status. Based on evaluation rules defined, you get metrics and reporting on the different connectors. You can also define actions to take in order to help try to recover the connectors, such as restart these.

Install ¶

            # Inside a python virtual environment
python3 -m venv venv
source venv/bin/activate
pip install pip -U
pip install kafka-connect-watcher

# For your user only, without virtualenv
pip install kafka-connect-watcher --user

           

Examples and Help

Examples

Modules and Source Code

kafka_connect_watcher

Indices and tables ¶