Kafka Data Collector
Introduction
The Kafka Data Collector O11ySource reads from the configured Kafka cluster and ingests the selected topics into vuSmartMaps. It is intended for centralized collection of Kafka-based telemetry and enables downstream analysis in the platform.
Getting Started
Compatibility
vuSmartMaps supports the collecting data from the configured topic from a kafka cluster
Data Collection Method
vuSmartMaps collects Kafka Data Collector metrics through a data collector deployed within the vuSmartMaps platform, which uses kafka_consumer plugin to retrieve performance and health metrics.
Prerequisites
Inputs for Configuring Data Source
- Name: Unique identifier for the source
- Client Kafka Bootstrap Server: Enter the client kafka bootstrap server. Separate multiple broker by comma.
- Client Kafka Topics: Enter the topics to be consumed. Use comma separated for multiple topics.
- Destination Kafka Topic: Enter the destination kafka topic
Applications
Firewall Requirement
To collect data from this O11ySource, ensure the following ports are opened:
| Source IP | Destination IP | Destination Port | Protocol | Direction |
|---|---|---|---|---|
| vuSmartMaps Collection/Ingress node IP | IP address or FQDN of the Kafka Data Collector service | 9092*/9093*/9094*/9096* | TCP | Outbound |
*Before providing the firewall requirements, please update the port based on the customer environment.
Configuring the Target
To verify that your Kafka broker is reachable from your vuSmartMaps cluster, run the following command. Replace <broker_ip> with the actual IP address of your Kafka broker.
nc -vz <broker_ip> 9092
If the connection is successful, you should see an output similar to:
Connection to <broker_ip> 9092 port [tcp/*] succeeded!
Configuration Steps
Metrics Collected
The metrics are based on the input data collected.
