Kafka

Kafka is primarily used to enable real-time data streaming and processing between systems. It ensures reliable, scalable, and fault-tolerant communication in data-driven environments, making it ideal for building event-driven architectures.

data-driven environment

A is a setting where decisions and operations are guided by data and analytics. It involves collecting, centralizing, and analyzing data to gain insights and drive evidence-based decisions. Examples include:

• Healthcare: Predicting disease outbreaks using patient data.
• Finance: Detecting fraud through transaction analysis.
• Retail: Optimizing inventory based on customer trends.
• Aviation: Monitoring aircraft telemetry for real-time safety and predictive maintenance.

Functionalities of Kafka

1. Publish-Subscribe Messaging:

   • Kafka allows applications to produce messages (events) to a topic.
   • Consumers can subscribe to these topics and receive the events in real-time.

2. Scalability:

   • Kafka is horizontally scalable. Topics are divided into partitions, which can be distributed across multiple brokers (nodes) in the Kafka cluster.
   • This ensures high throughput by parallelizing reads and writes.

3. Data Durability:

   • Kafka stores messages on disk, allowing consumers to retrieve them later.
   • Messages are retained for a configurable period, even after consumption, ensuring reliability.

4. Fault Tolerance:

   • Kafka replicates partitions across multiple brokers for redundancy.
   • If a broker fails, the replicas take over seamlessly.

5. High Throughput:

   • Designed to handle millions of events per second, Kafka ensures minimal performance impact even in high-traffic systems.

6. Event Storage:

   • Kafka acts as a durable log for event storage, allowing downstream systems to replay or process historical events.

7. Real-Time Processing:

   • Kafka supports stream processing using tools like Kafka Streams or external frameworks such as Apache Flink or Apache Spark.

8. Decoupling Systems:

   • Kafka acts as a mediator, enabling independent evolution of producers and consumers.
   • Producers and consumers can operate asynchronously, decoupling their lifecycles.

9. Integration:

   • Kafka connects heterogeneous systems by acting as a central hub for data exchange.
   • With Kafka Connect, it integrates with databases, storage systems, and other services for seamless data ingestion and export.

10. Replayability:

    • Consumers can read data from a specific point (offset) in the stream, making Kafka suitable for debugging, audits, and reprocessing.

Use Cases

1. Log Aggregation:

   • Collects logs from different services and systems into a central platform for processing.

2. Real-Time Analytics:

   • Enables event-driven analytics for dashboards, fraud detection, and recommendation systems.

3. Stream Processing:

   • Transforms or aggregates event data for complex processing pipelines.

4. Event Sourcing:

   • Captures the state changes in applications as events for replaying and recovering state.

5. Data Integration:

   • Acts as a bridge for real-time data movement between databases, cloud services, and applications.

6. IoT and Monitoring:

   • Processes streams from IoT devices or monitoring systems in real-time.

Kafka Components

1. Producer: Sends messages to Kafka topics.
2. Broker: Servers that store and serve messages.
3. Topic: A category where messages are published.
4. Partition: A subset of a topic for parallelism.
5. Consumer: Reads messages from topics.
6. ZooKeeper (Deprecated in newer versions): Manages cluster metadata and leader election (replaced by Kafka’s own metadata quorum in newer versions).

1. Core Components:

   • Producers: Applications that publish events/messages to Kafka topics.
   • Brokers: Kafka servers that store and serve messages to consumers.
   • Topics: Logical channels where messages are categorized.
   • Partitions: Subsets of topics for parallel processing and scalability.
   • Consumers: Applications that read messages from Kafka topics.
   • ZooKeeper/Metadata Quorum: Manages metadata and cluster coordination (newer versions replace ZooKeeper with Kafka's native quorum).

2. Data Flow:

   • Producers send messages to topics, and these messages are distributed across partitions.
   • Brokers store the messages persistently, retaining them for a specified duration.
   • Consumers fetch these messages, either in real-time or by replaying historical data.

3. Key Features:

   • High throughput, fault tolerance, and scalability.
   • Ability to replay messages and decouple producers and consumers.
   • Seamless integration with stream processing tools for analytics and transformation.

1. Producers:

   • Web Applications: Publish user interactions, transactions, or events to Kafka topics.
   • IoT Devices: Send sensor data for real-time processing.
   • Logs and Monitoring Tools: Centralize logs from various systems.
   • Database Change Events (CDC): Capture changes in relational databases (e.g., MySQL, PostgreSQL) to propagate updates.

2. Kafka Core:

   • Topics act as logical channels for communication.
   • Partitions enable parallelism and scalability.
   • Brokers store and serve messages.
   • Replication ensures data durability and fault tolerance.

3. Consumers:

   • Real-time dashboards for analytics.
   • Stream processing for data aggregation, filtering, or transformation.
   • Machine learning pipelines for real-time model training or inference.
   • Updates to relational databases.
   • Storing data in data warehouses for business intelligence.

4. Workflow:

   • Events flow from producers to topics.
   • Kafka retains messages in partitions for a configurable period.
   • Consumers retrieve messages for processing or storage.

5. Monitoring and Integration:

   • Use tools like Prometheus and Grafana for monitoring Kafka.
   • Integrate with external systems using Kafka Connect (e.g., ElasticSearch, Hadoop).

6. Features:

   • Kafka’s high throughput, low latency, scalability, and fault tolerance make it ideal for real-time data streaming use cases.

use case in the aviation industry, such as real-time flight monitoring and data streaming

1. Producers:

   • Aircraft Sensors: Send real-time telemetry data, including speed, altitude, and system health.
   • Air Traffic Control Systems: Provide updates on flight schedules, airspace usage, and routing.
   • Airline Operations Centers: Share crew assignments, gate information, and ground operations data.
   • Weather Data Providers: Deliver live weather updates, turbulence warnings, and wind patterns.

2. Kafka Core:

   • Topics like AircraftTelemetry, FlightSchedules, and WeatherUpdates organize data streams.
   • Partitions divide topics for scalability (e.g., telemetry by aircraft ID).
   • Brokers store messages, enabling real-time streaming and fault-tolerant data storage.

3. Consumers:

   • Airline Dashboards: Display real-time flight statuses and operational insights.
   • Real-Time Alerts: Notify pilots or ground staff of critical updates (e.g., turbulence or maintenance needs).
   • Maintenance Systems: Analyze telemetry for predictive maintenance and diagnostics.
   • Passenger Notification Systems: Inform passengers of delays or gate changes.
   • Predictive Analytics: Use data to optimize flight paths, fuel efficiency, and scheduling.

4. Workflow:

   • Data from producers (sensors, systems) is streamed to Kafka topics.
   • Consumers fetch this data for processing, visualization, and decision-making.
   • Kafka enables seamless communication between aviation systems.

5. Features:

   • Kafka ensures low latency for critical updates, fault tolerance for reliability, and scalability to handle data from thousands of flights.

Project Setup

1. Spring Boot Application (A)

Central application for handling real-time aviation telemetry data.

2. Producers (B)

• Telemetry Producer: Sends telemetry data to Kafka.
• REST API Endpoint: Receives telemetry data via HTTP.
• Kafka Template: Sends data to Kafka.
• Kafka Topic: AircraftTelemetry: The channel where data is published.

3. Kafka Core (C)

• Topics: Organizes data streams.
• AircraftTelemetry: Specific topic for aircraft data.
• Brokers: Kafka servers managing data.
• Partitions: Allows parallel processing.
• Replication: Ensures fault tolerance.

4. Consumers (D)

• Telemetry Consumer: Subscribes to the Kafka topic to process data.
• Kafka Listener: Listens for incoming data.
• Process Telemetry Data: Processes the received data.
• Log to Console: Logs data for monitoring.
• Save to Database: Stores data for future use.

5. External Systems

• Client Application: Sends data via the REST API.
• Real-Time Dashboard: Displays data in real-time.
• Alerting System: Notifies about anomalies.
• Data Analytics Pipeline: Analyzes telemetry data.

6. Enhancements

• Prometheus/Grafana: Monitoring and visualization.
• Kafka Streams: Stream processing.
• Database Integration: Stores historical data.

Create a Spring Boot Project

• Spring Web
• Spring Kafka
• Spring Boot Actuator (for monitoring)
• Spring DevTools (optional for development)

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-kafka</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-web</artifactId>
</dependency>

application.properties

spring.kafka.bootstrap-servers=localhost:9092
spring.kafka.consumer.group-id=aviation-group
spring.kafka.consumer.auto-offset-reset=earliest
spring.kafka.producer.key-serializer=org.apache.kafka.common.serialization.StringSerializer
spring.kafka.producer.value-serializer=org.apache.kafka.common.serialization.StringSerializer
spring.kafka.consumer.key-deserializer=org.apache.kafka.common.serialization.StringDeserializer
spring.kafka.consumer.value-deserializer=org.apache.kafka.common.serialization.StringDeserializer

1. spring.kafka.bootstrap-servers=localhost:9092

   • This specifies the Kafka broker (or cluster of brokers) the application connects to.
   • In this case, it’s a single Kafka broker running on localhost at port 9092.

2. spring.kafka.consumer.group-id=aviation-group

   • Kafka consumers are organized into groups. Each consumer in a group is assigned a subset of partitions to read messages.
   • Here, the consumer group is identified as aviation-group.

3. spring.kafka.consumer.auto-offset-reset=earliest

   • Determines what the consumer should do if there is no initial offset or if the current offset is invalid (e.g., messages are deleted).
   • earliest: Reads messages starting from the beginning of the topic's log.

4. spring.kafka.producer.key-serializer and spring.kafka.producer.value-serializer

   • Producers need to serialize the keys and values of messages into a format Kafka can store and transmit.
   • These properties specify the serializers for keys and values:
     • org.apache.kafka.common.serialization.StringSerializer is used to convert data to String format.

5. spring.kafka.consumer.key-deserializer and spring.kafka.consumer.value-deserializer

   • Consumers need to deserialize the keys and values of messages received from Kafka.
   • These properties specify the deserializers for keys and values:
     • org.apache.kafka.common.serialization.StringDeserializer is used to convert data from Kafka's format back to String.

Step 1: Define Models

package com.example.aviation.model;

public class AircraftTelemetry {
    private String aircraftId;
    private double latitude;
    private double longitude;
    private double altitude;
    private double speed;

    // Getters and Setters
    public String getAircraftId() { return aircraftId; }
    public void setAircraftId(String aircraftId) { this.aircraftId = aircraftId; }
    public double getLatitude() { return latitude; }
    public void setLatitude(double latitude) { this.latitude = latitude; }
    public double getLongitude() { return longitude; }
    public void setLongitude(double longitude) { this.longitude = longitude; }
    public double getAltitude() { return altitude; }
    public void setAltitude(double altitude) { this.altitude = altitude; }
    public double getSpeed() { return speed; }
    public void setSpeed(double speed) { this.speed = speed; }
}

Step 2: Producer Implementation

package com.example.aviation.producer;

import com.example.aviation.model.AircraftTelemetry;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.stereotype.Service;

@Service
public class TelemetryProducer {

    private static final String TOPIC = "AircraftTelemetry";

    @Autowired
    private KafkaTemplate kafkaTemplate;

    public void sendTelemetry(AircraftTelemetry telemetry) {
        kafkaTemplate.send(TOPIC, telemetry.getAircraftId(), telemetry);
    }
}

Step 3: Consumer Implementation

package com.example.aviation.consumer;

import com.example.aviation.model.AircraftTelemetry;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.stereotype.Service;

@Service
public class TelemetryConsumer {

    @KafkaListener(topics = "AircraftTelemetry", groupId = "aviation-group")
    public void consumeTelemetry(AircraftTelemetry telemetry) {
        System.out.println("Received telemetry: " + telemetry);
        // Add logic to process or save telemetry data
    }
}

Step 4: REST API for Sending Data

package com.example.aviation.controller;

import com.example.aviation.model.AircraftTelemetry;
import com.example.aviation.producer.TelemetryProducer;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.*;

@RestController
@RequestMapping("/api/telemetry")
public class TelemetryController {

    @Autowired
    private TelemetryProducer producer;

    @PostMapping("/send")
    public String sendTelemetry(@RequestBody AircraftTelemetry telemetry) {
        producer.sendTelemetry(telemetry);
        return "Telemetry sent successfully!";
    }
}

Kafka vs RabbitMQ