Kafka Security - Part 1

Custom Kafka Security - Part 1

In this series, my hope is to explain on how to configure Kafka Security and how to setup custom Kafka Security. The current documentation about configuring Kafka Security is basically limited to few pages. I hoping to fill in the gaps and provide more information about my findings. As a disclaimer, I do not pretend to be a security expert or Kafka expert.

Kafka supports various types of authentication methods as described in the Apache Kafka documentation

In this first case, we would like to secure Broker to Broker and Client to Broker using SASL/Plain and without TLS.

Authentication using SASL/PLAIN without TLS

This article provides the configuration for authentication of connections to brokers from clients (producers and consumers), using SASL without TLS. 

SASL/PLAIN is a simple username/password authentication mechanism to implement secure authentication. Kafka supports a default implementation for SASL/PLAIN. Apache Kafka documentation provides detail on how it can be extended for production here.The username is used as the authenticated Principal for configuration of ACLs etc.

The first step is to configure the broker to authenticate the in-coming call using SASL with PlainText. This call can come from other brokers or clients such as consumer or producer.

The second step is to configure the broker to call other brokers using SASL with PlainText. 

The third steps is to configure the clients (producer/consumer) to use SASL with PlainText.

"Configuring Kafka Brokers" section takes care of the first and second steps.

Configuring Kafka Brokers: 

1. Add a suitably modified JAAS file similar to the one below to each Kafka broker's config directory, let's call it kafka_server_jaas.conf for this example:

KafkaServer {
   org.apache.kafka.common.security.plain.PlainLoginModule required
   username="admin"
   password="admin-secret"
   user_admin="admin-secret"
   user_alice="alice-secret";
};

Let's try to understand this: 

The properties username and password in the KafkaServer section are used by the broker to initiate connections to other brokers. In this example, admin is the user for inter-broker communication. 

The set of properties user_userName defines the passwords for all users that connect to the broker. The broker validates all client connections including those from other brokers using these properties. This configuration defines two users (admin and alice). 

2. Pass the JAAS config file location as JVM parameter to each Kafka broker:

-Djava.security.auth.login.config=/etc/kafka/kafka_server_jaas.conf

3. Configure SASL port and SASL mechanisms in server.properties. These are properties that would be configured:

listeners=OUR_LISTENER://:9092
advertised.listeners= OUR_LISTENER://:9092
security.inter.broker.protocol= OUR_LISTENER
listener.security.protocol.map= OUR_LISTENER:SASL_PLAINTEXT
sasl.mechanism.inter.broker.protocol=PLAIN
sasl.enabled.mechanisms=PLAIN

Let's try to understand this:

listeners=OUR_LISTENER://:9092

advertised.listeners= OUR_LISTENER://:9092

# You are defining a listener with the name OUR_LISTENER.  You can give any name to the listener. 

security.inter.broker.protocol= OUR_LISTENER
#The listener to communicate with for Broker to Broker communication

listener.security.protocol.map= OUR_LISTENER:SASL_PLAINTEXT

#The actual mapping of security protocol to listener name.
#Here we are specifying that we are using SASL with PLAINTEXT (not SSL)

sasl.mechanism.inter.broker.protocol=PLAIN

#The inter broker SASL mechanism uses PLAIN text (not SSL)

sasl.enabled.mechanisms=PLAIN

#The client broker SASL mechanism uses PLAIN text (not SSL)

SASL with PlainTexy Configuration for Kafka clients

1. Add a suitably modified JAAS config file to one below to the Client directory

KafkaClient { 
     org.apache.kafka.common.security.plain.PlainLoginModule required
     username="alice"
     password="alice-secret";
};

Let's try to understand this :
The username used to call the broker is specified as "alice" with password "alice-secret". 

2. Pass the JAAS config file location as JVM parameter to each client JVM. For example:

-Djava.security.auth.login.config=/etc/kafka/kafka_client_jaas.conf

3. Instead of 1 and 2, you can provided the JAAS configuration in the Client properties

props.put("sasl.jaas.config", 
        "org.apache.kafka.common.security.plain.PlainLoginModule required\n" +
        "username=\"alice\"\n" +
        "password=\"alice-secret\";");

Here, we have configured to secure the Kafka Broker to use SASL with PlainText.
In the next blog kafka-security-part-2, I would explain on how to setup Kafka configuration for Broker to Broker communication and Client to Broker communication