Kafka Connection

The Kafka connection type enables automations to produce and consume messages from Apache Kafka topics. It is used by the plugin-kafka plugin, which delegates to the kafka-connector.

Connection Type

Property	Value
Type enum	KAFKA
Type ID	kafka
Category	Messaging
Entity class (shared lib)	KafkaConnectionDetails
Connector class	KafkaConnectionDetails (connector)

Friendly Example

What you enter	Example
Connection name	Kafka - Event Stream
Used for	Publishing and consuming business events between systems
Main details to collect	Bootstrap servers, topic name, security method, username, and password
Best person to provide it	Messaging or platform administrator

After it is saved, workflow builders select Kafka - Event Stream in a node instead of entering these details again.

Mandatory Fields

The shared library entity and the connector have slightly different field models. The connection is stored using the shared library fields; the plugin maps them to the connector model at runtime.

Shared Library Fields (stored in MongoDB)

Field	Type	Required	Description
brokerUrl	String	Yes	Comma-separated list of Kafka broker addresses in host:port format (e.g., broker1:9092,broker2:9092).
serializerKey	String	Yes	Fully qualified class name of the key serializer (e.g., org.apache.kafka.common.serialization.StringSerializer).
serializerValue	String	Yes	Fully qualified class name of the value serializer (e.g., org.apache.kafka.common.serialization.StringSerializer).

Connector Fields (used at runtime)

The connector's KafkaConnectionDetails accepts a richer set of fields for authentication and TLS:

Field	Type	Required	Description
bootstrapServers	String	Yes	Same as brokerUrl. Comma-separated broker addresses.
securityProtocol	String	Yes	Security protocol for broker communication. One of: PLAINTEXT, SSL, SASL_PLAINTEXT, SASL_SSL.
clientId	String	No	Client identifier for the Kafka producer/consumer.
username	String	No	SASL authentication username. Required when securityProtocol is SASL_PLAINTEXT or SASL_SSL.
password	String	No	SASL authentication password. Encrypted at rest. Required when using SASL.
saslMechanism	String	No	SASL mechanism. One of: PLAIN, SCRAM-SHA-256, SCRAM-SHA-512. Required when using SASL.
sslTruststoreLocation	String	No	Path to the SSL truststore file. Required when securityProtocol is SSL or SASL_SSL.
sslTruststorePassword	String	No	Password for the SSL truststore. Encrypted at rest.

Security Protocol Reference

Protocol	Encryption	Authentication	Use Case
PLAINTEXT	None	None	Development/testing only
SSL	TLS	Certificate-based	Encrypted without SASL
SASL_PLAINTEXT	None	SASL (username/password)	Authenticated but unencrypted (internal networks)
SASL_SSL	TLS	SASL (username/password)	Recommended for production. Encrypted + authenticated.

Setup Instructions

1. Identify your Kafka bootstrap servers. These are the initial broker addresses the client uses to discover the full cluster. Typically 2-3 brokers are listed for redundancy.

2. Determine the security protocol used by your Kafka cluster:

   • For Confluent Cloud: Use SASL_SSL with PLAIN mechanism and API key/secret as username/password.
   • For Amazon MSK: Use SASL_SSL with SCRAM-SHA-512 or SSL with IAM authentication.
   • For internal clusters: Check your broker's listeners configuration in server.properties.

3. Obtain credentials:

   • For SASL authentication, get the username and password from your Kafka administrator.
   • For SSL, obtain the truststore file and its password.
   • For Confluent Cloud, create an API key in the Confluent Cloud console.

4. Create the connection in the Global Configurator with the fields above.

5. Test the connection by verifying that the platform can reach the brokers and authenticate.

Troubleshooting

Symptom	Likely Cause
Connection to node -1 could not be established	Brokers are unreachable. Check hostname, port, and firewall rules.
SASL authentication failed	Wrong username/password or SASL mechanism mismatch
SSL handshake failed	Truststore is missing, expired, or the broker's certificate is not trusted
Topic not found	The topic does not exist. Kafka auto-create may be disabled on the cluster.
Not authorized to access group	Kafka ACLs are restricting the consumer group. Ask your admin to grant access.