Often the protocol is used for the listener name too, but here let’s make it nice and clear by using abstract names for the listeners: KAFKA_LISTENERS: LISTENER_BOB://kafka0:29092,LISTENER_FRED://localhost:9092 The actual host and IP that it will connect to for reading/writing data is based on the data that the broker passes back in that initial connection-even if it’s just a single node and the broker returned is the same as the one it’s connected to.įor configuring this correctly, you need to understand that Kafka brokers can have multiple listeners. The key thing is that when you run a client, the broker you pass to it is just where it’s going to go and get the metadata about brokers in the cluster from. You need to tell Kafka how the brokers can reach each other but also make sure that external clients (producers/consumers) can reach the broker they need to reach. Brokers in the cloud (e.g., AWS EC2) and on-premises machines locally (or even in another cloud).
Docker network internal plus#
Docker internal network(s) plus host machine.Let’s assume you have more than one network.
But once you move into more complex networking setups and multiple nodes, you have to pay more attention to it. On a single machine, running bare metal (no VMs, no Docker), everything might be the hostname (or just localhost), and it’s easy. It’s these endpoints that cause people trouble.
The metadata returned will include the endpoints available for the Leader broker for that partition, and the client will then use those endpoints to connect to the broker to read/write data as required.
When a client (producer/consumer) starts, it will request metadata about which broker is the leader for a partition-and it can do this from any broker. Data is read from and written to the leader for a given partition, which could be on any of the brokers in a cluster. Is anyone listening?Īpache Kafka is a distributed system.
Docker network internal how to#
In this post, I’ll talk about why this is necessary and then show how to do it based on a couple of scenarios-Docker and AWS. LISTENERS are what interfaces Kafka binds to.ĪDVERTISED_LISTENERS are how clients can connect. Put another way, courtesy of Spencer Ruport: Otherwise, they’ll try to connect to the internal host address-and if that’s not reachable, then problems ensue. Tl dr: You need to set advertised.listeners (or KAFKA_ADVERTISED_LISTENERS if you’re using Docker images) to the external address (host/IP) so that clients can correctly connect to it. If you are not using fully managed Apache Kafka ® in the Confluent Cloud, then this question on Kafka listener configuration comes up on Stack Overflow and such places a lot, so here’s something to try and help.