blob: 56e270da44305fe816d44eeef2743929e1a25b5c (
plain) (
blame)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
|
# Kafka on Kubernetes
Transparent Kafka setup that you can grow with.
Good for both experiments and production.
How to use:
* Run a Kubernetes cluster, [minikube](https://github.com/kubernetes/minikube) or real.
* To quickly get a small Kafka cluster running, use the `kubectl apply`s below.
* To start using Kafka for real, fork and have a look at [addon](https://github.com/Yolean/kubernetes-kafka/labels/addon)s.
* Join the discussion here in issues and PRs.
Why?
See for yourself. No single readable readme can properly introduce both Kafka and Kubernets.
Back when we read [Newman](http://samnewman.io/books/building_microservices/) we were beginners with both.
Now we read [Kleppmann](http://dataintensive.net/), [Confluent's blog](https://www.confluent.io/blog/) and [SRE](https://landing.google.com/sre/book.html) and enjoy this "Streaming Platform" lock-in :smile:.
## What you get
[Bootstrap servers](http://kafka.apache.org/documentation/#producerconfigs): `kafka-0.broker.kafka.svc.cluster.local:9092,kafka-1.broker.kafka.svc.cluster.local:9092,kafka-2.broker.kafka.svc.cluster.local:9092`
`
Zookeeper at `zookeeper.kafka.svc.cluster.local:2181`.
## Set up Zookeeper
The [Kafka book](https://www.confluent.io/resources/kafka-definitive-guide-preview-edition/) recommends that Kafka has its own Zookeeper cluster with at least 5 instances.
```
kubectl create -f ./zookeeper/
```
To support automatic migration in the face of availability zone unavailability :wink: we mix persistent and ephemeral storage.
## Start Kafka
```
kubectl create -f ./
```
You might want to verify in logs that Kafka found its own DNS name(s) correctly. Look for records like:
```
kubectl -n kafka logs kafka-0 | grep "Registered broker"
# INFO Registered broker 0 at path /brokers/ids/0 with addresses: PLAINTEXT -> EndPoint(kafka-0.broker.kafka.svc.cluster.local,9092,PLAINTEXT)
```
|
