diff options
author | Staffan Olsson <staffan@repos.se> | 2017-07-28 10:39:42 +0200 |
---|---|---|
committer | Staffan Olsson <staffan@repos.se> | 2017-07-28 10:39:42 +0200 |
commit | 21a94a36b8515308d098b216bdf785db1e6c6612 (patch) | |
tree | ae58b64c0a6029d46fd67a5b13e3d51db1c5761d /README.md | |
parent | 364c9b7316d764bfb1a43b7e6930505d624ced1f (diff) | |
parent | 91b4dde07d80ec2fb7e44bd8edfac28e381ba0c1 (diff) | |
download | kubernetes-kafka-21a94a36b8515308d098b216bdf785db1e6c6612.tar.gz kubernetes-kafka-21a94a36b8515308d098b216bdf785db1e6c6612.tar.bz2 kubernetes-kafka-21a94a36b8515308d098b216bdf785db1e6c6612.zip |
Use a lighter readme with more links, as readers' background will vary widely
Diffstat (limited to 'README.md')
-rw-r--r-- | README.md | 79 |
1 files changed, 30 insertions, 49 deletions
@@ -1,72 +1,53 @@ -# Kafka as Kubernetes StatefulSet -Example of three Kafka brokers depending on five Zookeeper instances. +# Kafka on Kubernetes -To get consistent service DNS names `kafka-N.broker.kafka`(`.svc.cluster.local`), run everything in a [namespace](http://kubernetes.io/docs/admin/namespaces/walkthrough/): -``` -kubectl create -f 00namespace.yml -``` +Transparent Kafka setup that you can grow with. +Good for both experiments and production. -## Set up Zookeeper +How to use: + * Run a Kubernetes cluster, [minikube](https://github.com/kubernetes/minikube) or real. + * To quickly get a small Kafka cluster running, use the `kubectl apply`s below. + * To start using Kafka for real, fork and have a look at [addon](https://github.com/Yolean/kubernetes-kafka/labels/addon)s. + * Join the discussion here in issues and PRs. -The Kafka book (Definitive Guide, O'Reilly 2016) recommends that Kafka has its own Zookeeper cluster with at least 5 instances. -We use the zookeeper build that comes with the Kafka distribution, and tweak the startup command to support StatefulSet. +Why? +See for yourself. No single readable readme can properly introduce both Kafka and Kubernets. +Back when we read [Newman](http://samnewman.io/books/building_microservices/) we were beginners with both. +Now we read [Kleppmann](http://dataintensive.net/), [Confluent's blog](https://www.confluent.io/blog/) and [SRE](https://landing.google.com/sre/book.html) and enjoy this "Streaming Platform" lock-in :smile:. -``` -kubectl create -f ./zookeeper/ -``` +## What you get -## Start Kafka +Keep an eye on `kubectl --namespace kafka get pods -w`. -Assuming you have your PVCs `Bound`, or enabled automatic provisioning (see above), go ahead and: +[Bootstrap servers](http://kafka.apache.org/documentation/#producerconfigs): `kafka-0.broker.kafka.svc.cluster.local:9092,kafka-1.broker.kafka.svc.cluster.local:9092,kafka-2.broker.kafka.svc.cluster.local:9092` +` -``` -kubectl create -f ./ -``` +Zookeeper at `zookeeper.kafka.svc.cluster.local:2181`. -You might want to verify in logs that Kafka found its own DNS name(s) correctly. Look for records like: -``` -kubectl -n kafka logs kafka-0 | grep "Registered broker" -# INFO Registered broker 0 at path /brokers/ids/0 with addresses: PLAINTEXT -> EndPoint(kafka-0.broker.kafka.svc.cluster.local,9092,PLAINTEXT) -``` +## Start Zookeeper -## Testing manually +The [Kafka book](https://www.confluent.io/resources/kafka-definitive-guide-preview-edition/) recommends that Kafka has its own Zookeeper cluster with at least 5 instances. -There's a Kafka pod that doesn't start the server, so you can invoke the various shell scripts. ``` -kubectl create -f test/99testclient.yml +kubectl create -f ./zookeeper/ ``` -See `./test/test.sh` for some sample commands. - -## Automated test, while going chaosmonkey on the cluster +To support automatic migration in the face of availability zone unavailability we mix persistent and ephemeral storage. -This is WIP, but topic creation has been automated. Note that as a [Job](http://kubernetes.io/docs/user-guide/jobs/), it will restart if the command fails, including if the topic exists :( -``` -kubectl create -f test/11topic-create-test1.yml -``` +## Start Kafka -Pods that keep consuming messages (but they won't exit on cluster failures) ``` -kubectl create -f test/21consumer-test1.yml +kubectl create -f ./ ``` -## Teardown & cleanup - -Testing and retesting... delete the namespace. PVs are outside namespaces so delete them too. +You might want to verify in logs that Kafka found its own DNS name(s) correctly. Look for records like: ``` -kubectl delete namespace kafka -rm -R ./data/ && kubectl -n kafka delete pv datadir-kafka-0 datadir-kafka-1 datadir-kafka-2 +kubectl -n kafka logs kafka-0 | grep "Registered broker" +# INFO Registered broker 0 at path /brokers/ids/0 with addresses: PLAINTEXT -> EndPoint(kafka-0.broker.kafka.svc.cluster.local,9092,PLAINTEXT) ``` -## Metrics, Prometheus style - -Is the metrics system up and running? -``` -kubectl logs -c metrics kafka-0 -kubectl exec -c broker kafka-0 -- /bin/sh -c 'apk add --no-cache curl && curl http://localhost:5556/metrics' -kubectl logs -c metrics zoo-0 -kubectl exec -c zookeeper zoo-0 -- /bin/sh -c 'apk add --no-cache curl && curl http://localhost:5556/metrics' -``` -Metrics containers can't be used for the curl because they're too short on memory. +That's it. Just add business value :wink:. +For clients we tend to use [librdkafka](https://github.com/edenhill/librdkafka)-based drivers like [node-rdkafka](https://github.com/Blizzard/node-rdkafka). +To use [Kafka Connect](http://kafka.apache.org/documentation/#connect) and [Kafka Streams](http://kafka.apache.org/documentation/streams/) you may want to take a look at our [sample](https://github.com/solsson/dockerfiles/tree/master/connect-files) [Dockerfile](https://github.com/solsson/dockerfiles/tree/master/streams-logfilter)s. +Don't forget the [addon](https://github.com/Yolean/kubernetes-kafka/labels/addon)s. |