From ba907647b9f1c2186d45265d595c0b0bdc110980 Mon Sep 17 00:00:00 2001 From: Staffan Olsson Date: Fri, 28 Jul 2017 09:28:18 +0200 Subject: Could be a pitch --- README.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/README.md b/README.md index 25ecff2..7841df0 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,10 @@ # Kafka as Kubernetes StatefulSet +Transparent Kafka setup that you can grow with, +extendable through [addon](https://github.com/Yolean/kubernetes-kafka/labels/addon)s. + + Example of three Kafka brokers depending on five Zookeeper instances. To get consistent service DNS names `kafka-N.broker.kafka`(`.svc.cluster.local`), run everything in a [namespace](http://kubernetes.io/docs/admin/namespaces/walkthrough/): -- cgit v1.2.3 From 36a36032acac13aa4b340557fcf4c1d1a9c053ee Mon Sep 17 00:00:00 2001 From: Staffan Olsson Date: Fri, 28 Jul 2017 09:31:39 +0200 Subject: Could be the motivation --- README.md | 27 +++++++++++++++++++++------ 1 file changed, 21 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index 7841df0..dd16bc7 100644 --- a/README.md +++ b/README.md @@ -1,17 +1,32 @@ -# Kafka as Kubernetes StatefulSet -Transparent Kafka setup that you can grow with, -extendable through [addon](https://github.com/Yolean/kubernetes-kafka/labels/addon)s. +# Kafka on Kubernetes +Transparent Kafka setup that you can grow with. +Good for both experiments and production. -Example of three Kafka brokers depending on five Zookeeper instances. +How to use: + * Run a Kubernetes cluster, [minikube](https://github.com/kubernetes/minikube) or real. + * To quickly get a small Kafka cluster running, use the `kubectl apply`s below. + * To start using Kafka for real, fork and have a look at [addon](https://github.com/Yolean/kubernetes-kafka/labels/addon)s. + * Join the discussion here in issues and PRs. -To get consistent service DNS names `kafka-N.broker.kafka`(`.svc.cluster.local`), run everything in a [namespace](http://kubernetes.io/docs/admin/namespaces/walkthrough/): +Why? +No single readable readme can properly introduce both Kafka and Kubernets. +We started this project as beginners with both, +and by now our microservices enjoy lock-in with this "Streaming Platform" :smile:. +We read [Designing Data-Intensive Applications](http://dataintensive.net/) +and the [Confluent blog](https://www.confluent.io/blog/). + +## What you get + +[Bootstrap servers](http://kafka.apache.org/documentation/#producerconfigs): ``` -kubectl create -f 00namespace.yml +kafka-0.broker.kafka.svc.cluster.local:9092,kafka-1.broker.kafka.svc.cluster.local:9092,kafka-2.broker.kafka.svc.cluster.local:9092` ``` +Zookeeper at `zookeeper.kafka.svc.cluster.local:2181`. + ## Set up Zookeeper The Kafka book (Definitive Guide, O'Reilly 2016) recommends that Kafka has its own Zookeeper cluster with at least 5 instances. -- cgit v1.2.3 From dfa82bd6e56e9edc3d7cbcfabe86427d68aa2d61 Mon Sep 17 00:00:00 2001 From: Staffan Olsson Date: Fri, 28 Jul 2017 09:38:42 +0200 Subject: Shorter --- README.md | 14 +++++--------- 1 file changed, 5 insertions(+), 9 deletions(-) diff --git a/README.md b/README.md index dd16bc7..534e7d9 100644 --- a/README.md +++ b/README.md @@ -12,18 +12,14 @@ How to use: * Join the discussion here in issues and PRs. Why? -No single readable readme can properly introduce both Kafka and Kubernets. -We started this project as beginners with both, -and by now our microservices enjoy lock-in with this "Streaming Platform" :smile:. -We read [Designing Data-Intensive Applications](http://dataintensive.net/) -and the [Confluent blog](https://www.confluent.io/blog/). +See for yourself. No single readable readme can properly introduce both Kafka and Kubernets. +Back when we read [Newman](http://samnewman.io/books/building_microservices/) we were beginners with both. +Now we read [Kleppmann](http://dataintensive.net/), [Confluent's blog](https://www.confluent.io/blog/) and [SRE](https://landing.google.com/sre/book.html) and enjoy this "Streaming Platform" lock-in :smile:. ## What you get -[Bootstrap servers](http://kafka.apache.org/documentation/#producerconfigs): -``` -kafka-0.broker.kafka.svc.cluster.local:9092,kafka-1.broker.kafka.svc.cluster.local:9092,kafka-2.broker.kafka.svc.cluster.local:9092` -``` +[Bootstrap servers](http://kafka.apache.org/documentation/#producerconfigs): `kafka-0.broker.kafka.svc.cluster.local:9092,kafka-1.broker.kafka.svc.cluster.local:9092,kafka-2.broker.kafka.svc.cluster.local:9092` +` Zookeeper at `zookeeper.kafka.svc.cluster.local:2181`. -- cgit v1.2.3 From 315317f5e3992a6a56be8e670fa571df41a6b506 Mon Sep 17 00:00:00 2001 From: Staffan Olsson Date: Fri, 28 Jul 2017 09:44:48 +0200 Subject: Shorter still --- README.md | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 534e7d9..6191eda 100644 --- a/README.md +++ b/README.md @@ -25,16 +25,15 @@ Zookeeper at `zookeeper.kafka.svc.cluster.local:2181`. ## Set up Zookeeper -The Kafka book (Definitive Guide, O'Reilly 2016) recommends that Kafka has its own Zookeeper cluster with at least 5 instances. -We use the zookeeper build that comes with the Kafka distribution, and tweak the startup command to support StatefulSet. +The [Kafka book](https://www.confluent.io/resources/kafka-definitive-guide-preview-edition/) recommends that Kafka has its own Zookeeper cluster with at least 5 instances. ``` kubectl create -f ./zookeeper/ ``` -## Start Kafka +To support automatic migration in the face of availability zone unavailability :wink: we mix persistent and ephemeral storage. -Assuming you have your PVCs `Bound`, or enabled automatic provisioning (see above), go ahead and: +## Start Kafka ``` kubectl create -f ./ -- cgit v1.2.3 From 7e6df7119c5dca05e69c09a6f5dad720c9b9889a Mon Sep 17 00:00:00 2001 From: Staffan Olsson Date: Fri, 28 Jul 2017 09:45:31 +0200 Subject: If you got this far you don't need Kubernetes intro --- README.md | 8 -------- 1 file changed, 8 deletions(-) diff --git a/README.md b/README.md index 6191eda..ae75c21 100644 --- a/README.md +++ b/README.md @@ -66,14 +66,6 @@ Pods that keep consuming messages (but they won't exit on cluster failures) kubectl create -f test/21consumer-test1.yml ``` -## Teardown & cleanup - -Testing and retesting... delete the namespace. PVs are outside namespaces so delete them too. -``` -kubectl delete namespace kafka -rm -R ./data/ && kubectl -n kafka delete pv datadir-kafka-0 datadir-kafka-1 datadir-kafka-2 -``` - ## Metrics, Prometheus style Is the metrics system up and running? -- cgit v1.2.3 From c4f3c412c8338d562e49a1ac77d4654c07c71ce1 Mon Sep 17 00:00:00 2001 From: Staffan Olsson Date: Fri, 28 Jul 2017 09:47:49 +0200 Subject: Reworked in https://github.com/Yolean/kubernetes-kafka/pull/51 --- README.md | 21 --------------------- 1 file changed, 21 deletions(-) diff --git a/README.md b/README.md index ae75c21..547ddcd 100644 --- a/README.md +++ b/README.md @@ -45,27 +45,6 @@ kubectl -n kafka logs kafka-0 | grep "Registered broker" # INFO Registered broker 0 at path /brokers/ids/0 with addresses: PLAINTEXT -> EndPoint(kafka-0.broker.kafka.svc.cluster.local,9092,PLAINTEXT) ``` -## Testing manually - -There's a Kafka pod that doesn't start the server, so you can invoke the various shell scripts. -``` -kubectl create -f test/99testclient.yml -``` - -See `./test/test.sh` for some sample commands. - -## Automated test, while going chaosmonkey on the cluster - -This is WIP, but topic creation has been automated. Note that as a [Job](http://kubernetes.io/docs/user-guide/jobs/), it will restart if the command fails, including if the topic exists :( -``` -kubectl create -f test/11topic-create-test1.yml -``` - -Pods that keep consuming messages (but they won't exit on cluster failures) -``` -kubectl create -f test/21consumer-test1.yml -``` - ## Metrics, Prometheus style Is the metrics system up and running? -- cgit v1.2.3 From a880307fa1eb7ec785f12d08a12d2e27ff669454 Mon Sep 17 00:00:00 2001 From: Staffan Olsson Date: Fri, 28 Jul 2017 09:48:29 +0200 Subject: Metrics intro belongs in ... https://github.com/Yolean/kubernetes-kafka/pull/49 but maybe with tests instead of talk --- README.md | 11 ----------- 1 file changed, 11 deletions(-) diff --git a/README.md b/README.md index 547ddcd..56e270d 100644 --- a/README.md +++ b/README.md @@ -44,14 +44,3 @@ You might want to verify in logs that Kafka found its own DNS name(s) correctly. kubectl -n kafka logs kafka-0 | grep "Registered broker" # INFO Registered broker 0 at path /brokers/ids/0 with addresses: PLAINTEXT -> EndPoint(kafka-0.broker.kafka.svc.cluster.local,9092,PLAINTEXT) ``` - -## Metrics, Prometheus style - -Is the metrics system up and running? -``` -kubectl logs -c metrics kafka-0 -kubectl exec -c broker kafka-0 -- /bin/sh -c 'apk add --no-cache curl && curl http://localhost:5556/metrics' -kubectl logs -c metrics zoo-0 -kubectl exec -c zookeeper zoo-0 -- /bin/sh -c 'apk add --no-cache curl && curl http://localhost:5556/metrics' -``` -Metrics containers can't be used for the curl because they're too short on memory. -- cgit v1.2.3 From 91b4dde07d80ec2fb7e44bd8edfac28e381ba0c1 Mon Sep 17 00:00:00 2001 From: Staffan Olsson Date: Fri, 28 Jul 2017 10:37:03 +0200 Subject: Where to go from here --- README.md | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 56e270d..97b6d46 100644 --- a/README.md +++ b/README.md @@ -18,12 +18,14 @@ Now we read [Kleppmann](http://dataintensive.net/), [Confluent's blog](https://w ## What you get +Keep an eye on `kubectl --namespace kafka get pods -w`. + [Bootstrap servers](http://kafka.apache.org/documentation/#producerconfigs): `kafka-0.broker.kafka.svc.cluster.local:9092,kafka-1.broker.kafka.svc.cluster.local:9092,kafka-2.broker.kafka.svc.cluster.local:9092` ` Zookeeper at `zookeeper.kafka.svc.cluster.local:2181`. -## Set up Zookeeper +## Start Zookeeper The [Kafka book](https://www.confluent.io/resources/kafka-definitive-guide-preview-edition/) recommends that Kafka has its own Zookeeper cluster with at least 5 instances. @@ -31,7 +33,7 @@ The [Kafka book](https://www.confluent.io/resources/kafka-definitive-guide-previ kubectl create -f ./zookeeper/ ``` -To support automatic migration in the face of availability zone unavailability :wink: we mix persistent and ephemeral storage. +To support automatic migration in the face of availability zone unavailability we mix persistent and ephemeral storage. ## Start Kafka @@ -44,3 +46,8 @@ You might want to verify in logs that Kafka found its own DNS name(s) correctly. kubectl -n kafka logs kafka-0 | grep "Registered broker" # INFO Registered broker 0 at path /brokers/ids/0 with addresses: PLAINTEXT -> EndPoint(kafka-0.broker.kafka.svc.cluster.local,9092,PLAINTEXT) ``` + +That's it. Just add business value :wink:. +For clients we tend to use [librdkafka](https://github.com/edenhill/librdkafka)-based drivers like [node-rdkafka](https://github.com/Blizzard/node-rdkafka). +To use [Kafka Connect](http://kafka.apache.org/documentation/#connect) and [Kafka Streams](http://kafka.apache.org/documentation/streams/) you may want to take a look at our [sample](https://github.com/solsson/dockerfiles/tree/master/connect-files) [Dockerfile](https://github.com/solsson/dockerfiles/tree/master/streams-logfilter)s. +Don't forget the [addon](https://github.com/Yolean/kubernetes-kafka/labels/addon)s. -- cgit v1.2.3