Skip to content

Quick Start: Scale Apache Kafka Streams

Warning

Make sure you have first followed the steps in Quick Start: Overview and Setup.

This guide shows you how to scale a simple Apache Kafka Streams application on the operator-level and on the deployment-level using the KTA Generic Kubernetes Scale Driver.

The guide uses a simple WordCount streaming query, which consists of two subtopologies, a tokenizer and a counter. To deploy Apache Kafka (which is needed to run Kafka Streams), you will use Strimzi. KTA will adjust the parallelism (= number of replicas of the deployments) based on the output of the sample algorithm.

Set up a Kafka Cluster

Install Strimzi using

kubectl apply -f https://raw.githubusercontent.com/dynatrace-oss/kubernetes-topology-autoscaler/refs/tags/v0.1.0-alpha.1/quickstart-examples/kta-quickstart-stream-processing-systems/kta-quickstart-kstreams/kubernetes/strimzi.yml

Next, deploy the Kafka Cluster using

kubectl apply -f https://raw.githubusercontent.com/dynatrace-oss/kubernetes-topology-autoscaler/refs/tags/v0.1.0-alpha.1/quickstart-examples/kta-quickstart-stream-processing-systems/kta-quickstart-kstreams/kubernetes/kafka-cluster.yml

You can verify that everything is running as expected by using the command below.

kubectl wait kafka/kta-kafka-cluster --for=condition=Ready --timeout=240s

Then, create a source topic and a sink topic.

# Source topic
cat << EOF | kubectl create -f -
apiVersion: kafka.strimzi.io/v1beta1
kind: KafkaTopic
metadata:
  name: kta-quickstart-source-topic
  labels:
    strimzi.io/cluster: kta-kafka-cluster
spec:
  partitions: 10
  replicas: 1
EOF

# Sink topic
cat << EOF | kubectl create -f -
apiVersion: kafka.strimzi.io/v1beta1
kind: KafkaTopic
metadata:
  name: kta-quickstart-sink-topic
  labels:
    strimzi.io/cluster: kta-kafka-cluster
spec:
  partitions: 10
  replicas: 1
EOF

Finally, start the producer.

kubectl apply -f https://raw.githubusercontent.com/dynatrace-oss/kubernetes-topology-autoscaler/refs/tags/v0.1.0-alpha.1/quickstart-examples/kta-quickstart-stream-processing-systems/kta-quickstart-kstreams/kubernetes/producer.yml

The producer writes to the source topic. The Kafka Streams application will use this topic as input to the streaming query. You can verify that everything is running as expected by using the command below.

kubectl run kafka-consumer -it --rm=true --restart=Never --image=quay.io/strimzi/kafka:0.47.0-kafka-4.0.0 -- bin/kafka-console-consumer.sh --bootstrap-server kta-kafka-cluster-kafka-bootstrap:9092 --topic kta-quickstart-source-topic --from-beginning

You should see text fragments from Franz Kafka's "Der Prozess", displayed line by line.

Close the Kafka console consumer using [CTRL] + [C] and wait until the application exits automatically.

Scaling Apache Kafka Streams on the Operator Level

To deploy the application on the operator level, you need to

  • explicitly create intermediate topics between subtopologies and
  • manually split the subtopologies of your Apache Kafka Streams application into separate applications.

Info

To simplify deployment, we are currently evaluating the feasibility of using a custom task assignor logic instead of manually splitting the application.

Create the intermediate topic using

cat << EOF | kubectl create -f -
apiVersion: kafka.strimzi.io/v1beta1
kind: KafkaTopic
metadata:
  name: kta-quickstart-intermediate-topic
  labels:
    strimzi.io/cluster: kta-kafka-cluster
spec:
  partitions: 10
  replicas: 1
EOF

Then deploy the tokenizer and the counter as separate applications using

kubectl apply -f https://raw.githubusercontent.com/dynatrace-oss/kubernetes-topology-autoscaler/refs/tags/v0.1.0-alpha.1/quickstart-examples/kta-quickstart-stream-processing-systems/kta-quickstart-kstreams/kubernetes/operator-level.yml

You can verify that everything is running as expected by using the commands below.

kubectl run kafka-consumer -it --rm=true --restart=Never --image=quay.io/strimzi/kafka:0.47.0-kafka-4.0.0 -- bin/kafka-console-consumer.sh --bootstrap-server kta-kafka-cluster-kafka-bootstrap:9092 --topic kta-quickstart-intermediate-topic --from-beginning --property value.deserializer=org.apache.kafka.common.serialization.IntegerDeserializer --property print.key=true --property key.separator=","

kubectl run kafka-consumer -it --rm=true --restart=Never --image=quay.io/strimzi/kafka:0.47.0-kafka-4.0.0 -- bin/kafka-console-consumer.sh --bootstrap-server kta-kafka-cluster-kafka-bootstrap:9092 --topic kta-quickstart-sink-topic --from-beginning --property value.deserializer=org.apache.kafka.common.serialization.IntegerDeserializer --property print.key=true --property key.separator=","

For the first Kafka console consumer, you should see key-value pairs of a string and an integer, where the values are always 1. This is the output of the tokenizer. For the second Kafka console consumer, you should see key value pairs of a string and an integer, where the values become eventually greater than 1. This is the (final) output of the counter.

When you run

kubectl get deployment kta-quickstart-kstreams-tokenizer kta-quickstart-kstreams-counter

you should see that both, the tokenizer and the counter run with a parallelism of 2.

Apply a KTAPolicy, Behold and See: Observe KTA's Scaling Behavior

Finally, to activate KTA, apply the corresponding KTAPolicy.

kubectl apply -f https://raw.githubusercontent.com/dynatrace-oss/kubernetes-topology-autoscaler/refs/tags/v0.1.0-alpha.1/quickstart-examples/kta-quickstart-kubernetes-operator/kubernetes/quickstart-policy-kstreams-operator-level.yml

The KTAPolicy uses KTA's Generic Kubernetes Scale Driver and a reconciliation interval of 30 seconds.

By default, the sample algorithm toggles the state of an operator (parallelism) on every second invocation, with possible states being a parallelism of 2 or 4.

When you run

watch -n 2 'kubectl get deployment kta-quickstart-kstreams-tokenizer kta-quickstart-kstreams-counter'

you should see that the operators changed their parallelism immedetialy upon applying the policy and then changes occur approximately every 60 seconds.

Since the application is deployed on the operator level, each operator can run with a different parallelism. If the first operator is running with parallelism 4, the second one should run with parallelism 2 and vice versa.

Clean Up and Next Steps

Congratulations! 🥳 You have successfully deployed KTA to scale an Apache Kafka Streams application on the operator level.

If you don't use the cluster anymore, you can tear it down using

k3d cluster delete kta-quickstart

To learn more about KTA and how to implement your own autoscaling algorithms, check out the User Guide and the API Reference.

Scaling Apache Kafka Streams on the Deployment Level (Kafka Streams Default)

To deploy the application on the deployment level use

kubectl apply -f https://raw.githubusercontent.com/dynatrace-oss/kubernetes-topology-autoscaler/refs/tags/v0.1.0-alpha.1/quickstart-examples/kta-quickstart-stream-processing-systems/kta-quickstart-kstreams/kubernetes/deployment-level.yml

You can verify that everything is running as expected by using the command below.

kubectl run kafka-consumer -it --rm=true --restart=Never --image=quay.io/strimzi/kafka:0.47.0-kafka-4.0.0 -- bin/kafka-console-consumer.sh --bootstrap-server kta-kafka-cluster-kafka-bootstrap:9092 --topic kta-quickstart-sink-topic --from-beginning --property value.deserializer=org.apache.kafka.common.serialization.IntegerDeserializer --property print.key=true --property key.separator=","

You should see key value pairs of a string and an integer, where the values become eventually greater than 1. This is the (final) output of the counter.

When you run

kubectl get deployment kta-quickstart-kstreams-tokenizer-counter

you should see that application is running with a parallelsim of 2.

Apply a KTAPolicy, Behold and See: Observe KTA's Scaling Behavior

Finally, to activate KTA, apply the corresponding KTAPolicy.

kubectl apply -f https://raw.githubusercontent.com/dynatrace-oss/kubernetes-topology-autoscaler/refs/tags/v0.1.0-alpha.1/quickstart-examples/kta-quickstart-kubernetes-operator/kubernetes/quickstart-policy-kstreams-deployment-level.yml

The KTAPolicy uses KTA's Generic Kubernetes Scale Driver and a reconciliation interval of 30 seconds.

By default, the sample algorithm toggles the state of an operator (parallelism) on every second invocation, with possible states being a parallelism of 2 or 4.

When you run

watch -n 2 'kubectl get deployment kta-quickstart-kstreams-tokenizer kta-quickstart-kstreams-counter'

you should see that the application changed its parallelism immedetialy upon applying the policy and then changes occur approximately every 60 seconds.

Since the application is deployed on the deployment level, both operators run with the same parallelism (either 2 or 4).

Clean Up and Next Steps

Congratulations! 🥳 You have successfully deployed KTA to scale an Apache Kafka Streams application on the deployment level.

If you don't use the cluster anymore, you can tear it down using

k3d cluster delete kta-quickstart

To learn more about KTA and how to implement your own autoscaling algorithms, check out the User Guide and the API Reference.