Big Data Passion

Big Data Passion

Strona tworzona przez pasjonatów i praktyków Big Data

Radosław Szmit

Poniżej lista przydatnych poleceń Apache Kafka. Dla ułatwienia korzystania, używane są odpowiednie zmienne systemowe.

export KAFKA_HOME=/usr/hdp/2.5.3.0-37/kafka
export ZOOKEEPER_HOME=/usr/hdp/2.5.3.0-37/zookeeper

export KAFKA_ZOOKEEPER=localhost:2181
export KAFKA_BROKER=localhost:9092,localhost:9093,localhost:9094

export TOPIC=test-topic

Jeśli korzystamy z Dockera:

export KAFKA_ZOOKEEPER=zookeeper.bigdatapassion.pl:2181
export KAFKA_BROKER=kafka.bigdatapassion.pl:9092
export TOPIC=test-topic

lub

export KAFKA_ZOOKEEPER=cluster_zookeeper:2181
export KAFKA_BROKER=cluster_kafka1:9092,cluster_kafka2:9092,cluster_kafka3:9092
export TOPIC=test-topic

Topiki

lista:

bin/kafka-topics.sh --list --zookeeper $KAFKA_ZOOKEEPER

tworzenie:

bin/kafka-topics.sh --create --zookeeper $KAFKA_ZOOKEEPER --replication-factor 3 --partitions 10 --topic $TOPIC

opis:

bin/kafka-topics.sh --describe --zookeeper $KAFKA_ZOOKEEPER
bin/kafka-topics.sh --describe --zookeeper $KAFKA_ZOOKEEPER --topic $TOPIC
bin/kafka-topics.sh --zookeeper $KAFKA_ZOOKEEPER --describe --under-replicated-partitions

bin/kafka-topics.sh --describe --zookeeper $KAFKA_ZOOKEEPER --topic __consumer_offsets

usuwanie

bin/kafka-topics.sh --delete --zookeeper $KAFKA_ZOOKEEPER --topic $TOPIC

Do usuwania należy włączyć tą możliwość w ustawieniach (delete.topic.enable), inaczej tylko oznacza topic jako usunięty.

edycja:

bin/kafka-topics.sh --alter --zookeeper $KAFKA_ZOOKEEPER --topic $TOPIC --partitions 20
bin/kafka-topics.sh --alter --zookeeper $KAFKA_ZOOKEEPER --topic $TOPIC --replication-factor 3
bin/kafka-topics.sh --alter --zookeeper $KAFKA_ZOOKEEPER --topic $TOPIC --replica-assignment 1003:1004:1005,0:1:2,0:1:2,2:1:0

bin/kafka-topics.sh --alter --zookeeper $KAFKA_ZOOKEEPER --topic __consumer_offsets --replication-factor 3

Producent i konsument w konsoli

bin/kafka-console-producer.sh --broker-list $KAFKA_BROKER --topic $TOPIC
bin/kafka-console-consumer.sh --bootstrap-server $KAFKA_BROKER --topic $TOPIC --from-beginning

bin/kafka-console-consumer.sh --bootstrap-server $KAFKA_BROKER \
    --topic test-topic-out \
    --from-beginning \
    --formatter kafka.tools.DefaultMessageFormatter \
    --property print.key=true \
    --property print.value=true \
    --property key.deserializer=org.apache.kafka.common.serialization.StringDeserializer \
    --property value.deserializer=org.apache.kafka.common.serialization.StringDeserializer

Grupy konsumentów

bin/kafka-consumer-groups.sh --zookeeper $KAFKA_ZOOKEEPER  --list
bin/kafka-consumer-groups.sh --bootstrap-server $KAFKA_BROKER  --list

bin/kafka-consumer-groups.sh --bootstrap-server $KAFKA_BROKER  --describe --group my-group

Zarządzanie partycjami

bin/kafka-reassign-partitions.sh --zookeeper $KAFKA_ZOOKEEPER --reassignment-json-file /tmp/increase-replication-factor.json --execute

Uruchamianie i zatrzymywanie

zookeeper:

bin/zookeeper-server-start.sh config/zookeeper.properties

broker:

bin/kafka-server-start.sh config/server.properties

ZooKeeper

instalacja oddzielna:

cd $ZOOKEEPER_HOME
bin/zkCli.sh

wersja z paczki Apache Kafka:

bin/zookeeper-shell.sh $KAFKA_ZOOKEEPER

polecenia konsoli ZooKeepera:

ls /brokers
ls /brokers/ids
ls /brokers/topics
ls /brokers/seqid

Narzędzia

bin/bin/kafka-consumer-groups.sh \
    --topic $TOPIC --zookeeper $KAFKA_ZOOKEEPER \
    --group test_group
comments powered by Disqus

Ostatnie wpisy

Zobacz więcej

Kategorie

About