Technology Sharing

한어Русский языкEnglishFrançaisIndonesianSanskrit日本語DeutschPortuguêsΕλληνικάespañolItalianoSuomalainenLatina

How to download Kafaka

Official website address
The latest version of Kafka is 3.7.1

We can see the following two versions of information. What do they mean?

• Scala 2.12 - kafka_2.12-3.7.1.tgz (asc, sha512)
• Scala 2.13 - kafka_2.13-3.7.1.tgz (asc, sha512)
  We should know that a complete Kafka instance consists of at least 3 parts:
• Producer
• Broker
• Producer-Consumer
  The producer and consumer use Java, while the broker uses Scala. 2.12 and 2.13 are actually the Scala versions, and 3.7.1 is the official version of Kafka.

Kafka Installation

We need to first ensure that our server has the Java environment installed:

java -version

Download Kafka:
Use -o to specify the path where we store the files.

wget -O /usr/local/kafka_2.13-3.7.1.tgz https://downloads.apache.org/kafka/3.7.1/kafka_2.13-3.7.1.tgz

Unzip:

tar -zxvf  kafka_2.13-3.7.1.tgz

Kafka configuration and startup

Tips
KafkaIt is a high-throughput distributed publish-subscribe messaging system that can handle all action stream data of consumers in the website.
KafkaIn previous versions, if there is noZooKeeper,Kafkawill not run.
UsedkafkaDevelopers should know that every time you startkafkaWhen serving, you need to firstZookeeperStart, then startkafka, the steps are quite complicated.
KafkaIn the published2.8In this version, users canZooKeeperRun underKafka, which will depend onZooKeeperThe controller was transformed into aKafka RaftofQuormController (KRaft mode).

Startup via ZooKeeper

View the configuration file:

connect-console-sink.properties
connect-console-source.properties
connect-distributed.properties
connect-file-sink.properties
connect-file-source.properties
connect-log4j.properties
connect-mirror-maker.properties
connect-standalone.properties
consumer.properties
kraft
log4j.properties
producer.properties
server.properties
tools-log4j.properties
trogdor.conf
zookeeper.properties

1: Modify ZooKeeper configuration

vim zookeeper.properties  
## 配置地址用于存放zookeeper数据存储位置，不存在会自动创建
dataDir=/usr/local/kafka_data/zookeeper_data

2: Start ZooKeeper
Enter the bin directory of kafka:

#启动zookeeper
../bin/zookeeper-server-start.sh -daemon ./config/zookeeper.properties 
# 查看zookeeper进程是否启动  
ps -ef | grep zookeeper

3: Start Kafka

vim ../config/server.properties  
#修改log.dirs地址
log.dirs=/usr/local/kafka_data/kafka_data
#启动Kafka
../bin/kafka-server-start.sh -daemon ../config/server.properties 
#通过jps命令可以查看是否启动成功，看到下面的结果表示我们的zookeeper和Kafka都启动成功了
[root@qingshan bin]# jps
7297 Jps
1592 QuorumPeerMain
7241 Kafka

View the startup log
If you do not see the Kafka process using the JPS command after executing the startup command, it may be that the startup failed. You can view the startup log in the corresponding path and replace the path with the path of your own Kafka.
/usr/local/kafka_2.13-3.7.1/bin/hs_err_pid***.log

Start Kafka in KRaft mode

Open our KafkaConfigIn the directory, there is a KRaft folder.
1: Modify the data storage location

vim config/kraft/server.properties
log.dirs=/usr/local/kafka_data/kafka_KRaft_data

2: Generate a unique ID for the storage directory
There is a kafka-storage.sh file in the bin directory of kafka

./kafka-storage.sh random-uuid 
#结果
bxoRVvPvR0qjT307GQ6Gag

3: Format the storage directory

# 将上一步生成的唯一ID替换成自己的，启动使用kraft下的配置文件
./kafka-storage.sh  format -t bxoRVvPvR0qjT307GQ6Gag   -c  /usr/local/kafka_2.13-3.7.1/config/kraft/server.properties

Results of the:

4: Start Kafka

#执行启动命令。
./kafka-server-start.sh -daemon ../config/kraft/server.properties