2
0
0

Kafka 日志分析报告20260721

2026-07-21
文章摘要
|

Kafka 日志分析报告

背景

运行了一周的kafka,突然没有日志了,于是看到这些错误:

kafka  | 	at java.base/java.lang.Thread.run(Unknown Source)
kafka  | [2026-07-20 10:36:16,017] WARN [SocketServer listenerType=ZK_BROKER, nodeId=1001] Unexpected error from /45.33.12.122; closing connection (org.apache.kafka.common.network.Selector)
kafka  | org.apache.kafka.common.network.InvalidReceiveException: Invalid receive (size = 302055471 larger than 104857600)
kafka  | 	at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:105)
kafka  | 	at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:452)
kafka  | 	at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:402)
kafka  | 	at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:674)
kafka  | 	at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:576)
kafka  | 	at org.apache.kafka.common.network.Selector.poll(Selector.java:481)
kafka  | 	at kafka.network.Processor.poll(SocketServer.scala:989)
kafka  | 	at kafka.network.Processor.run(SocketServer.scala:892)
kafka  | 	at java.base/java.lang.Thread.run(Unknown Source)
kafka  | [2026-07-20 10:36:21,281] ERROR Closing socket for 172.22.6.12:9092-45.33.12.122:55361-6659 because of error (kafka.network.Processor)
kafka  | org.apache.kafka.common.errors.InvalidRequestException: Error parsing request header. Our best guess of the apiKey is: -173
kafka  | Caused by: java.lang.IllegalArgumentException: Unexpected api key: -173
kafka  | 	at org.apache.kafka.common.protocol.ApiKeys.forId(ApiKeys.java:183)
kafka  | 	at org.apache.kafka.common.requests.RequestHeader.parse(RequestHeader.java:93)
kafka  | 	at kafka.network.Processor.parseRequestHeader(SocketServer.scala:999)
kafka  | 	at kafka.network.Processor.$anonfun$processCompletedReceives$1(SocketServer.scala:1012)
kafka  | 	at java.base/java.util.LinkedHashMap$LinkedValues.forEach(Unknown Source)
kafka  | 	at kafka.network.Processor.processCompletedReceives(SocketServer.scala:1008)
kafka  | 	at kafka.network.Processor.run(SocketServer.scala:893)
kafka  | 	at java.base/java.lang.Thread.run(Unknown Source)
kafka  | [2026-07-20 10:36:21,281] ERROR Exception while processing request from 172.22.6.12:9092-45.33.12.122:55361-6659 (kafka.network.Processor)
kafka  | org.apache.kafka.common.errors.InvalidRequestException: Error parsing request header. Our best guess of the apiKey is: -173
kafka  | Caused by: java.lang.IllegalArgumentException: Unexpected api key: -173
kafka  | 	at org.apache.kafka.common.protocol.ApiKeys.forId(ApiKeys.java:183)
kafka  | 	at org.apache.kafka.common.requests.RequestHeader.parse(RequestHeader.java:93)
kafka  | 	at kafka.network.Processor.parseRequestHeader(SocketServer.scala:999)
kafka  | 	at kafka.network.Processor.$anonfun$processCompletedReceives$1(SocketServer.scala:1012)
kafka  | 	at java.base/java.util.LinkedHashMap$LinkedValues.forEach(Unknown Source)
kafka  | 	at kafka.network.Processor.processCompletedReceives(SocketServer.scala:1008)
kafka  | 	at kafka.network.Processor.run(SocketServer.scala:893)
kafka  | 	at java.base/java.lang.Thread.run(Unknown Source)
kafka  | [2026-07-20 10:37:14,260] WARN [SocketServer listenerType=ZK_BROKER, nodeId=1001] Unexpected error from /45.33.12.122; closing connection (org.apache.kafka.common.network.Selector)
kafka  | org.apache.kafka.common.network.InvalidReceiveException: Invalid receive (size = 1937006964 larger than 104857600)
kafka  | 	at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:105)
kafka  | 	at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:452)
kafka  | 	at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:402)
kafka  | 	at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:674)
kafka  | 	at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:576)
kafka  | 	at org.apache.kafka.common.network.Selector.poll(Selector.java:481)
kafka  | 	at kafka.network.Processor.poll(SocketServer.scala:989)
kafka  | 	at kafka.network.Processor.run(SocketServer.scala:892)
kafka  | 	at java.base/java.lang.Thread.run(Unknown Source)
kafka  | [2026-07-20 10:37:15,096] WARN [SocketServer listenerType=ZK_BROKER, nodeId=1001] Unexpected error from /45.33.12.122; closing connection (org.apache.kafka.common.network.Selector)
kafka  | org.apache.kafka.common.network.InvalidReceiveException: Invalid receive (size = 1330664521 larger than 104857600)
kafka  | 	at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:105)
kafka  | 	at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:452)
kafka  | 	at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:402)
kafka  | 	at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:674)
kafka  | 	at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:576)
kafka  | 	at org.apache.kafka.common.network.Selector.poll(Selector.java:481)
kafka  | 	at kafka.network.Processor.poll(SocketServer.scala:989)
kafka  | 	at kafka.network.Processor.run(SocketServer.scala:892)
kafka  | 	at java.base/java.lang.Thread.run(Unknown Source)
kafka  | [2026-07-20 17:03:19,962] INFO [ProducerStateManager partition=LOG_APP-0] Writing producer snapshot at offset 16732667 (kafka.log.ProducerStateManager)
kafka  | [2026-07-20 17:03:19,965] INFO [Log partition=LOG_APP-0, dir=/kafka/kafka-logs-5cb6ac8d09b8] Rolled new log segment at offset 16732667 in 4 ms. (kafka.log.Log)
kafka  | [2026-07-20 17:03:37,795] INFO [ProducerStateManager partition=LOG_INF-0] Writing producer snapshot at offset 68253900 (kafka.log.ProducerStateManager)
kafka  | [2026-07-20 17:03:37,800] INFO [Log partition=LOG_INF-0, dir=/kafka/kafka-logs-5cb6ac8d09b8] Rolled new log segment at offset 68253900 in 6 ms. (kafka.log.Log)
kafka  | [2026-07-20 17:08:35,091] INFO [Log partition=LOG_INF-0, dir=/kafka/kafka-logs-5cb6ac8d09b8] Deleting segment LogSegment(baseOffset=56231686, size=1073502484, lastModifiedTime=1783962500085, largestRecordTimestamp=Some(1783962500071)) due to retention time 604800000ms breach based on the largest record timestamp in the segment (kafka.log.Log)
kafka  | [2026-07-20 17:08:35,091] INFO [Log partition=LOG_INF-0, dir=/kafka/kafka-logs-5cb6ac8d09b8] Incremented log start offset to 57563616 due to segment deletion (kafka.log.Log)
kafka  | [2026-07-20 17:09:35,092] INFO [Log partition=LOG_INF-0, dir=/kafka/kafka-logs-5cb6ac8d09b8] Deleting segment files LogSegment(baseOffset=56231686, size=1073502484, lastModifiedTime=1783962500085, largestRecordTimestamp=Some(1783962500071)) (kafka.log.Log)
kafka  | [2026-07-20 17:09:35,136] INFO Deleted log /kafka/kafka-logs-5cb6ac8d09b8/LOG_INF-0/00000000000056231686.log.deleted. (kafka.log.LogSegment)
kafka  | [2026-07-20 17:09:35,137] INFO Deleted offset index /kafka/kafka-logs-5cb6ac8d09b8/LOG_INF-0/00000000000056231686.index.deleted. (kafka.log.LogSegment)
kafka  | [2026-07-20 17:09:35,137] INFO Deleted time index /kafka/kafka-logs-5cb6ac8d09b8/LOG_INF-0/00000000000056231686.timeindex.deleted. (kafka.log.LogSegment)
kafka  | [2026-07-20 17:18:35,091] INFO [Log partition=LOG_APP-0, dir=/kafka/kafka-logs-5cb6ac8d09b8] Deleting segment LogSegment(baseOffset=13328159, size=1073733722, lastModifiedTime=1783962834936, largestRecordTimestamp=Some(1783962834913)) due to retention time 604800000ms breach based on the largest record timestamp in the segment (kafka.log.Log)
kafka  | [2026-07-20 17:18:35,091] INFO [Log partition=LOG_APP-0, dir=/kafka/kafka-logs-5cb6ac8d09b8] Incremented log start offset to 13825650 due to segment deletion (kafka.log.Log)
kafka  | [2026-07-20 17:19:35,091] INFO [Log partition=LOG_APP-0, dir=/kafka/kafka-logs-5cb6ac8d09b8] Deleting segment files LogSegment(baseOffset=13328159, size=1073733722, lastModifiedTime=1783962834936, largestRecordTimestamp=Some(1783962834913)) (kafka.log.Log)
kafka  | [2026-07-20 17:19:35,124] INFO Deleted log /kafka/kafka-logs-5cb6ac8d09b8/LOG_APP-0/00000000000013328159.log.deleted. (kafka.log.LogSegment)
kafka  | [2026-07-20 17:19:35,125] INFO Deleted offset index /kafka/kafka-logs-5cb6ac8d09b8/LOG_APP-0/00000000000013328159.index.deleted. (kafka.log.LogSegment)
kafka  | [2026-07-20 17:19:35,125] INFO Deleted time index /kafka/kafka-logs-5cb6ac8d09b8/LOG_APP-0/00000000000013328159.timeindex.deleted. (kafka.log.LogSegment)
kafka  | [2026-07-20 19:15:06,619] WARN [SocketServer listenerType=ZK_BROKER, nodeId=1001] Unexpected error from /205.210.31.73; closing connection (org.apache.kafka.common.network.Selector)
kafka  | org.apache.kafka.common.network.InvalidReceiveException: Invalid receive (size = 1195725856 larger than 104857600)
kafka  | 	at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:105)
kafka  | 	at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:452)
kafka  | 	at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:402)
kafka  | 	at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:674)
kafka  | 	at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:576)
kafka  | 	at org.apache.kafka.common.network.Selector.poll(Selector.java:481)
kafka  | 	at kafka.network.Processor.poll(SocketServer.scala:989)
kafka  | 	at kafka.network.Processor.run(SocketServer.scala:892)
kafka  | 	at java.base/java.lang.Thread.run(Unknown Source)
kafka  | [2026-07-21 07:33:35,091] INFO [Log partition=LOG_INF-0, dir=/kafka/kafka-logs-5cb6ac8d09b8] Deleting segment LogSegment(baseOffset=57563616, size=1073656676, lastModifiedTime=1784014379578, largestRecordTimestamp=Some(1784014378578)) due to retention time 604800000ms breach based on the largest record timestamp in the segment (kafka.log.Log)
kafka  | [2026-07-21 07:33:35,092] INFO [Log partition=LOG_INF-0, dir=/kafka/kafka-logs-5cb6ac8d09b8] Incremented log start offset to 59201750 due to segment deletion (kafka.log.Log)
kafka  | [2026-07-21 07:34:35,092] INFO [Log partition=LOG_INF-0, dir=/kafka/kafka-logs-5cb6ac8d09b8] Deleting segment files LogSegment(baseOffset=57563616, size=1073656676, lastModifiedTime=1784014379578, largestRecordTimestamp=Some(1784014378578)) (kafka.log.Log)
kafka  | [2026-07-21 07:34:35,141] INFO Deleted log /kafka/kafka-logs-5cb6ac8d09b8/LOG_INF-0/00000000000057563616.log.deleted. (kafka.log.LogSegment)
kafka  | [2026-07-21 07:34:35,141] INFO Deleted offset index /kafka/kafka-logs-5cb6ac8d09b8/LOG_INF-0/00000000000057563616.index.deleted. (kafka.log.LogSegment)
kafka  | [2026-07-21 07:34:35,141] INFO Deleted time index /kafka/kafka-logs-5cb6ac8d09b8/LOG_INF-0/00000000000057563616.timeindex.deleted. (kafka.log.LogSegment)
kafka  | [2026-07-21 08:15:11,592] WARN [SocketServer listenerType=ZK_BROKER, nodeId=1001] Unexpected error from /89.37.172.140; closing connection (org.apache.kafka.common.network.Selector)
kafka  | org.apache.kafka.common.network.InvalidReceiveException: Invalid receive (size = 218762506 larger than 104857600)
kafka  | 	at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:105)
kafka  | 	at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:452)
kafka  | 	at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:402)
kafka  | 	at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:674)
kafka  | 	at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:576)
kafka  | 	at org.apache.kafka.common.network.Selector.poll(Selector.java:481)
kafka  | 	at kafka.network.Processor.poll(SocketServer.scala:989)
kafka  | 	at kafka.network.Processor.run(SocketServer.scala:892)
kafka  | 	at java.base/java.lang.Thread.run(Unknown Source)

一、核心问题:外部恶意扫描/探测攻击 ⚠️

你的 Kafka 端口(9092)暴露在了公网上,正在遭受来自外部 IP 的扫描探测。以下是涉及的可疑 IP:

IP 地址时间错误类型说明
45.33.12.12207-20 10:36InvalidReceiveException + InvalidRequestException多次尝试,发送畸形数据
205.210.31.7307-20 19:15InvalidReceiveException单次探测
89.37.172.14007-21 08:15InvalidReceiveException单次探测

具体分析

1. InvalidReceiveException: Invalid receive (size = XXX larger than 104857600)

攻击者向 Kafka 端口发送了非 Kafka 协议的数据(如 HTTP 请求、SSH 探测等)。Kafka 将收到的前 4 字节解读为消息长度,得到荒谬的超大值(如 1937006964、302055471),远超 socket.request.max.bytes(默认 100MB),因此主动断开连接。

2. InvalidRequestException: Unexpected api key: -173

45.33.12.122 发来的数据被 Kafka 勉强尝试解析,但 API key 为 -173(非法值),说明这根本不是 Kafka 协议请求,而是攻击者在尝试识别服务类型(端口扫描指纹识别)。

🚨 风险等级:高

  • 你的 Kafka Broker(nodeId=1001)直接暴露在公网
  • 虽然 Kafka 自身的协议校验阻止了这些请求,但这会:
    • 消耗网络/线程资源
    • 日志被大量垃圾信息污染
    • 一旦存在已知漏洞(如旧版本 CVE),可能被攻破

🔧 建议措施

# 1. 最重要:不要将 Kafka 端口暴露到公网!
#    使用防火墙限制 9092 端口只允许内网 IP 访问

# iptables 示例(仅允许内网)
iptables -A INPUT -p tcp --dport 9092 -s 172.22.0.0/16 -j ACCEPT
iptables -A INPUT -p tcp --dport 9092 -j DROP

# 2. 如果使用 Docker,绑定端口到内网接口
# 修改 docker-compose.yml 中的端口映射
ports:
  - "172.22.6.12:9092:9092"   # 只绑定内网 IP,而不是 0.0.0.0

# 3. 如果必须公网访问,启用 SASL/SSL 认证
# 在 server.properties 中配置:
# listeners=SASL_SSL://0.0.0.0:9093
# sasl.mechanism.inter.broker.protocol=SCRAM-SHA-256

二、正常操作:日志清理(无需关注)✅

以下日志是 Kafka 正常的日志保留策略执行,retention.ms=604800000(7 天):

Topic操作详情
LOG_INF-0滚动新段offset 68253900
LOG_INF-0删除过期段baseOffset=56231686, 57563616
LOG_APP-0滚动新段offset 16732667
LOG_APP-0删除过期段baseOffset=13328159

这些是正常的日志生命周期管理,无需处理。


三、总结

项目状态优先级
公网扫描/探测🚨 需立即处理P0
日志清理✅ 正常无需操作

最关键的行动:立即通过防火墙或 Docker 端口绑定,将 Kafka 9092 端口限制为仅内网可访问。如果你使用的是云服务商,优先配置安全组规则。

支持与分享

如果这篇文章对你有帮助,欢迎分享给更多人或者给予支持!

Kafka 日志分析报告20260721
/archives/kafka-ri-zhi-fen-xi-bao-gao-20260721
Author
Administrator
Published at
2026-07-21

欢迎来到我的博客!