---
title: Dynamic Kafka
weight: 3
type: docs
aliases:
- /dev/connectors/dynamic-kafka.html
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements.  See the NOTICE file
distributed with this work for additional information
regarding copyright ownership.  The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License.  You may obtain a copy of the License at

  http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied.  See the License for the
specific language governing permissions and limitations
under the License.
-->

# Dynamic Kafka Source _`Experimental`_

Flink provides an [Apache Kafka](https://kafka.apache.org) connector for reading data from Kafka topics from one or more Kafka clusters.
The Dynamic Kafka connector discovers the clusters and topics using a Kafka metadata service and can achieve reading in a dynamic fashion, facilitating changes in
topics and/or clusters, without requiring a job restart. This is especially useful when you need to read a new Kafka cluster/topic and/or stop reading
an existing Kafka cluster/topic (cluster migration/failover/other infrastructure changes) and when you need direct integration with Hybrid Source. The solution
makes these operations automated so that they are transparent to Kafka consumers.

## Dependency

For details on Kafka compatibility, please refer to the official [Kafka documentation](https://kafka.apache.org/protocol.html#protocol_compatibility).

{{< connector_artifact flink-connector-kafka kafka >}}

Flink's streaming connectors are not part of the binary distribution.
See how to link with them for cluster execution [here]({{< ref "docs/dev/configuration/overview" >}}).

## Dynamic Kafka Source
{{< hint info >}}
This part describes the Dynamic Kafka Source based on the new
[data source]({{< ref "docs/dev/datastream/sources.md" >}}) API.
{{< /hint >}}

### Usage

Dynamic Kafka Source provides a builder class to initialize the DynamicKafkaSource. The code snippet
below shows how to build a DynamicKafkaSource to consume messages from the earliest offset of the
stream "input-stream" and deserialize only the value of the
ConsumerRecord as a string, using "MyKafkaMetadataService" to resolve the cluster(s) and topic(s)
corresponding to "input-stream".

{{< tabs "DynamicKafkaSource" >}}
{{< tab "Java" >}}
```java

DynamicKafkaSource<String> source = DynamicKafkaSource.<String>builder()
    .setKafkaMetadataService(new MyKafkaMetadataService())
    .setStreamIds(Collections.singleton("input-stream"))
    .setEnumeratorMode(DynamicKafkaSourceOptions.EnumeratorMode.PER_CLUSTER)
    .setStartingOffsets(OffsetsInitializer.earliest())
    .setDeserializer(KafkaRecordDeserializationSchema.valueOnly(StringDeserializer.class))
    .setProperties(properties)
    .build();

env.fromSource(source, WatermarkStrategy.noWatermarks(), "Dynamic Kafka Source");
```
{{< /tab >}}
{{< tab "Python" >}}
```python
metadata_service = SingleClusterTopicMetadataService(
    "cluster-a",
    {"bootstrap.servers": "localhost:9092"})

source = DynamicKafkaSource.builder() \
    .set_kafka_metadata_service(metadata_service) \
    .set_stream_ids({"input-stream"}) \
    .set_starting_offsets(KafkaOffsetsInitializer.earliest()) \
    .set_value_only_deserializer(SimpleStringSchema()) \
    .set_properties(properties) \
    .build()

env.from_source(source, WatermarkStrategy.no_watermarks(), "Dynamic Kafka Source")
```
{{< /tab >}}
{{< /tabs >}}
The following properties are **required** for building a DynamicKafkaSource:

The Kafka metadata service, configured by setKafkaMetadataService(KafkaMetadataService)
The stream ids to subscribe, see the following Kafka stream subscription section for more details.
Deserializer to parse Kafka messages, see the [Kafka Source Documentation]({{< ref "docs/connectors/datastream/kafka" >}}#deserializer) for more details.

### Offsets 初始化

可以通过 builder 配置全局的起始/停止 offsets。起始 offsets 同时适用于有界与无界模式；停止 offsets 仅在有界模式下生效。Cluster metadata 可以可选地携带每个集群的起始/停止 offsets initializer；如果存在，将覆盖全局默认配置。

示例：在元数据中为不同集群设置偏移量初始化规则。

{{< tabs "DynamicKafkaSourceOffsets" >}}
{{< tab "Java" >}}
```java
Properties cluster0Props = new Properties();
cluster0Props.setProperty(
    CommonClientConfigs.BOOTSTRAP_SERVERS_CONFIG, "cluster0:9092");
Properties cluster1Props = new Properties();
cluster1Props.setProperty(
    CommonClientConfigs.BOOTSTRAP_SERVERS_CONFIG, "cluster1:9092");

KafkaStream stream =
    new KafkaStream(
        "input-stream",
        Map.of(
            "cluster0",
            new ClusterMetadata(
                Set.of("topic-a"),
                cluster0Props,
                OffsetsInitializer.earliest(),
                OffsetsInitializer.latest()),
            "cluster1",
            new ClusterMetadata(
                Set.of("topic-b"),
                cluster1Props,
                OffsetsInitializer.latest(),
                null)));

DynamicKafkaSource<String> source =
    DynamicKafkaSource.<String>builder()
        .setStreamIds(Set.of(stream.getStreamId()))
        .setKafkaMetadataService(new MockKafkaMetadataService(Set.of(stream)))
        .setDeserializer(KafkaRecordDeserializationSchema.valueOnly(StringDeserializer.class))
        // 如果元数据中包含每个集群的起始 offsets，将覆盖此处设置。
        .setStartingOffsets(OffsetsInitializer.earliest())
        .setBounded(OffsetsInitializer.latest())
        .build();
```
{{< /tab >}}
{{< /tabs >}}

### Watermark 对齐

当启用 watermark alignment 且一个 Dynamic Kafka reader 持有多个 split 时，Flink 可能会暂停或恢复单个 split，以便将各 split 的 watermark 控制在配置的 drift 范围内。Dynamic Kafka Source 会将这些针对 split 的 pause/resume 请求转发给底层 Kafka source readers，因此 split 级别的 watermark 对齐行为与常规 Kafka Source 一致。

### Kafka Stream Subscription
The Dynamic Kafka Source provides 2 ways of subscribing to Kafka stream(s).
* A set of Kafka stream ids. For example:
  {{< tabs "DynamicKafkaSource#setStreamIds" >}}
  {{< tab "Java" >}}
  ```java
  DynamicKafkaSource.builder().setStreamIds(Set.of("stream-a", "stream-b"));
  ```
  {{< /tab >}}
  {{< tab "Python" >}}
  ```python
  DynamicKafkaSource.builder().set_stream_ids({"stream-a", "stream-b"})
  ```
  {{< /tab >}}
  {{< /tabs >}}
* A regex pattern that subscribes to all Kafka stream ids that match the provided regex. For example:
  {{< tabs "DynamicKafkaSource#setStreamPattern" >}}
  {{< tab "Java" >}}
  ```java
  DynamicKafkaSource.builder().setStreamPattern(Pattern.of("stream.*"));
  ```
  {{< /tab >}}
  {{< tab "Python" >}}
  ```python
  DynamicKafkaSource.builder().set_stream_pattern("stream.*")
  ```
  {{< /tab >}}
  {{< /tabs >}}

### Split 分配模式

Dynamic Kafka Source 支持两种 split 分配模式：

* `per_cluster`（默认）：在每个 Kafka 集群内独立进行 split 分配。
* `global`：在所有已发现的 Kafka 集群范围内统一做全局负载均衡分配。

你可以通过 builder API 或 source properties 来配置该模式。

新发现 split 的 owner 计算方式如下：

* `per_cluster`：使用 `KafkaSourceEnumerator` 相同的 owner 逻辑。
  对于 topic 分区 `(topic, partition)`，令 `numReaders = P`：
  `startIndex = ((topic.hashCode() * 31) & 0x7FFFFFFF) % P`，
  `owner = (startIndex + partition) % P`。
* `global`：在所有集群共享一个全局 owner 游标：
  `owner = knownActiveSplitIds.size() % numReaders`，
  然后将该 split id 加入 `knownActiveSplitIds`。
  （当 split 通过 `addSplitsBack` 返回时，若原 owner 仍然有效，则优先复用该 owner。）

在 `global` 模式下，均衡策略是**前向增量（forward-only）**的：新发现 split 会尽量保证后续分配均衡，
但不会仅为重平衡主动迁移已分配且仍在消费的 active split。

如果因为缩容/移除导致 global 分配出现倾斜，enumerator 不会自行重排已在运行的 split。
如需对已有 ownership 做重平衡，可通过并行度变化后的恢复流程（例如 savepoint/checkpoint + rescale restore），
让 Flink Runtime 对 source reader 的 operator state 进行重分区。

{{< tabs "DynamicKafkaSourceEnumeratorMode" >}}
{{< tab "Java" >}}
```java
DynamicKafkaSource<String> source =
    DynamicKafkaSource.<String>builder()
        .setKafkaMetadataService(new MyKafkaMetadataService())
        .setStreamIds(Set.of("input-stream"))
        .setEnumeratorMode(DynamicKafkaSourceOptions.EnumeratorMode.GLOBAL)
        .setDeserializer(KafkaRecordDeserializationSchema.valueOnly(StringDeserializer.class))
        .build();
```
{{< /tab >}}
{{< /tabs >}}

### Kafka Metadata Service

An interface is provided to resolve the logical Kafka stream(s) into the corresponding physical
topic(s) and cluster(s). Typically, these implementations are based on services that align well
with internal Kafka infrastructure--if that is not available, an in-memory implementation
would also work. An example of in-memory implementation can be found in our tests.

This source achieves its dynamic characteristic by periodically polling this Kafka metadata service
for any changes to the Kafka stream(s) and reconciling the reader tasks to subscribe to the new
Kafka metadata returned by the service. For example, in the case of a Kafka migration, the source would
swap from one cluster to the new cluster when the service makes that change in the Kafka stream metadata.

Cluster metadata 可以包含每个集群的起始/停止 offsets initializer，用于覆盖全局 builder 配置。

### Additional Properties
There are configuration options in DynamicKafkaSourceOptions that can be configured in the properties through the builder:
<table class="table table-bordered">
    <thead>
      <tr>
      <th class="text-left" style="width: 25%">Option</th>
      <th class="text-center" style="width: 8%">Required</th>
      <th class="text-center" style="width: 7%">Default</th>
      <th class="text-center" style="width: 10%">Type</th>
      <th class="text-center" style="width: 50%">Description</th>
    </tr>
    </thead>
    <tbody>
    <tr>
      <td><h5>stream-metadata-discovery-interval-ms</h5></td>
      <td>required</td>
      <td style="word-wrap: break-word;">-1</td>
      <td>Long</td>
      <td>The interval in milliseconds for the source to discover the changes in stream metadata. A non-positive value disables the stream metadata discovery.</td>
    </tr>
    <tr>
      <td><h5>stream-metadata-discovery-failure-threshold</h5></td>
      <td>required</td>
      <td style="word-wrap: break-word;">1</td>
      <td>Integer</td>
      <td>The number of consecutive failures before letting the exception from Kafka metadata service discovery trigger jobmanager failure and global failover. The default is one to at least catch startup failures.</td>
    </tr>
    <tr>
      <td><h5>stream-enumerator-mode</h5></td>
      <td>required</td>
      <td style="word-wrap: break-word;">per_cluster</td>
      <td>String</td>
      <td>Dynamic Kafka split 分配所使用的 Enumerator 实现。支持 <code>per_cluster</code>（集群内独立分配）和 <code>global</code>（跨集群全局均衡分配）。</td>
    </tr>
    </tbody>
</table>


In addition to this list, see the [regular Kafka connector]({{< ref "docs/connectors/datastream/kafka" >}}#additional-properties) for
a list of applicable properties.

### Metrics

<table class="table table-bordered">
  <thead>
    <tr>
      <th class="text-left" style="width: 15%">Scope</th>
      <th class="text-left" style="width: 18%">Metrics</th>
      <th class="text-left" style="width: 18%">User Variables</th>
      <th class="text-left" style="width: 39%">Description</th>
      <th class="text-left" style="width: 10%">Type</th>
    </tr>
  </thead>
  <tbody>
    <tr>
        <th rowspan="8">Operator</th>
        <td>currentEmitEventTimeLag</td>
        <td>n/a</td>
        <td>The time span from the record event timestamp to the time the record is emitted by the source connector¹: <code>currentEmitEventTimeLag = EmitTime - EventTime.</code></td>
        <td>Gauge</td>
    </tr>
    <tr>
        <td>watermarkLag</td>
        <td>n/a</td>
        <td>The time span that the watermark lags behind the wall clock time: <code>watermarkLag = CurrentTime - Watermark</code></td>
        <td>Gauge</td>
    </tr>
    <tr>
        <td>sourceIdleTime</td>
        <td>n/a</td>
        <td>The time span that the source has not processed any record: <code>sourceIdleTime = CurrentTime - LastRecordProcessTime</code></td>
        <td>Gauge</td>
    </tr>
    <tr>
        <td>pendingRecords</td>
        <td>n/a</td>
        <td>The number of records that have not been fetched by the source. e.g. the available records after the consumer offset in a Kafka partition.</td>
        <td>Gauge</td>
    </tr>
    <tr>
      <td>kafkaClustersCount</td>
      <td>n/a</td>
      <td>The total number of Kafka clusters read by this reader.</td>
      <td>Gauge</td>
    </tr>
  </tbody>
</table>

In addition to this list, see the [regular Kafka connector]({{< ref "docs/connectors/datastream/kafka" >}}#monitoring) for
the KafkaSourceReader metrics that are also reported.

### Additional Details

For additional details on deserialization, event time and watermarks, idleness, consumer offset
committing, security, and more, you can refer to the [Kafka Source documentation]({{< ref "docs/connectors/datastream/kafka" >}}#kafka-source). This is possible because the
Dynamic Kafka Source leverages components of the Kafka Source, and the implementation will be
discussed in the next section.

### Behind the Scene
{{< hint info >}}
If you are interested in how Kafka source works under the design of new data source API, you may
want to read this part as a reference. For details about the new data source API,
[documentation of data source]({{< ref "docs/dev/datastream/sources.md" >}}) and
<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface">FLIP-27</a>
provide more descriptive discussions.
{{< /hint >}}


Under the abstraction of the new data source API, Dynamic Kafka Source consists of the following components:
#### Source Split
A source split in Dynamic Kafka Source represents a partition of a Kafka topic, with cluster information. It
consists of:
* A Kafka cluster id that can be resolved by the Kafka metadata service.
* A Kafka Source Split (TopicPartition, starting offset, stopping offset).

You can check the class `DynamicKafkaSourceSplit` for more details.

#### Split Enumerator

This enumerator is responsible for discovering and assigning splits from one or more clusters. At startup, the
enumerator will discover metadata belonging to the Kafka stream ids. Using the metadata, it can
initialize KafkaSourceEnumerators to handle the functions of assigning splits to the readers. In addition,
source events will be sent to the source reader to reconcile the metadata. This enumerator has the ability to poll the
KafkaMetadataService, periodically for stream discovery. In addition, restarting enumerators when metadata changes involve
clearing outdated metrics since clusters may be removed and so should their metrics.

#### Source Reader

This reader is responsible for reading from one or more clusters and using the KafkaSourceReader to fetch
records from topics and clusters based on the metadata. When new metadata is discovered by the enumerator,
the reader will reconcile metadata changes to possibly restart the KafkaSourceReader to read from the new
set of topics and clusters.

#### Kafka Metadata Service

This interface represents the source of truth for the current metadata for the configured Kafka stream ids.
Metadata that is removed in between polls is considered non-active (e.g. removing a cluster from the
return value, means that a cluster is non-active and should not be read from). The cluster metadata
contains an immutable Kafka cluster id, the set of topics, and properties needed to connect to the
Kafka cluster.

#### FLIP 246

To understand more behind the scenes, please read [FLIP-246](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=217389320)
for more details and discussion.

{{< top >}}
