
---
title: HTTP
weight: 3
type: docs
aliases:
- /dev/table/connectors/http.html
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements.  See the NOTICE file
distributed with this work for additional information
regarding copyright ownership.  The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License.  You may obtain a copy of the License at

  http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied.  See the License for the
specific language governing permissions and limitations
under the License.
-->

# Apache HTTP SQL Connector

{{< label "Sink: Streaming Append Mode" >}}
{{< label "Lookup Source: Sync Mode" >}}
{{< label "Lookup Source: Async Mode" >}}
{{< label "Sink: Batch" >}}

The HTTP connector allows for pulling data from an external system via HTTP methods and HTTP Sink that allows for sending data to an external system via HTTP requests.

The HTTP source connector supports [Lookup Joins](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sourcessinks/#lookup-table-source) in [Table API and SQL](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/overview/).

<!-- TOC -->
* [HTTP Connector](#http-connector)
  * [Dependencies](#dependencies)
  * [Migration from GetInData HTTP connector](#migration-from-getindata-http-connector)
  * [Working with HTTP lookup source tables](#working-with-http-lookup-source-tables)
    * [HTTP Lookup Table API and SQL Source example](#http-lookup-table-api-and-sql-source-example)
    * [Using a HTTP Lookup Source in a lookup join](#using-a-http-lookup-source-in-a-lookup-join)
    * [Lookup Source Connector Options](#lookup-source-connector-options)
    * [Query Creators](#query-creators)
    * [http-generic-json-url Query Creator](#http-generic-json-url-query-creator)
    * [Format considerations](#format-considerations)
      * [For HTTP requests](#for-http-requests)
      * [For HTTP responses](#for-http-responses)
    * [Default Query Creator Implementation](#default-query-creator-implementation)
    * [Http headers](#http-headers)
    * [Timeouts](#timeouts)
    * [Source table HTTP status code](#source-table-http-status-code)
    * [Retries and handling errors (Lookup source)](#retries-and-handling-errors-lookup-source)
        * [Retry strategy](#retry-strategy)
      * [Lookup multiple results](#lookup-multiple-results)
  * [Working with HTTP sink tables](#working-with-http-sink-tables)
    * [HTTP Sink](#http-sink)
    * [Sink Connector Options](#sink-connector-options)
    * [Sink table HTTP status codes](#sink-table-http-status-codes)
    * [Request submission](#request-submission)
    * [Batch submission mode](#batch-submission-mode)
    * [Single submission mode](#single-submission-mode)
  * [Available Metadata](#available-metadata)
    * [http-completion-state possible values](#http-completion-state-possible-values)
  * [HTTP status code handler](#http-status-code-handler)
  * [Security considerations](#security-considerations)
    * [TLS (more secure replacement for SSL) and mTLS support](#tls-more-secure-replacement-for-ssl-and-mtls-support)
    * [Basic Authentication](#basic-authentication)
    * [OIDC Bearer Authentication](#oidc-bearer-authentication)
  * [Logging the HTTP content](#logging-the-http-content)
      * [Restrictions at this time](#restrictions-at-this-time)
<!-- TOC -->
## Dependencies

{{< sql_connector_download_table "http" >}}

The HTTP connector is not part of the binary distribution.
See how to link with it for cluster execution [here]({{< ref "docs/dev/configuration/overview" >}}).

## Migration from GetInData HTTP connector

The GetInData HTTP connector was donated to Flink in [FLIP-532](https://cwiki.apache.org/confluence/display/FLINK/FLIP-532%3A+Donate+GetInData+HTTP+Connector+to+Flink). The Flink connector has the same capabilities as the original connector.
The Flink connector does have some changes that you need to be aware of if you are migrating from using the original connector:

* Existing java applications will need to be recompiled to pick up the new flink package names.
* Existing application and SQL need to be amended to use the new connector option names. The new option names do not have
the _com.getindata.http_ prefix, the prefix is now _http_.
* The name of the connector and the identifiers of components that are discovered have been changed, so that the GetInData jar file can co-exist
with this connector's jar file. Be aware that if you have created custom pluggable components; you will need to recompile against this connector.
* Note that the `http-generic-json-url` query creator now processes HTTP bodies differently using `http.request.body-template`.
* Note that if you were incorrectly using `gid.connector.http.request.query-param-fields` with POST or PUT did not give an error. This connector corrects the behaviour so specifying `http.request.query-param-fields` with POST or PUT does give an error.
* The GetInData HTTP connector was built against Flink version 1, so works with that level of Flink and also Flink version 2. This connector is built against and supports Flink 2.2.

## Working with HTTP lookup source tables

### HTTP Lookup Table API and SQL Source example
Here is an example Flink SQL Enrichment Lookup Table definition:

```roomsql
CREATE TABLE Customers (
    id STRING,
    id2 STRING,
    msg STRING,
    uuid STRING,
    details ROW<
      isActive BOOLEAN,
      nestedDetails ROW<
        balance STRING
      >
    >
) WITH (
'connector' = 'http',
'format' = 'json',
'url' = 'http://localhost:8080/client',
'asyncPolling' = 'true'
)
```

### Using a HTTP Lookup Source in a lookup join

To easily see how the lookup enrichment works, we can define a data source using datagen:
```roomsql
CREATE TABLE Orders (
    id STRING,
    id2 STRING,
    proc_time AS PROCTIME()
) WITH (
'connector' = 'datagen',
'rows-per-second' = '1',
'fields.id.kind' = 'sequence',
'fields.id.start' = '1',
'fields.id.end' = '120',
'fields.id2.kind' = 'sequence',
'fields.id2.start' = '2',
'fields.id2.end' = '120'
);
```

Then we can enrich the _Orders_ table with the _Customers_ HTTP table with the following SQL:

```roomsql
SELECT o.id, o.id2, c.msg, c.uuid, c.details.isActive, c.details.nestedDetails.balance FROM Orders AS o
JOIN Customers FOR SYSTEM_TIME AS OF o.proc_time AS c ON o.id = c.id AND o.id2 = c.id2
```

The columns and their values used for JOIN `ON` condition will be used as HTTP GET parameters where the column name will be used as a request parameter name.

For Example:
``
http://localhost:8080/client/service?id=1&uuid=2
``

Or for REST POST method they will be converted to Json and used as request body. In this case, json request body will look like this:
```json
{
    "id": "1",
    "uuid": "2"
}
```

### Lookup Source Connector Options

Note the options with the prefix _http_ are the HTTP connector specific options, the others are Flink options.

| Option                                                                 | Required | Description/Value                                                                                                                                                                                                                                                                                                                                                                                      |
|:-----------------------------------------------------------------------|----------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| connector                                                              | required | The Value should be set to _http_                                                                                                                                                                                                                                                                                                                                                                      |
| format                                                                 | required | Flink's format name that should be used to decode REST response, Use `json` for a typical REST endpoint.                                                                                                                                                                                                                                                                                               |
| url                                                                    | required | The base URL that should be used for GET requests. For example _http://localhost:8080/client_                                                                                                                                                                                                                                                                                                          |
| asyncPolling                                                           | optional | true/false - determines whether Async Polling should be used. Mechanism is based on Flink's Async I/O.                                                                                                                                                                                                                                                                                                 |
| lookup-method                                                          | optional | GET/POST/PUT (and any other) - determines what REST method should be used for lookup REST query. If not specified, `GET` method will be used.                                                                                                                                                                                                                                                          |
| lookup.cache                                                           | optional | Enum possible values: `NONE`, `PARTIAL`. The cache strategy for the lookup table. Currently supports `NONE` (no caching) and `PARTIAL` (caching entries on lookup operation in external API).                                                                                                                                                                                                          |
| lookup.partial-cache.max-rows                                          | optional | The max number of rows of lookup cache, over this value, the oldest rows will be expired. `lookup.cache` must be set to `PARTIAL` to use this option. See the following <a href="#lookup-cache">Lookup Cache</a> section for more details.                                                                                                                                                             |
| lookup.partial-cache.expire-after-write                                | optional | The max time to live for each rows in lookup cache after writing into the cache. Specify as a [Duration](https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/deployment/config/#duration).  `lookup.cache` must be set to `PARTIAL` to use this option. See the following <a href="#lookup-cache">Lookup Cache</a> section for more details.                                               |
| lookup.partial-cache.expire-after-access                               | optional | The max time to live for each rows in lookup cache after accessing the entry in the cache. Specify as a [Duration](https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/deployment/config/#duration). `lookup.cache` must be set to `PARTIAL` to use this option. See the following <a href="#lookup-cache">Lookup Cache</a> section for more details.                                      |
| lookup.partial-cache.cache-missing-key                                 | optional | This is a boolean that defaults to true. Whether to store an empty value into the cache if the lookup key doesn't match any rows in the table. `lookup.cache` must be set to `PARTIAL` to use this option. See the following <a href="#lookup-cache">Lookup Cache</a> section for more details.                                                                                                        |
| lookup.max-retries                                                     | optional | The max retry times if the lookup failed; default is 3. See the following <a href="#lookup-cache">Lookup Cache</a> section for more detail. Set value 0 to disable retries.                                                                                                                                                                                                                            |
| http.security.cert.server                                              | optional | Comma separated paths to trusted HTTP server certificates that should be added to the connectors trust store.                                                                                                                                                                                                                                                                                          |
| http.security.cert.client                                              | optional | Path to trusted certificate that should be used by connector's HTTP client for mTLS communication.                                                                                                                                                                                                                                                                                                     |
| http.security.key.client                                               | optional | Path to trusted private key that should be used by connector's HTTP client for mTLS communication.                                                                                                                                                                                                                                                                                                     |
| http.security.cert.server.allowSelfSigned                              | optional | Accept untrusted certificates for TLS communication.                                                                                                                                                                                                                                                                                                                                                   |
| http.security.oidc.token.request                                       | optional | OIDC `Token Request` body in `application/x-www-form-urlencoded` encoding                                                                                                                                                                                                                                                                                                                              |
| http.security.oidc.token.endpoint.url                                  | optional | OIDC `Token Endpoint` url, to which the token request will be issued                                                                                                                                                                                                                                                                                                                                   |
| http.security.oidc.token.expiry.reduction                              | optional | OIDC tokens will be requested if the current time is later than the cached token expiry time minus this value.                                                                                                                                                                                                                                                                                         |
| http.source.lookup.continue-on-error                                   | optional | When true, the flow will continue on errors, returning row content. When false (the default) the job ends on errors.                                                                                                                                                                                                                                                                                   |
| http.source.lookup.request.timeout                                     | optional | Sets HTTP request timeout for the lookup source as a Duration (e.g. `'30s'`, `'1min'`). Controls how long the HTTP client waits for a response before timing out a single request. If not specified, the default value of `30s` will be used.                                                                                                                                                          |
| http.source.lookup.request.thread-pool.size                            | optional | Sets the size of pool thread for HTTP lookup request processing. Increasing this value would mean that more concurrent requests can be processed in the same time. If not specified, the default value of 8 threads will be used.                                                                                                                                                                      |
| http.source.lookup.response.thread-pool.size                           | optional | Sets the size of pool thread for HTTP lookup response processing. Increasing this value would mean that more concurrent requests can be processed in the same time. If not specified, the default value of 4 threads will be used.                                                                                                                                                                     |
| http.source.lookup.use-raw-authorization-header                        | optional | If set to `'true'`, uses the raw value set for the `Authorization` header, without transformation for Basic Authentication (base64, addition of "Basic " prefix). If not specified, defaults to `'false'`.                                                                                                                                                                                             |
| http.source.lookup.request-callback                                    | optional | Specify which `HttpLookupPostRequestCallback` implementation to use. By default, it is set to `slf4j-lookup-logger` corresponding to `Slf4jHttpLookupPostRequestCallback`.                                                                                                                                                                                                                             |
| http.source.lookup.connection.timeout                                  | optional | Source table connection timeout. Default - no value.                                                                                                                                                                                                                                                                                                                                                   |
| http.source.lookup.http-version                                        | optional | Version of HTTP to use for lookup http requests. The valid values are HTTP_1_1 and HTTP_2, which specify HTTP 1.1 or 2 respectively. This option may be required as HTTP_1_1, if the endpoint is HTTP 1.1, because some http 1.1 endpoints reject HTTP Version 2 calls, with 'Invalid HTTP request received' and 'HTTP/2 upgrade not supported'.                                                       |
| http.source.lookup.success-codes                                       | optional | Comma separated http codes considered as success response. Use [1-5]XX for groups and '!' character for excluding. The default is 2XX.                                                                                                                                                                                                                                                                 |
| http.source.lookup.retry-codes                                         | optional | Comma separated http codes considered as transient errors. Use [1-5]XX for groups and '!' character for excluding. The default is 500,503,504.                                                                                                                                                                                                                                                         |
| http.source.lookup.ignored-response-codes                              | optional | Comma separated http codes. Content for these responses will be ignored. Use [1-5]XX for groups and '!' character for excluding. Ignored responses together with `http.source.lookup.success-codes` are considered as successful.                                                                                                                                                                      |
| http.source.lookup.retry-strategy.type                                 | optional | Auto retry strategy type: fixed-delay (default) or exponential-delay.                                                                                                                                                                                                                                                                                                                                  |
| http.source.lookup.retry-strategy.fixed-delay.delay                    | optional | Fixed-delay interval between retries. Default 1 second. Use with`lookup.max-retries` parameter.                                                                                                                                                                                                                                                                                                        |
| http.source.lookup.retry-strategy.exponential-delay.initial-backoff    | optional | Exponential-delay initial delay. Default 1 second.                                                                                                                                                                                                                                                                                                                                                     |
| http.source.lookup.retry-strategy.exponential-delay.max-backoff        | optional | Exponential-delay maximum delay. Default 1 minute. Use with `lookup.max-retries` parameter.                                                                                                                                                                                                                                                                                                            |
| http.source.lookup.retry-strategy.exponential-delay.backoff-multiplier | optional | Exponential-delay multiplier. Default value 1.5                                                                                                                                                                                                                                                                                                                                                        |
| http.source.lookup.proxy.host                                          | optional | Specify the hostname of the proxy.                                                                                                                                                                                                                                                                                                                                                                     |
| http.source.lookup.proxy.port                                          | optional | Specify the port of the proxy.                                                                                                                                                                                                                                                                                                                                                                         |
| http.source.lookup.proxy.username                                      | optional | Specify the username used for proxy authentication.                                                                                                                                                                                                                                                                                                                                                    |
| http.source.lookup.proxy.password                                      | optional | Specify the password used for proxy authentication.                                                                                                                                                                                                                                                                                                                                                    |
| http.request.body-template                                             | optional | Used for the `http-generic-json-url` query creator. A JSON template string for constructing the request body for PUT and POST operations. Use `{{fieldName}}` placeholders to reference top-level columns from the lookup table. Supports creating complex nested JSON structures with both placeholders and literal values. See the [Body Template](#body-template) section for details and examples. |
| http.request.url-map                                                   | optional | Used for the `http-generic-json-url` query creator. The map of insert names to column names used as url segments or query parameters.                                                                                                                                                                                                                                                                  |

### Query Creators

In the above example we see that HTTP GET operations and HTTP POST operations result in different mapping of the columns to the
HTTP request content. In reality, you will want to have more control over how the SQL columns are mapped to the HTTP content.
The HTTP connector supplies a number of Query Creators that you can use to define these mappings.

<table class="table table-bordered">
    <thead>
    <tr>
      <th class="text-left" style="width: 76%">Name</th>
      <th class="text-center" style="width: 8%">Query param mapping</th>
  <th class="text-center" style="width: 8%">URL path mapping</th>
  <th class="text-center" style="width: 8%">Body mapping</th>
    </tr>
    </thead>
    <tbody>
    <tr>
      <td><h5>http-generic-json-url</h5></td>
      <td>✓</td>
      <td>✓</td>
      <td>✓</td>
    </tr>
    <tr>
      <td><h5>http-generic-get-query</h5></td>
      <td>✓ for GETs</td>
      <td></td>
      <td>✓ for PUTs and POSTs</td>
    </tr>

    </tbody>
</table>

### http-generic-json-url Query Creator

The recommended Query creator for json is called _http-generic-json-url_, which allows column content to be mapped as URL, path, body and query parameter request values; it supports
POST, PUT and GET operations. This query creator allows you to issue json requests without needing to code
your own custom http connector. The mappings from columns to the json request are supplied in the query creator configuration
parameters `http.request.body-template` and `http.request.url-map`.
#### Mapping the URL

The `http.request.url-map` option provides a flexible way to map table columns to parts of the URL, either URL segments or HTTP query parameters.
 Parses a string as a map of strings. For example if there are table columns called `customerId` and `orderId`,
then specifying value `customerId:cid,orderID:oid` and a url of https://myendpoint/customers/{{cid}}?orders={{oid}} will mean that the url used for the
lookup query will dynamically pickup the values for `customerId`, `orderId` and use them in the url e.g. https://myendpoint/customers/cid1?orders=oid1.
The expected format of the map is: `key1:value1,key2:value2`.

As these values are being supplied as URL segments or part or query parameters, the connector url encodes that content so characters like spaces
do not appear invalidly in the URL. In the case where the complete url is the insert then url encoding is not performed; the url needs to be valid and already
properly url encoded as appropriate.

**Example Scenario around clashing request and response columns:**

If your API response has a field `customer` defined as an object (complex type), but you need to send a customer ID as a query parameter with the same name `customer`, you can:

1. Define a new string type column `qp_customer` in your table for the request parameter
2. Keep the `customer` column for the response object
3. Map the request column to the query parameter:

```sql
CREATE TABLE CustomerLookup (
    qp_customer STRING,           -- Request: customer ID as string
    qp_order STRING,              -- Request: order ID as string
    customer ROW<                 -- Response: customer object
        id STRING,
        name STRING
    >,
    order ROW<                    -- Response: order object
        id STRING,
        total DECIMAL
    >
) WITH (
    'connector' = 'http',
    'format' = 'json',
    'url' = 'http://api.example.com/lookup?customer={{qp_customer}}&order={{qp_order}}',
    'lookup-method' = 'GET',
    'http.request.url-map' = 'qp_customer:qp_customer,qp_order:qp_order'
)
```

In this example, when you join with `qp_customer='C123'` and `qp_order='O456'`, the HTTP request will be:
```
GET http://api.example.com/lookup?customer=C123&order=O456
```

The response will populate the `customer` and `order` complex objects.

#### Body Template

The `http.request.body-template` option provides a flexible way to construct HTTP request bodies for PUT and POST operations using a JSON template with placeholders.

**Template Syntax:**
- Use `{{fieldName}}` placeholders to reference top-level columns from the lookup table
- Placeholders are replaced with actual values from the lookup row at runtime
- Supports creating complex nested JSON structures with both placeholders and literal values
- All placeholders must reference existing top-level columns in the table schema

**Simple Example:**
```sql
'http.request.body-template' = '{"id": {{id}}, "name": {{name}}}'
```
If the lookup row has `id=123` and `name="John"`, the request body will be `{"id": "123", "name": "John"}`.

**Nested Structure Example:**
```sql
'http.request.body-template' = '{
  "user": {
    "id": {{userId}},
    "profile": {
      "name": {{userName}},
      "email": {{userEmail}}
    }
  },
  "source": "flink",
  "version": "1.0"
}'
```
This creates a nested JSON structure where `{{userId}}`, `{{userName}}`, and `{{userEmail}}` are replaced with values from the lookup table, while `"source"` and `"version"` are literal values.

**Array Example:**
```sql
'http.request.body-template' = '{
  "request": {
    "ids": [{{id1}}, {{id2}}],
    "metadata": {
      "tags": ["lookup", "flink"]
    }
  }
}'
```

**Important Notes:**
- Placeholders must reference top-level columns only (e.g., `{{userId}}` not `{{user.id}}`)
- The template must be valid JSON with placeholders
- All referenced fields must exist in the table schema
- Field values are properly JSON-encoded (strings are quoted, numbers/booleans are not)
- Can only be used with POST/PUT methods (not GET - use query parameters for GET requests)
- Note that all API response fields should match the name structure and type of Table defined columns.

**Using http-generic-json-url Query Creator in your flow**

This query creator allows you to populate API calls very flexibly. To do this effectively follow the below guidance:

1) All lookup join keys need to be at the top level.
2) The lookup connector will only see runtime content that is a lookup join key.
3) Everything we want to put into the body, query params, or paths needs to be lookup join keys.
4) For paths and query params, these columns need to be defined at the top level of the HTTP table.
5) For constants that might be required in the query params, define them as query parameters in the URL, e.g., suffix the URL with `?myParam=myvalue`.
6) For constants required in the path, hard code them as URL segments.
7) The join key may be needed in the body. The `http.request.body-template` allows you to populate the body as required, including nested levels, using the template. It also allows you to specify content required to be the same for every call.
8) If you start from an OpenAPI specification that contains nested content required as a lookup join key, then use `http.request.body-template` to map top-level columns into that structure.
9) Response content is mapped to matching named top-level columns in the lookup table. You should arrange your table columns so that some are request columns (all top level) and some are response columns.
10) Use single quotes for the value of `http.request.body-template` so you do not need to escape the double quotes, and add newline characters for readability.
11) If you want to enrich every event with the same API content, you can specify a placeholder as the complete URL the `url`, then use `http.request.url-map` to map it. In this scenario switching on caching is advised to avoid repeated identical API calls.
12) Note that columns in SQL tables (the DDL) do not have a natural way to distinguish between request and response fields. Where possible, use the API field name as column names in the DDL; this minimizes the number of columns you need to define.
13) The exception to 12) is when a response API field name is the same as a request API field **and** they have incompatible types. In this case, define the request column with a different name, then use `http.request.query-param-fields-with-key`, `http.request.body-template`, and/or `http.request.url-map` to provide the mapping to the API field.
14) Note the columns representing the response are those that should be used for enrichment.

### Format considerations

#### For HTTP requests
In order to use a custom format, users have to specify the option `'lookup-request.format' = '{customFormatName}'`, where `{customFormatName}` is the identifier of the custom format factory.
Additionally, it is possible to pass custom query format options from table's DDL.
This can be done by: `'lookup-request.format.{customFormatName}.{customFormatProperty}' = '{propertyValue}'`, where {customFormatProperty} is the name of a custom
property and {propertyValue} is the property value.
For example:
`'lookup-request.format.myCustomFormatName.foo' = 'baa'`.

With the default configuration, flink-Json format is used for `GenericGetQueryCreator`; all options defined in [json-format](https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/formats/json/) can be passed through the table DDL.
For example:
`'lookup-request.format.json.fail-on-missing-field' = 'true'`.

#### For HTTP responses
Specify your format options at the top level. For example:
```roomsql
       'format' = 'json',
       'json.ignore-parse-errors' = 'true',
```

### Default Query Creator Implementation

The default Query Creator is called _http-generic-json-url_. For body based queries such as POST/PUT requests, the GenericGetQueryCreator is provided as a default query creator. This implementation uses Flink's [json-format](https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/formats/json/) to convert RowData object into Json String.
For GET requests it can be used for query parameter based queries.

The _http-generic-json-url_ allows for using custom formats that will perform serialization to Json. Thanks to this, users can create their own logic for converting RowData to Json Strings suitable for their HTTP endpoints and use this logic as custom format
with HTTP Lookup connector and SQL queries.
To create a custom format user has to implement Flink's `SerializationSchema` and `SerializationFormatFactory` interfaces and register a custom format factory alongside other factories in
`resources/META-INF.services/org.apache.flink.table.factories.Factory` file. This is common Flink mechanism for providing custom implementations for various factories.

### Http headers
It is possible to set HTTP headers that will be added to HTTP request sent by lookup source connector.
Headers are defined via property key `http.source.lookup.header.HEADER_NAME = header value` for example:
`http.source.lookup.header.X-Content-Type-Options = nosniff`.

Headers can be set using http lookup source table DDL. In example below, HTTP request done for `http-lookup` table will contain three headers:
- `Origin`
- `X-Content-Type-Options`
- `Content-Type`

```roomsql
CREATE TABLE http-lookup (
  id bigint,
  some_field string
) WITH (
  'connector' = 'http',
  'format' = 'json',
  'url' = 'http://localhost:8080/client',
  'asyncPolling' = 'true',
  'http.source.lookup.header.Origin' = '*',
  'http.source.lookup.header.X-Content-Type-Options' = 'nosniff',
  'http.source.lookup.header.Content-Type' = 'application/json'
)
```

Note that when using OIDC, it adds an `Authentication` header with the bearer token; this will override
an existing `Authorization` header specified in configuration.

### Timeouts
Lookup Source is guarded by two timeout timers. First one is specified by Flink's AsyncIO operator that executes `AsyncTableFunction`.
The default value of this timer is set to 3 minutes and can be changed via `table.exec.async-lookup.timeout` [option](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/config/#table-exec-async-lookup-timeout).

The second one is set per individual HTTP requests by HTTP client. Its default value is `30s` and can be changed via `http.source.lookup.request.timeout` option (accepts a Duration value, e.g. `'30s'` or `'1min'`).

Flink's current implementation of `AsyncTableFunction` does not allow specifying custom logic for handling Flink AsyncIO timeouts as it is for Java API.
Because of that, if AsyncIO timer passes, Flink will throw TimeoutException which will cause job restart.

### Source table HTTP status code
The source table categorizes HTTP responses into three groups based on status codes:
- Retry codes (`http.source.lookup.retry-codes`):
  Responses in this group indicate a temporary issue (it can be e.g., HTTP 503 Service Unavailable). When such a response is received, the request should be retried.
- Success codes (`http.source.lookup.success-codes`):
  These are expected responses that should be processed by table function.
- Ignored responses (`http.source.lookup.ignored-response-codes`):
  Successful response, but its content will be ignored. For example, an HTTP 404 Not Found response is valid and indicates that the requested item does not exist, so its content can be ignored.
- Error codes:
  Any response code that is not classified as a retry or success code falls into this category. Receiving such a response will result in a job failure.

### Retries and handling errors (Lookup source)
Lookup source handles auto-retries for two scenarios:
1. IOException occurs (e.g. temporary network outage)
2. The response contains a HTTP error code that indicates a retriable error. These codes are defined in the table configuration (see `http.source.lookup.retry-codes`).
   Retries are executed silently, without restarting the job.

Notice that HTTP codes are categorized into 3 groups:
- successful responses - response is returned immediately for further processing
- temporary errors - request will be retried up to the retry limit
- error responses - unexpected responses are not retried. Any HTTP error code which is not configured as successful or temporary error is treated as an unretriable error.

For temporary errors that have reached max retries attempts (per request) and error responses, the operation will
succeed if `http.source.lookup.continue-on-error` is true, otherwise the job will fail.

##### Retry strategy
Users can choose a retry strategy type for source table:
- fixed-delay - http request will be re-sent after specified delay.
- exponential-delay - request will be re-sent with exponential backoff strategy, limited by `lookup.max-retries` attempts. The delay for each retry is calculated as the previous attempt's delay multiplied by the backoff multiplier (parameter `http.source.lookup.retry-strategy.exponential-delay.backoff-multiplier`) up to `http.source.lookup.retry-strategy.exponential-delay.max-backoff`. The initial delay value is defined in the table configuration as `http.source.lookup.retry-strategy.exponential-delay.initial-backoff`.


#### Lookup multiple results

Typically, join can return zero, one or more results. What is more, there are lots of possible REST API designs and
pagination methods. Currently, the connector supports only two simple approaches (`http.source.lookup.result-type`):

- `single-value` - REST API returns single object.
- `array` - REST API returns array of objects. Pagination is not supported yet.

## Working with HTTP sink tables

### HTTP Sink
The following example shows the minimum Table API example to create a sink:

```roomsql
CREATE TABLE http (
  id bigint,
  some_field string
) WITH (
  'connector' = 'http-sink',
  'url' = 'http://example.com/myendpoint',
  'format' = 'json'
)
```

Then use `INSERT` SQL statement to send data to your HTTP endpoint:

```roomsql
INSERT INTO http VALUES (1, 'Ninette'), (2, 'Hedy')
```


When `'format' = 'json'` is specified on the table definition, the HTTP sink sends json payloads. It is possible to change the format of the payload by specifying
another format name.

### Sink Connector Options

| Option                                    | Required | Description/Value                                                                                                                                                                                                                  |
|-------------------------------------------|----------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| connector                                 | required | Specify what connector to use. For HTTP Sink it should be set to _'http-sink'_.                                                                                                                                                    |
| format                                    | required | Specify what format to use.                                                                                                                                                                                                        |
| url                                       | required | The base URL that should be used for HTTP requests. For example _http://localhost:8080/client_.                                                                                                                                     |
| insert-method                             | optional | Specify which HTTP method to use in the request. The value should be set either to `POST` or `PUT`.                                                                                                                                |
| sink.batch.max-size                       | optional | Maximum number of elements that may be passed in a batch to be written downstream.                                                                                                                                                 |
| sink.requests.max-inflight                | optional | The maximum number of in flight requests that may exist, if any more in flight requests need to be initiated once the maximum has been reached, then it will be blocked until some have completed.                                 |
| sink.requests.max-buffered                | optional | Maximum number of buffered records before applying backpressure.                                                                                                                                                                   |
| sink.flush-buffer.size                    | optional | The maximum size of a batch of entries that may be sent to the HTTP endpoint measured in bytes.                                                                                                                                    |
| sink.flush-buffer.timeout                 | optional | Threshold time in milliseconds for an element to be in a buffer before being flushed.                                                                                                                                              |
| http.logging.level                        | optional | Logging levels for HTTP content. Valid values are `MIN` (the default), `REQ_RESP` and `MAX`.                                                                                                                                       |
| http.sink.request-callback                | optional | Specify which `HttpPostRequestCallback` implementation to use. By default, it is set to `slf4j-logger` corresponding to `Slf4jHttpPostRequestCallback`.                                                                            |
| http.sink.error.code                      | optional | List of HTTP status codes that should be treated as errors by HTTP Sink, separated with comma.                                                                                                                                     |
| http.sink.error.code.exclude              | optional | List of HTTP status codes that should be excluded from the `http.sink.error.code` list, separated with comma.                                                                                                                      |
| http.security.cert.server                 | optional | Path to trusted HTTP server certificate that should be added to connectors key store. More than one path can be specified using `,` as path delimiter.                                                                               |
| http.security.cert.client                 | optional | Path to trusted certificate that should be used by connector's HTTP client for mTLS communication.                                                                                                                                 |
| http.security.key.client                  | optional | Path to trusted private key that should be used by connector's HTTP client for mTLS communication.                                                                                                                                 |
| http.security.cert.server.allowSelfSigned | optional | Accept untrusted certificates for TLS communication.                                                                                                                                                                               |
| http.sink.request.timeout                 | optional | Sets HTTP request timeout for the HTTP sink as a Duration (e.g. `'30s'`, `'1min'`). If not specified, the default value of `30s` will be used.                                                                                                                              |
| http.sink.writer.thread-pool.size         | optional | Sets the size of pool thread for HTTP Sink request processing. Increasing this value would mean that more concurrent requests can be processed in the same time. If not specified, the default value of 1 thread will be used.     |
| http.sink.writer.request.mode             | optional | Sets the Http Sink request submission mode. Two modes are available: `single` and `batch`. Defaults to `batch` if not specified. |
| http.sink.request.batch.size              | optional | Applicable only for `http.sink.writer.request.mode = batch`. Sets number of individual events/requests that will be submitted as one HTTP request by HTTP sink. The default value is 500 which is same as HTTP Sink `maxBatchSize` |

### Sink table HTTP status codes
You can configure a list of HTTP status codes that should be treated as errors for HTTP sink table.
By default all 400 and 500 response codes will be interpreted as an error code.

This behavior can be changed by using the below properties in the table definition. The property names are:
- `http.sink.error.code` used to define HTTP status code value that should be treated as error for example 404.
  Many status codes can be defined in one value, where each code should be separated with comma, for example:
  `401, 402, 403`. User can use this property also to define a type code mask. In that case, all codes from given HTTP response type will be treated as errors.
  An example of such a mask would be `3XX, 4XX, 5XX`. In this case, all 300s, 400s and 500s status codes will be treated as errors.
- `http.sink.error.code.exclude` used to exclude a HTTP code from error list.
  Many status codes can be defined in one value, where each code should be separated with comma, for example:
  `401, 402, 403`. In this example, codes 401, 402 and 403 would not be interpreted as error codes.


### Request submission
HTTP Sink by default submits events in batch. The submission mode can be changed using `http.sink.writer.request.mode` property using `single` or `batch` as property value.

#### Batch submission mode
In batch mode, a number of events (processed elements) will be batched and submitted in one HTTP request.
In this mode, HTTP PUT/POST request's body contains a Json array, where every element of this array represents
individual event.

An example of Http Sink batch request body containing data for three events:
```json
[
  {
    "id": 1,
    "first_name": "Ninette",
    "last_name": "Clee",
    "gender": "Female",
    "stock": "CDZI",
    "currency": "RUB",
    "tx_date": "2021-08-24 15:22:59"
  },
  {
    "id": 2,
    "first_name": "Rob",
    "last_name": "Zombie",
    "gender": "Male",
    "stock": "DGICA",
    "currency": "GBP",
    "tx_date": "2021-10-25 20:53:54"
  },
  {
    "id": 3,
    "first_name": "Adam",
    "last_name": "Jones",
    "gender": "Male",
    "stock": "DGICA",
    "currency": "PLN",
    "tx_date": "2021-10-26 20:53:54"
  }
]
```

The HTTP Sink uses a two-stage batching mechanism that decouples the rate and size of incoming records from how they are sent as HTTP requests.

**Stage 1 - Flink Runtime Buffering**: Controlled by `sink.batch.max-size` property (default: 500). Flink buffers records internally until reaching `sink.batch.max-size` records, `sink.flush-buffer.size`, or timeout `sink.flush-buffer.timeout`. When triggered, Flink flushes the buffered records to the HTTP Sink.

**Stage 2 - HTTP Request Batching**: Controlled by `http.sink.request.batch.size` property (default: 500). The HTTP Sink receives the flushed records and groups them into HTTP requests. Each HTTP request contains up to `http.sink.request.batch.size` records as a JSON array `[record1, record2, ...]`. If more records are flushed than this size, multiple HTTP requests are created.

By default, both values are 500, creating a 1:1 mapping where 500 buffered records result in 1 HTTP request.

```roomsql
CREATE TABLE http (
  id bigint,
  some_field string
) WITH (
  'connector' = 'http-sink',
  'url' = 'http://example.com/myendpoint',
  'format' = 'json',
  'http.sink.request.batch.size' = '50'
)
```

In this example, the default `sink.batch.max-size` of 500 is used. When Flink flushes 500 records, the HTTP Sink splits them into 10 HTTP POST requests (50 records each).

#### Single submission mode
In this mode every processed event is submitted as an individual HTTP POST/PUT request.

SQL:
```roomsql
CREATE TABLE http (
  id bigint,
  some_field string
) WITH (
  'connector' = 'http-sink',
  'url' = 'http://example.com/myendpoint',
  'format' = 'json',
  'http.sink.writer.request.mode' = 'single'
)
```

## Available Metadata

The metadata column `http-status-code`, if specified in the table definition, will get the HTTP status code.
The metadata column `http-headers-map`, if specified in the table definition, will get a map of the HTTP headers.

HTTP requests can fail either immediately or after temporary error retries. The usual behaviour after such failures is to end the job. If you would like to continue
processing after these failures then specify `http.source.lookup.continue-on-error` as true. The lookup join will complete without content in the expected enrichment columns from the http call,
this means that these columns will be null for nullable columns and hold a default value for the type for non-nullable columns.

When using `http.source.lookup.continue-on-error` as true, consider adding extra metadata columns that will surface information about failures into your stream.

Note that if metadata columns are specified and the status code is ignored, then a row containing metadata columns will be produced. If
the status code is ignored and there are no metadata columns defined, then no row will be emitted; this ensures that the expected
inner join behaviour still occurs.

Metadata columns can be specified and hold http information. They are optional read-only columns that must be declared VIRTUAL to exclude them during an INSERT INTO operation.

| Key                   | Data Type                        | Description                            |
|-----------------------|----------------------------------|----------------------------------------|
| error-string          | STRING NULL                      | A string associated with the error     |
| http-status-code      | INT NULL                         | The HTTP status code                   |
| http-headers-map      | MAP <STRING, ARRAY<STRING>> NULL | The headers returned with the response |
| http-completion-state | STRING NULL                      | The completion state of the http call. |

### http-completion-state possible values

| Value                          | Description                         |
|:-------------------------------|-------------------------------------|
| SUCCESS                        | Success                             |
| HTTP_ERROR_STATUS              | HTTP error status code              |
| EXCEPTION                      | An Exception occurred               |
| UNABLE_TO_DESERIALIZE_RESPONSE | Unable to deserialize HTTP response |
| IGNORE_STATUS_CODE             | Status code is ignored              |

If the `error-string` metadata column is defined on the table and the call succeeds then it will have a null value.
When the HTTP response cannot be deserialized, then the `http-completion-state` will be `UNABLE_TO_DESERIALIZE_RESPONSE`
and the `error-string` will be the response body.
When the HTTP status code is in the `http.source.lookup.ignored-response-codes`, then the `http-completion-state` will
be `IGNORE_STATUS_CODE`and no data is returned; any metadata columns contain information about the API call that
occurred.

When a HTTP lookup call fails and populates the metadata columns with the error information, the expected enrichment columns from the HTTP call
are not populated, this means that they will be null for nullable columns and hold a default value for the type for non-nullable columns.

If you are using the Table API `TableResult` and have an `await` with a timeout, this Timeout exception will cause the job to terminate,
even if there are metadata columns defined.

## HTTP status code handler

Above parameters support include lists and exclude lists. A sample configuration may look like this:
`2XX,404,!203` - meaning all codes from group 2XX (200-299), with 404 and without 203 ('!' character). Group exclude listing e.g. !2XX is not supported.

The same format is used in parameter `http.source.lookup.retry-codes`.

Example with explanation:
```roomsql
CREATE TABLE [...]
WITH (
  [...],
  'http.source.lookup.success-codes' = '2XX',
  'http.source.lookup.retry-codes' = '5XX,!501,!505,!506',
  'http.source.lookup.ignored-response-codes' = '404'
)
```
All 200s codes and 404 are considered as successful (`success-codes`, `ignored-response-codes`). These responses won't cause retry or job failure. 404 response is listed in `ignored-response-codes` parameter, which means content body will be ignored. Http with 404 code will produce just empty record.
When server returns response with 500s code except 501, 505 and 506 then connector will re-send request based on configuration in `http.source.lookup.retry-strategy` parameters. By default it's fixed-delay with 1 second delay, up to 3 times per request (parameter `lookup.max-retries`). After exceeding max-retries limit the job will fail.
A response with any other code than specified in params `success-codes` and `retry-codes` e.g. 400, 505, 301 will cause job failure.


```roomsql
CREATE TABLE [...]
WITH (
  [...],
  'http.source.lookup.success-codes' = '2XX',
  'http.source.lookup.retry-codes' = '',
  'http.source.lookup.ignored-response-codes' = '1XX,3XX,4XX,5XX'
)
```
In this configuration, all HTTP responses are considered successful because the sets `success-codes` and `ignored-response-codes` together cover all possible status codes. As a result, no retries will be triggered based on HTTP response codes. However, only responses with status code 200 will be parsed and processed by the Flink operator. Responses with status codes in the 1xx, 3xx, 4xx, and 5xx ranges are classified under `ignored-response-codes`.
Note that retries remain enabled and will still occur on IOException.
To disable retries, set `'lookup.max-retries' = '0'`.

## Security considerations

### TLS (more secure replacement for SSL) and mTLS support

Both HTTP Sink and Lookup Source connectors support HTTPS communication using TLS 1.2 and mTLS.
To enable HTTPS communication simply use `https` protocol in endpoint's URL.

To specify certificate(s) to be used by the server, use `http.security.cert.server` connector property;
the value is a comma separated list of paths to certificate(s), for example you can use your organization's CA
Root certificate, or a self-signed certificate.

Note that if there are no security properties for a `https` url then, the JVMs default certificates are
used - allowing use of globally recognized CAs without the need for configuration.

You can also configure the connector to use mTLS. For this simply use `http.security.cert.client`
and `http.security.key.client` connector properties to specify paths to the certificate and
private key. The key MUST be in `PKCS8` format. Both PEM and DER keys are
allowed.

For non production environments it is sometimes necessary to use HTTPS connection and accept all certificates.
In this special case, you can configure connector to trust all certificates without adding them to keystore.
To enable this option use `http.security.cert.server.allowSelfSigned` property setting its value to `true`.

### Basic Authentication
The connector supports Basic Authentication using a HTTP `Authorization` header.
The header value can be set via properties, similarly as for other headers. The connector converts the passed value to Base64 and uses it for the request.
If the used value starts with the prefix `Basic`, or `http.source.lookup.use-raw-authorization-header`
is set to `'true'`, it will be used as header value as is, without any extra modification.

### OIDC Bearer Authentication
The connector supports Bearer Authentication using a HTTP `Authorization` header. The [OAuth 2.0 RFC](https://datatracker.ietf.org/doc/html/rfc6749) mentions [Obtaining Authorization](https://datatracker.ietf.org/doc/html/rfc6749#section-4)
and an authorization grant. OIDC makes use of this [authorisation grant](https://datatracker.ietf.org/doc/html/rfc6749#section-1.3) in a [Token Request](https://openid.net/specs/openid-connect-core-1_0.html#TokenRequest) by including a [OAuth grant type](https://oauth.net/2/grant-types/) and associated properties, the response is the [token response](https://openid.net/specs/openid-connect-core-1_0.html#TokenResponse).

If you want to use this authorization then you should supply the `Token Request` body in `application/x-www-form-urlencoded` encoding
in configuration property `http.security.oidc.token.request`. See [grant extension](https://datatracker.ietf.org/doc/html/rfc6749#section-4.5) for
an example of a customised grant type token request. The supplied `token request` will be issued to the
[token end point](https://datatracker.ietf.org/doc/html/rfc6749#section-3.2), whose url should be supplied in configuration property
`http.security.oidc.token.endpoint.url`. The returned `access token` is then cached and used for subsequent requests; if the token has expired then
a new one is requested. There is a property `http.security.oidc.token.expiry.reduction`, that defaults to 1 second; new tokens will
be requested if the current time is later than the cached token expiry time minus `http.security.oidc.token.expiry.reduction`.

## Logging the HTTP content
Debug level logging has been added for class `org.apache.flink.connector.http.HttpLogger`. To enable this, alter the log4j properties.
This logging puts out log entries for the HTTP requests and responses. This can be useful for diagnostics to confirm that HTTP requests have been issued and what
that HTTP responses or an exception has occurred (for example connection Refused).

Logging HTTP may not be appropriate for production systems; where sensitive information is not allowed into the logs. But in development environments it is useful
to be able to see HTTP content. Sensitive information can occur in the headers for example authentication tokens and passwords. Also the HTTP request and response bodies
could be sensitive. The default minimal logging should be used in production. For development, you can specify config option `http.logging.level`.
This dictates the amount of content that debug logging will show around HTTP calls; the valid values are:

| log level | Request method | URI | HTTP Body | Response status code | Headers |
|-----------|----------------|-----|-----------|----------------------|---------|
| MIN       | Y              | Y   | N         | Y                    | N       |
| REQ_RESP  | Y              | Y   | Y         | Y                    | N       |
| MAX       | Y              | Y   | Y         | Y                    | Y       |

Notes:
- you can customize what is traced for lookups using the `http.source.lookup.request-callback`.
- where there is an N in the table the output is obfuscated.

#### Restrictions at this time
* No authentication is applied to the token request.
* The processing does not use the refresh token if it is present.
  {{< top >}}
