---
title: "Manifest"
weight: 4
type: docs
aliases:
- /concepts/spec/manifest.html
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements.  See the NOTICE file
distributed with this work for additional information
regarding copyright ownership.  The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License.  You may obtain a copy of the License at

  http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied.  See the License for the
specific language governing permissions and limitations
under the License.
-->

# Manifest

## Manifest List

```shell
├── manifest
    └── manifest-list-51c16f7b-421c-4bc0-80a0-17677f343358-1
```

Manifest List includes meta of several manifest files. Its name contains UUID, it is a avro file, the schema is:

1. _FILE_NAME: STRING, manifest file name.
2. _FILE_SIZE: BIGINT, manifest file size.
3. _NUM_ADDED_FILES: BIGINT, number added files in manifest.
4. _NUM_DELETED_FILES: BIGINT, number deleted files in manifest.
5. _PARTITION_STATS: SimpleStats, partition stats, the minimum and maximum values of partition fields in this manifest are beneficial
   for skipping certain manifest files during queries, it is a SimpleStats.
6. _SCHEMA_ID: BIGINT, schema id when writing this manifest file.

## Manifest

Manifest includes meta of several data files or changelog files or table-index files. Its name contains UUID, it is an
avro file.

The changes of the file are saved in the manifest, and the file can be added or deleted. Manifests should be in
an orderly manner, and the same file may be added or deleted multiple times. The last version should be read. This
design can make commit lighter to support file deletion generated by compaction.

### Data Manifest

Data Manifest includes meta of several data files or changelog files.

```shell
├── manifest
    └── manifest-6758823b-2010-4d06-aef0-3b1b597723d6-0
```

The schema is:

1. _KIND: TINYINT, ADD or DELETE,
2. _PARTITION: BYTES, partition spec, a BinaryRow.
3. _BUCKET: INT, bucket of this file.
4. _TOTAL_BUCKETS: INT, total buckets when write this file, it is used for verification after bucket changes.
5. _FILE: data file meta.

The data file meta is:

1. _FILE_NAME: STRING, file name.
2. _FILE_SIZE: BIGINT, file size.
3. _ROW_COUNT: BIGINT, total number of rows (including add & delete) in this file.
4. _MIN_KEY: STRING, the minimum key of this file.
5. _MAX_KEY: STRING, the maximum key of this file.
6. _KEY_STATS: SimpleStats, the statistics of the key.
7. _VALUE_STATS: SimpleStats, the statistics of the value.
8. _MIN_SEQUENCE_NUMBER: BIGINT, the minimum sequence number.
9. _MAX_SEQUENCE_NUMBER: BIGINT, the maximum sequence number.
10. _SCHEMA_ID: BIGINT, schema id when write this file.
11. _LEVEL: INT, level of this file, in LSM.
12. _EXTRA_FILES: ARRAY<STRING>, extra files for this file, for example, data file index file.
13. _CREATION_TIME: TIMESTAMP_MILLIS, creation time of this file.
14. _DELETE_ROW_COUNT: BIGINT, rowCount = addRowCount + deleteRowCount.
15. _EMBEDDED_FILE_INDEX: BYTES, if data file index is too small, store the index in manifest.
16. _FILE_SOURCE: TINYINT, indicate whether this file is generated as an APPEND or COMPACT file.
17. _VALUE_STATS_COLS: ARRAY<STRING>, statistical column in metadata.
18. _EXTERNAL_PATH: external path of this file, null if it is in warehouse.

### Index Manifest

Index Manifest includes meta of several [table-index]({{< ref "concepts/spec/tableindex" >}}) files.

```shell
├── manifest
    └── index-manifest-5d670043-da25-4265-9a26-e31affc98039-0
```

The schema is:

1. _KIND: TINYINT, ADD or DELETE,
2. _PARTITION: BYTES, partition spec, a BinaryRow.
3. _BUCKET: INT, bucket of this file.
4. _INDEX_TYPE: STRING, "HASH" or "DELETION_VECTORS".
5. _FILE_NAME: STRING, file name.
6. _FILE_SIZE: BIGINT, file size.
7. _ROW_COUNT: BIGINT, total number of rows.
8. _DELETIONS_VECTORS_RANGES: Metadata only used by "DELETION_VECTORS", is an array of deletion vector meta, the schema of each deletion vector meta is:
   1. f0: the data file name corresponding to this deletion vector.
   2. f1: the starting offset of this deletion vector in the index file.
   3. f2: the length of this deletion vector in the index file.
   4. _CARDINALITY: the number of deleted rows.

## Appendix

### SimpleStats

SimpleStats is nested row, the schema is:

1. _MIN_VALUES: BYTES, BinaryRow, the minimum values of the columns.
2. _MAX_VALUES: BYTES, BinaryRow, the maximum values of the columns.
3. _NULL_COUNTS: ARRAY<BIGINT>, the number of nulls of the columns.

### BinaryRow

BinaryRow is backed by bytes instead of Object. It can significantly reduce the serialization/deserialization of Java
objects.

A Row has two part: Fixed-length part and variable-length part. Fixed-length part contains 1 byte header and null bit
set and field values. Null bit set is used for null tracking and is aligned to 8-byte word boundaries. `Field values`
holds fixed-length primitive types and variable-length values which can be stored in 8 bytes inside. If it do not fit
the variable-length field, then store the length and offset of variable-length part.
