Lucene90StoredFieldsFormat (Lucene 10.5.0 core API)
Description: declaration: package: org.apache.lucene.codecs.lucene90, class: Lucene90StoredFieldsFormat

Package org.apache.lucene.codecs.lucene90

# Class Lucene90StoredFieldsFormat

java.lang.Object
org.apache.lucene.codecs.StoredFieldsFormat
org.apache.lucene.codecs.lucene90.Lucene90StoredFieldsFormat

public class Lucene90StoredFieldsFormat extends StoredFieldsFormat
Lucene 9.0 stored fields format.

Principle

This `StoredFieldsFormat` compresses blocks of documents in order to improve the compression ratio compared to document-level compression. It uses the LZ4 compression algorithm by default in 8KB blocks and shared dictionaries, which is fast to compress and very fast to decompress data. Although the default compression method that is used (`BEST_SPEED`) focuses more on speed than on compression ratio, it should provide interesting compression ratios for redundant inputs (such as log files, HTML or plain text). For higher compression, you can choose (`BEST_COMPRESSION`), which uses the DEFLATE algorithm with 48KB blocks and shared dictionaries for a better ratio at the expense of slower performance. These two options can be configured like this:

       // the default: for high performance
       indexWriterConfig.setCodec(new Lucene100Codec(Mode.BEST_SPEED));
       // instead for higher performance (but slower):
       // indexWriterConfig.setCodec(new Lucene100Codec(Mode.BEST_COMPRESSION));

File formats

Stored fields are represented by three files:

1. A fields data file (extension `.fdt`). This file stores a compact representation of documents in compressed blocks of 8KB or more. When writing a segment, documents are appended to an in-memory `byte[]` buffer. When its size reaches 80KB or more, some metadata about the documents is flushed to disk, immediately followed by a compressed representation of the buffer using the LZ4 compression format.

   Notes

   - When at least one document in a chunk is large enough so that the chunk is larger than 80KB, the chunk will actually be compressed in several LZ4 blocks of 8KB. This allows `StoredFieldVisitor`s which are only interested in the first fields of a document to not have to decompress 10MB of data if the document is 10MB, but only 8-16KB(may cross the block).
   - Given that the original lengths are written in the metadata of the chunk, the decompressor can leverage this information to stop decoding as soon as enough data has been decompressed.
   - In case documents are incompressible, the overhead of the compression format is less than 0.5%.

2. A fields index file (extension `.fdx`). This file stores two `monotonic arrays`, one for the first doc IDs of each block of compressed documents, and another one for the corresponding offsets on disk. At search time, the array containing doc IDs is binary-searched in order to find the block that contains the expected doc ID, and the associated offset on disk is retrieved from the second array.

3. A fields meta file (extension `.fdm`). This file stores metadata about the monotonic arrays stored in the index file.

Known limitations

This `StoredFieldsFormat` does not support individual documents larger than (`231 - 214`) bytes.

WARNING: This API is experimental and might change in incompatible ways in the next release.

- ## Nested Class Summary

  Nested Classes
  Modifier and Type
  Class
  Description
  `static enum`
  `Lucene90StoredFieldsFormat.Mode`
  Configuration option for stored fields.

- ## Field Summary

  Fields
  Modifier and Type
  Field
  Description
  `static final CompressionMode`
  `BEST_COMPRESSION_MODE`
  Compression mode for `Lucene90StoredFieldsFormat.Mode.BEST_COMPRESSION`
  `static final CompressionMode`
  `BEST_SPEED_MODE`
  Compression mode for `Lucene90StoredFieldsFormat.Mode.BEST_SPEED`
  `static final String`
  `MODE_KEY`
  Attribute key for compression mode.

- ## Constructor Summary

  Constructors
  Constructor
  Description
  `Lucene90StoredFieldsFormat()`
  Stored fields format with default options
  `Lucene90StoredFieldsFormat(Lucene90StoredFieldsFormat.Mode mode)`
  Stored fields format with specified mode

- ## Method Summary

  Modifier and Type
  Method
  Description
  `StoredFieldsReader`
  `fieldsReader(Directory directory, SegmentInfo si, FieldInfos fn, IOContext context)`
  Returns a `StoredFieldsReader` to load stored fields.
  `StoredFieldsWriter`
  `fieldsWriter(Directory directory, SegmentInfo si, IOContext context)`
  Returns a `StoredFieldsWriter` to write stored fields.

  ### Methods inherited from class java.lang.Object

  `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- ## Field Details

  - ### MODE_KEY

    public static final String MODE_KEY
    Attribute key for compression mode.

  - ### BEST_COMPRESSION_MODE

    public static final CompressionMode BEST_COMPRESSION_MODE
    Compression mode for `Lucene90StoredFieldsFormat.Mode.BEST_COMPRESSION`

  - ### BEST_SPEED_MODE

    public static final CompressionMode BEST_SPEED_MODE
    Compression mode for `Lucene90StoredFieldsFormat.Mode.BEST_SPEED`

- ## Constructor Details

  - ### Lucene90StoredFieldsFormat

    public Lucene90StoredFieldsFormat()
    Stored fields format with default options

  - ### Lucene90StoredFieldsFormat

    public Lucene90StoredFieldsFormat(Lucene90StoredFieldsFormat.Mode mode)
    Stored fields format with specified mode

- ## Method Details

  - ### fieldsReader

    public StoredFieldsReader fieldsReader(Directory directory, SegmentInfo si, FieldInfos fn, IOContext context) throws IOException
    Description copied from class: `StoredFieldsFormat`
    Returns a `StoredFieldsReader` to load stored fields.
    Specified by:
      `fieldsReader` in class `StoredFieldsFormat`
    Throws:
      `IOException`

  - ### fieldsWriter

    public StoredFieldsWriter fieldsWriter(Directory directory, SegmentInfo si, IOContext context) throws IOException
    Description copied from class: `StoredFieldsFormat`
    Returns a `StoredFieldsWriter` to write stored fields.
    Specified by:
      `fieldsWriter` in class `StoredFieldsFormat`
    Throws:
      `IOException`