FSTTermsWriter (Lucene 10.5.0 codecs API)
Description: declaration: package: org.apache.lucene.codecs.memory, class: FSTTermsWriter

Package org.apache.lucene.codecs.memory

# Class FSTTermsWriter

java.lang.Object
org.apache.lucene.codecs.FieldsConsumer
org.apache.lucene.codecs.memory.FSTTermsWriter

All Implemented Interfaces:
  `Closeable`, `AutoCloseable`

public class FSTTermsWriter extends FieldsConsumer
FST-based term dict, using metadata as FST output.

The FST directly holds the mapping between <term, metadata>.

Term metadata consists of three parts: 1. term statistics: docFreq, totalTermFreq; 2. monotonic long[], e.g. the pointer to the postings list for that term; 3. generic byte[], e.g. other information need by postings reader.

File:

- `.tst`: Term Dictionary

## Term Dictionary

The .tst contains a list of FSTs, one for each field. The FST maps a term to its corresponding statistics (e.g. docfreq) and metadata (e.g. information for postings list reader like file pointer to postings list).

Typically the metadata is separated into two parts:

- Monotonical long array: Some metadata will always be ascending in order with the corresponding term. This part is used by FST to share outputs between arcs.
- Generic byte array: Used to store non-monotonic metadata.

File format:

- TermsDict(.tst) --> Header, PostingsHeader, FieldSummary, DirOffset
- FieldSummary --> NumFields, <FieldNumber, NumTerms, SumTotalTermFreq?, SumDocFreq, DocCount, LongsSize, TermFST >NumFields
- TermFST --> `FST<TermData>`
- TermData --> Flag, BytesSize?, LongDeltaLongsSize?, ByteBytesSize?, < DocFreq[Same?], (TotalTermFreq-DocFreq) > ?
- Header --> `IndexHeader`
- DirOffset --> `Uint64`
- DocFreq, LongsSize, BytesSize, NumFields, FieldNumber, DocCount --> `VInt`
- TotalTermFreq, NumTerms, SumTotalTermFreq, SumDocFreq, LongDelta --> `VLong`

Notes:

- The format of PostingsHeader and generic meta bytes are customized by the specific postings implementation: they contain arbitrary per-file data (such as parameters or versioning information), and per-term data (non-monotonic ones like pulsed postings data).
- The format of TermData is determined by FST, typically monotonic metadata will be dense around shallow arcs, while in deeper arcs only generic bytes and term statistics exist.
- The byte Flag is used to indicate which part of metadata exists on current arc. Specially the monotonic part is omitted when it is an array of 0s.
- Since LongsSize is per-field fixed, it is only written once in field summary.

WARNING: This API is experimental and might change in incompatible ways in the next release.

- ## Field Summary

  Fields
  Modifier and Type
  Field
  Description
  `static final int`
  `TERMS_VERSION_CURRENT`
  `static final int`
  `TERMS_VERSION_START`

- ## Constructor Summary

  Constructors
  Constructor
  Description
  `FSTTermsWriter(SegmentWriteState state, PostingsWriterBase postingsWriter)`

- ## Method Summary

  Modifier and Type
  Method
  Description
  `void`
  `close()`
  `void`
  `write(Fields fields, NormsProducer norms)`

  ### Methods inherited from class org.apache.lucene.codecs.FieldsConsumer

  `merge`

  ### Methods inherited from class java.lang.Object

  `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- ## Field Details

  - ### TERMS_VERSION_START

    public static final int TERMS_VERSION_START
    See Also:
      - Constant Field Values

  - ### TERMS_VERSION_CURRENT

    public static final int TERMS_VERSION_CURRENT
    See Also:
      - Constant Field Values

- ## Constructor Details

  - ### FSTTermsWriter

    public FSTTermsWriter(SegmentWriteState state, PostingsWriterBase postingsWriter) throws IOException
    Throws:
      `IOException`

- ## Method Details

  - ### write

    public void write(Fields fields, NormsProducer norms) throws IOException
    Specified by:
      `write` in class `FieldsConsumer`
    Throws:
      `IOException`

  - ### close

    public void close() throws IOException
    Specified by:
      `close` in interface `AutoCloseable`
    Specified by:
      `close` in interface `Closeable`
    Specified by:
      `close` in class `FieldsConsumer`
    Throws:
      `IOException`