TokenStreamToAutomaton (Lucene 10.5.0 core API)
Description: declaration: package: org.apache.lucene.analysis, class: TokenStreamToAutomaton

Package org.apache.lucene.analysis

# Class TokenStreamToAutomaton

java.lang.Object
org.apache.lucene.analysis.TokenStreamToAutomaton

public class TokenStreamToAutomaton extends Object
Consumes a TokenStream and creates an `Automaton` where the transition labels are UTF8 bytes (or Unicode code points if unicodeArcs is true) from the `TermToBytesRefAttribute`. Between tokens we insert POS_SEP and for holes we insert HOLE.

WARNING: This API is experimental and might change in incompatible ways in the next release.

- ## Field Summary

  Fields
  Modifier and Type
  Field
  Description
  `static final int`
  `HOLE`
  We add this arc to represent a hole.
  `static final int`
  `POS_SEP`
  We create transition between two adjacent tokens.

- ## Constructor Summary

  Constructors
  Constructor
  Description
  `TokenStreamToAutomaton()`
  Sole constructor.

- ## Method Summary

  Modifier and Type
  Method
  Description
  `protected BytesRef`
  `changeToken(BytesRef in)`
  Subclass and implement this if you need to change the token (such as escaping certain bytes) before it's turned into a graph.
  `void`
  `setFinalOffsetGapAsHole(boolean finalOffsetGapAsHole)`
  If true, any final offset gaps will result in adding a position hole.
  `void`
  `setPreservePositionIncrements(boolean enablePositionIncrements)`
  Whether to generate holes in the automaton for missing positions, `true` by default.
  `void`
  `setUnicodeArcs(boolean unicodeArcs)`
  Whether to make transition labels Unicode code points instead of UTF8 bytes, `false` by default
  `Automaton`
  `toAutomaton(TokenStream in)`
  Pulls the graph (including `PositionLengthAttribute`) from the provided `TokenStream`, and creates the corresponding automaton where arcs are bytes (or Unicode code points if unicodeArcs = true) from each term.

  ### Methods inherited from class java.lang.Object

  `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- ## Field Details

  - ### POS_SEP

    public static final int POS_SEP
    We create transition between two adjacent tokens.
    See Also:
      - Constant Field Values

  - ### HOLE

    public static final int HOLE
    We add this arc to represent a hole.
    See Also:
      - Constant Field Values

- ## Constructor Details

  - ### TokenStreamToAutomaton

    public TokenStreamToAutomaton()
    Sole constructor.

- ## Method Details

  - ### setPreservePositionIncrements

    public void setPreservePositionIncrements(boolean enablePositionIncrements)
    Whether to generate holes in the automaton for missing positions, `true` by default.

  - ### setFinalOffsetGapAsHole

    public void setFinalOffsetGapAsHole(boolean finalOffsetGapAsHole)
    If true, any final offset gaps will result in adding a position hole.

  - ### setUnicodeArcs

    public void setUnicodeArcs(boolean unicodeArcs)
    Whether to make transition labels Unicode code points instead of UTF8 bytes, `false` by default

  - ### changeToken

    protected BytesRef changeToken(BytesRef in)
    Subclass and implement this if you need to change the token (such as escaping certain bytes) before it's turned into a graph.

  - ### toAutomaton

    public Automaton toAutomaton(TokenStream in) throws IOException
    Pulls the graph (including `PositionLengthAttribute`) from the provided `TokenStream`, and creates the corresponding automaton where arcs are bytes (or Unicode code points if unicodeArcs = true) from each term.
    Throws:
      `IOException`