TokenFilter (Lucene 10.5.0 core API)
Description: declaration: package: org.apache.lucene.analysis, class: TokenFilter

Package org.apache.lucene.analysis

# Class TokenFilter

java.lang.Object
org.apache.lucene.util.AttributeSource
org.apache.lucene.analysis.TokenStream
org.apache.lucene.analysis.TokenFilter

All Implemented Interfaces:
  `Closeable`, `AutoCloseable`, `Unwrappable<TokenStream>`

Direct Known Subclasses:
  `CachingTokenFilter`, `FilteringTokenFilter`, `GraphTokenFilter`, `LowerCaseFilter`

public abstract class TokenFilter extends TokenStream implements Unwrappable<TokenStream>
A TokenFilter is a TokenStream whose input is another TokenStream.

This is an abstract class; subclasses must override `TokenStream.incrementToken()`.

See Also:
  - `TokenStream`

- ## Nested Class Summary

  ## Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource

  `AttributeSource.State`

- ## Field Summary

  Fields
  Modifier and Type
  Field
  Description
  `protected final TokenStream`
  `input`
  The source of tokens for this filter.

  ### Fields inherited from class org.apache.lucene.analysis.TokenStream

  `DEFAULT_TOKEN_ATTRIBUTE_FACTORY`

- ## Constructor Summary

  Constructors
  Modifier
  Constructor
  Description
  `protected`
  `TokenFilter(TokenStream input)`
  Construct a token stream filtering the given input.

- ## Method Summary

  Modifier and Type
  Method
  Description
  `void`
  `close()`
  Releases resources associated with this stream.
  `void`
  `end()`
  This method is called by the consumer after the last token has been consumed, after `TokenStream.incrementToken()` returned `false` (using the new `TokenStream` API).
  `void`
  `reset()`
  This method is called by a consumer before it begins consumption using `TokenStream.incrementToken()`.
  `TokenStream`
  `unwrap()`
  Unwraps this instance

  ### Methods inherited from class org.apache.lucene.analysis.TokenStream

  `incrementToken`

  ### Methods inherited from class org.apache.lucene.util.AttributeSource

  `addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, endAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, removeAllAttributes, restoreState, toString`

  ### Methods inherited from class java.lang.Object

  `clone, finalize, getClass, notify, notifyAll, wait, wait, wait`

- ## Field Details

  - ### input

    protected final TokenStream input
    The source of tokens for this filter.

- ## Constructor Details

  - ### TokenFilter

    protected TokenFilter(TokenStream input)
    Construct a token stream filtering the given input.

- ## Method Details

  - ### end

    public void end() throws IOException
    This method is called by the consumer after the last token has been consumed, after `TokenStream.incrementToken()` returned `false` (using the new `TokenStream` API). Streams implementing the old API should upgrade to use this feature.

    This method can be used to perform any end-of-stream operations, such as setting the final offset of a stream. The final offset of a stream might differ from the offset of the last token eg in case one or more whitespaces followed after the last token, but a WhitespaceTokenizer was used.

    Additionally any skipped positions (such as those removed by a stopfilter) can be applied to the position increment, or any adjustment of other attributes where the end-of-stream value may be important.

    If you override this method, always call `super.end()`.

    NOTE: The default implementation chains the call to the input TokenStream, so be sure to call `super.end()` first when overriding this method.

    Overrides:
      `end` in class `TokenStream`
    Throws:
      `IOException` - If an I/O error occurs

  - ### close

    public void close() throws IOException
    Releases resources associated with this stream.

    If you override this method, always call `super.close()`, otherwise some internal state will not be correctly reset (e.g., `Tokenizer` will throw `IllegalStateException` on reuse).

    NOTE: The default implementation chains the call to the input TokenStream, so be sure to call `super.close()` when overriding this method.

    Specified by:
      `close` in interface `AutoCloseable`
    Specified by:
      `close` in interface `Closeable`
    Overrides:
      `close` in class `TokenStream`
    Throws:
      `IOException`

  - ### reset

    public void reset() throws IOException
    This method is called by a consumer before it begins consumption using `TokenStream.incrementToken()`.

    Resets this stream to a clean state. Stateful implementations must implement this method so that they can be reused, just as if they had been created fresh.

    If you override this method, always call `super.reset()`, otherwise some internal state will not be correctly reset (e.g., `Tokenizer` will throw `IllegalStateException` on further usage).

    NOTE: The default implementation chains the call to the input TokenStream, so be sure to call `super.reset()` when overriding this method.

    Overrides:
      `reset` in class `TokenStream`
    Throws:
      `IOException`

  - ### unwrap

    public TokenStream unwrap()
    Description copied from interface: `Unwrappable`
    Unwraps this instance
    Specified by:
      `unwrap` in interface `Unwrappable<TokenStream>`