UnicodeWhitespaceTokenizer (Lucene 10.5.0 common API)
Description: declaration: package: org.apache.lucene.analysis.core, class: UnicodeWhitespaceTokenizer

Package org.apache.lucene.analysis.core

# Class UnicodeWhitespaceTokenizer

java.lang.Object
org.apache.lucene.util.AttributeSource
org.apache.lucene.analysis.TokenStream
org.apache.lucene.analysis.Tokenizer
org.apache.lucene.analysis.util.CharTokenizer
org.apache.lucene.analysis.core.UnicodeWhitespaceTokenizer

All Implemented Interfaces:
  `Closeable`, `AutoCloseable`

public final class UnicodeWhitespaceTokenizer extends CharTokenizer
A UnicodeWhitespaceTokenizer is a tokenizer that divides text at whitespace. Adjacent sequences of non-Whitespace characters form tokens (according to Unicode's WHITESPACE property).

For Unicode version see: `UnicodeProps`

- ## Nested Class Summary

  ## Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource

  `AttributeSource.State`

- ## Field Summary

  ### Fields inherited from class org.apache.lucene.analysis.util.CharTokenizer

  `DEFAULT_MAX_WORD_LEN`

  ### Fields inherited from class org.apache.lucene.analysis.Tokenizer

  `input`

  ### Fields inherited from class org.apache.lucene.analysis.TokenStream

  `DEFAULT_TOKEN_ATTRIBUTE_FACTORY`

- ## Constructor Summary

  Constructors
  Constructor
  Description
  `UnicodeWhitespaceTokenizer()`
  Construct a new UnicodeWhitespaceTokenizer.
  `UnicodeWhitespaceTokenizer(AttributeFactory factory)`
  Construct a new UnicodeWhitespaceTokenizer using a given `AttributeFactory`.
  `UnicodeWhitespaceTokenizer(AttributeFactory factory, int maxTokenLen)`
  Construct a new UnicodeWhitespaceTokenizer using a given `AttributeFactory`.

- ## Method Summary

  Modifier and Type
  Method
  Description
  `protected boolean`
  `isTokenChar(int c)`
  Collects only characters which do not satisfy Unicode's WHITESPACE property.

  ### Methods inherited from class org.apache.lucene.analysis.util.CharTokenizer

  `end, fromSeparatorCharPredicate, fromSeparatorCharPredicate, fromTokenCharPredicate, fromTokenCharPredicate, incrementToken, reset`

  ### Methods inherited from class org.apache.lucene.analysis.Tokenizer

  `close, correctOffset, setReader, setReaderTestPoint`

  ### Methods inherited from class org.apache.lucene.util.AttributeSource

  `addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, endAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, removeAllAttributes, restoreState, toString`

  ### Methods inherited from class java.lang.Object

  `clone, finalize, getClass, notify, notifyAll, wait, wait, wait`

- ## Constructor Details

  - ### UnicodeWhitespaceTokenizer

    public UnicodeWhitespaceTokenizer()
    Construct a new UnicodeWhitespaceTokenizer.

  - ### UnicodeWhitespaceTokenizer

    public UnicodeWhitespaceTokenizer(AttributeFactory factory)
    Construct a new UnicodeWhitespaceTokenizer using a given `AttributeFactory`.
    Parameters:
      `factory` - the attribute factory to use for this `Tokenizer`

  - ### UnicodeWhitespaceTokenizer

    public UnicodeWhitespaceTokenizer(AttributeFactory factory, int maxTokenLen)
    Construct a new UnicodeWhitespaceTokenizer using a given `AttributeFactory`.
    Parameters:
      `factory` - the attribute factory to use for this `Tokenizer`
      `maxTokenLen` - maximum token length the tokenizer will emit. Must be greater than 0 and less than MAX_TOKEN_LENGTH_LIMIT (1024*1024)
    Throws:
      `IllegalArgumentException` - if maxTokenLen is invalid.

- ## Method Details

  - ### isTokenChar

    protected boolean isTokenChar(int c)
    Collects only characters which do not satisfy Unicode's WHITESPACE property.
    Specified by:
      `isTokenChar` in class `CharTokenizer`