MinHashFilter (Lucene 10.5.0 common API)
Description: declaration: package: org.apache.lucene.analysis.minhash, class: MinHashFilter

Package org.apache.lucene.analysis.minhash

# Class MinHashFilter

java.lang.Object
org.apache.lucene.util.AttributeSource
org.apache.lucene.analysis.TokenStream
org.apache.lucene.analysis.TokenFilter
org.apache.lucene.analysis.minhash.MinHashFilter

All Implemented Interfaces:
  `Closeable`, `AutoCloseable`, `Unwrappable<TokenStream>`

public class MinHashFilter extends TokenFilter
Generate min hash tokens from an incoming stream of tokens. The incoming tokens would typically be 5 word shingles.

The number of hashes used and the number of minimum values for each hash can be set. You could have 1 hash and keep the 100 lowest values or 100 hashes and keep the lowest one for each. Hashes can also be bucketed in ranges over the 128-bit hash space,

A 128-bit hash is used internally. 5 word shingles from 10e5 words generate 10e25 combinations So a 64 bit hash would have collisions (1.8e19)

When using different hashes 32 bits are used for the hash position leaving scope for 8e28 unique hashes. A single hash will use all 128 bits.

- ## Nested Class Summary

  ## Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource

  `AttributeSource.State`

- ## Field Summary

  Fields
  Modifier and Type
  Field
  Description
  `static final int`
  `DEFAULT_BUCKET_COUNT`
  `static final int`
  `DEFAULT_HASH_COUNT`
  `static final int`
  `DEFAULT_HASH_SET_SIZE`

  ### Fields inherited from class org.apache.lucene.analysis.TokenFilter

  `input`

  ### Fields inherited from class org.apache.lucene.analysis.TokenStream

  `DEFAULT_TOKEN_ATTRIBUTE_FACTORY`

- ## Constructor Summary

  Constructors
  Constructor
  Description
  `MinHashFilter(TokenStream input, int hashCount, int bucketCount, int hashSetSize, boolean withRotation)`
  create a MinHash filter

- ## Method Summary

  Modifier and Type
  Method
  Description
  `void`
  `end()`
  `final boolean`
  `incrementToken()`
  `void`
  `reset()`

  ### Methods inherited from class org.apache.lucene.analysis.TokenFilter

  `close, unwrap`

  ### Methods inherited from class org.apache.lucene.util.AttributeSource

  `addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, endAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, removeAllAttributes, restoreState, toString`

  ### Methods inherited from class java.lang.Object

  `clone, finalize, getClass, notify, notifyAll, wait, wait, wait`

- ## Field Details

  - ### DEFAULT_HASH_COUNT

    public static final int DEFAULT_HASH_COUNT
    See Also:
      - Constant Field Values

  - ### DEFAULT_HASH_SET_SIZE

    public static final int DEFAULT_HASH_SET_SIZE
    See Also:
      - Constant Field Values

  - ### DEFAULT_BUCKET_COUNT

    public static final int DEFAULT_BUCKET_COUNT
    See Also:
      - Constant Field Values

- ## Constructor Details

  - ### MinHashFilter

    public MinHashFilter(TokenStream input, int hashCount, int bucketCount, int hashSetSize, boolean withRotation)
    create a MinHash filter
    Parameters:
      `input` - the token stream
      `hashCount` - the no. of hashes
      `bucketCount` - the no. of buckets for hashing
      `hashSetSize` - the no. of min hashes to keep
      `withRotation` - whether rotate or not hashes while incrementing tokens

- ## Method Details

  - ### incrementToken

    public final boolean incrementToken() throws IOException
    Specified by:
      `incrementToken` in class `TokenStream`
    Throws:
      `IOException`

  - ### end

    public void end() throws IOException
    Overrides:
      `end` in class `TokenFilter`
    Throws:
      `IOException`

  - ### reset

    public void reset() throws IOException
    Overrides:
      `reset` in class `TokenFilter`
    Throws:
      `IOException`