IndexedDISI (Lucene 10.5.0 core API)
Description: declaration: package: org.apache.lucene.codecs.lucene90, class: IndexedDISI

Package org.apache.lucene.codecs.lucene90

# Class IndexedDISI

java.lang.Object
org.apache.lucene.search.DocIdSetIterator
org.apache.lucene.search.AbstractDocIdSetIterator
org.apache.lucene.codecs.lucene90.IndexedDISI

public final class IndexedDISI extends AbstractDocIdSetIterator
Disk-based implementation of a `DocIdSetIterator` which can return the index of the current document, i.e. the ordinal of the current document among the list of documents that this iterator can return. This is useful to implement sparse doc values by only having to encode values for documents that actually have a value.

Implementation-wise, this `DocIdSetIterator` is inspired of `roaring bitmaps` and encodes ranges of `65536` documents independently and picks between 3 encodings depending on the density of the range:

- `ALL` if the range contains 65536 documents exactly,
- `DENSE` if the range contains 4096 documents or more; in that case documents are stored in a bit set,
- `SPARSE` otherwise, and the lower 16 bits of the doc IDs are stored in a `short`.

Only ranges that contain at least one value are encoded.

This implementation uses 6 bytes per document in the worst-case, which happens in the case that all ranges contain exactly one document.

To avoid O(n) lookup time complexity, with n being the number of documents, two lookup tables are used: A lookup table for block offset and index, and a rank structure for DENSE block index lookups.

The lookup table is an array of `int`-pairs, with a pair for each block. It allows for direct jumping to the block, as opposed to iteration from the current position and forward one block at a time.

Each int-pair entry consists of 2 logical parts:

The first 32 bit int holds the index (number of set bits in the blocks) up to just before the wanted block. The maximum number of set bits is the maximum number of documents, which is less than 2^31.

The next int holds the offset in bytes into the underlying slice. As there is a maximum of 2^16 blocks, it follows that the maximum size of any block must not exceed 2^15 bytes to avoid overflow (2^16 bytes if the int is treated as unsigned). This is currently the case, with the largest block being DENSE and using 2^13 + 36 bytes.

The cache overhead is numDocs/1024 bytes.

Note: There are 4 types of blocks: ALL, DENSE, SPARSE and non-existing (0 set bits). In the case of non-existing blocks, the entry in the lookup table has index equal to the previous entry and offset equal to the next non-empty block.

The block lookup table is stored at the end of the total block structure.

The rank structure for DENSE blocks is an array of byte-pairs with an entry for each sub-block (default 512 bits) out of the 65536 bits in the outer DENSE block.

Each rank-entry states the number of set bits within the block up to the bit before the bit positioned at the start of the sub-block. Note that that the rank entry of the first sub-block is always 0 and that the last entry can at most be 65536-2 = 65634 and thus will always fit into an byte-pair of 16 bits.

The rank structure for a given DENSE block is stored at the beginning of the DENSE block. This ensures locality and keeps logistics simple.

NOTE: This API is for internal purposes only and might change in incompatible ways in the next release.

- ## Field Summary

  Fields
  Modifier and Type
  Field
  Description
  `static final byte`
  `DEFAULT_DENSE_RANK_POWER`

  ### Fields inherited from class org.apache.lucene.search.AbstractDocIdSetIterator

  `doc`

  ### Fields inherited from class org.apache.lucene.search.DocIdSetIterator

  `NO_MORE_DOCS`

- ## Constructor Summary

  Constructors
  Constructor
  Description
  `IndexedDISI(IndexInput in, long offset, long length, int jumpTableEntryCount, byte denseRankPower, long cost)`
  This constructor always creates a new blockSlice and a new jumpTable from in, to ensure that operations are independent from the caller.

- ## Method Summary

  Modifier and Type
  Method
  Description
  `int`
  `advance(int target)`
  Advances to the first beyond the current whose document number is greater than or equal to target, and returns the document number itself.
  `boolean`
  `advanceExact(int target)`
  `static KnnVectorValues.DocIndexIterator`
  `asDocIndexIterator(IndexedDISI disi)`
  Returns an iterator that delegates to the IndexedDISI.
  `long`
  `cost()`
  Returns the estimated cost of this `DocIdSetIterator`.
  `static IndexInput`
  `createBlockSlice(IndexInput slice, String sliceDescription, long offset, long length, int jumpTableEntryCount)`
  Helper method for using `IndexedDISI(IndexInput, RandomAccessInput, int, byte, long)`.
  `static RandomAccessInput`
  `createJumpTable(IndexInput slice, long offset, long length, int jumpTableEntryCount)`
  Helper method for using `IndexedDISI(IndexInput, RandomAccessInput, int, byte, long)`.
  `int`
  `docIDRunEnd()`
  Returns the end of the run of consecutive doc IDs that match this `DocIdSetIterator` and that contains the current `DocIdSetIterator.docID()`, that is: one plus the last doc ID of the run.
  `int`
  `index()`
  `void`
  `intoBitSet(int upTo, FixedBitSet bitSet, int offset)`
  Load doc IDs into a `FixedBitSet`.
  `int`
  `nextDoc()`
  Advances to the next document in the set and returns the doc it is currently on, or `DocIdSetIterator.NO_MORE_DOCS` if there are no more docs in the set.
  NOTE: after the iterator has exhausted you should not call this method, as it may result in unpredicted behavior.
  `static short`
  `writeBitSet(DocIdSetIterator it, IndexOutput out, byte denseRankPower)`
  Writes the docIDs from it to out, in logical blocks, one for each 65536 docIDs in monotonically increasing gap-less order.

  ### Methods inherited from class org.apache.lucene.search.AbstractDocIdSetIterator

  `docID`

  ### Methods inherited from class org.apache.lucene.search.DocIdSetIterator

  `all, empty, range, slowAdvance`

  ### Methods inherited from class java.lang.Object

  `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- ## Field Details

  - ### DEFAULT_DENSE_RANK_POWER

    public static final byte DEFAULT_DENSE_RANK_POWER
    See Also:
      - Constant Field Values

- ## Constructor Details

  - ### IndexedDISI

    public IndexedDISI(IndexInput in, long offset, long length, int jumpTableEntryCount, byte denseRankPower, long cost) throws IOException
    This constructor always creates a new blockSlice and a new jumpTable from in, to ensure that operations are independent from the caller. See `IndexedDISI(IndexInput, RandomAccessInput, int, byte, long)` for re-use of blockSlice and jumpTable.
    Parameters:
      `in` - backing data.
      `offset` - starting offset for blocks in the backing data.
      `length` - the number of bytes holding blocks and jump-table in the backing data.
      `jumpTableEntryCount` - the number of blocks covered by the jump-table. This must match the number returned by `writeBitSet(DocIdSetIterator, IndexOutput, byte)`.
      `denseRankPower` - the number of docIDs covered by each rank entry in DENSE blocks, expressed as `2^denseRankPower`. This must match the power given in `writeBitSet(DocIdSetIterator, IndexOutput, byte)`
      `cost` - normally the number of logical docIDs.
    Throws:
      `IOException`

- ## Method Details

  - ### writeBitSet

    public static short writeBitSet(DocIdSetIterator it, IndexOutput out, byte denseRankPower) throws IOException
    Writes the docIDs from it to out, in logical blocks, one for each 65536 docIDs in monotonically increasing gap-less order. The caller must keep track of the number of jump-table entries (returned by this method) as well as the denseRankPower and provide them when constructing an IndexedDISI for reading.
    Parameters:
      `it` - the document IDs.
      `out` - destination for the blocks.
      `denseRankPower` - for `IndexedDISI.Method.DENSE` blocks, a rank will be written every `2^denseRankPower` docIDs. Values < 7 (every 128 docIDs) or > 15 (every 32768 docIDs) disables DENSE rank. Recommended values are 8-12: Every 256-4096 docIDs or 4-64 longs. `DEFAULT_DENSE_RANK_POWER` is 9: Every 512 docIDs. This should be stored in meta and used when creating an instance of IndexedDISI.
    Returns:
      the number of jump-table entries following the blocks, -1 for no entries. This should be stored in meta and used when creating an instance of IndexedDISI.
    Throws:
      `IOException` - if there was an error writing to out.

  - ### createBlockSlice

    public static IndexInput createBlockSlice(IndexInput slice, String sliceDescription, long offset, long length, int jumpTableEntryCount) throws IOException
    Helper method for using `IndexedDISI(IndexInput, RandomAccessInput, int, byte, long)`. Creates a disiSlice for the IndexedDISI data blocks, without the jump-table.
    Parameters:
      `slice` - backing data, holding both blocks and jump-table.
      `sliceDescription` - human readable slice designation.
      `offset` - relative to the backing data.
      `length` - full length of the IndexedDISI, including blocks and jump-table data.
      `jumpTableEntryCount` - the number of blocks covered by the jump-table.
    Returns:
      a jumpTable containing the block jump-data or null if no such table exists.
    Throws:
      `IOException` - if a RandomAccessInput could not be created from slice.

  - ### createJumpTable

    public static RandomAccessInput createJumpTable(IndexInput slice, long offset, long length, int jumpTableEntryCount) throws IOException
    Helper method for using `IndexedDISI(IndexInput, RandomAccessInput, int, byte, long)`. Creates a RandomAccessInput covering only the jump-table data or null.
    Parameters:
      `slice` - backing data, holding both blocks and jump-table.
      `offset` - relative to the backing data.
      `length` - full length of the IndexedDISI, including blocks and jump-table data.
      `jumpTableEntryCount` - the number of blocks covered by the jump-table.
    Returns:
      a jumpTable containing the block jump-data or null if no such table exists.
    Throws:
      `IOException` - if a RandomAccessInput could not be created from slice.

  - ### asDocIndexIterator

    public static KnnVectorValues.DocIndexIterator asDocIndexIterator(IndexedDISI disi)
    Returns an iterator that delegates to the IndexedDISI. Advancing this iterator will advance the underlying IndexedDISI, and vice-versa.

  - ### advance

    public int advance(int target) throws IOException
    Description copied from class: `DocIdSetIterator`
    Advances to the first beyond the current whose document number is greater than or equal to target, and returns the document number itself. Exhausts the iterator and returns `DocIdSetIterator.NO_MORE_DOCS` if target is greater than the highest document number in the set.

    The behavior of this method is undefined when called with `target ≤ current` , or after the iterator has exhausted. Both cases may result in unpredicted behavior.

    When `target > current` it behaves as if written:

         int advance(int target) {
           int doc;
           while ((doc = nextDoc()) < target) {
           }
           return doc;
         }

    Some implementations are considerably more efficient than that.

    NOTE: this method may be called with `DocIdSetIterator.NO_MORE_DOCS` for efficiency by some Scorers. If your implementation cannot efficiently determine that it should exhaust, it is recommended that you check for that value in each call to this method.

    Specified by:
      `advance` in class `DocIdSetIterator`
    Throws:
      `IOException`

  - ### intoBitSet

    public void intoBitSet(int upTo, FixedBitSet bitSet, int offset) throws IOException
    Description copied from class: `DocIdSetIterator`
    Load doc IDs into a `FixedBitSet`. This should behave exactly as if implemented as below, which is the default implementation:

         for (int doc = docID(); doc < upTo; doc = nextDoc()) {
           bitSet.set(doc - offset);
         }

    Note: `offset` must be less than or equal to the `current doc ID`. Behaviour is undefined if this iterator is unpositioned.

    Note: It is important not to clear bits from `bitSet` that may be already set.

    Note: `offset` may be negative.

    Overrides:
      `intoBitSet` in class `DocIdSetIterator`
    Throws:
      `IOException`

  - ### advanceExact

    public boolean advanceExact(int target) throws IOException
    Throws:
      `IOException`

  - ### nextDoc

    public int nextDoc() throws IOException
    Description copied from class: `DocIdSetIterator`
    Advances to the next document in the set and returns the doc it is currently on, or `DocIdSetIterator.NO_MORE_DOCS` if there are no more docs in the set.
    NOTE: after the iterator has exhausted you should not call this method, as it may result in unpredicted behavior.
    Specified by:
      `nextDoc` in class `DocIdSetIterator`
    Throws:
      `IOException`

  - ### index

    public int index()

  - ### cost

    public long cost()
    Description copied from class: `DocIdSetIterator`
    Returns the estimated cost of this `DocIdSetIterator`.

    This is generally an upper bound of the number of documents this iterator might match, but may be a rough heuristic, hardcoded value, or otherwise completely inaccurate.

    Specified by:
      `cost` in class `DocIdSetIterator`

  - ### docIDRunEnd

    public int docIDRunEnd() throws IOException
    Description copied from class: `DocIdSetIterator`
    Returns the end of the run of consecutive doc IDs that match this `DocIdSetIterator` and that contains the current `DocIdSetIterator.docID()`, that is: one plus the last doc ID of the run.
    1. The returned doc is greater than `DocIdSetIterator.docID()`.
    2. All docs in range `[docID(), docIDRunEnd())` match this iterator.
    3. The current position of this iterator is not affected by calling `DocIdSetIterator.docIDRunEnd()`.

    Note: It is illegal to call this method when the iterator is exhausted or not positioned.

    The default implementation assumes runs of a single doc ID and returns `DocIdSetIterator.docID()`) + 1.

    Overrides:
      `docIDRunEnd` in class `DocIdSetIterator`
    Throws:
      `IOException`