/*
 * Licensed to the Apache Software Foundation (ASF) under one or more
 * contributor license agreements.  See the NOTICE file distributed with
 * this work for additional information regarding copyright ownership.
 * The ASF licenses this file to You under the Apache License, Version 2.0
 * (the "License"); you may not use this file except in compliance with
 * the License.  You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

/**
 * Text analysis.
 *
 * <p>API and code to convert text into indexable/searchable tokens. Covers {@link
 * org.apache.lucene.analysis.Analyzer} and related classes.
 *
 * <h2>Parsing? Tokenization? Analysis!</h2>
 *
 * <p>Lucene, an indexing and search library, accepts only plain text input.
 *
 * <h2>Parsing</h2>
 *
 * <p>Applications that build their search capabilities upon Lucene may support documents in various
 * formats &ndash; HTML, XML, PDF, Word &ndash; just to name a few. Lucene does not care about the
 * <i>Parsing</i> of these and other document formats, and it is the responsibility of the
 * application using Lucene to use an appropriate <i>Parser</i> to convert the original format into
 * plain text before passing that plain text to Lucene.
 *
 * <h2>Tokenization</h2>
 *
 * <p>Plain text passed to Lucene for indexing goes through a process generally called tokenization.
 * Tokenization is the process of breaking input text into small indexing elements &ndash; tokens.
 * The way input text is broken into tokens heavily influences how people will then be able to
 * search for that text. For instance, sentences beginnings and endings can be identified to provide
 * for more accurate phrase and proximity searches (though sentence identification is not provided
 * by Lucene).
 *
 * <p>In some cases simply breaking the input text into tokens is not enough &ndash; a deeper
 * <i>Analysis</i> may be needed. Lucene includes both pre- and post-tokenization analysis
 * facilities.
 *
 * <p>Pre-tokenization analysis can include (but is not limited to) stripping HTML markup, and
 * transforming or removing text matching arbitrary patterns or sets of fixed strings.
 *
 * <p>There are many post-tokenization steps that can be done, including (but not limited to):
 *
 * <ul>
 *   <li><a href="http://en.wikipedia.org/wiki/Stemming">Stemming</a> &ndash; Replacing words with
 *       their stems. For instance with English stemming "bikes" is replaced with "bike"; now query
 *       "bike" can find both documents containing "bike" and those containing "bikes".
 *   <li><a href="http://en.wikipedia.org/wiki/Stop_words">Stop Words Filtering</a> &ndash; Common
 *       words like "the", "and" and "a" rarely add any value to a search. Removing them shrinks the
 *       index size and increases performance. It may also reduce some "noise" and actually improve
 *       search quality.
 *   <li><a href="http://en.wikipedia.org/wiki/Text_normalization">Text Normalization</a> &ndash;
 *       Stripping accents and other character markings can make for better searching.
 *   <li><a href="http://en.wikipedia.org/wiki/Synonym">Synonym Expansion</a> &ndash; Adding in
 *       synonyms at the same token position as the current word can mean better matching when users
 *       search with words in the synonym set.
 * </ul>
 *
 * <h2>Core Analysis</h2>
 *
 * <p>The analysis package provides the mechanism to convert Strings and Readers into tokens that
 * can be indexed by Lucene. There are four main classes in the package from which all analysis
 * processes are derived. These are:
 *
 * <ul>
 *   <li>{@link org.apache.lucene.analysis.Analyzer} &ndash; An <code>Analyzer</code> is responsible
 *       for supplying a {@link org.apache.lucene.analysis.TokenStream} which can be consumed by the
 *       indexing and searching processes. See below for more information on implementing your own
 *       {@link org.apache.lucene.analysis.Analyzer}. Most of the time, you can use an anonymous
 *       subclass of {@link org.apache.lucene.analysis.Analyzer}.
 *   <li>{@link org.apache.lucene.analysis.CharFilter} &ndash; <code>CharFilter</code> extends
 *       {@link java.io.Reader} to transform the text before it is tokenized, while providing
 *       corrected character offsets to account for these modifications. This capability allows
 *       highlighting to function over the original text when indexed tokens are created from <code>
 *       CharFilter</code>-modified text with offsets that are not the same as those in the original
 *       text. {@link org.apache.lucene.analysis.Tokenizer#setReader(java.io.Reader)} accept <code>
 *       CharFilter</code>s. <code>CharFilter</code>s may be chained to perform multiple
 *       pre-tokenization modifications.
 *   <li>{@link org.apache.lucene.analysis.Tokenizer} &ndash; A <code>Tokenizer</code> is a {@link
 *       org.apache.lucene.analysis.TokenStream} and is responsible for breaking up incoming text
 *       into tokens. In many cases, an {@link org.apache.lucene.analysis.Analyzer} will use a
 *       {@link org.apache.lucene.analysis.Tokenizer} as the first step in the analysis process.
 *       However, to modify text prior to tokenization, use a {@link
 *       org.apache.lucene.analysis.CharFilter} subclass (see above).
 *   <li>{@link org.apache.lucene.analysis.TokenFilter} &ndash; A <code>TokenFilter</code> is a
 *       {@link org.apache.lucene.analysis.TokenStream} and is responsible for modifying tokens that
 *       have been created by the <code>Tokenizer</code>. Common modifications performed by a <code>
 *       TokenFilter</code> are: deletion, stemming, synonym injection, and case folding. Not all
 *       <code>Analyzer</code>s require <code>TokenFilter</code>s.
 * </ul>
 *
 * <h2>Hints, Tips and Traps</h2>
 *
 * <p>The relationship between {@link org.apache.lucene.analysis.Analyzer} and {@link
 * org.apache.lucene.analysis.CharFilter}s, {@link org.apache.lucene.analysis.Tokenizer}s, and
 * {@link org.apache.lucene.analysis.TokenFilter}s is sometimes confusing. To ease this confusion,
 * here are some clarifications:
 *
 * <ul>
 *   <li>The {@link org.apache.lucene.analysis.Analyzer} is a <strong>factory</strong> for analysis
 *       chains. <code>Analyzer</code>s don't process text, <code>Analyzer</code>s construct <code>
 *       CharFilter</code>s, <code>Tokenizer</code>s, and/or <code>TokenFilter</code>s that process
 *       text. An <code>Analyzer</code> has two tasks: to produce {@link
 *       org.apache.lucene.analysis.TokenStream}s that accept a reader and produces tokens, and to
 *       wrap or otherwise pre-process {@link java.io.Reader} objects.
 *   <li>The {@link org.apache.lucene.analysis.CharFilter} is a subclass of {@link java.io.Reader}
 *       that supports offset tracking.
 *   <li>The{@link org.apache.lucene.analysis.Tokenizer} is only responsible for <u>breaking</u> the
 *       input text into tokens.
 *   <li>The{@link org.apache.lucene.analysis.TokenFilter} modifies a stream of tokens and their
 *       contents.
 *   <li>{@link org.apache.lucene.analysis.Tokenizer} is a {@link
 *       org.apache.lucene.analysis.TokenStream}, but {@link org.apache.lucene.analysis.Analyzer} is
 *       not.
 *   <li>{@link org.apache.lucene.analysis.Analyzer} is "field aware", but {@link
 *       org.apache.lucene.analysis.Tokenizer} is not. {@link org.apache.lucene.analysis.Analyzer}s
 *       may take a field name into account when constructing the {@link
 *       org.apache.lucene.analysis.TokenStream}.
 * </ul>
 *
 * <p>If you want to use a particular combination of <code>CharFilter</code>s, a <code>Tokenizer
 * </code>, and some <code>TokenFilter</code>s, the simplest thing is often to create an anonymous
 * subclass of {@link org.apache.lucene.analysis.Analyzer}, provide {@link
 * org.apache.lucene.analysis.Analyzer#createComponents(String)} and perhaps also {@link
 * org.apache.lucene.analysis.Analyzer#initReader(String, java.io.Reader)}. However, if you need the
 * same set of components over and over in many places, you can make a subclass of {@link
 * org.apache.lucene.analysis.Analyzer}. In fact, Apache Lucene supplies a large family of <code>
 * Analyzer</code> classes that deliver useful analysis chains. The most common of these is the <a
 * href="{@docRoot}/org/apache/lucene/analysis/standard/StandardAnalyzer.html">StandardAnalyzer</a>.
 * Many applications will have a long and industrious life with nothing more than the <code>
 * StandardAnalyzer</code>. The <a
 * href="{@docRoot}/../analysis/common/overview-summary.html">analysis-common</a> library provides
 * many pre-existing analyzers for various languages. The analysis-common library also allows to
 * configure a custom Analyzer without subclassing using the <a
 * href="{@docRoot}/../analysis/common/org/apache/lucene/analysis/custom/CustomAnalyzer.html">CustomAnalyzer</a>
 * class.
 *
 * <p>Aside from the <code>StandardAnalyzer</code>, Lucene includes several components containing
 * analysis components, all under the 'analysis' directory of the distribution. Some of these
 * support particular languages, others integrate external components. The 'common' subdirectory has
 * some noteworthy general-purpose analyzers, including the <a
 * href="{@docRoot}/../analysis/common/org/apache/lucene/analysis/miscellaneous/PerFieldAnalyzerWrapper.html">PerFieldAnalyzerWrapper</a>.
 * Most <code>Analyzer</code>s perform the same operation on all {@link
 * org.apache.lucene.document.Field}s. The PerFieldAnalyzerWrapper can be used to associate a
 * different <code>Analyzer</code> with different {@link org.apache.lucene.document.Field}s. There
 * is a great deal of functionality in the analysis area, you should study it carefully to find the
 * pieces you need.
 *
 * <p>Analysis is one of the main causes of slow indexing. Simply put, the more you analyze the
 * slower the indexing (in most cases). Perhaps your application would be just fine using the simple
 * WhitespaceTokenizer combined with a StopFilter. The benchmark/ library can be useful for testing
 * out the speed of the analysis process.
 *
 * <h2>Invoking the Analyzer</h2>
 *
 * <p>Applications usually do not invoke analysis &ndash; Lucene does it for them. Applications
 * construct <code>Analyzer</code>s and pass then into Lucene, as follows:
 *
 * <ul>
 *   <li>At indexing, as a consequence of {@link
 *       org.apache.lucene.index.IndexWriter#addDocument(Iterable) addDocument(doc)}, the <code>
 *       Analyzer</code> in effect for indexing is invoked for each indexed field of the added
 *       document.
 *   <li>At search, a <code>QueryParser</code> may invoke the Analyzer during parsing. Note that for
 *       some queries, analysis does not take place, e.g. wildcard queries.
 * </ul>
 *
 * <p>However an application might invoke Analysis of any text for testing or for any other purpose,
 * something like:
 *
 * <pre class="prettyprint" id="analysis-workflow">
 *     Version matchVersion = Version.LUCENE_XY; // Substitute desired Lucene version for XY
 *     Analyzer analyzer = new StandardAnalyzer(matchVersion); // or any other analyzer
 *     TokenStream ts = analyzer.tokenStream("myfield", new StringReader("some text goes here"));
 *     // The Analyzer class will construct the Tokenizer, TokenFilter(s), and CharFilter(s),
 *     //   and pass the resulting Reader to the Tokenizer.
 *     OffsetAttribute offsetAtt = ts.addAttribute(OffsetAttribute.class);
 *
 *     try {
 *       ts.reset(); // Resets this stream to the beginning. (Required)
 *       while (ts.incrementToken()) {
 *         // Use {@link org.apache.lucene.util.AttributeSource#reflectAsString(boolean)}
 *         // for token stream debugging.
 *         System.out.println("token: " + ts.reflectAsString(true));
 *
 *         System.out.println("token start offset: " + offsetAtt.startOffset());
 *         System.out.println("  token end offset: " + offsetAtt.endOffset());
 *       }
 *       ts.end();   // Perform end-of-stream operations, e.g. set the final offset.
 *     } finally {
 *       ts.close(); // Release resources associated with this stream.
 *     }
 * </pre>
 *
 * <h2>Indexing Analysis vs. Search Analysis</h2>
 *
 * <p>Selecting the "correct" analyzer is crucial for search quality, and can also affect indexing
 * and search performance. The "correct" analyzer for your application will depend on what your
 * input text looks like and what problem you are trying to solve. Lucene java's wiki page <a
 * href="http://wiki.apache.org/lucene-java/AnalysisParalysis">AnalysisParalysis</a> provides some
 * data on "analyzing your analyzer". Here are some rules of thumb:
 *
 * <ol>
 *   <li>Test test test... (did we say test?)
 *   <li>Beware of too much analysis &ndash; it might hurt indexing performance.
 *   <li>Start with the same analyzer for indexing and search, otherwise searches would not find
 *       what they are supposed to...
 *   <li>In some cases a different analyzer is required for indexing and search, for instance:
 *       <ul>
 *         <li>Certain searches require more stop words to be filtered. (i.e. more than those that
 *             were filtered at indexing.)
 *         <li>Query expansion by synonyms, acronyms, auto spell correction, etc.
 *       </ul>
 *       This might sometimes require a modified analyzer &ndash; see the next section on how to do
 *       that.
 * </ol>
 *
 * <h2>Implementing your own Analyzer and Analysis Components</h2>
 *
 * <p>Creating your own Analyzer is straightforward. Your Analyzer should subclass {@link
 * org.apache.lucene.analysis.Analyzer}. It can use existing analysis components &mdash;
 * CharFilter(s) <i>(optional)</i>, a Tokenizer, and TokenFilter(s) <i>(optional)</i> &mdash; or
 * components you create, or a combination of existing and newly created components. Before pursuing
 * this approach, you may find it worthwhile to explore the <a
 * href="{@docRoot}/../analysis/common/overview-summary.html">analysis-common</a> library and/or ask
 * on the <a href="http://lucene.apache.org/core/discussion.html">java-user@lucene.apache.org
 * mailing list</a> first to see if what you need already exists. If you are still committed to
 * creating your own Analyzer, have a look at the source code of any one of the many samples located
 * in this package.
 *
 * <p>The following sections discuss some aspects of implementing your own analyzer.
 *
 * <h3>Field Section Boundaries</h3>
 *
 * <p>When {@link org.apache.lucene.document.Document#add(org.apache.lucene.index.IndexableField)
 * document.add(field)} is called multiple times for the same field name, we could say that each
 * such call creates a new section for that field in that document. In fact, a separate call to
 * {@link org.apache.lucene.analysis.Analyzer#tokenStream(java.lang.String, java.io.Reader)
 * tokenStream(field,reader)} would take place for each of these so called "sections". However, the
 * default Analyzer behavior is to treat all these sections as one large section. This allows phrase
 * search and proximity search to seamlessly cross boundaries between these "sections". In other
 * words, if a certain field "f" is added like this:
 *
 * <PRE class="prettyprint">
 *     document.add(new Field("f","first ends",...);
 *     document.add(new Field("f","starts two",...);
 *     indexWriter.addDocument(document);
 * </PRE>
 *
 * <p>Then, a phrase search for "ends starts" would find that document. Where desired, this behavior
 * can be modified by introducing a "position gap" between consecutive field "sections", simply by
 * overriding {@link org.apache.lucene.analysis.Analyzer#getPositionIncrementGap(java.lang.String)
 * Analyzer.getPositionIncrementGap(fieldName)}:
 *
 * <pre class="prettyprint">
 *   Version matchVersion = Version.LUCENE_XY; // Substitute desired Lucene version for XY
 *   Analyzer myAnalyzer = new StandardAnalyzer(matchVersion) {
 *     public int getPositionIncrementGap(String fieldName) {
 *       return 10;
 *     }
 *   };
 * </pre>
 *
 * <h3>End of Input Cleanup</h3>
 *
 * <p>At the ends of each field, Lucene will call the {@link
 * org.apache.lucene.analysis.TokenStream#end()}. The components of the token stream (the tokenizer
 * and the token filters) <strong>must</strong> put accurate values into the token attributes to
 * reflect the situation at the end of the field. The Offset attribute must contain the final offset
 * (the total number of characters processed) in both start and end. Attributes like PositionLength
 * must be correct.
 *
 * <p>The base method{@link org.apache.lucene.analysis.TokenStream#end()} sets PositionIncrement to
 * 0, which is required. Other components must override this method to fix up the other attributes.
 *
 * <h3>Token Position Increments</h3>
 *
 * <p>By default, TokenStream arranges for the {@link
 * org.apache.lucene.analysis.tokenattributes.PositionIncrementAttribute#getPositionIncrement()
 * position increment} of all tokens to be one. This means that the position stored for that token
 * in the index would be one more than that of the previous token. Recall that phrase and proximity
 * searches rely on position info.
 *
 * <p>If the selected analyzer filters the stop words "is" and "the", then for a document containing
 * the string "blue is the sky", only the tokens "blue", "sky" are indexed, with position("sky") = 3
 * + position("blue"). Now, a phrase query "blue is the sky" would find that document, because the
 * same analyzer filters the same stop words from that query. But the phrase query "blue sky" would
 * not find that document because the position increment between "blue" and "sky" is only 1.
 *
 * <p>If this behavior does not fit the application needs, the query parser needs to be configured
 * to not take position increments into account when generating phrase queries.
 *
 * <p>Note that a filter that filters <strong>out</strong> tokens <strong>must</strong> increment
 * the position increment in order not to generate corrupt tokenstream graphs. Here is the logic
 * used by StopFilter to increment positions when filtering out tokens:
 *
 * <pre class="prettyprint">
 *   public TokenStream tokenStream(final String fieldName, Reader reader) {
 *     final TokenStream ts = someAnalyzer.tokenStream(fieldName, reader);
 *     TokenStream res = new TokenStream() {
 *       CharTermAttribute termAtt = addAttribute(CharTermAttribute.class);
 *       PositionIncrementAttribute posIncrAtt = addAttribute(PositionIncrementAttribute.class);
 *
 *       public boolean incrementToken() throws IOException {
 *         int extraIncrement = 0;
 *         while (true) {
 *           boolean hasNext = ts.incrementToken();
 *           if (hasNext) {
 *             if (stopWords.contains(termAtt.toString())) {
 *               extraIncrement += posIncrAtt.getPositionIncrement(); // filter this word
 *               continue;
 *             }
 *             if (extraIncrement &gt; 0) {
 *               posIncrAtt.setPositionIncrement(posIncrAtt.getPositionIncrement()+extraIncrement);
 *             }
 *           }
 *           return hasNext;
 *         }
 *       }
 *     };
 *     return res;
 *   }
 * </pre>
 *
 * <p>A few more use cases for modifying position increments are:
 *
 * <ol>
 *   <li>Inhibiting phrase and proximity matches in sentence boundaries &ndash; for this, a
 *       tokenizer that identifies a new sentence can add 1 to the position increment of the first
 *       token of the new sentence.
 *   <li>Injecting synonyms &ndash; synonyms of a token should be created at the same position as
 *       the original token, and the output order of the original token and the injected synonym is
 *       undefined as long as they both leave from the same position. As result, all synonyms of a
 *       token would be considered to appear in exactly the same position as that token, and so
 *       would they be seen by phrase and proximity searches. For multi-token synonyms to work
 *       correctly, you should use {@code SynoymGraphFilter} at search time only.
 * </ol>
 *
 * <h3>Token Position Length</h3>
 *
 * <p>By default, all tokens created by Analyzers and Tokenizers have a {@link
 * org.apache.lucene.analysis.tokenattributes.PositionLengthAttribute#getPositionLength() position
 * length} of one. This means that the token occupies a single position. This attribute is not
 * indexed and thus not taken into account for positional queries, but is used by eg. suggesters.
 *
 * <p>The main use case for positions lengths is multi-word synonyms. With single-word synonyms,
 * setting the position increment to 0 is enough to denote the fact that two words are synonyms, for
 * example:
 *
 * <table>
 * <caption>table showing position increments of 1 and 0 for red and magenta, respectively</caption>
 * <tr><td>Term</td><td>red</td><td>magenta</td></tr>
 * <tr><td>Position increment</td><td>1</td><td>0</td></tr>
 * </table>
 *
 * <p>Given that position(magenta) = 0 + position(red), they are at the same position, so anything
 * working with analyzers will return the exact same result if you replace "magenta" with "red" in
 * the input. However, multi-word synonyms are more tricky. Let's say that you want to build a
 * TokenStream where "IBM" is a synonym of "International Business Machines". Position increments
 * are not enough anymore:
 *
 * <table>
 * <caption>position increments where international is zero</caption>
 * <tr><td>Term</td><td>IBM</td><td>International</td><td>Business</td><td>Machines</td></tr>
 * <tr><td>Position increment</td><td>1</td><td>0</td><td>1</td><td>1</td></tr>
 * </table>
 *
 * <p>The problem with this token stream is that "IBM" is at the same position as "International"
 * although it is a synonym with "International Business Machines" as a whole. Setting the position
 * increment of "Business" and "Machines" to 0 wouldn't help as it would mean that "International"
 * is a synonym of "Business". The only way to solve this issue is to make "IBM" span across 3
 * positions, this is where position lengths come to rescue.
 *
 * <table>
 * <caption>position lengths where IBM is three</caption>
 * <tr><td>Term</td><td>IBM</td><td>International</td><td>Business</td><td>Machines</td></tr>
 * <tr><td>Position increment</td><td>1</td><td>0</td><td>1</td><td>1</td></tr>
 * <tr><td>Position length</td><td>3</td><td>1</td><td>1</td><td>1</td></tr>
 * </table>
 *
 * <p>This new attribute makes clear that "IBM" and "International Business Machines" start and end
 * at the same positions. <a id="corrupt"></a>
 *
 * <h3>How to not write corrupt token streams</h3>
 *
 * <p>There are a few rules to observe when writing custom Tokenizers and TokenFilters:
 *
 * <ul>
 *   <li>The first position increment must be &gt; 0.
 *   <li>Positions must not go backward.
 *   <li>Tokens that have the same start position must have the same start offset.
 *   <li>Tokens that have the same end position (taking into account the position length) must have
 *       the same end offset.
 *   <li>Tokenizers must call {@link org.apache.lucene.util.AttributeSource#clearAttributes()} in
 *       incrementToken().
 *   <li>Tokenizers must override {@link org.apache.lucene.analysis.TokenStream#end()}, and pass the
 *       final offset (the total number of input characters processed) to both parameters of {@link
 *       org.apache.lucene.analysis.tokenattributes.OffsetAttribute#setOffset(int, int)}.
 * </ul>
 *
 * <p>Although these rules might seem easy to follow, problems can quickly happen when chaining
 * badly implemented filters that play with positions and offsets, such as synonym or n-grams
 * filters. Here are good practices for writing correct filters:
 *
 * <ul>
 *   <li>Token filters should not modify offsets. If you feel that your filter would need to modify
 *       offsets, then it should probably be implemented as a tokenizer.
 *   <li>Token filters should not insert positions. If a filter needs to add tokens, then they
 *       should all have a position increment of 0.
 *   <li>When they add tokens, token filters should call {@link
 *       org.apache.lucene.util.AttributeSource#clearAttributes()} first.
 *   <li>When they remove tokens, token filters should increment the position increment of the
 *       following token.
 *   <li>Token filters should preserve position lengths.
 * </ul>
 *
 * <h2>TokenStream API</h2>
 *
 * <p>"Flexible Indexing" summarizes the effort of making the Lucene indexer pluggable and
 * extensible for custom index formats. A fully customizable indexer means that users will be able
 * to store custom data structures on disk. Therefore the analysis API must transport custom types
 * of data from the documents to the indexer. (It also supports communications amongst the analysis
 * components.)
 *
 * <h3>Attribute and AttributeSource</h3>
 *
 * <p>Classes {@link org.apache.lucene.util.Attribute} and {@link
 * org.apache.lucene.util.AttributeSource} serve as the basis upon which the analysis elements of
 * "Flexible Indexing" are implemented. An Attribute holds a particular piece of information about a
 * text token. For example, {@link org.apache.lucene.analysis.tokenattributes.CharTermAttribute}
 * contains the term text of a token, and {@link
 * org.apache.lucene.analysis.tokenattributes.OffsetAttribute} contains the start and end character
 * offsets of a token. An AttributeSource is a collection of Attributes with a restriction: there
 * may be only one instance of each attribute type. TokenStream now extends AttributeSource, which
 * means that one can add Attributes to a TokenStream. Since TokenFilter extends TokenStream, all
 * filters are also AttributeSources.
 *
 * <p>Lucene provides seven Attributes out of the box:
 *
 * <table class="padding3">
 *   <caption>common bundled attributes</caption>
 *   <tbody style="border: 1px solid">
 *   <tr>
 *     <td>{@link org.apache.lucene.analysis.tokenattributes.CharTermAttribute}</td>
 *     <td>
 *       The term text of a token.  Implements {@link java.lang.CharSequence}
 *       (providing methods length() and charAt(), and allowing e.g. for direct
 *       use with regular expression {@link java.util.regex.Matcher}s) and
 *       {@link java.lang.Appendable} (allowing the term text to be appended to.)
 *     </td>
 *   </tr>
 *   <tr>
 *     <td>{@link org.apache.lucene.analysis.tokenattributes.OffsetAttribute}</td>
 *     <td>The start and end offset of a token in characters.</td>
 *   </tr>
 *   <tr>
 *     <td>{@link org.apache.lucene.analysis.tokenattributes.PositionIncrementAttribute}</td>
 *     <td>See above for detailed information about position increment.</td>
 *   </tr>
 *   <tr>
 *     <td>{@link org.apache.lucene.analysis.tokenattributes.PositionLengthAttribute}</td>
 *     <td>The number of positions occupied by a token.</td>
 *   </tr>
 *   <tr>
 *     <td>{@link org.apache.lucene.analysis.tokenattributes.PayloadAttribute}</td>
 *     <td>The payload that a Token can optionally have.</td>
 *   </tr>
 *   <tr>
 *     <td>{@link org.apache.lucene.analysis.tokenattributes.TypeAttribute}</td>
 *     <td>The type of the token. Default is 'word'.</td>
 *   </tr>
 *   <tr>
 *     <td>{@link org.apache.lucene.analysis.tokenattributes.FlagsAttribute}</td>
 *     <td>Optional flags a token can have.</td>
 *   </tr>
 *   <tr>
 *     <td>{@link org.apache.lucene.analysis.tokenattributes.KeywordAttribute}</td>
 *     <td>
 *       Keyword-aware TokenStreams/-Filters skip modification of tokens that
 *       return true from this attribute's isKeyword() method.
 *     </td>
 *   </tr>
 *   </tbody>
 * </table>
 *
 * <h3>More Requirements for Analysis Component Classes</h3>
 *
 * Due to the historical development of the API, there are some perhaps less than obvious
 * requirements to implement analysis components classes.
 *
 * <h4 id="analysis-lifetime">Token Stream Lifetime</h4>
 *
 * The code fragment of the <a href="#analysis-workflow">analysis workflow protocol</a> above shows
 * a token stream being obtained, used, and then left for garbage. However, that does not mean that
 * the components of that token stream will, in fact, be discarded. The default is just the
 * opposite. {@link org.apache.lucene.analysis.Analyzer} applies a reuse strategy to the tokenizer
 * and the token filters. It will reuse them. For each new input, it calls {@link
 * org.apache.lucene.analysis.Tokenizer#setReader(java.io.Reader)} to set the input. Your components
 * must be prepared for this scenario, as described below.
 *
 * <h4>Tokenizer</h4>
 *
 * <ul>
 *   <li>You should create your tokenizer class by extending {@link
 *       org.apache.lucene.analysis.Tokenizer}.
 *   <li>Your tokenizer <strong>must</strong> override {@link
 *       org.apache.lucene.analysis.TokenStream#end()}. Your implementation <strong>must</strong>
 *       call <code>super.end()</code>. It must set a correct final offset into the offset
 *       attribute, and finish up and other attributes to reflect the end of the stream.
 *   <li>If your tokenizer overrides {@link org.apache.lucene.analysis.TokenStream#reset()} or
 *       {@link org.apache.lucene.analysis.TokenStream#close()}, it <strong>must</strong> call the
 *       corresponding superclass method.
 * </ul>
 *
 * <h4>Token Filter</h4>
 *
 * You should create your token filter class by extending {@link
 * org.apache.lucene.analysis.TokenFilter}. If your token filter overrides {@link
 * org.apache.lucene.analysis.TokenStream#reset()}, {@link
 * org.apache.lucene.analysis.TokenStream#end()} or {@link
 * org.apache.lucene.analysis.TokenStream#close()}, it <strong>must</strong> call the corresponding
 * superclass method.
 *
 * <h4>Creating delegates</h4>
 *
 * Forwarding classes (those which extend {@link org.apache.lucene.analysis.Tokenizer} but delegate
 * selected logic to another tokenizer) must also set the reader to the delegate in the overridden
 * {@link org.apache.lucene.analysis.Tokenizer#reset()} method, e.g.:
 *
 * <pre class="prettyprint">
 *     public class ForwardingTokenizer extends Tokenizer {
 *        private Tokenizer delegate;
 *        ...
 *        {@literal @Override}
 *        public void reset() {
 *           super.reset();
 *           delegate.setReader(this.input);
 *           delegate.reset();
 *        }
 *     }
 * </pre>
 *
 * <h3>Testing Your Analysis Component</h3>
 *
 * <p>The lucene-test-framework component defines <a
 * href="{@docRoot}/../test-framework/org/apache/lucene/tests/analysis/BaseTokenStreamTestCase.html">BaseTokenStreamTestCase</a>.
 * By extending this class, you can create JUnit tests that validate that your Analyzer and/or
 * analysis components correctly implement the protocol. The checkRandomData methods of that class
 * are particularly effective in flushing out errors.
 *
 * <h3>Using the TokenStream API</h3>
 *
 * There are a few important things to know in order to use the new API efficiently which are
 * summarized here. You may want to walk through the example below first and come back to this
 * section afterwards.
 *
 * <ol>
 *   <li>Please keep in mind that an AttributeSource can only have one instance of a particular
 *       Attribute. Furthermore, if a chain of a TokenStream and multiple TokenFilters is used, then
 *       all TokenFilters in that chain share the Attributes with the TokenStream.
 *   <li>Attribute instances are reused for all tokens of a document. Thus, a TokenStream/-Filter
 *       needs to update the appropriate Attribute(s) in incrementToken(). The consumer, commonly
 *       the Lucene indexer, consumes the data in the Attributes and then calls incrementToken()
 *       again until it returns false, which indicates that the end of the stream was reached. This
 *       means that in each call of incrementToken() a TokenStream/-Filter can safely overwrite the
 *       data in the Attribute instances.
 *   <li>For performance reasons a TokenStream/-Filter should add/get Attributes during
 *       instantiation; i.e., create an attribute in the constructor and store references to it in
 *       an instance variable. Using an instance variable instead of calling
 *       addAttribute()/getAttribute() in incrementToken() will avoid attribute lookups for every
 *       token in the document.
 *   <li>All methods in AttributeSource are idempotent, which means calling them multiple times
 *       always yields the same result. This is especially important to know for addAttribute(). The
 *       method takes the <b>type</b> (<code>Class</code>) of an Attribute as an argument and
 *       returns an <b>instance</b>. If an Attribute of the same type was previously added, then the
 *       already existing instance is returned, otherwise a new instance is created and returned.
 *       Therefore TokenStreams/-Filters can safely call addAttribute() with the same Attribute type
 *       multiple times. Even consumers of TokenStreams should normally call addAttribute() instead
 *       of getAttribute(), because it would not fail if the TokenStream does not have this
 *       Attribute (getAttribute() would throw an IllegalArgumentException, if the Attribute is
 *       missing). More advanced code could simply check with hasAttribute(), if a TokenStream has
 *       it, and may conditionally leave out processing for extra performance.
 * </ol>
 *
 * <h3>Example</h3>
 *
 * <p>In this example we will create a WhiteSpaceTokenizer and use a LengthFilter to suppress all
 * words that have only two or fewer characters. The LengthFilter is part of the Lucene core and its
 * implementation will be explained here to illustrate the usage of the TokenStream API.
 *
 * <p>Then we will develop a custom Attribute, a PartOfSpeechAttribute, and add another filter to
 * the chain which utilizes the new custom attribute, and call it PartOfSpeechTaggingFilter.
 *
 * <h4>Whitespace tokenization</h4>
 *
 * <pre class="prettyprint">
 * public class MyAnalyzer extends Analyzer {
 *
 *   private Version matchVersion;
 *
 *   public MyAnalyzer(Version matchVersion) {
 *     this.matchVersion = matchVersion;
 *   }
 *
 *   {@literal @Override}
 *   protected TokenStreamComponents createComponents(String fieldName) {
 *     return new TokenStreamComponents(new WhitespaceTokenizer(matchVersion));
 *   }
 *
 *   public static void main(String[] args) throws IOException {
 *     // text to tokenize
 *     final String text = "This is a demo of the TokenStream API";
 *
 *     Version matchVersion = Version.LUCENE_XY; // Substitute desired Lucene version for XY
 *     MyAnalyzer analyzer = new MyAnalyzer(matchVersion);
 *     TokenStream stream = analyzer.tokenStream("field", new StringReader(text));
 *
 *     // get the CharTermAttribute from the TokenStream
 *     CharTermAttribute termAtt = stream.addAttribute(CharTermAttribute.class);
 *
 *     try {
 *       stream.reset();
 *
 *       // print all tokens until stream is exhausted
 *       while (stream.incrementToken()) {
 *         System.out.println(termAtt.toString());
 *       }
 *
 *       stream.end();
 *     } finally {
 *       stream.close();
 *     }
 *   }
 * }
 * </pre>
 *
 * In this easy example a simple white space tokenization is performed. In main() a loop consumes
 * the stream and prints the term text of the tokens by accessing the CharTermAttribute that the
 * WhitespaceTokenizer provides. Here is the output:
 *
 * <pre>
 * This
 * is
 * a
 * demo
 * of
 * the
 * new
 * TokenStream
 * API
 * </pre>
 *
 * <h4>Adding a LengthFilter</h4>
 *
 * We want to suppress all tokens that have 2 or less characters. We can do that easily by adding a
 * LengthFilter to the chain. Only the <code>createComponents()</code> method in our analyzer needs
 * to be changed:
 *
 * <pre class="prettyprint">
 *   {@literal @Override}
 *   protected TokenStreamComponents createComponents(String fieldName) {
 *     final Tokenizer source = new WhitespaceTokenizer(matchVersion);
 *     TokenStream result = new LengthFilter(true, source, 3, Integer.MAX_VALUE);
 *     return new TokenStreamComponents(source, result);
 *   }
 * </pre>
 *
 * Note how now only words with 3 or more characters are contained in the output:
 *
 * <pre>
 * This
 * demo
 * the
 * new
 * TokenStream
 * API
 * </pre>
 *
 * Now let's take a look at how the LengthFilter is implemented:
 *
 * <pre class="prettyprint">
 * public final class LengthFilter extends FilteringTokenFilter {
 *
 *   private final int min;
 *   private final int max;
 *
 *   private final CharTermAttribute termAtt = addAttribute(CharTermAttribute.class);
 *
 *   &#47;**
 *    * Create a new LengthFilter. This will filter out tokens whose
 *    * CharTermAttribute is either too short
 *    * (&lt; min) or too long (&gt; max).
 *    * {@literal @param} version the Lucene match version
 *    * {@literal @param} in      the TokenStream to consume
 *    * {@literal @param} min     the minimum length
 *    * {@literal @param} max     the maximum length
 *    *&#47;
 *  public LengthFilter(Version version, TokenStream in, int min, int max) {
 *     super(version, in);
 *     this.min = min;
 *     this.max = max;
 *   }
 *
 *   {@literal @Override}
 *   public boolean accept() {
 *     final int len = termAtt.length();
 *     return (len &gt;= min &amp;&amp; len &lt;= max);
 *   }
 *
 * }
 * </pre>
 *
 * <p>In LengthFilter, the CharTermAttribute is added and stored in the instance variable <code>
 * termAtt</code>. Remember that there can only be a single instance of CharTermAttribute in the
 * chain, so in our example the <code>addAttribute()</code> call in LengthFilter returns the
 * CharTermAttribute that the WhitespaceTokenizer already added.
 *
 * <p>The tokens are retrieved from the input stream in FilteringTokenFilter's <code>
 * incrementToken()</code> method (see below), which calls LengthFilter's <code>accept()</code>
 * method. By looking at the term text in the CharTermAttribute, the length of the term can be
 * determined and tokens that are either too short or too long are skipped. Note how <code>accept()
 * </code> can efficiently access the instance variable; no attribute lookup is necessary. The same
 * is true for the consumer, which can simply use local references to the Attributes.
 *
 * <p>LengthFilter extends FilteringTokenFilter:
 *
 * <pre class="prettyprint">
 * public abstract class FilteringTokenFilter extends TokenFilter {
 *
 *   private final PositionIncrementAttribute posIncrAtt = addAttribute(PositionIncrementAttribute.class);
 *
 *   &#47;**
 *    * Create a new FilteringTokenFilter.
 *    * {@literal @param} in      the TokenStream to consume
 *    *&#47;
 *   public FilteringTokenFilter(Version version, TokenStream in) {
 *     super(in);
 *   }
 *
 *   &#47;** Override this method and return if the current input token should be returned by incrementToken. *&#47;
 *   protected abstract boolean accept() throws IOException;
 *
 *   {@literal @Override}
 *   public final boolean incrementToken() throws IOException {
 *     int skippedPositions = 0;
 *     while (input.incrementToken()) {
 *       if (accept()) {
 *         if (skippedPositions != 0) {
 *           posIncrAtt.setPositionIncrement(posIncrAtt.getPositionIncrement() + skippedPositions);
 *         }
 *         return true;
 *       }
 *       skippedPositions += posIncrAtt.getPositionIncrement();
 *     }
 *     // reached EOS -- return false
 *     return false;
 *   }
 *
 *   {@literal @Override}
 *   public void reset() throws IOException {
 *     super.reset();
 *   }
 *
 * }
 * </pre>
 *
 * <h4>Adding a custom Attribute</h4>
 *
 * Now we're going to implement our own custom Attribute for part-of-speech tagging and call it
 * consequently <code>PartOfSpeechAttribute</code>. First we need to define the interface of the new
 * Attribute:
 *
 * <pre class="prettyprint">
 *   public interface PartOfSpeechAttribute extends Attribute {
 *     public enum PartOfSpeech {
 *       Noun, Verb, Adjective, Adverb, Pronoun, Preposition, Conjunction, Article, Unknown
 *     }
 *
 *     public void setPartOfSpeech(PartOfSpeech pos);
 *
 *     public PartOfSpeech getPartOfSpeech();
 *   }
 * </pre>
 *
 * <p>Now we also need to write the implementing class. The name of that class is important here: By
 * default, Lucene checks if there is a class with the name of the Attribute with the suffix 'Impl'.
 * In this example, we would consequently call the implementing class <code>
 * PartOfSpeechAttributeImpl</code>.
 *
 * <p>This should be the usual behavior. However, there is also an expert-API that allows changing
 * these naming conventions: {@link org.apache.lucene.util.AttributeFactory}. The factory accepts an
 * Attribute interface as argument and returns an actual instance. You can implement your own
 * factory if you need to change the default behavior.
 *
 * <p>Now here is the actual class that implements our new Attribute. Notice that the class has to
 * extend {@link org.apache.lucene.util.AttributeImpl}:
 *
 * <pre class="prettyprint">
 * public final class PartOfSpeechAttributeImpl extends AttributeImpl
 *                                   implements PartOfSpeechAttribute {
 *
 *   private PartOfSpeech pos = PartOfSpeech.Unknown;
 *
 *   public void setPartOfSpeech(PartOfSpeech pos) {
 *     this.pos = pos;
 *   }
 *
 *   public PartOfSpeech getPartOfSpeech() {
 *     return pos;
 *   }
 *
 *   {@literal @Override}
 *   public void clear() {
 *     pos = PartOfSpeech.Unknown;
 *   }
 *
 *   {@literal @Override}
 *   public void copyTo(AttributeImpl target) {
 *     ((PartOfSpeechAttribute) target).setPartOfSpeech(pos);
 *   }
 * }
 * </pre>
 *
 * <p>This is a simple Attribute implementation has only a single variable that stores the
 * part-of-speech of a token. It extends the <code>AttributeImpl</code> class and therefore
 * implements its abstract methods <code>clear()</code> and <code>copyTo()</code>. Now we need a
 * TokenFilter that can set this new PartOfSpeechAttribute for each token. In this example we show a
 * very naive filter that tags every word with a leading upper-case letter as a 'Noun' and all other
 * words as 'Unknown'.
 *
 * <pre class="prettyprint">
 *   public static class PartOfSpeechTaggingFilter extends TokenFilter {
 *     PartOfSpeechAttribute posAtt = addAttribute(PartOfSpeechAttribute.class);
 *     CharTermAttribute termAtt = addAttribute(CharTermAttribute.class);
 *
 *     protected PartOfSpeechTaggingFilter(TokenStream input) {
 *       super(input);
 *     }
 *
 *     public boolean incrementToken() throws IOException {
 *       if (!input.incrementToken()) {return false;}
 *       posAtt.setPartOfSpeech(determinePOS(termAtt.buffer(), 0, termAtt.length()));
 *       return true;
 *     }
 *
 *     // determine the part of speech for the given term
 *     protected PartOfSpeech determinePOS(char[] term, int offset, int length) {
 *       // naive implementation that tags every uppercased word as noun
 *       if (length &gt; 0 &amp;&amp; Character.isUpperCase(term[0])) {
 *         return PartOfSpeech.Noun;
 *       }
 *       return PartOfSpeech.Unknown;
 *     }
 *   }
 * </pre>
 *
 * <p>Just like the LengthFilter, this new filter stores references to the attributes it needs in
 * instance variables. Notice how you only need to pass in the interface of the new Attribute and
 * instantiating the correct class is automatically taken care of.
 *
 * <p>Now we need to add the filter to the chain in MyAnalyzer:
 *
 * <pre class="prettyprint">
 *   {@literal @Override}
 *   protected TokenStreamComponents createComponents(String fieldName) {
 *     final Tokenizer source = new WhitespaceTokenizer(matchVersion);
 *     TokenStream result = new LengthFilter(true, source, 3, Integer.MAX_VALUE);
 *     result = new PartOfSpeechTaggingFilter(result);
 *     return new TokenStreamComponents(source, result);
 *   }
 * </pre>
 *
 * Now let's look at the output:
 *
 * <pre>
 * This
 * demo
 * the
 * new
 * TokenStream
 * API
 * </pre>
 *
 * Apparently it hasn't changed, which shows that adding a custom attribute to a TokenStream/Filter
 * chain does not affect any existing consumers, simply because they don't know the new Attribute.
 * Now let's change the consumer to make use of the new PartOfSpeechAttribute and print it out:
 *
 * <pre class="prettyprint">
 *   public static void main(String[] args) throws IOException {
 *     // text to tokenize
 *     final String text = "This is a demo of the TokenStream API";
 *
 *     MyAnalyzer analyzer = new MyAnalyzer();
 *     TokenStream stream = analyzer.tokenStream("field", new StringReader(text));
 *
 *     // get the CharTermAttribute from the TokenStream
 *     CharTermAttribute termAtt = stream.addAttribute(CharTermAttribute.class);
 *
 *     // get the PartOfSpeechAttribute from the TokenStream
 *     PartOfSpeechAttribute posAtt = stream.addAttribute(PartOfSpeechAttribute.class);
 *
 *     try {
 *       stream.reset();
 *
 *       // print all tokens until stream is exhausted
 *       while (stream.incrementToken()) {
 *         System.out.println(termAtt.toString() + ": " + posAtt.getPartOfSpeech());
 *       }
 *
 *       stream.end();
 *     } finally {
 *       stream.close();
 *     }
 *   }
 * </pre>
 *
 * The change that was made is to get the PartOfSpeechAttribute from the TokenStream and print out
 * its contents in the while loop that consumes the stream. Here is the new output:
 *
 * <pre>
 * This: Noun
 * demo: Unknown
 * the: Unknown
 * new: Unknown
 * TokenStream: Noun
 * API: Noun
 * </pre>
 *
 * Each word is now followed by its assigned PartOfSpeech tag. Of course this is a naive
 * part-of-speech tagging. The word 'This' should not even be tagged as noun; it is only spelled
 * capitalized because it is the first word of a sentence. Actually this is a good opportunity for
 * an exercise. To practice the usage of the new API the reader could now write an Attribute and
 * TokenFilter that can specify for each word if it was the first token of a sentence or not. Then
 * the PartOfSpeechTaggingFilter can make use of this knowledge and only tag capitalized words as
 * nouns if not the first word of a sentence (we know, this is still not a correct behavior, but
 * hey, it's a good exercise). As a small hint, this is how the new Attribute class could begin:
 *
 * <pre class="prettyprint">
 *   public class FirstTokenOfSentenceAttributeImpl extends AttributeImpl
 *                               implements FirstTokenOfSentenceAttribute {
 *
 *     private boolean firstToken;
 *
 *     public void setFirstToken(boolean firstToken) {
 *       this.firstToken = firstToken;
 *     }
 *
 *     public boolean getFirstToken() {
 *       return firstToken;
 *     }
 *
 *     {@literal @Override}
 *     public void clear() {
 *       firstToken = false;
 *     }
 *
 *   ...
 * </pre>
 *
 * <h4>Adding a CharFilter chain</h4>
 *
 * Analyzers take Java {@link java.io.Reader}s as input. Of course you can wrap your Readers with
 * {@link java.io.FilterReader}s to manipulate content, but this would have the big disadvantage
 * that character offsets might be inconsistent with your original text.
 *
 * <p>{@link org.apache.lucene.analysis.CharFilter} is designed to allow you to pre-process input
 * like a FilterReader would, but also preserve the original offsets associated with those
 * characters. This way mechanisms like highlighting still work correctly. CharFilters can be
 * chained.
 *
 * <p>Example:
 *
 * <pre class="prettyprint">
 * public class MyAnalyzer extends Analyzer {
 *
 *   {@literal @Override}
 *   protected TokenStreamComponents createComponents(String fieldName) {
 *     return new TokenStreamComponents(new MyTokenizer());
 *   }
 *
 *   {@literal @Override}
 *   protected Reader initReader(String fieldName, Reader reader) {
 *     // wrap the Reader in a CharFilter chain.
 *     return new SecondCharFilter(new FirstCharFilter(reader));
 *   }
 * }
 * </pre>
 */
package org.apache.lucene.analysis;
