UNIQ(1)                    POSIX Programmer's Manual                   UNIQ(1)



PROLOG
       This manual page is part of the POSIX Programmer's Manual.  The OS
       implementation of this interface may differ (consult the corresponding
       OS manual page for details of OS behavior), or the interface may not be
       implemented on OS.

NAME
       uniq -- report or filter out repeated lines in a file

SYNOPSIS
       uniq [[-c|-d|-u]] [[-f fields]] [[-s char]] [[input__file [[output__file]]]]

DESCRIPTION
       The uniq utility shall read an input file comparing adjacent lines, and
       write one copy of each input line on the output. The second and
       succeeding copies of repeated adjacent input lines shall not be
       written.  The trailing <newline> of each line in the input shall be
       ignored when doing comparisons.

       Repeated lines in the input shall not be detected if they are not
       adjacent.

OPTIONS
       The uniq utility shall conform to the Base Definitions volume of
       POSIX.1-2017, Section 12.2, Utility Syntax Guidelines, except that ''++''
       may be recognized as an option delimiter as well as ''--''.

       The following options shall be supported:

       -c        Precede each output line with a count of the number of times
                 the line occurred in the input.

       -d        Suppress the writing of lines that are not repeated in the
                 input.

       -f fields Ignore the first fields fields on each input line when doing
                 comparisons, where fields is a positive decimal integer. A
                 field is the maximal string matched by the basic regular
                 expression:


                     [[:blank:]]*[^[:blank:]]*

                 If the fields option-argument specifies more fields than
                 appear on an input line, a null string shall be used for
                 comparison.

       -s chars  Ignore the first chars characters when doing comparisons,
                 where chars shall be a positive decimal integer. If specified
                 in conjunction with the -f option, the first chars characters
                 after the first fields fields shall be ignored. If the chars
                 option-argument specifies more characters than remain on an
                 input line, a null string shall be used for comparison.

       -u        Suppress the writing of lines that are repeated in the input.

OPERANDS
       The following operands shall be supported:

       input__file
                 A pathname of the input file. If the input__file operand is
                 not specified, or if the input__file is ''--'', the standard
                 input shall be used.

       output__file
                 A pathname of the output file. If the output__file operand is
                 not specified, the standard output shall be used. The results
                 are unspecified if the file named by output__file is the file
                 named by input__file.

STDIN
       The standard input shall be used only if no input__file operand is
       specified or if input__file is ''--''.  See the INPUT FILES section.

INPUT FILES
       The input file shall be a text file.

ENVIRONMENT VARIABLES
       The following environment variables shall affect the execution of uniq:

       LANG      Provide a default value for the internationalization
                 variables that are unset or null. (See the Base Definitions
                 volume of POSIX.1-2017, Section 8.2, Internationalization
                 Variables for the precedence of internationalization
                 variables used to determine the values of locale categories.)

       LC__ALL    If set to a non-empty string value, override the values of
                 all the other internationalization variables.

       LC__CTYPE  Determine the locale for the interpretation of sequences of
                 bytes of text data as characters (for example, single-byte as
                 opposed to multi-byte characters in arguments and input
                 files) and which characters constitute a <blank> in the
                 current locale.

       LC__MESSAGES
                 Determine the locale that should be used to affect the format
                 and contents of diagnostic messages written to standard
                 error.

       NLSPATH   Determine the location of message catalogs for the processing
                 of LC__MESSAGES.

ASYNCHRONOUS EVENTS
       Default.

STDOUT
       The standard output shall be used if no output__file operand is
       specified, and shall be used if the output__file operand is ''--'' and the
       implementation treats the ''--'' as meaning standard output. Otherwise,
       the standard output shall not be used.  See the OUTPUT FILES section.

STDERR
       The standard error shall be used only for diagnostic messages.

OUTPUT FILES
       If the -c option is specified, the output file shall be empty or each
       line shall be of the form:


           "%d %s", <number of duplicates>, <line>

       otherwise, the output file shall be empty or each line shall be of the
       form:


           "%s", <line>

EXTENDED DESCRIPTION
       None.

EXIT STATUS
       The following exit values shall be returned:

        0    The utility executed successfully.

       >0    An error occurred.

CONSEQUENCES OF ERRORS
       Default.

       The following sections are informative.

APPLICATION USAGE
       If the collating sequence of the current locale has a total ordering of
       all characters, the sort utility can be used to cause repeated lines to
       be adjacent in the input file. If the collating sequence does not have
       a total ordering of all characters, the sort utility should still do
       this but it might not. To ensure that all duplicate lines are
       eliminated, and have the output sorted according the collating sequence
       of the current locale, applications should use:


           LC_ALL=C sort -u | sort

       instead of:


           sort | uniq

       To remove duplicate lines based on whether they collate equally instead
       of whether they are identical, applications should use:


           sort -u

       instead of:


           sort | uniq

       When using uniq to process pathnames, it is recommended that LC_ALL, or
       at least LC_CTYPE and LC_COLLATE, are set to POSIX or C in the
       environment, since pathnames can contain byte sequences that do not
       form valid characters in some locales, in which case the utility's
       behavior would be undefined. In the POSIX locale each byte is a valid
       single-byte character, and therefore this problem is avoided.

EXAMPLES
       The following input file data (but flushed left) was used for a test
       series on uniq:


           #01 foo0 bar0 foo1 bar1
           #02 bar0 foo1 bar1 foo1
           #03 foo0 bar0 foo1 bar1
           #04
           #05 foo0 bar0 foo1 bar1
           #06 foo0 bar0 foo1 bar1
           #07 bar0 foo1 bar1 foo0

       What follows is a series of test invocations of the uniq utility that
       use a mixture of uniq options against the input file data. These tests
       verify the meaning of adjacent.  The uniq utility views the input data
       as a sequence of strings delimited by '\n'.  Accordingly, for the
       fieldsth member of the sequence, uniq interprets unique or repeated
       adjacent lines strictly relative to the fields+1th member.

        1. This first example tests the line counting option, comparing each
           line of the input file data starting from the second field:


               uniq -c -f 1 uniq_0I.t
                   1 #01 foo0 bar0 foo1 bar1
                   1 #02 bar0 foo1 bar1 foo1
                   1 #03 foo0 bar0 foo1 bar1
                   1 #04
                   2 #05 foo0 bar0 foo1 bar1
                   1 #07 bar0 foo1 bar1 foo0

           The number ''22'', prefixing the fifth line of output, signifies that
           the uniq utility detected a pair of repeated lines. Given the input
           data, this can only be true when uniq is run using the -f 11 option
           (which shall cause uniq to ignore the first field on each input
           line).

        2. The second example tests the option to suppress unique lines,
           comparing each line of the input file data starting from the second
           field:


               uniq -d -f 1 uniq_0I.t
               #05 foo0 bar0 foo1 bar1

        3. This test suppresses repeated lines, comparing each line of the
           input file data starting from the second field:


               uniq -u -f 1 uniq_0I.t
               #01 foo0 bar0 foo1 bar1
               #02 bar0 foo1 bar1 foo1
               #03 foo0 bar0 foo1 bar1
               #04
               #07 bar0 foo1 bar1 foo0

        4. This suppresses unique lines, comparing each line of the input file
           data starting from the third character:


               uniq -d -s 2 uniq_0I.t

           In the last example, the uniq utility found no input matching the
           above criteria.

RATIONALE
       Some historical implementations have limited lines to be 1080 bytes in
       length, which does not meet the implied {LINE_MAX} limit.

       Earlier versions of this standard allowed the --number and ++number
       options. These options are no longer specified by POSIX.1-2008 but may
       be present in some implementations.

FUTURE DIRECTIONS
       None.

SEE ALSO
       comm, sort

       The Base Definitions volume of POSIX.1-2017, Chapter _8, Environment
       Variables, Section 12.2, Utility Syntax Guidelines

COPYRIGHT
       Portions of this text are reprinted and reproduced in electronic form
       from IEEE Std 1003.1-2017, Standard for Information Technology --
       Portable Operating System Interface (POSIX), The Open Group Base
       Specifications Issue 7, 2018 Edition, Copyright (C) 2018 by the
       Institute of Electrical and Electronics Engineers, Inc and The Open
       Group.  In the event of any discrepancy between this version and the
       original IEEE and The Open Group Standard, the original IEEE and The
       Open Group Standard is the referee document. The original Standard can
       be obtained online at http://www.opengroup.org/unix/online.html .

       Any typographical or formatting errors that appear in this page are
       most likely to have been introduced during the conversion of the source
       files to man page format. To report such errors, see
       https://www.kernel.org/doc/man-pages/reporting_bugs.html .



IEEE/The Open Group                  2017                              UNIQ(1)
