BS ISO IEC 14651:2019 download free.Information technology — International string ordering and comparison — Method for comparing character strings and description of the common template tailorable ordering.
BS ISO IEC 14651 defines the following.
— A reference comparison method. This method is applicable to two character strings to determine their collating order In a sorted list. The method can be applied to strings containing characters from the full repertoire of ISO/IEC 10646. This method Is also applicable to subsets of that repertoire, such as those of the different ISO/IEC 8-bit standard character sets, or any other character set, standardized or not, to produce ordering results valid (after tailoring) for a given set of languages for each script. This method uses collation tables derived either from the Common Template Table defined In BS ISO IEC 14651 or from one of its tailorings. This method provides a reference format. The format Is described using the Backus-Naur Form (BNF). This format is used to describe the Common Template Table, The format is used normatively within this document.
— A Common Template Table. A given tailoring of the Common Template Table is used by the reference comparison method. The Common Template Table describes an order for all characters encoded in the LJnlcode 10,0 standardl2.Zl. Included In lSO/IEC 10646:2017. It allows for a specification of a fully deterministic ordering. This table enables the specification of a string ordering adapted to local ordering rules, without requiring an implementer to have knowledge of all the different scripts already encoded In the Universal Coded Character Set (UCS).
NOTE I This Common Template Table Is to be modified to suit the needs of a local environment. The main worldwide benefit is that, for other scripts, often no modification is required and the order will remain as consistent as possible and predictable from an International point of view.
NOTE 2 The character repertoire used in this document Is equivalent to that of the Unicode Standard version l0.OL2.i.
— A reference name. The reference name refers to this particular version of the Common Template Table, for use as a reference when tailoring. In particular, this name implies that the table is linked to a particular stage of development of the ISO/IEC 10646 Universal coded character set.
— Requirements for a declaration of the differences (delta) between the collation table and the Corn mon Template Table.
BS ISO IEC 14651 does not mandate the following.
— A specific comparison method; any equivalent method giving the same results is acceptable.
— A specific format for describing or tailoring tables in a given Implementation.
6 String comparison
6.1 Preparation of character strings prior to comparison
It can be necessary to transform character strings before the reference comparison method is applied to them. Although not part of the Scope of this document, preparation can be an important part of the ordering process. See Annex C for some examples of preparation.
Characters of the input string shall be encoded according to ISO/IEC 10646 (UCS) or a mapping to ISO, IEC 10646 shall be provided if another encoding scheme is used.
Therefore it can he an important part of the preparation phase to map characters from a non-UCS encoding scheme to the UCS for input to the comparison method. This task can, amongst other things, encompass the correct handling of escape sequences in the originating encoding scheme, the mapping of characters without an allocated UCS codepoint to an application-defined codepoint in the private zone area and change the sequence of characters in strings that are not stored in logical order. For example, for visual order Arabic code sets, input strings shall be put into logical order; and for some bibliographic code sets, strings with combining accents stored before their respective base character require that the combining accents be put after their base character. The resulting string sequence may then have to he remapped into its original encoding scheme.
The Common Template Table is designed so that combining sequences and corresponding single characters (precomposed) will have precisely the same ordering. To avoid inadvertently breaking this invariant (and in the process breaking Unicode conformance), tailoring should reorder combining sequences when corresponding precomposed characters are reordered. For example, if A is reordered after Z, then the sequence <A>+ccombining diaeresis> should also be reordered. To avoid exposing encoding differences that can be invisible to the end-user, it is recommended that strings be normalized according to Unicode normalization form NFD to achieve this equivalence (see [2]).
Escape sequences and control characters constitute very sensitive data to Interpret, and it is highly recommended that preparation should filter out or transform these sequences.
NOTE Since the reference method is a logical statement for the mechanism for string comparison, it does not preclude an implementation from using a non-UCS character encoding only, as long as it produces results as if it were usint the reference comnarison method.
6.22.1 General
When two strings arc to be compared to determine their relative order, the two strings are first parsed into a sequence of collating elements taking Into account the multi-character collating.clemcnt” statements declared and used in a tailored table (if the syntax of 6.12 is used). For the syntax used for expressing the Common Template Table, the name of a collating element consisting of a single character, is formed by the tICS value of the character, expressed as a hexadecimal string, prefixed with “U. For multi-character collating elements, the name and association to characters can be found via the collating elements declarations.
NOTE Collating elements with more characters have preference over shorter ones. As an example, if a multlcharacter collating clement is defined for abc and another one Is defined for ab or for Thc, then Ifabc is encountered, the collating element [or abc will apply and not the one for ab or bc.
Then, a sequence of m intermediary subkeys is formed out of a character string, where m Is the number of levels described In a tailored collation weighting table.
Each ordering key isa sequence of subkeys. Each subkey isa list of numeric weights. A subkeyis formed by successively appending the list of the weights assigned, at the level of the subkey, to each collating element of the string. The keyword IGNORE in the Common Template Table at the place of a sequence of collating symbols at a level Indicates that the sequence of weights at that level for that collating element is an empty sequence of weights.
6.2.2.2 Weighting elements to be ignored
When forming a sort key, collating elements ignored at the first level or at the two first levels and that follow a collation element ignored at all levels but the last one, do not keep their weights as defined in the common template table (or a tailored table); each of these weights shall be zeroed (this means that “IGNORE” shall he assigned to each non-nil weight).
6.2.2.3 Implicit weights computing
If there is no entry in the tailored table for a character of the input string, then the character’s weights are undefined. In this case, one shall compute an implicit primary weight consisting ui a pair of 16- bit words — call them aaaa and bbbb — and assume that lines like the following were added to the weighting table.