iPedia Free Information Center

Information about Character Set

A character encoding consists of a code that pairs a sequence of characters from a given character set (sometimes referred to as code page) with something else, such as a sequence of natural numbers, octets or electrical pulses, in order to facilitate the storage of text in computers and the transmission of text through telecommunication networks. Common examples include Morse code, which encodes letters of the Latin alphabet as series of long and short depressions of a telegraph key; and ASCII, which encodes letters, numerals, and other symbols, both as integers and as 7-bit binary versions of those integers, generally extended with an extra zero-bit to facilitate storage in 8-bit bytes (octets).

In earlier days of computing, the introduction of character sets such as ASCII (1963) and EBCDIC (1964) began the process of standardization. The limitations of such sets soon became apparent, and a number of ad-hoc methods developed to extend them. The need to support multiple writing systems (Languages), including the CJK family of East Asian scripts, required support for a far larger number of characters and demanded a systematic approach to character encoding rather than the previous ad hoc approaches.

Simple character sets

Conventionally character set and character encoding were considered synonymous, as the same standard would specify both what characters were available and how they were to be encoded into a stream of code units (usually with a single character per code unit). For historical reasons, MIME and systems based on it use the term charset to refer to the complete system for encoding a sequence of characters into a sequence of octets.

Modern encoding model

Unicode and its parallel standard, ISO 10646 Universal Character Set, which together constitute the most modern character encoding, broke away from this idea, and instead separated the ideas of what characters are available, their numbering, how those numbers are encoded as a series of "code units" (limited-size numbers), and finally how those units are encoded as a stream of octets (bytes). The idea behind this decomposition is to establish a universal set of characters that can be encoded in a variety of ways. To correctly describe this model needs more precise terms than "character set" and "character encoding". The terms used in the modern model follow:

A character repertoire is the full set of abstract characters that a system supports. The repertoire may be closed, that is no additions are allowed without creating a new standard (as is the case with ASCII and most of the ISO-8859 series), or it may be open, allowing additions (as is the case with Unicode and to a limited extent the Windows code pages). The characters in a given repertoire reflect decisions that have been made about how to divide writing systems into linear information units. The basic variants of the Latin, Greek, and Cyrillic alphabets, can be broken down into letters, digits, punctuation, and a few special characters like the space, which can all be arranged in simple linear sequences that are displayed in the same order they are read. Even with these alphabets however diacritics pose a complication: they can be regarded either as part of a single character containing a letter and diacritic (known in modern terminology as a precomposed character), or as separate characters. The former allows a far simpler text handling system but the latter allows any letter/diacritic combination to be used in text. Other writing systems, such as Arabic and Hebrew, are represented with more complex character repertoires due to the need to accommodate things like bidirectional text and glyphs that are joined together in different ways for different situations.

A coded character set specifies how to represent a repertoire of characters using a number of non-negative integer codes called code points. For example, in a given repertoire, a character representing the capital letter "A" in the Latin alphabet might be assigned to the integer 65, the character for "B" to 66, and so on. A complete set of characters and corresponding integers is a coded character set. Multiple coded character sets may share the same repertoire; for example ISO-8859-1 and IBM code pages 037 and 500 all cover the same repertoire but map them to different codes. In a coded character set, each code point only represents one character.

A character encoding form (CEF) specifies the conversion of a coded character set's integer codes into a set of limited-size integer code values that facilitate storage in a system that represents numbers in binary form using a fixed number of bits (e.g., virtually any computer system). For example, a system that stores numeric information in 16-bit units would only be able to directly represent integers from 0 to 65,535 in each unit, but larger integers could be represented if more than one 16-bit unit could be used. This is what a CEF accommodates: it defines a way of mapping single code point from a range of, say, 0 to 1.4 million, to a series of one or more code values from a range of, say, 0 to 65,535.

The simplest CEF system is simply to choose large enough units that the values from the coded character set can be encoded directly (one code point to one code value). This works well for coded character sets that fit in 8 bits (as most legacy non-CJK encodings do) and reasonably well for coded character sets that fit in 16 bits (such as early versions of Unicode). However, as the size of the coded character set increases (e.g. modern Unicode requires at least 21 bits/character), this becomes less and less efficient, and it is difficult to adapt existing systems to use larger code values. Therefore, most systems working with later versions of Unicode use either UTF-8, which maps Unicode code points to variable-length sequences of octets, or UTF-16, which maps Unicode code points to variable-length sequences of 16-bit words.

Next, a character encoding scheme (CES) specifies how the fixed-size integer codes should be mapped into an octet sequence suitable for saving on an octet-based file system or transmitting over an octet-based network. With Unicode, a simple character encoding scheme is used in most cases, simply specifying if the bytes for each integer should be in big-endian or little-endian order (even this isn't needed with UTF-8). However, there are also compound character encoding schemes, which use escape sequences to switch between several simple schemes (such as ISO 2022), and compressing schemes, which try to minimise the number of bytes used per code unit (such as SCSU, BOCU, and Punycode).

Finally, there may be a higher level protocol which supplies additional information that can be used to select the particular variant of a Unicode character, particularly where there are regional variants that have been 'unified' in Unicode as the same character. An example is the XML attribute xml:lang.

History of character encodings

Popular character encodings

Character conversion tools

Cross-platform: Linux:

See also

  • — articles related to character encoding in general
  • — articles detailing specific character encodings
  • Code page — various character set encodings used by Microsoft
  • Windows code page — various character set encodings used by Microsoft Windows
  • Mojibake — character set mismap.

External links

code is a function mapping an alphabet to non-negative real numbers, satisfying a generalization of Kraft's inequality. A code page, a type of character encoding table, is one such code.
..... Click the link for more information.
character is a unit of information that roughly corresponds to a grapheme, grapheme-like unit, or symbol, such as in an alphabet or syllabary in the written form of a natural language.

An example of a character is a letter, numeral, or punctuation mark.
..... Click the link for more information.
Code page is the traditional IBM term used for a specific character encoding table: a mapping in which a sequence of bits, usually a single octet representing integer values 0 through 255, is associated with a specific character.
..... Click the link for more information.
number is an abstract idea used in counting and measuring. A symbol which represents a number is called a numeral, but in common usage the word number is used for both the idea and the symbol.
..... Click the link for more information.
In computing, an octet is a grouping of eight bits.

In France, French Canada and Romania, the word octet usually means byte; a megabyte (MB) is called a megaoctet in France, Romania and also French Canada. Bit and Byte are homophones in the French language.
..... Click the link for more information.
Computer data storage, computer memory, and often casually storage or memory refer to computer components, devices and recording media that retain digital data used for computing for some interval of time.
..... Click the link for more information.
character is a unit of information that roughly corresponds to a grapheme, grapheme-like unit, or symbol, such as in an alphabet or syllabary in the written form of a natural language.

An example of a character is a letter, numeral, or punctuation mark.
..... Click the link for more information.
computer is a machine which manipulates data according to a list of instructions.

Computers take numerous physical forms. The first devices that resemble modern computers date to the mid-20th century (around 1940 - 1941), although the computer concept and various machines
..... Click the link for more information.
Morse code is a method for transmitting telegraphic information, using standardized sequences of short and long elements to represent the letters, numerals, punctuation and special characters of a message.
..... Click the link for more information.
Latin alphabet
Child systems Numerous: see Alphabets derived from the Latin
Sister systems Cyrillic
Coptic
Armenian
Runic/Futhark
Unicode range See Latin characters in Unicode
ISO 15924 Latn

Note
..... Click the link for more information.
Telegraph key, also known as a Morse key, are generic terms for any switching device used primarily to send Morse code. Similar keys are used for all forms of manual telegraphy, such as in electrical telegraph and radio telegraphy.
..... Click the link for more information.
American Standard Code for Information Interchange (ASCII), generally pronounced ask-ee IPA: /ˈæski/ ( [1] ), is a character encoding based on the English alphabet.
..... Click the link for more information.
The integers (from the Latin integer, which means with untouched integrity, whole, entire) are the set of numbers including the whole numbers (0, 1, 2, 3, …) and their negatives (0, −1, −2, −3, …).
..... Click the link for more information.
BIT is an acronym for:
  • Bannari amman Institute of Technology
  • Bangalore Institute of Technology
  • Beijing Institute of Technology
  • Benzisothiazolinone
  • Bilateral Investment Treaty
  • Bhilai Institute of Technology - Durg

..... Click the link for more information.
binary numeral system, or base-2 number system, is a numeral system that represents numeric values using two symbols, usually 0 and 1. More specifically, the usual base-2 system is a positional notation with a radix of 2.
..... Click the link for more information.
byte (pronounced /baɪt/) is a unit of measurement of information storage, most often consisting of eight bits. In many computer architectures it is a unit of memory addressing.
..... Click the link for more information.
American Standard Code for Information Interchange (ASCII), generally pronounced ask-ee IPA: /ˈæski/ ( [1] ), is a character encoding based on the English alphabet.
..... Click the link for more information.
Extended Binary Coded Decimal Interchange Code (EBCDIC) is an 8-bit character encoding (code page) used on IBM mainframe operating systems, like z/OS, OS/390, VM and VSE, as well as IBM minicomputer operating systems like OS/400 and i5/OS (see also Binary Coded Decimal).
..... Click the link for more information.
For the human rights organization, see .
Ad hoc is a Latin phrase which means "for this purpose". It generally signifies a solution that has been custom designed for a specific problem, is non-generalizable, and cannot be adapted to other purposes.
..... Click the link for more information.
writing system is a type of symbolic system used to represent elements or statements expressible in language.

General properties

Writing systems are distinguished from other possible symbolic communication systems in that one must usually understand something of the
..... Click the link for more information.
Mime or pantomime is a theatrical medium or performance art, involving the acting out of a story by a mime artist through body motions, without use of speech.

History


..... Click the link for more information.
Unicode is an industry standard allowing computers to consistently represent and manipulate text expressed in any of the world's writing systems. Developed in tandem with the Universal Character Set standard and published in book form as The Unicode Standard
..... Click the link for more information.
The international standard ISO/IEC 10646 defines the Universal Character Set (UCS) as a character set on which many encodings are based. It contains nearly a hundred thousand abstract characters, each identified by an unambiguous name and an integer number called its
..... Click the link for more information.
Windows code pages are sets of characters or code pages (known as character encodings in other operating systems) used in Microsoft Windows systems from the 1980s and 1990s.
..... Click the link for more information.
A diacritical mark or diacritic, also called an accent, is a small sign added to a letter to alter pronunciation or to distinguish between similar words.
..... Click the link for more information.
glyph is the shape given in a particular typeface to a specific grapheme or symbol.

The term for the abstract entity represented by a glyph is character: a typographical character may be a grapheme (an element of a writing system), but also a numeral, a punctuation
..... Click the link for more information.
ISO 8859-1, more formally cited as ISO/IEC 8859-1 is part 1 of ISO/IEC 8859, a standard character encoding of the Latin alphabet. It is less formally called as Latin-1. It was originally developed by the ISO, but later jointly maintained by the ISO and the IEC.
..... Click the link for more information.
UTF-8 (8-bit UCS/Unicode Transformation Format) is a variable-length character encoding for Unicode. It is able to represent any character in the Unicode standard, yet the initial encoding of byte codes and character assignments for UTF-8 is backwards compatible with ASCII.
..... Click the link for more information.
Endian may refer to:
  • Endian Firewall, a Linux Routing/Firewall distribution
  • Endianess, the byte (and sometimes bit) ordering in memory used to represent some kind of data in computing

..... Click the link for more information.
ISO 2022, more formally ISO/IEC 2022, is an ISO standard (equivalent to the ECMA standard ECMA-35) specifying
  • a technique for including multiple character sets in a single character encoding, and

..... Click the link for more information.


This article is copied from an article on Wikipedia.org - the free encyclopedia created and edited by online user community. The text was not checked or edited by anyone on our staff. Although the vast majority of the wikipedia encyclopedia articles provide accurate and timely information please do not assume the accuracy of any particular article. This article is distributed under the terms of GNU Free Documentation License.
Herod_Archelaus


page counter