Unicode
1. Definition
Unicode is a universal character encoding standard (Unicode Consortium, 1991 — superseding ASCII and ISO-8859) that assigns a unique number, called a code point, to every character of every writing system in the world, plus mathematical symbols, technical symbols and emoji. A code point is written as U+ followed by its hexadecimal value: U+0041 = 'A', U+20B9 = ₹, U+0905 = अ, U+1F600 = 😀.
Its goal: one code for every character, every platform, every language, so that text no longer depends on a machine's national code page.
2. Range, Planes and Coverage
Unicode defines code points from U+0000 to U+10FFFF — that is
organised into 17 planes of 65,536 code points each:
| Plane | Range | Contents |
|---|---|---|
| 0 — BMP (Basic Multilingual Plane) | U+0000-U+FFFF | Latin, Greek, Cyrillic, Devanagari (U+0900-U+097F), Bengali, Tamil, Telugu, Arabic, Hebrew, CJK ideographs, punctuation, most symbols |
| 1 — SMP | U+10000-U+1FFFF | Historic scripts, musical notation, emoji (U+1F600 😀), mathematical alphanumerics |
| 2 — SIP | U+20000-U+2FFFF | Rare CJK extension ideographs |
| 3-13 | up to U+DFFFF | CJK extensions, unassigned/reserved |
| 14 — SSP | U+E0000-U+EFFFF | Tags, variation selectors |
| 15-16 — PUA | U+F0000-U+10FFFF | Private use areas |
Of the 1.1 million code points about 155,000 are currently assigned; the rest are reserved so new scripts can be added without breaking existing text. ASCII occupies U+0000-U+007F, which is why an ASCII file is already a valid Unicode file.
3. Unicode Encoding Forms (UTF) — How Code Points Become Bytes
Unicode itself is a character set (a numbering); the UTF forms are the encodings (byte representations):
| Form | Bytes per character | Units | ASCII compatible? | Typical use |
|---|---|---|---|---|
| UTF-8 | 1-4 (variable) | 1-byte units | Yes (0-127 identical to ASCII) | Web (HTML/JSON), Linux, MySQL, most modern APIs |
| UTF-16 | 2 or 4 | 2-byte code units, surrogate pairs beyond the BMP | No | Windows internals, Java/C# strings |
| UTF-32 | Always 4 | 4-byte code units | No | Internal use where fixed width simplifies indexing |
UTF-8 encoding rules:
| Code point range | Byte pattern |
|---|---|
| U+0000-U+007F (7 bits) | 0xxxxxxx |
| U+0080-U+07FF (11 bits) | 110xxxxx 10xxxxxx |
| U+0800-U+FFFF (16 bits) | 1110xxxx 10xxxxxx 10xxxxxx |
| U+10000-U+10FFFF (21 bits) | 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx |
Examples
'A' = U+0041→ 41 (1 byte, same as ASCII)'₹' = U+20B9→ E2 82 B9 (3 bytes)अ = U+0905→ E0 A4 85 (3 bytes)'😀' = U+1F600→ F0 9F 98 80 (4 bytes)
Advantages of UTF-8: backwards compatible with ASCII (English text takes no extra space), self-synchronising (a byte of the form 10xxxxxx is always a continuation byte, so text can be recovered after corruption), byte-order independent (no BOM needed), and safe with legacy software that expects ASCII.
4. Unicode vs ASCII
| Aspect | ASCII | Unicode |
|---|---|---|
| Introduced | 1963-68 | 1991 onwards |
| Bits per code | 7 (stored in 8) | 21 (encoded in 8/16/32 bits depending on the UTF form) |
| Characters | 128 | 1,114,112 code points (~155,000 assigned) |
| Scripts | English/Latin only, plus control codes | All scripts of the world incl. Devanagari, CJK, Arabic, emoji |
| Bytes per character | 1 | 1-4 (UTF-8), 2-4 (UTF-16) |
| Compatibility | — | First 128 code points identical to ASCII |
| Collation/case/combining rules | Not specified | Specified (normalisation NFC/NFD, case folding, collation) |
5. Practical Points to Remember
- Code point is not the same as glyph: several code points may combine into one visible character (द + ि produces the syllable दि;
e+ combining acute U+0301 gives é). Hence normalisation — NFC (compose) and NFD (decompose) — which produces canonically equivalent strings. - BOM (Byte Order Mark)
U+FEFFis used by UTF-16 to signal byte order; in UTF-8 it is optional and best avoided. - Escapes: a code point can be written as
U+XXXXin documentation and as a backslash-u escape (uXXXXprefixed with a backslash) in source code and JSON strings. - India's earlier 8-bit standard ISCII (IS 13194) handled Devanagari and other Indic scripts but has been replaced by Unicode UTF-8/UTF-16 in every modern OS and web framework.
- Storage comparison: an English page in UTF-8 has exactly the same size as in ASCII, while a Hindi page costs ~3 bytes per Devanagari character — a 1000-character Hindi document is about 3 KB.
6. Exam Points
- What is Unicode? Why was it needed over ASCII? — 5 marks, almost certain.
- How many code points does Unicode define, and into how many planes? — 3 marks.
- Differentiate UTF-8, UTF-16 and UTF-32 (tabular) — 5 marks, frequent.
- Encode a given character (e.g. ₹, अ or 😀) into UTF-8 bytes — 3-5 marks.
- Compare ASCII and Unicode — 5 marks.
- What are code point, glyph and normalisation? — 3 marks.