Guide
Character Encoding is a 14-lesson interactive course that answers questions which come up constantly in development and are rarely explained clearly: how many bytes is a CJK character really? Why is "😀".length 2? What is the = at the end of Base64 for? Is mojibake a broken file or something that can be rescued? Each lesson gives you a piece of sample text (which you can change freely) and several views — the bits, bytes and Unicode code point of every character, a byte-by-byte breakdown of UTF-8, the 6-bit groups of Base64, URL percent-encoding, and “what this looks like read as another encoding” — and the goal is to read off a number, write a hexadecimal string, or type a character that satisfies a condition, with the checker ticking it off immediately.
Updated 2026-09-095 sources14 min read
Character Encoding is a 14-lesson interactive course that answers questions which come up constantly in development and are rarely explained clearly: how many bytes is a CJK character really? Why is "😀".length 2? What is the = at the end of Base64 for? Is mojibake a broken file or something that can be rescued? Each lesson gives you a piece of sample text (which you can change freely) and several views — the bits, bytes and Unicode code point of every character, a byte-by-byte breakdown of UTF-8, the 6-bit groups of Base64, URL percent-encoding, and “what this looks like read as another encoding” — and the goal is to read off a number, write a hexadecimal string, or type a character that satisfies a condition, with the checker ticking it off immediately.
The course follows the history of encoding: bits and bytes → the 128 ASCII conventions → Unicode numbering every character → how UTF-8 turns a number into one to four bytes → UTF-16 and JavaScript string length → Base64 carrying bytes into plain text → URL encoding → and two final lessons on mojibake: how it happens and how to work back from garbled text to the original.
It takes about 40 minutes. It is for back-end and front-end developers, people handling data, and anyone who has been puzzled by mojibake, Base64 or the length of an emoji; all you need is the ability to count in binary.
%XX; and “read as another encoding” shows the same bytes treated as GBK, Latin-1 or UTF-16.?lesson=n jumps straight to a lesson.The course itself is above this page: change the sample text, read the views, fill in an answer.
Lesson 6, “UTF-8 two-byte”, uses café as its sample and asks for the two bytes of é:
Lesson 14, “reverse the mojibake”, gives the text 浣犲ソ as it appears when read as GBK and asks you to find the original in the sample box:
Char Code point Binary code point UTF-8 template Bytes
c U+0063 1100011 0xxxxxxx 63
a U+0061 1100001 0xxxxxxx 61
f U+0066 1100110 0xxxxxxx 66
é U+00E9 000 1110 1001 110xxxxx 10xxxxxx C3 A9
→ 11 payload bits 00011 101001 into the template → 11000011 10101001
Answer C3 A9 ✓Sample 中文 UTF-8 bytes E4 B8 AD E6 96 87 read as GBK → 涓枃
Sample 你好 UTF-8 bytes E4 BD A0 E5 A5 BD read as GBK → 浣犲ソ ← matches the prompt
Answer 你好 ✓A computer has only 0 and 1. A bit is one switch; eight bits make a byte, which can hold different values. Adding up place values turns binary into decimal: 01000001 = 64 + 1 = 65. All text ends up as a string of bytes, and a “character encoding” is a table saying which number stands for which character — different tables are different encodings.
ASCII, defined in 1963, used 7 bits for 128 characters: 0–31 are control characters such as newline, tab and bell; 48–57 are the digits 0–9; 65–90 are the uppercase letters and 97–122 the lowercase. The table is not arbitrary: uppercase and lowercase differ by 32 (the 6th bit), so c | 0x20 lowercases a letter, and a digit character minus 48 is its numeric value. Hexadecimal is the usual way to read bytes, two digits to a byte: 0x48 0x69 = Hi. ASCII used only 7 bits and left the 8th free — which various countries later filled for themselves (Latin-1, GBK, Shift_JIS and so on) in mutually incompatible ways, and that is the historical root of mojibake.
Unicode numbers the world's characters in one scheme, called a code point, written as U+ plus hexadecimal: A is U+0041, 中 is U+4E2D and 😀 is U+1F600. The space runs from U+0000 to U+10FFFF in 17 “planes”, with the characters in everyday use almost all in the first, the basic multilingual plane (U+0000–U+FFFF) and emoji and rare CJK characters in the supplementary planes. The crucial point: a code point is only a number, and how it becomes bytes is decided by an “encoding form” such as UTF-8, UTF-16 or UTF-32. “Unicode” is not an encoding; utf-8 is.
UTF-8 (RFC 3629) uses one to four bytes per code point, and the rules are all written in the first byte's prefix:
| Code point range | Bytes | Template | Payload bits |
|---|---|---|---|
| U+0000–U+007F | 1 | 0xxxxxxx |
7 |
| U+0080–U+07FF | 2 | 110xxxxx 10xxxxxx |
11 |
| U+0800–U+FFFF | 3 | 1110xxxx 10xxxxxx 10xxxxxx |
16 |
| U+10000–U+10FFFF | 4 | 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx |
21 |
Pad the code point's binary on the left to the payload width, fill the x positions in order, and you have the bytes. é = U+00E9 = 11101001, padded to 11 bits that is 00011 101001, and filling the two-byte template gives 11000011 10101001 = C3 A9.
The design has three benefits: ASCII text encoded as UTF-8 is byte for byte unchanged (fully backward compatible); every continuation byte starts with 10, so the start of the next character can be found from any position in the stream (self-synchronising, so losing one byte does not desynchronise everything after it); and encodings of different lengths never overlap, so a zero byte such as 00 only ever means U+0000 (safe for C strings). The cost is that a CJK character takes 3 bytes, 50% more than GBK's 2 — but English, punctuation and code are unaffected, which is why UTF-8 is the encoding of over 98% of web pages.
Lessons 7 and 8 have you type a 3-byte character and then a 4-byte one. The latter exposes a trap that is still common: MySQL's utf8 character set in fact only supports up to 3 bytes, so storing an emoji either errors or truncates, and utf8mb4 is required.
.lengthJavaScript, Java, C# and the Windows API use UTF-16 internally: each code unit is 16 bits. A character in the basic plane takes one code unit; a code point in a supplementary plane is split into two code units called a surrogate pair, in the ranges D800–DBFF and DC00–DFFF. That is why "😀".length === 2, why "😀".split("") cuts it into two meaningless halves and why str[0] gives you the high surrogate. To count real characters use [...str].length (iterating by code point); to count what a user perceives as one character (including combined emoji and diacritics), use Intl.Segmenter.
Base64 (RFC 4648) is not encryption: it turns arbitrary bytes into the 64 printable ASCII characters A–Z a–z 0–9 + / so that binary can travel through places that only accept text — mail attachments, JSON fields, data: URLs, JWTs. The method is to take 3 bytes = 24 bits, cut them into four 6-bit groups () and look each one up: Man = 4D 61 6E = 01001101 01100001 01101110 → 010011 010110 000101 101110 → T W F u. The size therefore grows to 4/3 of the original.
When the byte count is not a multiple of 3 the last group is short: one missing byte adds a single =, two missing bytes add two, so the output length is always a multiple of 4. A single byte A is 8 bits padded to 12, cut into 010000 01(0000) → QQ, followed by ==. The URL-safe variant swaps + / for - _ (so nothing has to be escaped in a URL) and usually drops the =, which the decoder restores from the length.
Only a small subset of ASCII may appear as-is in a URL (RFC 3986 §2). Spaces, and reserved characters with syntactic meaning such as &, =, ? and /, along with every non-ASCII character, have to be written as % plus the hexadecimal of each byte. So URL encoding is UTF-8 first, then % per byte: 中 → E4 B8 AD → %E4%B8%AD, turning one CJK character into nine characters. In JavaScript, encodeURIComponent encodes most thoroughly (even / ? & =) and suits a parameter value, while encodeURI keeps the structural characters and suits a whole address. A form submitted as application/x-www-form-urlencoded also writes a space as + — an HTML form convention rather than part of RFC 3986.
Mojibake is not corrupted data; it is written with encoding A and read with encoding B. The UTF-8 for 中 is E4 B8 AD:
ä¸ — a run of letters with diacritics is the signature of “UTF-8 read as a Western encoding”.E4 B8 is one CJK character and AD is left as half of one — “odd CJK characters plus a half” is the signature of “UTF-8 read as GBK”, and 浣犲ソ and 涓枃 are both of that kind.� (U+FFFD).As long as the original bytes survive, reading them again with the right encoding restores the text perfectly. The first rule of fixing mojibake is therefore to keep the original file and not convert it repeatedly: each conversion can replace an illegal byte with ? or �, and that is the point at which information is genuinely lost. Lesson 14 has you reverse one: read the mojibake 浣犲ソ as GBK to get the bytes E4 BD A0 E5 A5 BD, then read those as UTF-8 to get the original text back. The modern answer is to declare UTF-8 consistently in the HTTP header, the HTML <meta charset>, the database connection and the save dialogue, so that a mismatch never gets the chance to happen.
UTF-16 has big-endian and little-endian byte orders, and files often start with a byte order mark, FE FF or FF FE. UTF-8 has no byte-order problem, but Notepad on Windows used to write the UTF-8 BOM EF BB BF by default, which breaks the first line of a shell script and puts an invisible character in front of the first CSV column name. If the UTF-8 breakdown view shows those three bytes at the start, you know where the file came from.
TextDecoder, and the internals of GB18030, Big5 and Shift_JIS are not covered.length is the character count”: in JavaScript it is the number of UTF-16 code units, and an emoji or a rare CJK character counts as 2; the n in a database VARCHAR(n) may be bytes or characters depending on the database.encodeURIComponent, or &, # and spaces will break the structure.Recognise %E4%B8%AD%E6%96%87 immediately as two UTF-8 CJK characters; know how to turn the three dot-separated Base64 parts of a JWT back into JSON; and understand why + and %20 in a parameter are treated as different things by the back end.
Use the method from lessons 13–14 to decide whether it is “UTF-8 read as GBK” or the reverse, then use the editor's “reopen with encoding” rather than “convert encoding” — keeping the original bytes is the first rule.
Store emoji with utf8mb4; estimate UTF-8 storage with 3 bytes per CJK character; and find out whether a VARCHAR length is counted in bytes or characters.
Truncate a user's display name with [...str].slice(0, n) rather than str.slice(0, n), so an emoji is not cut in half; and decide whether to count code units, code points or grapheme clusters.
An emoji with a skin-tone modifier or combined with zero-width joiners (such as 👍🏽 or 👨👩👧) is made of several code points, and the code point view lists them one by one. Lesson 8 asks for exactly one 4-byte character, so use a plain emoji such as 😀.
Neither. c3a9, C3 A9 and c3 a9 are all accepted.
The course uses the browser's built-in TextDecoder('gbk') (in practice a superset of GB18030), which outputs � for an illegal sequence; some tools output ? or skip it. The glyph differs but the meaning is the same — “this cannot be decoded here”.
If the original bytes are intact and only the reading was wrong, yes: reopen it in an editor with the correct encoding. If the file has already been “converted” and shows many ?, that information is gone and the course cannot bring it back.
In the current browser's localStorage; nothing is uploaded. The “share this lesson” link carries only the lesson number, not your progress.
The whole course runs locally in the browser: encoding, decoding, Base64 and URL encoding of the sample text are done by the page's own TextEncoder / TextDecoder and pure functions and are never sent to a server; progress is stored in local localStorage.
TextDecoder supports and their decoding algorithms, the specification behind the GBK and windows-1252 decoders used in the mojibake demos: https://encoding.spec.whatwg.org/（访问日期：2026-09-09）Updated 2026-09-09
14 lessons on bits, bytes, ASCII, Unicode code points, UTF-8 variable-length encoding, Base64 groups, URL encoding and where mojibake comes from
Goal试验文本是字母 A，它的一个字节显示为 01000001。把这个二进制数换算成十进制填入答案框。
| Character | Code point | UTF-8 bytes | UTF-16 code units | Bytes |
|---|---|---|---|---|
| A | U+0041 | 41 | 0041 | 1 |
计算机只存 0 和 1。一个「位」（bit）是一个开关，8 个位组成一个「字节」（byte），能表示 0–255 共 256 个值。二进制按位权求和：01000001 = 64 + 1 = 65。所有文字、图片、声音最终都是一串字节，「编码」就是规定「哪个数代表哪个字符」。