{"id":298,"date":"2014-11-06T23:15:05","date_gmt":"2014-11-06T23:15:05","guid":{"rendered":"http:\/\/sonny.cslu.ohsu.edu\/~gormanky\/blog\/?p=298"},"modified":"2018-09-06T21:54:11","modified_gmt":"2018-09-06T21:54:11","slug":"understanding-text-encoding-in-python-2-and-python-3","status":"publish","type":"post","link":"https:\/\/www.wellformedness.com\/blog\/understanding-text-encoding-in-python-2-and-python-3\/","title":{"rendered":"Understanding text encoding in Python 2 and Python 3"},"content":{"rendered":"

Computers were rather late to the word processing game. The founding mothers and fathers of computing were primarily interested in numbers. This is\u00a0fortunate: after all,\u00a0computers only know about numbers<\/em>. But as Brian Kunde explains in his brief history of word processing<\/a>, word processing existed long before digital computing, and the text processing has\u00a0always been something of an afterthought.<\/p>\n

Humans think of text as consisting of ordered sequence of “characters” (an\u00a0ill-defined Justice-Stewart-type concept which I won’t attempt to clarify\u00a0here). To manipulate text in digital computers, we have to have a mapping between the character set (a finite list of the characters the system recognizes) and numbers. Encoding <\/em>is the process of converting characters to numbers, and\u00a0decoding\u00a0<\/em>is (naturally) the process of converting numbers to characters. Before we get to Python, a bit of history.<\/p>\n

ASCII and Unicode<\/h1>\n

There are only a few\u00a0character sets that have any relevance to life in 2014. The first is ASCII<\/a> (American Standard Code for Information Interchange), which was first published\u00a0in 1963. This character set consists of 128 characters intended for use by an English audience. Of these 95 are\u00a0printable<\/em>, meaning that they correspond to lay-human notions about\u00a0characters. On a US keyboard, these are (approximately) the alphanumeric and punctuation characters that can be typed with a single keystroke, or with a single keystroke while holding down the Shift key, space, tab, the two newline characters (which you get when you type return), and a few apocrypha. The remaining 33 are\u00a0<\/em>non-printable “control characters”. For instance, the first character in the ASCII table is the “null byte”. This is indicated by a\u00a0''<\/code> in C and other languages, but there’s no\u00a0standard way to render it. Many control characters were\u00a0designed for earlier, more innocent times; for instance, character #7 'a'<\/code> tells the receiving device to ring a cute little bell (which were apparently attached to teletype terminals); today your computer might make a beep, or the terminal window might flicker once, but either way, nothing is printed.<\/p>\n

Of course, this is completely inadequate for anything but English (not to mention those users of\u00a0superfluous\u00a0diaresis<\/a>…e.g., the editors of the New Yorker, Mot\u00f6rhead). However, each ASCII\u00a0character takes up only 7 bits, leaving room for another 128 characters (since\u00a0a byte has an integer value between\u00a00-255, inclusive), and so engineers could\u00a0exploited the remaining 128 characters to write the characters from different\u00a0alphabets, alphasyllabaries, or\u00a0syllabaries. Of these ASCII-based character sets, the best-known are ISO\/IEC 8859-1<\/a>, also known as Latin-1, and\u00a0Windows-1252<\/a>, also known as CP-1252. Unfortunately, this created more problems than it solved. That\u00a0last bit just didn’t leave enough space for the many languages which need a larger character set (Japanese kanji<\/i> being an obvious example). And\u00a0even when there are technically enough code points left over, engineers working in different languages didn’t see eye-to-eye about what to do with them. As a result, the state of affairs made it impossible to, for example,\u00a0write\u00a0in<\/em> French\u00a0(ISO\/IEC 8859-1) about<\/em>\u00a0Ukrainian (ISO\/IEC 8859-5<\/a>, at least before the 1990 orthography\u00a0reform<\/a>).<\/p>\n

Clearly, fighting over scraps\u00a0isn’t going to cut it in the global village. Enter the\u00a0Unicode<\/a>\u00a0standard and its\u00a0Universal Character Set<\/a>\u00a0(UCS), first published in 1991. Unicode is\u00a0<\/em>the\u00a0platonic ideal of an character encoding,\u00a0abstracting away from the need to efficiently<\/em> convert all characters to numbers. Each character is represented by a single code with various metadata (e.g., A<\/code> is an “Uppercase Letter” from the “Latin” script). ASCII and its extensions map onto a small subset of this code.<\/p>\n

Fortunately,\u00a0not\u00a0all\u00a0encodings<\/em>\u00a0are merely shadows on the walls of a cave<\/a>. The One True Encoding is\u00a0UTF-8<\/a>, which implements the entire UCS\u00a0using an 8-bit code. There are other encodings, of course, but this one is ours, and I am not alone in feeling strongly that UTF-8 is the chosen encoding. At the risk of getting too far afield, here are two arguments for why\u00a0you and everyone you know should just use UTF-8. First off, it is hardly matters much which UCS-compatible encoding we all use (the differences between them are largely arbitrary), but what\u00a0does<\/em> matter is that we all choose the same one<\/a>. There is no\u00a0general procedure for “sniffing” out the encoding of a file, and \u00a0there’s nothing preventing you from coming up with a file that’s a French cookbook in one encoding, and a top-secret message in another. This is good for\u00a0steganographers<\/a>, but bad for the rest of us, since so many\u00a0text files\u00a0lack encoding\u00a0metadata.\u00a0When it comes to encodings, there’s no question that UTF-8 is the most popular\u00a0Unicode encoding scheme worldwide, and is on its way to becoming the de-facto standard. Secondly,\u00a0ASCII is valid UTF-8<\/em>, because UTF-8 and ASCII encode the ASCII characters in exactly the same way. What this means, practically speaking, is you can achieve nearly complete\u00a0coverage of the world’s languages simply\u00a0by assuming\u00a0that all the inputs to your software are UTF-8. This is a big, big win for us all.<\/p>\n

Decode\u00a0early, encode late<\/h1>\n

A general rule of\u00a0thumb for developers is “decode early” (convert inputs to their Unicode representation), “encode late” (convert back to bytestrings). The reason for this is that in nearly any programming language, Unicode strings behave the way our monkey brains expect them to, but bytestrings do not. To see why, try iterating over non-ASCII bytestring in Python (more on the syntax later).<\/p>\n

>>> for byte in b\"a\u00f1o\":\r\n...     print(byte)\r\n...\r\na\r\n?\r\n?\r\no<\/code><\/pre>\n
There are two surprising things here: iterating over the bytestring returned more bytes then there are “characters” (goodbye, indexing), and furthermore the 2nd\u00a0“character” failed to render properly. This is what happens when you let computers dictate the semantics to our monkey brains, rather than the other way around. Here’s\u00a0what happens when we try the same with a Unicode string:<\/p>\n
>>> for byte in u\"a\u00f1o\":\r\n...     print(byte)\r\n...\r\na\r\n\u00f1\r\no<\/code><\/pre>\n
The Python 2 & 3 string models<\/h1>\n
Before you put this all into practice,\u00a0it is important to note that Python 2 and Python 3 use very different string models. The familiar Python 2 str<\/code> class is a bytestring. To convert it to a Unicode string, use the\u00a0str.decode<\/code> instance method, which returns a copy of the string as an instance of the unicode<\/code>\u00a0class.\u00a0Similarly, you can make a\u00a0str<\/code> copy of a unicode<\/code> instance with unicode.encode<\/code>. Both of these functions take a single argument: a string (either kind!) representing the encoding.<\/p>\n
Python 2 provides specific syntax for Unicode string literals (which you saw above): the a lower-case u<\/code>\u00a0prefix before the initial quotation mark (as in u\"a\u00f1o\"<\/code>).<\/p>\n
When it comes to Unicode-awareness, Python 3 has totally flipped the script; in my opinion, it’s for the best. Instances of str<\/code> are now Unicode strings (the u\"\"<\/code> syntax still works, but is vacuous). The (reduced) functionality of the old-style strings is now just available for instances of the class bytes<\/code>. As you might expect, you can create a bytes<\/code> instance by using the encode<\/code> method of a new-style str<\/code>. Python 3 decodes bytestrings as soon as they are created, and (re)encodes Unicode strings only at the interfaces; in other words, it gets the “early\/late” stuff right by default. Your APIs probably won’t need to change much, because Python 3\u00a0treats\u00a0UTF-8 (and thus ASCII) as the default encoding, and this assumption is valid\u00a0more often than not.<\/p>\n
If for some reason, you want a bytestring literal, Python has syntax for that, too: prefix the quotation marks delimiting the string with a lower-case b<\/code>\u00a0(as in b\"a\u00f1o\"<\/code>; see above also).<\/p>\n
tl;dr<\/h1>\n
Strings are ordered sequences of characters. But computers only know about numbers, so they are encoded as\u00a0byte arrays; there are many ways to do this, but UTF-8 is the One True Encoding. To get the strings to have the semantics you expect as a human, decode a string to\u00a0Unicode as early as possible, and encode it as bytes as late as possible. You have to do this explicitly in Python 2; it happens automatically in Python 3.<\/p>\n
Further reading<\/h1>\n
For more of the historical angle, see Joel Spolsky’s\u00a0The absolute minimum every software developer absolutely, positively must know About Unicode and character sets (no excuses!)<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"
Computers were rather late to the word processing game. The founding mothers and fathers of computing were primarily interested in numbers. This is\u00a0fortunate: after all,\u00a0computers only know about numbers. But as Brian Kunde explains in his brief history of word processing, word processing existed long before digital computing, and the text processing has\u00a0always been something … Continue reading “Understanding text encoding in Python 2 and Python 3”<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_crdt_document":"","footnotes":""},"categories":[3,4,5,8],"tags":[],"class_list":["post-298","post","type-post","status-publish","format-standard","hentry","category-dev","category-language","category-nlp","category-python"],"_links":{"self":[{"href":"https:\/\/www.wellformedness.com\/blog\/wp-json\/wp\/v2\/posts\/298","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.wellformedness.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.wellformedness.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.wellformedness.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.wellformedness.com\/blog\/wp-json\/wp\/v2\/comments?post=298"}],"version-history":[{"count":1,"href":"https:\/\/www.wellformedness.com\/blog\/wp-json\/wp\/v2\/posts\/298\/revisions"}],"predecessor-version":[{"id":591,"href":"https:\/\/www.wellformedness.com\/blog\/wp-json\/wp\/v2\/posts\/298\/revisions\/591"}],"wp:attachment":[{"href":"https:\/\/www.wellformedness.com\/blog\/wp-json\/wp\/v2\/media?parent=298"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.wellformedness.com\/blog\/wp-json\/wp\/v2\/categories?post=298"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.wellformedness.com\/blog\/wp-json\/wp\/v2\/tags?post=298"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}