How does Python handle Unicode internally, and what is the difference between str and bytes?
str is Unicode text; bytes is raw binary; encode/decode at boundaries
In Python 3, str represents Unicode code points, while bytes represents raw 8-bit values. You decode bytes to str and encode str to bytes. CPython stores str internally using the most compact representation that fits the widest character: latin-1, UCS-2, or UCS-4. That means len(str) returns the number of code points, not bytes and not grapheme clusters. Text should be decoded at input boundaries and encoded at output boundaries; keep bytes for binary protocols, files, and network payloads.
- 1
str + str is text concatenation; bytes + bytes is binary concatenation. Mixing them raises TypeError.
- 2
Encoding is str -> bytes; decoding is bytes -> str. Always specify encoding, usually utf-8, at boundaries.
- 3
Common mistake: assuming len(str) counts user-perceived characters. Emoji and combining marks can span multiple code points.
- 4
Trade-off: UTF-8 bytes are compact for ASCII but variable-width; UTF-32 is fixed-width but memory-heavy.
- 5
Version note: Python 3.15 is expected to move toward UTF-8 mode by default via PEP 686; explicit encoding is still best practice.
Scenario Questions
0-2 years experience
- 1You read a text file. Do you get str or bytes?
- 2You need to send text over network. What conversion?
2-5 years experience
- 1You decode UTF-8 bytes and get UnicodeDecodeError. How do you debug?
- 2Why is len('é') sometimes 1 and sometimes 2?
5-8 years experience
- 1You process multilingual text and need grapheme clusters. What library?
- 2You compare str and bytes for equality. Why false?
8+ years experience
- 1Design an encoding boundary strategy for a system that stores bytes but serves Unicode text.
- 2You need to handle invalid UTF-8 without data loss. What approach?
Follow-up Questions
- Why can len('é') be 1 or 2 depending on normalization?
- How do you handle invalid UTF-8 without losing data?