From Writing to Typing to Writing: The UX Evolution in Language Interaction

Have you ever thought how people across the world use their language while interacting with computers? A while ago, when computers were invented, it wasn’t natural for humans to write in a way that is different than a pencil. Throughout the years, the writing (typing) interaction was far from the original pencil, but recently, technology has allowed us to go back to our roots, and made typing feel much more like the pencil experience. In this article I’ll outline this experience through a special use-case: typing in Chinese in general, and specifically in Mainland China.

Lately, Apple launched new gestures for Apple pencil. My first interaction with these gestures is one of these moments I live for, really. It’s important to say — it’s not about the technology, as we can find it in many products, and many forms. It’s the experience of having a digital product in my hand, but the interaction is familiar, as it’s an actual integration of the physical world into the digital one.

Apple Pencil (by Suganth on Unsplash)

Throughout the years we can see that digital experiences work well when they give us an analogy of the real world. As the digital world is flat and complicated, a great way to create an intuitive usability is to use what’s most familiar to us — the physical world.

Apple did it perfectly this time. I was about to tap on Google search bar with Apple Pencil, and suddenly it’s sketched something. I did it again and the penny dropped — It’s a Pencil!

Apple Pencil New Gesture: Using OCR to convert writing-hand to digital letters. (Taken by Inbar Kofman)

Apple wasn’t the first to harness this advanced OCR (Optical Character Recognition) technology to make a better user experience in writing languages. Have you ever thought how logogram writing systems, that are based on characters, in languages like Chinese or Japanese, are used in the digital world?

Well, to have a better understanding of that, I’ll give a short introduction about logogram writing systems by focusing on Chinese.

Phonetic writing systems, like English, are based on letters that act like building blocks: Each letter has its own syllable and symbol, and a combination of letters creates a word. As a result, humans can read new words by using their preliminary knowledge about each single letter.

On the contrary, logogram writing systems, like Chinese, are structured differently. In Chinese, a character (hanzi) doesn’t represent a syllable, but a polysyllabic of a word. The Chinese characters are often composed of parts that may represent physical objects, abstract notions, or pronunciation. An attachment of two characters (or more) together is a radical. This combination provides a different meaning. Literacy requires the memorization of a great number of characters: Educated Chinese is familiar with about 4,000+ characters(!)

Evolution of “Mountain” character (Wikipedia), the character represents the mountains shape

Now, back to the digital world. How are logogram writing systems typed?

The computer was invented in the USA, and this can be a reason that keyboards are built for phonetic languages. Phonetic languages have a small amount of letters that can easily fit a keyboard, and each key represents a letter.

So how the Chinese characters can be typed using a keyboard?

This historical Chinese typewriter can reflect how complicated designing a keyboard for logogram writing systems can be. Computers can’t contain such a huge keyboard like this historical one.

Chinese typewriter (Stanford University). “To use a Chinese typewriter, the typist moves a character-selection lever over a tray bed filled with metal character slugs. The typist then presses a type bar, and the lever picks up the character, inks it, types it and returns it to its place.”

The modernism forced Chinese speakers to find new solutions. Here are two (out of many) popular methods for typing Chinese in computers, and how technologies like OCR made Chinese speakers’ lives easier.

  1. Shaped based — Wubi: This method is based on the structure of characters, making it possible to input characters even when the user does not know the pronunciation. In short, the keyboard is divided into regions, and the user is able to build a character by selecting its parts.
Wubi Keyboard (Wikimedia)

2. Phonetic based — Pinyin: This method is more similar to phonetic systems, as it’s based on pronunciation using latin letters and as a result — the user needs to select the relevant Chinese characters.

Pinyin: User types in Pinyin and selects a Chinese character (cheng-tsui)
Pinyin: A user types a complete sentence, and gets the most relevant results in order to select it in Chinese (Koh Li-Na)

We can see how the Chinese writing system was adjusted to fit computers interaction. The UX aspect pushed the experience of writing Chinese in computers some levels up.

When I visited China for the first time — I started wondering about this writing-typing interaction. I found some interesting facts about how Chinese speakers tend to use their language while interacting with their mobiles. The first thing I came across is that they actually prefer to use voice messages.

In case they do write, I found a lot of Pinyin users. Pinyin works with Auto-correct technology to help users find the Chinese characters faster. “Pinyin is much more forgiving when you know fewer hanzi as it doesn’t require you to remember exactly how a specific character is written, you just need to know what it sounds like and how it’s used in context. I might forget how to write 精彩 stroke-for-stroke, but I can easily type in “jing cai” in pinyin and recognize and select it from the suggested option”. (Koh Li-Na)

Using hand drawing on touch screen. (Photo taken by Inbar Kofman)

In addition, I saw users that actually hand draw the characters on their mobile touch screen. This technology uses OCR to convert their hand drawing to Chinese characters instead of using a keyboard.

This hand draw option can help closing the gap of non-Pinyin speakers (like some of the older generation) to write how they used to writing using their smartphone.

After putting all of this together, Apple Pencil new gestures seems to me like a benefit, a wow moment, while for Chinese speakers it’s another development of the typing interaction.

We tend to think that technology and user experience are two sides of an equation. However, when we make them work together it makes wonderful solutions for complicated pain-points. These examples represent the ideal UX work and we haven’t spoken about voice interaction yet. :)

Thanks for reading! Hope you enjoyed it. If you have anything to comment or add, feel free to drop a line and develop the knowledge of this article.

Product Designer (UX UI)