6. Text input systems

Today's topics

Text input methods (IME) for Japanese

Various keyboards and input devices

Text input in various environments

Text input in the mobile computing age

Text input on small devices

Historical CHI Video Project

https://www.youtube.com/playlist?list=PLqhXYFYmZ-VeryE_-sJuc0oo1e_szTkTa

https://www.youtube.com/watch?v=2C5iIxssa3I&list=PLqhXYFYmZ-VeryE_-sJuc0oo1e_szTkTa

Masui's text input system system

https://www.youtube.com/watch?v=OOviDNZ9vAE&list=PLqhXYFYmZ-VeryE_-sJuc0oo1e_szTkTa&index=201&t=4s

Text input

Important for intellectual activities

Required for text-based search

SIG name changes in IPSJ

IPSJ = Information Processing Society in Japan (情報処理学会)

Japanese text input methods (JI) 1981～1984

Japsnese text processing (JDP) 1985～1986

Text processing and human interface (DPHI) 1987～1988

Human interface (HI) 1989～2006

Human-computer interaction (HCI) 2007～

Problems of various input devices

Keyboards

Key layouts (Dvorak, Qwerty, ...)

Tablet devices

Multi-touch

Swipe operation

Very small display

Input devices for VR

Trends of input devices

Special keyboards

Pen devices / touch devices

Character recognition

Ambiguous input

Keyboards with small number of keys

Self-made keyboards

(Speech input)

Problems of Japanese text input systems

For personal computers

Too many buttons and function keys

For smartphones

Many keys and menus

State transition difficult

Different methods for different machines

PC / smartphone / PDA / ...

What styleis good for digital signages?

Personalization difficult

Word registration difficult

No shared dictionary

Why Japanese input is difficult

Phrase-based conversion (連文節変換)

Somehow very popular

Inline conversion on apps

Apps should take care of IME

Misc features

Katakana conversion, Greek conversion, ...

History of Japanese text input

Kanji tablet (digitizer)

2-stroke Kanji input

Clause-based conversion

Phrase-based conversion

Predictive text input

Kanji tablet

Use "Kanji table" and pick up characters

Similar to typesetting

https://gyazo.com/da7d130e342a009e26faafdec7c038fa

2-stroke Kanji input

e.g. TUT-code

Type "L" "D" for "指"

Kana-kanji conversion

Use pronunciation for selecting Kanji characters

e.g. "masui" => "増井"

Now very popular in Japan

Lawsuit against Toshiba

Issue: Who invented Kana-Kanji conversion?

Prof. Amano sued Toshiba

Settled in 2012/5 6,430,000 yen

https://gyazo.com/d6247e9e851b8575774d128b37524078

Problems of phrase-based Japanese text input (1)

Accurate input required

Mistakes not allowed

Not ideal for IoT age

Problems of phrase-based Japanese text input (2)

All the pronunciation should be entered

shounandaiekimade for "湘南台駅まで"

"駅", "まで", "湘南台" can be predicted

Entering a long text without mistake is difficult

Only experts can do that on the desk

Almost impossible in mobile environment

Problems of phrase-based Japanese text input (3)

Error correction inevitable

kyouhaishaniitta

今日は医者に行った? (Did you meet a doctor today?)

今日歯医者に行った? (Did you meet a dentist today?)

Special error correction methods required

Ambiguous pattern matching difficult

Problems of phrase-based Japanese text input (4)

Customization difficult

Dictionary sharing difficult

Problems of phrase-based Japanese text input (5)

Not usable with various languages

Not usable for dialects

Problems of phrase-based Japanese text input

No mistake allowed

Many special operations

Error correction required

Customization difficult

Valid only for "standard" Japanese notation

Not ideal at all

SKK

https://www.youtube.com/watch?v=EfULaLEVjAI

Capital letter for entering Kanji

Kanji ⇒ 漢字

Small letter for entering Hiragana

kanji ⇒ かんじ

Capital letter for okurigana

KaeRanai ⇒ 帰らない

SKK

Simple algorithm

Simple dictionary

Quick operation possible

× Useful only for Japanese language

Text input for the mobile age

Problems of standard keyboards

Too large

Difficult to use on a train

Stable desk and chair required

Two hands required

Long-term practices required

Special keyboards

Separate keyboard

One-hand keyboard

Special shape keyboard

https://s3-ap-northeast-1.amazonaws.com/masui.org/6/6/666a3d8f888c4455e8d49fcf04618dc2.gif https://s3-ap-northeast-1.amazonaws.com/masui.org/7/c/7c8a8405f08c6830e901d44dcab6f0dd.gif https://s3-ap-northeast-1.amazonaws.com/masui.org/9/2/92a27f85561116354aebae6e071462b5.gif https://s3-ap-northeast-1.amazonaws.com/masui.org/4/4/447bce426d96c28a2129da3e92b348b5.gif https://s3-ap-northeast-1.amazonaws.com/masui.org/9/7/9705c76e7ffbd7e8c663d1b175accef6.gif https://s3-ap-northeast-1.amazonaws.com/masui.org/6/7/67be8ee9844462e7c67f4b43efd8e5a5.gif https://s3-ap-northeast-1.amazonaws.com/masui.org/1/2/12815d37e2df20e8b0ec3fc91ba59305.gif https://s3-ap-northeast-1.amazonaws.com/masui.org/0/d/0d75fa654ced71cb69f31a1249f08f53.gif https://s3-ap-northeast-1.amazonaws.com/masui.org/c/6/c64b9e9943417cc7fe9c5a5edfccb84d.gif https://s3-ap-northeast-1.amazonaws.com/masui.org/d/d/dd37db63d67c16d28b522f5dfbc6dd06.gif https://s3-ap-northeast-1.amazonaws.com/masui.org/f/0/f07886b328bf67cdb076fedbd83c4398.gif https://s3-ap-northeast-1.amazonaws.com/masui.org/4/d/4d26720a3a9d1460e00319e499463c90.gif https://s3-ap-northeast-1.amazonaws.com/masui.org/4/1/41024bdd240ddc932d5903a8aee3f1af.gif https://s3-ap-northeast-1.amazonaws.com/masui.org/d/7./d76d63f9fc7beca2992fec2e2c821018.gif https://s3-ap-northeast-1.amazonaws.com/masui.org/8/7/8700499e2e3cf3973a940783c9b8b230.gif https://s3-ap-northeast-1.amazonaws.com/masui.org/8/d/8bd257004420ffee336a041661b6ae3d.gif https://s3-ap-northeast-1.amazonaws.com/masui.org/d/8/d825d423c7ee8db8894c2635341a5824.gif https://s3-ap-northeast-1.amazonaws.com/masui.org/a/c/ac6895ddca8fcd557b81fa19875e8993.gif https://s3-ap-northeast-1.amazonaws.com/masui.org/9/a/9ab0352f3475c0664e2e6a3b2e4e294b.gif https://s3-ap-northeast-1.amazonaws.com/masui.org/3/b/3b6cc5006b3fa01a73e5b41a101e18e0.gif https://s3-ap-northeast-1.amazonaws.com/masui.org/1/f/1fba8099ea050d5b1dd39018219cd4b3.gif https://s3-ap-northeast-1.amazonaws.com/masui.org/0/9/09fa6289584ccc086f169098a4fd648d.gif https://s3-ap-northeast-1.amazonaws.com/masui.org/e/b/eba76e9884d38856cb7ab50fcf530583.gif https://gyazo.com/2db3ad0073f457265727945c21e0ffa8

Keyboard info on Scrapbox

http://scrapbox.io/MECHKEYS https://gyazo.com/fbbad2c73ed616b96bc0cff33f9eb599

Self-made keyboard shop

https://yushakobo.jp/ https://gyazo.com/7013bed04853569dcb704ad4e39c5dc8

SHK

Use multiple keys for entering a character

Used for PalmPilot

http://gyazo.com/5943f3ce1d8f62a214ca8b16e62c11ec.png

http://gyazo.com/ebadee4528193e87e94fc4651f832595.png

Half-Qwerty

Fold the right side of ASCII KB to left

http://gyazo.com/2e5d62def5daf7c29b26bf8c747af977.png

http://gyazo.com/0c71febcb90afd97e1bf43ceae9de0f3.png

FrogPad

Lay out frquently-used 15 keys

http://gyazo.com/b7e0e862c533631d8946d1b619dd9ec6.png

CUT Key

From Misawa home

Combination of consonant and vowel

http://gyazo.com/eb53e043f5b9f09f9cbb1e4fe6b5774f.png

Fujitsu keyboard

Combination of consonant and vowel

http://gyazo.com/77f87713c5b8be59255422638248c441.png

MessageEase

Tap same key twice for frequently-used character

Use key combination for other keys

http://gyazo.com/e986e9dd2375b81d45853c395b45b014.png

TagType

Special layout for handicapped

Consonant and vowel

http://gyazo.com/8faf8ba53fed494930cbd4dc3a513425.png

ThumbSync

Use a stick for entering characters

http://gyazo.com/f277a0e61c6ac571f94761c2f25ab56b.png

Fingering (Fukumoto 1996)

Detect finger movement by motion sensors

Use multiple fingers for character selection

http://gyazo.com/f2de3995fee893293d9a411932a91057.png

http://gyazo.com/7e70e8269fe83db9755bbaa60c31ce83.png

TelemetRing (Takahashi, Fukumoto 2020)

https://gyazo.com/2127b6bb0be2979791b30a418985a793

Accuracy: 90%

HoriKeys

Use tilt for selecting left/center/right of QWERTY KB

https://gyazo.com/5322b62d52aec14e4ae5f6ef366e07bc

Video: HoriKeys

https://s3-ap-northeast-1.amazonaws.com/masui.org/3/c/3c49897310bdad35b072ff5f98a25f4a.mp4

4-key Japanese text input

Map all keys to a, s, d, f

http://gyazo.com/57b0d7f9b73f6df792f06ba9ac9c9059.png

Video: 4-key input

https://s3-ap-northeast-1.amazonaws.com/masui.org/4/2/42dd5991a7c96eef43363d1e4fd974cb.mp4

10-key character input

Automatic disambiguation

e.g. "TGD" => "THE"

Widely used on old mobile phones in Europe

http://gyazo.com/ce05f7102f1de15ccd32dc88b49818a6.png

TouchMeKey

Use on ly consonant keys

10 keys / 4 keys

Prediction based on PPM (Prediction by Partial Match)

http://gyazo.com/ca722272f527efceb2677dd4ce2e4019.png

WordWise

Use a aux key (「1」)

「2」～「9」 => main characters

Aux key + 「2」～「9」 => sub characters

Ambiguity resolution often works

http://gyazo.com/bf0ffdab3e311e50bafa2f13387d6530.png

Thumbscript

Use special KB for entering characters

http://gyazo.com/cde0ba32727a2fb07871a61ae7391864.png

http://gyazo.com/cee7b7ef579e80b241e8acf9cef3ef04.png

ThumbType

Keyboard on a tablet

http://gyazo.com/9b0d180897cbdbbee9d323ca5d359642.png

Sea otter keyboard

http://gyazo.com/2805a8654f5d2bcef6b819951126d049.png

http://gyazo.com/04f94e0a51ccb54159ed2d507d30ab45.png

http://gyazo.com/898285f27c43b5a45f5bec65a12a969d.png

H4Writer

Key combination based on Huffman code

http://gyazo.com/4cedb881d0b704eda2e2e7f9d9900746.png

http://gyazo.com/55f1541b70e60dcd0161788f52be9662.png

H4Writer

Key assignment

http://gyazo.com/e8564e25ff19b5bb67b7eaea83b3c607.png

1 Line KB

1-line Qwrety KB

枠をタップして候補選択

http://gyazo.com/22ec12c2dd3a54cfbaf20605ca6192fb.png

http://gyazo.com/8f9fb004fbd4380647b008702b65bc09.png

http://gyazo.com/e4cd564b4f2f5603eb0a142d08d4a898.png

Various pen/touch text input methods

T-Cube

Unistroke

Graffiti

Quikwriting

Cirrin

Octave

Oval layout

Fitaly layout

Metropolis layout

Shark

Swype

ShapeWriter

Pen computers

Can be used while standing

Small device

Can be hidden while not in use

Can be put on a wall

More useful than finger-based tablets

T-Cube

http://gyazo.com/eedbbd982f2000f08ebb30ac66425e89.png

Pie-menu + tap location

1 character entry by tap+move

T-Cube for Japanese

http://gyazo.com/f9f4f36de68637d5b8695d6bc9183c8e.png

Enter Hiragana with T-Cube

Similar to "Flick input method" for iPhone

"Flick" text input

http://gyazo.com/1691febad27439d3bf44232c54dcb1e8.png

Hanabi

For Newton PDA

http://www.clubnewton.com/newton/Packages/Hanabi00.GIF

Unistroke

Developed at Xerox PARC

1-stroke character recognition

Map alphabets to special strokes

Not easily remembered

http://gyazo.com/8e8a1730a34c375b415ba2cabb6bbce3.png

http://gyazo.com/c77ac54c04a97ee553f19909c0c9614f.png

Graffiti

http://gyazo.com/c6fbb6277395e94f6294239ab0cd3f99.png

Available on old PDAs (Zoomer, OmniGo, PalmPilot)

1 stroke for 1 character

Easier to remember than Unistroke

Lawsuits on Graffiti

Xerox sued Palm

Xerox finally won, but no impact at that time

TOO LATE!

Meaningless lawsuit!

Quikwriting

Divide the area with 8 pie-shaped areas

Select a character based on pen movement

center / right / center → t

center /right / upper right / center → u

Multiple character input possible with one stroke

http://gyazo.com/21b5807b089323a36d51cf32216a4093.png

http://gyazo.com/f733ecaa576d69296ba3a7143d50a021.png

8pen

Use pen direction and rotation

http://www.8pen.com/ https://gyazo.com/e390375d7e9b46ff9dbba002ffa20313.png

https://www.youtube.com/watch?v=99vsUF4NuLk

Cirrin

Characters on circle

Select characters witout pen-up

http://gyazo.com/b336c4868b7c26ded5b5635bd035bfa2.png

EdgeWrite

Use rectangle edges for character input

Goog for people who cannot control fingers and hands

http://gyazo.com/ff817c8994e1adfbcc1a6b1ca90bb7e0.png

https://gyazo.com/79ad6b6806c7f59fd33edf8b065ecbd3.png

Octave

Quikwritingと同様のペンストローク

ペンを8方向になぞって字形で文字を選択していくと

候補単語が予測され、

正しい単語が予測されたところでペンを離す。

http://gyazo.com/1037fc48dc2ffdd73f16074062d959ca.png

NeoPad

Character entry based on character shapes

http://gyazo.com/ec1ec213d21d92bf21622994f2daff95.png

http://gyazo.com/f86869de83c6a264127d0f462a538536.png

Oval layout

Faster than rectangular layout when used with fingers

http://gyazo.com/df30356eed760973bb50b3c655fe5e43.png

Fitaly keyboard

Pen movement minimized

http://gyazo.com/33f56be9e72711ed6d5b79c6e867a87d.png

Metropolis layout

Pen movement minimized

http://gyazo.com/164091346c6ad0cf17a13fcfcf06e862.png

Shark

Draw a line on software keyboard

http://gyazo.com/3c7edf27cb9d08e5b6be87eb30d90f58.png

Swype

http://www.youtube.com/watch?v=1bibCui3lFM

http://gyazo.com/0569a1f2f190195e07f344b1484f857c.png

ShapeWriter

http://gyazo.com/88b989d4aa02b82c767ae6b059438fa8.png

Modern Touchscreen Keyboards as Intelligent User Interfaces - A Research Review

by Shumin Zhai

https://www.youtube.com/watch?v=0Cwrd2mDRjA

Phone call from Google

SplitBoard

https://gyazo.com/e4fd61c6d7bac98d8e79a9db03c64948.png

TapBoard

https://www.youtube.com/watch?v=oyLC5UeCbsA

Bimanual gesture keyboard

http://gyazo.com/4ed0d03c6209fe86da91fb1de038463e.png

DualKey

https://gyazo.com/2fd1eff85c49cfc2fb68d6e65c1a55b2.png

Change characters based on finger

Mima's system

https://gyazo.com/47770ef0b83cb571a580e900bebcaad9.png

https://www.youtube.com/watch?v=XeXOqMjZYSA

TiltType

https://gyazo.com/ee708874dab3c9171749eda21ee43224.png

TiltText

http://gyazo.com/bb7edd016cc5b82ff60f57175cfe26dd.png

SWiM

https://www.youtube.com/watch?v=8BuXIlQ7_XM

GesText

https://gyazo.com/b5804cf05d22e1ef15934fd8c311b7d4.png

https://www.youtube.com/watch?v=_dJROxk9nNs

Swipeboard

https://gyazo.com/179d619880bd0b0b2eab72dc99350467.png

https://www.youtube.com/watch?v=QfwReAyF3MY

ZoomBoard

https://gyazo.com/288280299e4b22f27bef8bef5102d327.png

https://www.youtube.com/watch?v=ivEzWporlhg

SwipeZone

Use SwipeBoard on HMD

https://gyazo.com/3520401c46ab2bbc63fc224d74f1f313.png

https://www.youtube.com/watch?v=hN3zPVXQjTI

Ideal text input systems

Universal

Eerybody can use

Available for all languages

Ubiquitous

Usable anywhere , at any time

Use resources on the net

Simple functions

Independent from apps, OSs

Minimul features

Minimul practice required

Use resources on the net and in the real world

Put dictionaries on the net and share

Put the conversion system on the net

e.g. Google transliterate api

Use various info on the net

Use time, location, etc. for conversion

Japanese dictionary on the net

https://scrapbox.io/Gictionary https://gyazo.com/d8301056b9c5fabb3de3887369be9284

POBox (Masui)

http://gyazo.com/ac2b347a7042f920edd576ee07c4b7f4.png

Prediction-based Japanese text input system

Select a candidate from the list

Give small conditions to get the list

Approximate pattern matching

POBox algorithm

Specify pronunciation / handwriting

Show candidate words

Use corpuses for predicting next word

Approximate pattern matching if no candidate found

Products

http://gyazo.com/1822ff6c32df6b726808bf0c3b447408.pnghttp://gyazo.com/59bb09888b26a2942ada1804c7de555a.pnghttp://gyazo.com/8c94d0ac06c695bdc694ce72903a4b32.pnghttp://gyazo.com/f2c255399d4b1fbc8ab7dbac48b2f088.pnghttp://gyazo.com/8d993e684403bb2d6291bbdc77f86520.png https://gyazo.com/045acf4df7c17ba92f64f784fb375f4c

Demo: POBox

Integration with handwriting recognition

Move to recognition mode after drawing a line

http://gyazo.com/a26a5a4f16b345fc3fbf48e2d17bbd98.png

https://www.youtube.com/watch?v=VNYCLOQQxXs

Gyaim

IME on Mac

Implemented in RubyMotion

No natural language processing

Just search words and show

Gyaim

https://gyazo.com/648245adc6252b716db7af9f7b2022ab.gif