11. Real-world I/F

http://gyazo.com/de8b1f08b7b7cdb65dda4cf3dd0bd71c.png

https://gyazo.com/423084ecae6c8a425097fce366d9b4b3

http://gyazo.com/de8b1f08b7b7cdb65dda4cf3dd0bd71c.png

Today's topics

Real-world computing

Augmented Reality

Mobile computing

;; Wearable computing

Ubiquitous computing

Changes of computer usage

Before the mobile age

In the computer room

On the desk

Mobile age

In the commuter train

On the sidewalk

Whereever, Whenever, Anyone

Integration betweein computers and real world

Changes of computer usage

http://gyazo.com/47c7e8a93135af6fe602a77487a6f0a0.png

Old-style computers

CUI, GUI

Computers in the real world

Real-world computing

Invisible Computer

Ubiquitous Computing

Real-world computing

Use real-world objects for computing

Desks, papers, places, time,

Computers are invisible

No distinction between computer data and real-world data

No distinction between written numbers and data in the computer

Drastic I/F changes

How to buy a train ticket

No ticket vending machine is required if Suica is available

Cash register in a super market

No need for a cash register if everything in the shopping cart is recognized

Automatic door

Technologies for real-word computing

Infrastructure

Wireless LAN

Small computers

Input devices / sensors

Sensors

Cameras

Barcode readers and RFID tags

Location sensors (GPS, WiFi)

Displays

Thin LCD display

Projector

Research trends

Ideas on real-world interface

Integration of computers and real-world

Sensing technologies

Sensors

Application for sensors

Text input /display in mobile environment

Text input methods

Display techniques for small devices

Life in the uniquitous computing age

Search techniques

Communication techniques

Terms

Ubiquitous Computing (Ubicomp)

Pervasive Computing

Cyber Physical System (CPS)

Augmented Reality (AR)

Virtualized Reality ( = Augmented Virtuality )

Mixed Reality ( = AR + AV )

Wearable Computer

Tangible Bits

Ubiquitous Computing

Coined by Mark Weiser

Many computers in the real world

Computers don't have to be portable / mobile

The third wave

https://gyazo.com/25e19294a5d5882df7bbab0b50ab68dd

"Calm Technology"

Mark Weiser

https://gyazo.com/bb0efafe130a31f6a66ad01e3b5e319c

The computer for the 21st century

https://gyazo.com/371440a5011fa05f176559d89440d3cd

Mark Weiser proposed "Ubiquitous computing"

"Calm" technology

By Mark Weiser

Calm technology or Calm design is a type of information technology where the interaction between the technology and its user is designed to occur in the user's periphery rather than constantly at the center of attention. I

ParcTab

PDA

https://s3-ap-northeast-1.amazonaws.com/masui.org/9/0/9050bc20d3b437a7297658a1d6fb3aed.gif

Forget-me-Not

Record all the events around the terminal

Active Badge

http://www.cl.cam.ac.uk/research/dtg/attarchive/ab.html http://gyazo.com/c025505f930e5e4c37eb656ce4c2014d.png

Pioneering work on ubiquitous computing

by Andy Hopper, etc.

"Bat" system for getting location

Use ultrasonic signal

Cambridge university and AT&T research

Andy Hopper

https://gyazo.com/12fe17cf959a31b6740e5343d14b39a3

Various systems

http://gyazo.com/109d773c66c6e1b997befb7afd6775ad.png

System taxonomy

Augmented reality

Paper-based

House-based

Desk-based

Wearable computers

People-based

Distribution-based

Image/movie distribution

Photos

Digital signage

Time-based systems

Map

Weather

Traffic

Examples of Augmented Reality systems

Baseball broadcast / Kabuki commentary

Add information to what people are watching

Car navigation systems

Add information to the view

Barcodes

Provide information to products

Augmented Reality

c.f. Virtual Reality

Narrow sense

Adding CG to real-world image

Became poppular by 「Denno-Coil」animation

Libraries Became polular by 「AR Toolkit」

ARKit

Wider sense

Add extra info to any object in the real world

Not restricted to CG

Augmented reality (in an narrow sense)

http://gyazo.com/773ae2462931560d4e3b4e93f0c672c5.png

Add CG to real images

Image overlay

Position sensing

Compensation to errors

Image overlay

HMD(Head Mount Display) - based

See-through HMD

Head-mount monitor

Using a handheld monitor

See-through HMD

http://gyazo.com/ec080c958162c33b69cde0ae023a722b.png

Head-mount monitor

http://gyazo.com/8d372110dd7fc20f090d12941c2db7cc.png

Handheld monitor

http://gyazo.com/154d11ef7331f9ab587bc0123b1e2edd.png

Wiring in an airplane (Boeing)

http://gyazo.com/b8b941749e6bf7fe72355e2da1b42405.png

Training for physical operation

http://gyazo.com/4780ba7625b1fe66f4bfc865df0d1015.png

http://gyazo.com/09ad4d27593bc6d7b784138fc583fa94.png

Karma

Dislaying help information for a printer

Overlay a CG image to the real image

NaviCam

https://www.youtube.com/watch?v=S6XKPEexRbU

NaviCam with 2D barcode

Use 2D barcode to get the info of real-world object

http://gyazo.com/c1e08d234e0b3777cb515b429c2b67b8.png

http://gyazo.com/b8bb2b373fb8679ff0f0adbe4cd1032c.png

Video: AR based on barcode

https://www.youtube.com/watch?v=cNFwEMkLK4A

Den-no Coil (NHK animation drama)

People learned about AR here

http://gyazo.com/cdbde78694cbedaa95b2a21469c128d0.png

Video: Den-no Coil

https://www.youtube.com/watch?v=L2QgAGGk9ow

Developed by Hirokazu Kato, Mark Billinghurst氏 @ Washington Univ.

1999〜

http://hondana.org/%E5%A2%97%E4%BA%95/4048673610 http://gyazo.com/8565dd98143823ef91b3032ed02b38fa.png

http://gyazo.com/788ed6a2f3313d09ab84acb2894c5b3e.png

Video: Den-no figure ARis

https://www.youtube.com/watch?v=yCCx7zANsGE

Parallel Tracking and Mapping for Small AR Workspaces

by George Klein

Sources available

Video: PTAM

https://www.youtube.com/watch?v=Y9HMn6bd-v8

Video: Hatsune Miku

https://www.youtube.com/watch?v=PkmjN6eyyHg

Pokemon Go

https://www.youtube.com/watch?v=7ZrmPTPgY3I

ARKit

https://www.youtube.com/watch?v=khavGQ7Dy3c

ARKit

https://www.youtube.com/watch?v=GBds_Y9buxY

ARKit3

https://www.youtube.com/watch?v=vkS-VqAss4s

Soccer On Your Tabletop (HoloLens)

https://www.youtube.com/watch?v=eRGAB4QBS6U

Oculus Quest (VR)

https://www.youtube.com/watch?v=Di7dIhUFsbw

Paper-based systems

Integrate paper / wall with computer data

Data conversion between them

Paper interfaces

XAX

Paper WWW

PaperLink

PaperIcon

Ultra Magic Key

IconSticker

CrossPad

XAX

Convert computer data ⇔ paper data

「DataGlyph」(Inconspicuous barcode)

Cover sheet with barcode

http://gyazo.com/d0243f8f84a08319331ecb2595bcf7bd.png

DataGlyph

http://gyazo.com/ca3a55e9f9c318992e646aa509398540.png

PaperLink

VideoPen

Small camera and fluorescence pen

Use paper document as hypertext

Specify reading region by drawing a line

Actions can be defined

http://gyazo.com/cfc6d873934c2c6cee86c5535801479e.png

PaperLink product「LinkStick」

http://gyazo.com/260b8917fc87cd493574ebf6a7a08206.png

Audio Notebook

Link the text on the notebook to the voice

Record the movement of pen and voice

http://gyazo.com/aed599c86abf855bcf0e944ca5fc7142.png

Pick and Drop

Copy/paste data between different machines

"Pick up" data from a pen commputer, and paste it on another macine

c.f. Drag and Drop

Paper Icon

Copy data from paper to computer by "Pick and Drop"

http://gyazo.com/c4b3996190e594d3873edf118bf15e76.png

https://www.youtube.com/watch?v=rFw9aMubL-Y

Real-world copy/paste using NFC on smartphone

Use NFC reader on Android

RFID recognithion + twist operation

Video: Real-world copy/paste

https://www.youtube.com/watch?v=1di7_2J8X0Y

IconSticker (Siio)

Print barcord icon in the real world

Use the real-world icons just like icons on desktop

http://gyazo.com/4a53f0ab6cc40a4954ca83a1a4892d2b.png

PICO

Toy from Sega

Write something on the tablet

Connected to TV

http://gyazo.com/49ae2f2daedd4b10a944316ccace33db.png

CrossPad

Cross/IBM product (1997)

Drawings on paper are saved on the machine

http://gyazo.com/3da99a0dd3617337881b8e692678c44e.png

Decrio

Integrating PalmPilot and CrossPad (1999)

http://gyazo.com/be188671940e6f954f343dd31c6b4f43.png

Bamboo Spark

2015

https://gyazo.com/bc077ee50c07400bbade56df314f9494

Bamboo Folio

https://gyazo.com/c3f7b25f56098ed2e321f7cf7bceb3de

House-based systems

TRON House

PAPI

AwareHome

Ocha house

Ubiquitous home (NICT)

TRON House

http://monotsukuri.net/mirai98/tron/tron.htm http://gyazo.com/7cadad163bb89f8a8125f5420df4c244.png

Experimental house at Roppongi (1989 - 1993)

Created by Ken Sakamura

Home automation experiments

Prof. Ken Sakamura

https://gyazo.com/7467a092ee5c0f35c896636b83d86b33

TRON House

Automatically serve salt / pepper / seasoning

https://gyazo.com/c1e6505e7f5412831e32c4000d2ec910

TRON House

Underground container

https://www.youtube.com/watch?v=7jPKEyM44GU

https://www.youtube.com/watch?v=Th_vyQVTlcE

RFID-based display in the folklore museum

Explanation of the object is displayed when put on th table

RFID tags in the objects

Difficult to find tags in the objects

https://gyazo.com/5679b58c78c8cbf6b530bca86b9732e1

AwareHome

Project at Georgia Tech

Various experiments in the real house

http://gyazo.com/ded4c3f883c28a5783813b304e8a315b.png

AwareHome

https://www.youtube.com/watch?v=70HrRnGCNWc

AwareHome

http://gyazo.com/29c4274bb0bd26327bb1d997ff4f0519.png

AwareHome

http://gyazo.com/25d67ff021bf8b9256bb5b80737daed7.png

AwareHome

http://gyazo.com/448cddccc02d09445f51cf870a9d85b0.png

http://gyazo.com/67e06c7bcd39af9e536d0098235c2d78.png

Video: PAPI

https://www.youtube.com/watch?v=vyhO6y2nUOU&NR=1

http://gyazo.com/b3e7bc9ece60d5524372ddb2f318e2b0.png

Kitchen @ Ubiquitous Home

http://gyazo.com/6a3f6d4ab7f0372546be9571bf69d4c4.png

Floor @ Ubiquitous Home

http://gyazo.com/0589d3d94fe0514fa5558a3fc1c4ff73.png

Bath @ Ubiquitous Home

http://gyazo.com/88c9d8970127c3c40aebe763e4c029e2.png

http://www.kyoto-su.ac.jp/liaison/kenkyu/images/message56_04_full.jpg

A project at Kyoto Sangyo University

http://gyazo.com/e2460f2a2401a70858deac084942fe0f.png

Video: Large wall display

https://www.youtube.com/watch?v=Sc8dkeFjlWM

Project at Ochanomizu University

Various experiments in a small house

http://gyazo.com/c3eadc92f8e5945ff0076c9d3faa188d.png

http://gyazo.com/0cf5821611616846092fdeda2a4ae56f.png

Kitchen of the future

http://gyazo.com/31144a22f923a8e74b518190fa2f3346.png

Home hacking people

https://www.facebook.com/groups/ouch.hack/ https://gyazo.com/a241d1df775d252dcbdff6a0796ca512

MakeupMirror

http://gyazo.com/918099c634a13ee7a3fe0e70a9542d65.png

SyncTrash

http://gyazo.com/8deb4f91d4d97e7eb8662465f1fa2ba5.png

Desk--based systems

Project computer display on desk

Integrate computer data and paper data

Examples

DigitalDesk

InteractiveDESK

InfoBinder

DigitalDesk

Projector and camera on ceiling

Recognize papers and fingers on the desk

http://gyazo.com/1fcaced1228e3d10eb0402e2a3e99d30.png

InteractiveDESK

机上に投射した計算機画面をペンで操作

キーボードを机上に置くと前面のモニタが使えるようになる

InfoBinder (Siio)

Use a special device for handling real-world data

http://gyazo.com/4405fdc3481de5bea2091f2a3c3cad50.png

MetaDesk

Use "Phicon"s (= Physical Icon) for input

Projection from under the table

http://gyazo.com/f341c5536bd26aad812719092003d870.png

http://gyazo.com/6e28f9a4e0dcb328727be90997ae2aff.png

PingPongPlus

Project images on ping-pong table

Get the location using four microphones at the corners

http://gyazo.com/eb4b2106eb602f4e043c769fdbb725a1.png

https://www.youtube.com/watch?v=AZO8sfmpKIQ

Simulation of light and lens

Combination of projector and lens

http://gyazo.com/0c28614690d3b2d676e059bab0bc14e1.png

Piano

https://www.youtube.com/watch?v=4mM04UscQxc

inFORM

https://www.youtube.com/watch?v=lvtfD_rJ2hE

https://www.youtube.com/watch?v=lCARHatJQJA

Wearable Computer

"Wear" a computer

Computers are never turned off (like a watch)

Using wearing computers

PDA approach

Carrying a small computer

AR approach

Empowering people

Never forgetting people

Hidden information become visible

Use location information

Wearing computers fashion show

People at MIT

Thad, Steve, Sandy, Brad, Lenny, Rehmi

People at Univ. Toronto

http://gyazo.com/0036204ca26c10a9ad5f45035def0b0e.png

Google Glass

http://gyazo.com/72bc27b1ab8a51cc308edb241a8c2d58.png

Video: Google Glass

http://www.youtube.com/watch?v=9c6W4CCU9M4

Prof. Tsukamoto at Kobe university

http://gyazo.com/6913ff3ae749f74631c0f96726b828ad.png

http://gyazo.com/6f292ee4983d3ee055fd071b25fe69de.png

Other wearable computers

People data in the real world

Social network

https://gyazo.com/0d63cc3dc2c47ab840897f492766ba8f

Distribution services

FedEx, Yamato, etc.

Checking packet location

Image/movie distribution

ustream, YouTube

Webinar

Webcam

Photo sharing

Sharing photos

Integration with maps

Digital signages

Displays in the public space

https://gyazo.com/340e944519b6f489886166baed87f2f1

Time-based systems

Party scheduling support

Schedule support

Map information services

Google Maps, Yahoo Maps

Google Street View

Remote sensing

Location informationi system

Google maps

Checking my location

Using GPS

Using WiFi data

Geocoding

Maybe usable for authentication

Weather info

Weather forecast

Weather info

Traffic information

Road traffic

Train schedule

Summary

http://gyazo.com/109d773c66c6e1b997befb7afd6775ad.png