10. Real-world programming

Standard programming

Programs in computers

Data in computers

Computers on the net

Standard programming

Can't handle real-world data

Home lock status

Temperature

Smell

Feeling

Weight

People in restaurants

Real-world programming

Programs in the real world

Handle objects in the real world

c.f. Real-world interface

Use computers without keyboards and monitors

Walls, desks, etc.

Ubiquitous computing

Wearable computing

Tangible Bits

Augmented Reality

MetaDesk

Example ofTangible Bits

Use Phicons (= physical icons) for input

Projected display on the desk

http://gyazo.com/f341c5536bd26aad812719092003d870.png

http://gyazo.com/6e28f9a4e0dcb328727be90997ae2aff.png

Augmented Reality

http://gyazo.com/788ed6a2f3313d09ab84acb2894c5b3e.png

ARKit

https://www.youtube.com/watch?v=khavGQ7Dy3c

Computing environment in the near future

Computers everywhere

⇒ Computers work everywhere

⇒ Programs developed everywhere

Real-world programming becomes important

Taxonomy

Deal with real-world objects from standard programs

Many UbiComp systems

Use text-based programming

Deal with real-world objects in the real-world

Programming by example

Visual programing

Existing real-world programming

Alarm clock

Coffee machine with timer

Rube Goleberg machine

Analog alarm clock

Mapping between time and hand is obvious

Programming == execution

Programming: moving hand by human

Execution: moving hand automatically

c.f. "rectangle" and "drawing rectangle" is very different

Coffee maker with timer

Cookingn and programming are different

Digital timer is difficult to set

Not for everyone

Time bomb

Fuse is easy

http://gyazo.com/75e5c90e25d20e582c45d39aea837ffd.png

Digital time bomb is difficult

Rube Goldberg machine

Not difficult to understand

http://gyazo.com/210eaff521cdc11af3bbb37592f054c5.png

c.f. State transition machine

Similar to Rube Goldberg machine

http://gyazo.com/c0d7354659a4eaa51af15edfa5c542b7.png

Goals of real-world programming

Ring the bell at 7:00

Shut down TV when talking on the phone

Change the time zone when travelling

Show timetable at station

Open a document file when the document is put on table

Goals of real-world programming

Get travel info when mountains become white

Remember the letter when near a post office

Remember buying a present when birthday is close

Print an urgent email message

Turn on TV when touching a wall

Goals of real-world programming

Copy data from a book

Receive message when it starts to rain

Automatic macro definition after repetitive operations

Use a "copy stick" to duplicate a document

Contents displayed if put something on the desk

Web service with real-world information

FeDex, Yamato, etc.

Map

Weather forcast

Train information

Integration of real world and the Web

http://gyazo.com/82174232cc357d0afb5b8e6b21afbc92.png

http://www.water.go.jp/yoshino/ikeda/sameura/same_top.html http://gyazo.com/b94bcb2f2efcbcc06250d2704bb7dcd0.png

Advantages of real-world programming

Concrete

No symbol required

Target objects visible

Related programming paradigms

Programming by example

Visual programming

World-wide programming

Requirements for real-wrld programming

Network

Sensors / input devices

Programming element

Integration of the above

Sensors and microprocessors

mbed

Galileo

Edison

Phidgets

USB-based sensors / actuators

http://gyazo.com/f380ac7de41e16cb8635243956ae886c.png

Arduino

USB/Serial + AVR

http://gyazo.com/ba0f1f2dd89edcd0b5d9d2630fec4e04.png

Arduino programming

Processingと同じIDEを利用

http://gyazo.com/4229d5702f06eeeba00c9fc7dba14b7f.png

LilyPad

Wearable Arduino

http://gyazo.com/d69ad7cad9a36bcd1e9cd4e315b7dd1a.png

LilyPad

http://gyazo.com/54cd28dda035ebee0eff4efc33ee8d51.png

mbed

Web-programmable one-board computer

http://gyazo.com/d21d725a11af5e9cd6cdba2de538b7e8.png

Raspberry Pi

ARM-based one-board computer

Linux

http://gyazo.com/8eec26e91d9b97c806de1e32c6cabbcf.png

Intel Galileo

Linux + Arduino

http://gyazo.com/022769af99c411731730a31e3bec67d0.png

Intel Edison

http://gyazo.com/990178c9f83a96d1d2d779f40af7dbd0.png

XBee

ZigBee module

http://gyazo.com/a71ddd6ded18a47281748008ec9b5006.png

WiPort

WiFi + CPU + server

http://gyazo.com/a4ed7e2872f8ebfea6d01d6be65c2bab.png

Sun SPOT

Device for sensor network

Java

Various sensors

http://gyazo.com/4ed9e5514f39c526a5f83669bec4c9de.png

iBeacon

Bluetooth Low Energy利用

Sends ID with low power

http://gyazo.com/0848c9a63fc580b748d9db6f89e6e065.png

http://gyazo.com/8921282d50d982e7e42f8d91301a4267.png

Programming models for real-world programming

Exmple-based

Event-driven

Real-world visual programming

Systems

Sensor network

Event-based system with AhroD

ifttt

Phidgets

Stanfordのd.tools

FieldMouse, MouseField

Linda-based world-wide programming

"Ubiquitous system architecture" by Prof. Tokuda

Hayakawa, Tsukamoto, Terada @ Kobe Univ.

Programming with "ECA rules"

http://gyazo.com/fdbe87d65d516e38708f39556dfa6c2c.png

http://gyazo.com/2be95ac8bfa17b5ccf1933d1f1cd6298.png

ECA rules

code:eca.txt

DEFINE Rule-ID

WHEN Event_type (E)

IF Condition (C)

THEN Action (A)

Goals of real-world programming

Ring a bell at 7:00

Turn off TV when talking on the phone

...

ECA rules works for most cases

"On-the-spot programming" by Prof. Terada

"On-the-spot programming" example 1

"I found an interesting book at a book shop, but I had no money. I want to remember the book next time I visit the shop."

"On-the spot programming" example 2

"I want to see the map of my destination, but I only want to see it on my HMD when I'm not walking."

"On-the-spot programming" example 3

"I spent too much money this month, and my wallet should warn me when picked up."

Required technologies

Simple programming model

Dynamic program modification

Context definition

ifttt

"IF This, Then do That"

Use "recepes"

Recepe examples

Save YouTube data to Evernote if the user wants to see it later

Forward a message to GMail if it seems to rain tomorrow

Phidgets

Developed at Univ. Calgary

Linking sensors and GUI widgets

Sensors sold from phidgets.com

http://gyazo.com/076035dc841ef41a9099e8e71e133477.png

System at Stanford

Scott Klemmer, Björn Hartmann

Visual programming + Programming by example

d.tools

Linking computers and real-world objects

PBE + visual programming

http://gyazo.com/ef285a1ee1411fd4c98d4aada8e37ef1.png

Video: d.tools

http://masui.org/195975bab17731fe3728b4e1eeaa1845.mov

FieldMouse

ID recognizer + Motion sensor

Mouse-like usage on a wall or on a desk

Real-world GUI with FieldMouse

http://gyazo.com/2593d8fafcf5b4074439079cca1f9d66.png

MouseFiel

Simple real-world I/F device

RFID + motion sensor

http://gyazo.com/dcb16fd6194a15b459d2502ca627d2d1.png

MouseFieldの使用方法

Put&Move interaction

http://gyazo.com/e98258a8a47de176282e2ed148e7db18.png

MouseField in a study

http://gyazo.com/f475336f971c564980a6368947e15180.png

MouseField

http://gyazo.com/c42e22616d9c306acab92c742416a886.png

PlayStand++

http://gyazo.com/8abd64e12f203964e582cdc675b3fe45.png

PlayStand++の操作

http://gyazo.com/024b85c0aed15a9308436eaf2fd448fc.png

Video: MouseField

https://www.youtube.com/watch?v=w-nEPbptLO8

IT Whitebox (2011/6/19)

http://gyazo.com/d3e79a8b457bc1544f0f5dc399e7c10b.png

Video: Real-world copy/paste

https://www.youtube.com/watch?v=1di7_2J8X0Y

http://gyazo.com/18b298f2b2a2851978504fcd46503938.png

GoldFish server

like TinyURL, Bit.ly

Ex1: Showing a manual

http://gyazo.com/a2dcd1bfa983b4c8b81e406085e59c47.png

http://gyazo.com/de0fe7498b3a1fe856e07002f9a7ae1b.png

Ex3: Rotation interface

http://gyazo.com/b06aee214b542952125f0d8a0ecb7abf.png

http://gyazo.com/f38d032e1c317e1bac30d52581b8317b.png

http://gyazo.com/ab301f469b9266c3f2d2ed07db3f7c22.png

http://gyazo.com/f00aef03e85ef6b1a838c527825a21fc.png

Door opener

http://gyazo.com/129c69de3b13912a532dbdec458ed5c3.png

Real-world copy/paste

http://gyazo.com/96ed656fe1f1007c299ec3817f4fe8d1.png

World-wide programming

Watch all the information in the world

Seashore status, stock prices, ...

Control anything in the real world

Lights, net-based appliances, ...

All the people in the world use it

Hobbists, artists, ...

Example 1

Alarm clock

Coffee maker time setting

Bath time setting

Example 2

Turn off TV when nobody is watching

Turn of alarm if there's nobody

Order beer if glass is empty

Register URL when pressed hard

Turn off lights when in bed

Turn off air conditioner at night

Clean the toilet seat if dirty

Example 3

Check parents living far away

Check water supply

Check burglers

Check winds and water condition

Example 4

Set alarm if something wrong detected on the net

Change BGM based on weather

World-wide Rube Goldberg machine

....

Phidgets体重計 {Phidgets-based scale}

http://masui.org/6c3e200cc1b7ea1d332ac9a4680fe0c9.png

Phidgets programming

code:phidgets.java

import Phidgets.*;

public class IFKex1 extends _IPhidgetInterfaceKitEventsAdapter

{

public void OnSensorChange(_IPhidgetInterfaceKitEvents_OnSensorChangeEvent ke) {

System.out.println("SensorChange: " + ke.get_SensorValue());

}

public void OnInputChange(_IPhidgetInterfaceKitEvents_OnInputChangeEvent ke) {

System.out.println("InputChange: " + ke.get_Index() + " " + ke.get_NewState());

}

public void OnDetach(_IPhidgetInterfaceKitEvents_OnDetachEvent ke) {

System.out.println("FINISHED!");

}

public static void main(String[] args) {

new IFKex1();

}

public IFKex1()

{

PhidgetInterfaceKit phid = new PhidgetInterfaceKit();

phid.add_IPhidgetInterfaceKitEventsListener(this);

if (phid.Open(false) == false)

{

System.out.println("Could not find a PhidgetInterfaceKit");

return;

}

System.out.println(phid.GetDeviceType());

System.out.println("Serial Number " + phid.GetSerialNumber());

System.out.println("Device Version " + phid.GetDeviceVersion());

phid.SetSensorChangeTrigger(7, 1);

phid.start();

System.out.println("Looping...\n");

for(int i = 0; i < 1000; ++i)

{

phid.SetOutputState(0,true);

}

// Turn the output on, and rudely leave it turned on!

phid.Close();

System.out.println("Closed and exitting...");

}

}

Problems of Phidgets programming

Require development env

Libraries for every language

Not available for minor languages

Can't handle remote phidgets

Phidgets Server

Control Phidgets via TCP/IP

Accessible rom any language

by Prof. Tsukada at Hakodate Mirai Univ.

MobiServer

Phidgets, X10, KURO-RS, XBeeを制御

by Prof. Tsukada at Hakodate Mirai Univ.

http://gyazo.com/094eb1ca6d685cc52c624ccaf664e00f.png

Bookmark by weight

code:bookmark.rb

require "socket"

require "delicious"

PORT = 4321

HOST = "phidget.server.host"

USER = "masui"

server = TCPSocket.open(HOST,PORT)

while true

s = server.gets

break if s.nil?

a = s.split(/,/)

if weight > 4.0 then

register(USER)

end

end

end

Presentation program

code:presentation.rb

require "socket"

require "dl/import"

module Win32

extend DL::Importable

typealias "DWORD", "unsigned long"

typealias "BYTE", "unsigned char"

typealias "ULONG", "unsigned long"

KEYEVENTF_KEYUP = 0x02

VK_SHIFT = 0x10

VK_A = 0x41

VK_SPACE = 0x20

VK_BACKSPACE = 0x08

dlload "user32.dll"

extern "void keybd_event(BYTE, BYTE, DWORD, ULONG)"

def send_keybd_event(bVk, dwFlags = 0)

keybd_event(bVk, 0, dwFlags,0)

end

module_function :send_keybd_event

end

serverhost = 'localhost'

port = 4321

socket = TCPSocket.new(serverhost, port)

prevval0 = 0

prevval2 = 0

while line = socket.gets

(io,device,ad,portno,val) = line.chomp.split(/,/)

if io == 'In' && device == 'InterfaceKit' &&

ad = 'Analog' && portno.to_i == 0 then

if val.to_i > 100 && prevval0 < 100 then

Win32.send_keybd_event(Win32::VK_SPACE)

Win32.send_keybd_event(Win32::VK_SPACE,

Win32::KEYEVENTF_KEYUP)

end

prevval0 = val.to_i

end

...

Problems of Phidgets server

Address and port should be known

Hard to use many sensors

Difficult to use multiple servers

Linda

http://masui.org/5c79a78e3de1dcec20999cedfc5ddfad.png

Advantages of using Linda

IP address not required

Various machines share one shared space

Parallel processing

Various sensors can be used

Tell location

Wind sensor

X10

IR remote

Fingerprint center

Accelerometer (tilting sensor)

Rinda

Ruby-based Linda implementation

by Seki-san

Simple Linda on Ruby

× Other languages cannot be used

Programming Rinda

code:rinda.rb

require 'rinda/rinda'

require 'delicious'

TS_URI = 'druby://localhost:12345'

USER = 'masui'

ts = DRbObject.new(nil,TS_URI) # Connect to tuple space

while true

register(USER) if val > 4.0

end

Linda programming on the Web server

Data sharing / syncing on the Web server

Communication between devices and Web information}

Web-based communication via Linda

Real-world programming in Linda

Available systems

PC + Phidgets

A little old?

PC + Arduino/Firmata

Standard

Android + Arduino

各種センサをArduinoで読んでAndroidで制御

http://shokai.github.com/ArduinoFirmata-Android/images/servo.gif

Raspberry Pi + Arduino

Galileo

Edison

Conclusion

Real-world programming is promising

Not widely used at all

Programming environments

Sensors and hardware

Let's enjoy real-world programming!