8. Visual programming
CLI and GUI
CLI == Command Line Interface
Use texts everywhere
GUI = Graphical User Interface
Concrete / intuitive / easy to understand
Direct manipulation
Everything is "visual"
Visual programming
Use "visual" objects for programming
Visual == non-textual
Easy program construction / understanding
History of visual programming
Research topic over 30 years
Still not very popular
VLHCC2017
https://sites.google.com/site/vlhcc2017/ https://sites.google.com/site/vlhcc2017/_/rsrc/1472558702221/config/customLogo.gif
Definition of visual programming
Narrow sense: Do everything with visual objects
Wider sense: Includes visual objects in the programming process
Q: How about "Visual Basic"?
Requires program texts
VP in a wider sense
VP is effective for handling visual representations
What is visually represented
Execution results?
Execution process?
Programming environment?
Program itself?
What is visually represented
Execution results?
Execution process?
Programming environment?
Program itself?
Visual representations of textual programs
Traditional methods
Flowcharts
NS charts
PAD
Newer methoes
UML diagram
Flowchart
"Spaghetti" programming
http://gyazo.com/ba6a0d78000d76428b935d3a1d63c4af.png
NS chart
'Structured programming'
http://gyazo.com/82bd29da9867ec20105a3d7de374441d.png
Structured programming
No GOTO statement
Use while and if
PAD
Structured programming
http://gyazo.com/127b8cf6de5ffa9915efb63ba2a95520.png
http://gyazo.com/7e50d657a9eade1a7af627a4315fd9c0.png
HCP chart
http://gyazo.com/b220896bd7c83977b8a3afa3bd2fdc37.png
UML diagram
Object-oriented design
http://gyazo.com/f8b6742d4413cb75a3a0fc8639dd39f7.png
State transition machine
http://gyazo.com/c0d7354659a4eaa51af15edfa5c542b7.png
StateChart
http://gyazo.com/65ff0aa39a149e6b44605730b0b5e2e3.png
Petri net
http://gyazo.com/ae0ec7c3b9aae443e9c5ca813f095297.png
What is visually represented
Execution results?
Execution process?
Programming environment?
Program itself?
Visualization of program execution
Algorithm animation
Debugging
Step execution, etc.
Example: algirithm animation
http://gyazo.com/a03bd77248265bbf22945b02898b914d.png
What is visually represented
Execution results?
Execution process?
Programming environment?
Program itself?
Visual programming environments
IDE (Integrated development environment)
Visual C++
Eclipse
Xcode
IntelliJ
Android Studio
Advantages of visual programmming
Concrete expression
Conform to GUI
Conform to PBE
Suitable for representint structures and relations
Easy to understand the whole structure
Suitable for showing parallelism
Suitable for declarative definitions
Suitable for understanding data flow
Suitable for showing multiple viewpoints
Suitable for end-user programming}
Concrete expression
Easily represent number / mass / color / shape
"5 pieces" and "5 meters" are treated as same in textual representation
■■■■■ is easier to understand than "5 rectangles"
Thumbnail is better than filename
Operations can be visually represented
Move, zoom, etc
Conform to GUI
Easier to use figures for writing figure-based program
"red rectangle" vs ■
Conform to PBE (Programmng by Exmple)
Program generated by directly by handling concrete objects
Programming elements == operation targets
Suitable for representing complex structures and relations
Tables and graphs can be clearly represented
e.g. state transition machine
State transition table is difficult to understand
Visual representations of arrays, structures, links are easy to understand
Easy to understand the whole structure
Readable codes
Indentation
Prettyprintings
2D reperesentations
Easily browsable
Prettyprinting
http://gyazo.com/d7467ed332b6d227da1446646fce9e0e.png
Prettyprinting
Literate programming on CWEB
https://cdn-images-1.medium.com/max/2000/1*kU1JmZlhWgJNRhAcV4ideg.jpeg
Visualization of parallel processing
No explicit execution order
Text-based programs run from top to bottom
Syncronization representation
e.g. Petri nets
Visual declarative representation
Layout declarations on the display
Rules and constraings can be visualized
Dataflow representation
Data flow rather than control flow
Multiple data flow can be displayed
Difficult using CUI
Simple pipe
a | b | c
Multiple viewpoints
Axises can have different meanings
e.g. Calendar
X axis = day of week
Y axis = week
End-user programming
Good for GUI users
GUI-based PBE
Taxonomy of visual languages (Burnett)
Parallel language
Constraint language
Data flow language
Spreadsheet language
Functional language
Imperative language
Logic language
Multi-paradigm language
Object oriented language
Example-based language
Rule-based language
Taxonomy based on display
Diagram language
Icon language
Storyboard Language
More ways possible
Examples of visual languages
Spreadsheets
Computing as Editing
Cinderella
Interface builder
MAX/PD
Hi-Visual
Triggers
KidSim / Cocoa
Viscuit
AgentSheet
Visulan
SimTunes
MindStorms
Automator
Yahoo! Pipes
Quartz Composer
Grasshopper
Spreadsheets
Most successful visual language
Direct manipulation
Constraint programming
"free-format spreadsheet" on Emacs
Use a text editor for various calculation
http://nicosia.is.s.u-tokyo.ac.jp/pub/staff/hagiya/pro95/ipsj-pro.jpeg
Example
code:computing.txt
3 ←rate 10000 ←capital
↓capital ↓+
year 1995 :: 10000 * rate / 100 = 300 ←interest
♂ ∫+interest
year 1996 :: 10300 * rate / 100 = 309 ←interest
♀ ∫+interest
year 1997 :: 10609 * rate / 100 = 318 ←interest
∫+interest
year 1998 :: 10927 * rate / 100 = 327 ←interest
∫+interest
year 1999 :: 11254 * rate / 100 = 337 ←interest
∫+interest
year 2000 :: 11591 * rate / 100 = 347 ←interest
∫+interest
year 2001 :: 11938 * rate / 100 = 358 ←interest
∫+interest
year 2002 :: 12296 * rate / 100 = 368 ←interest
∫+interest
year 2003 :: 12664 * rate / 100 = 379 ←interest
∫+interest
year 2004 :: 13043 * rate / 100 = 391 ←interest
∫+interest
year 2005 :: 13434 * rate / 100 = 403 ←interest
-------------------------------------------------------
3837
Cinderella
https://www.youtube.com/watch?v=9bEFCPwMWyE
http://pitecan.com/Cinderella/Kempe/ https://gyazo.com/2e7e705c07fd081b840c2417dee733e6.png
Interface builder
Lay out GUI objects
Automatic generation of textual programs
Test without compilation
Available on modern IDEs
Interface builder on NeXT computer
http://gyazo.com/26628fafb9d8cddd83ada076705b68fe.png
Interface builder on Mac
http://gyazo.com/fc72e526450de45e308ef0e211a8525b.png
Android Studio
http://gyazo.com/ed1b662f8914a53e601ed73015fd471e.png
Max
Most popular diagram-based dataflow language
Signals, values flow on lines
Processing elements are represented as boxes
Developed at IRCAM (in Paris)
Multiple signals flow in parallel
Max
http://gyazo.com/4183cf453704af9be7c536e9c9233a66.png
Max
http://gyazo.com/e169f00f71e0bc1cce768b9f6a9b2469.png
'Public Domain' version of Max
http://gyazo.com/8e86aaeadc101fa06ef22e3904c9e257.png
http://gyazo.com/95c660387beee7e0d52e95ef6512d944.png
http://gyazo.com/9c744c355475466bceb11df5f4273806.png
Reason
Music studio simulator
http://gyazo.com/990c5ee73e964878be155059d9543696.png
Reason
http://gyazo.com/9e2d500195d4e31fc4c49eebeb83b5a8.png
VVVV
Programming system similar to PD
http://gyazo.com/5eb0012f42944ed591b37608377c2a7d.png
http://vimeo.com/23316783
Triggers
PBE using bitmap pattern matching
e.g. Click a button when "OK" is displayed
Run an operation when a bitmap pattern is detected
e.g. Changing colors of all the negative valures
http://gyazo.com/cd25caa5d14b850f9a1d9da2d6beb0c9.png
Video (Triggers)
http://gyazo.com/e3811d24bde71596f7aedef782f60d90.png
KidSIM
Animation generation based on PBE
Pairs of before-after figures
Graphical Rewrite Rule (GRR)
Screen changes based on rules
Commercial version of KidSIM
http://gyazo.com/ac046dc993ee300f647599f3ba9e988f.png
Programming environment for kids
General Rewriting Rule (GRR)
http://gyazo.com/66a451fefd3be1832a7418b017065fd9.png
http://www.youtube.com/watch?v=0N8cpLHZ41I
https://www.youtube.com/watch?v=kxeaeCajuFc
https://www.youtube.com/watch?v=L0Gg76f60ao
Using GRR
Filling the gap between operations and textual representation
http://gyazo.com/42c5d4b68b126fe6e6eb496167208781.png
Visulan
Bitmap rewriting (c.f. Triggers, KidSim)
http://gyazo.com/fe06d6e48f31568080383529d51565e1.png
http://gyazo.com/22836333d8861f8ea1a1a41b67ce5fb1.png
Visulan
http://gyazo.com/420a8925b90d75c8f224484c53c0c96d.png
http://gyazo.com/8867677d545cf15dd7d57015f02d2f91.png
SimTunes
Music games based on firing rules
Programmig 'bugs' on the display
http://gyazo.com/94f543505e8da08895db29ac25d2f595.png
http://gyazo.com/4d7ba0cc27d429d91a6e7f53a532abfb.png
https://www.youtube.com/watch?v=GchIgaSmFI0
TENORI-ON
http://gyazo.com/1f4b2b7d6989571a5faec6704356800a.png
https://www.youtube.com/watch?v=pU_rG0w0bsA
MindStorms
LEGO roboto with Visual programming
Easy programming of parallel behavior
Handling sensors and actuators simultaneously
http://gyazo.com/3b55f3632dc08f29e573bf137be13b91.png
Sensor-based program of Mindstorms
http://gyazo.com/a31f6c41ce3ef3131858044ce47ed7d1.png
Automator
Visual programming for Mac
http://gyazo.com/0a7f3224bcc395fc919f84a3ef4b36e2.png
Web service mashup
Finished in 2015
http://gyazo.com/fc4471a1378102c027ed7de43fac4e16.png
Quartz Composer
Mac graphic programming
http://gyazo.com/37f11b2c7428f29781af5969a40b2ee0.png
http://gyazo.com/0317269567ae67b98b8c0cc28df9b8da.png
Issues of visual programming
Generality
Systems for larger programs
Automatic layout
Portability
Superiority to textual programs