9. Programming and Information Visualization
Programming and information visualization
Visualizatin of execution result
= Normal visualization
Visualization of program code
Visualization of program execution
Visual programming environment
Visualization of programs
Old techniques
Flowcharts
NS chafts
PAD
More recent methods
UML chart
Flowcharts
"Spaghetti programming"
https://gyazo.com/ba6a0d78000d76428b935d3a1d63c4af.png
NS chart
For structured programmig
https://gyazo.com/82bd29da9867ec20105a3d7de374441d.png
PAD
For structured programming
https://gyazo.com/127b8cf6de5ffa9915efb63ba2a95520.png
Bisection method
https://gyazo.com/7034c8335d2398b07ea0b1f92a86e4fc
https://gyazo.com/7e50d657a9eade1a7af627a4315fd9c0.png
UML charts
Unified Modeling Language
Object-oriented design
https://gyazo.com/f8b6742d4413cb75a3a0fc8639dd39f7.png
State transition machine
https://gyazo.com/c0d7354659a4eaa51af15edfa5c542b7.png
Petri nets
https://gyazo.com/ae0ec7c3b9aae443e9c5ca813f095297.png
StateChart
https://gyazo.com/65ff0aa39a149e6b44605730b0b5e2e3.png
SeeSoft
Various visualization tools
https://gyazo.com/31164d1dc490ccc68827bf6dd30fe2ff
Visualization of program execution
Algorithm animation
Debugger
Step execution, etc.
Example: Visualization of sorting algorithm
http://www.sorting-algorithms.com/ https://gyazo.com/a03bd77248265bbf22945b02898b914d.png
Example: Visualization of sorting algorithm
http://www.cs.ubc.ca/~harrison/Java/sorting-demo.html https://gyazo.com/b09850c1848d0fd448e3ab3f78d4fd5b.png
Example: Visualization of Dijkstra algorithm
Used in Norikae-annai
http://tech.nitoyon.com/ja/blog/2010/01/26/dijkstra-aster-visualize/ https://gyazo.com/c83fc776e0aa60e57b4d3b725210737a.png
Visual programming
Use shapes for programming
"Visual" means "non-textual"
Sometimes programming becomes easier
Text-based programming
https://gyazo.com/003fe30526cc98f8a09348bebc7c6a7b.png
Definition of visual programming
Narrow sense: All programs described ith figures
Wider sense: Programming environment where figures are used
How about 「VisualBasic」 and 「Visual C++」?
Text-based reequired
VP in wider sense
Visual programming useful for generating visual output
Visual programming environment
aka IDE (Integrated develoment environment)
Visual C++
Eclipse
Xcode
IntelliJ
VSCode
Advantages of visual programming
Concreteness
Conform to GUI
Conform to PBE
Easy to understand structures and relations
Good visibility
Good for representing parallel programs
Conform to declarative programming
Easy to show data stream
Good to represent multiple viewpoints
Good for end-user programming
Concreteness
Numbers and values can be concretely represented
"5" can mean 5 pieces, 5m, etc.
Images are easier to understand than image filenames
Concrete operations can be specified
Move, zoom, etc.
Conform to GUI
GUI programs should be represented by GUI-based programs
"#ff0000" doesn't look red
"https://gyazo.com/2476716ac76a8c76ab70c54109382c9b.png" is better
Conform to PBE (Programming by Example)
Programs can be generated using direct maniplation
Directly manipulate programming elements
Representation of complicated relations and structures
Tables and graphs can be visually represented
e.g. state transition diagram
State transition table is difficult to understand
Arrays, struct, links, can be visually represented
Suitable for understanding whole structure
Readable
Indentation
Prettyprintings
2D reperesentations
Easily browsable
Visualization of parallel processing
No explicit execution order
Text-based programs run from top to bottom
Syncronization representation
e.g. Petri nets
Visual declarative representation
Layout declarations on the display
Rules, constraings can be visualized
Data flow representation
Data flow rather than control flow
Multiple data flow can be displayed
Difficult using CUI
Simple pipe
% ls | grep | wc
Execute ls, grep, wc sequentially
Multiple viewpoints
X, Y axises can have different meanings
Example: calendar
X axis = day of week
Y axis = week
End-user programming
Good for GUI users
GUI-based PBE
Visual languages taxonomy (Burnett)
Parallel language
Constraint language
Data flow language
Spreadsheet language
Functional language
Imperative language
Logic language
Multi-paradigm language
Object oriented language
Example-based language
Rule-based language
Taxonomy based on display
Diagram language
Icon language
Storyboard Language
More ways possible
Examples of visual languages
Spreadsheets
Interface builder
MAX/PD
Hi-Visual
Triggers
KidSim / Cocoa
Viscuit
AgentSheet
Visulan
SimTunes
MindStorms
Automator
Yahoo! Pipes
Quartz Composer
Spreadsheets
https://gyazo.com/2f0d4d7ab4b845ad97823c784379fdcf
Most successful visual language
VisiCalc
Direct manipulation
Constraint programming
Interface builder
Lay out GUI objects
Automatic generation of textual programs
Test without compilation
Available as IDEs
Interface builder on NeXT computer
https://gyazo.com/26628fafb9d8cddd83ada076705b68fe.png
Interface builder on Mac (XCode)
https://gyazo.com/fc72e526450de45e308ef0e211a8525b.png
Android Studio
https://gyazo.com/ed1b662f8914a53e601ed73015fd471e.png
Demo: Interface builder
Max/MSP
Diagram-based dataflow language
Signals, values flow on lines
Processing elements are represented as boxes
Developed at IRCAM (Paris)
Multiple signals flow in parallel
Max/MSP
https://gyazo.com/4183cf453704af9be7c536e9c9233a66.png
Max/MSP
https://gyazo.com/e169f00f71e0bc1cce768b9f6a9b2469.png
Video: Max/MSP
http://www.youtube.com/watch?v=zrOo4fOlKfA https://gyazo.com/b81249976ffee1a5212635e79a495f50.png
デモ: Max/MSP/Jitter
https://gyazo.com/5fdaf463c8a51af980532714d5c71074.png
Successor of Max
Public Domain
https://gyazo.com/8e86aaeadc101fa06ef22e3904c9e257.png
https://gyazo.com/95c660387beee7e0d52e95ef6512d944.png
https://gyazo.com/9c744c355475466bceb11df5f4273806.png
Reason
Music studio simulator
https://gyazo.com/990c5ee73e964878be155059d9543696.png
Reason
https://gyazo.com/9e2d500195d4e31fc4c49eebeb83b5a8.png
VVVV
Programming system similar to PD
https://gyazo.com/5eb0012f42944ed591b37608377c2a7d.png
Running VVVV
http://vimeo.com/23316783
Triggers
Bitmap pattern matching
e.g. Click a button when "OK" is displayed
Run an operation when a bitmap pattern is detected
e.g. Changing colors of all the negative valures
Checking all the "-" signs and
https://gyazo.com/cd25caa5d14b850f9a1d9da2d6beb0c9.png\
Video (Triggers)
https://s3-ap-northeast-1.amazonaws.com/masui.org/b/f/bf29228663ace081a0bb209a9b6372f0.mp4
KidSIM
http://gyazo.com/e3811d24bde71596f7aedef782f60d90.png
KidSIM
Animation generation based on PBE
Pairs of before-after figures
Graphical Rewrite Rule (GRR)
Screenchanged based on rules
Commercial version of Stagecast
https://gyazo.com/ac046dc993ee300f647599f3ba9e988f.png
Video: Stagecast
https://www.youtube.com/watch?v=Wq9uC58hF20
http://www.stagecast.com/movies/HowToMovie.html https://gyazo.com/ac046dc993ee300f647599f3ba9e988f.png
Stagecast
https://www.youtube.com/watch?v=51H1OnACvy8
Programming environment for kids
General Rewriting Rule
https://gyazo.com/66a451fefd3be1832a7418b017065fd9.png
http://www.youtube.com/watch?v=0N8cpLHZ41I https://gyazo.com/41993024607c465b43771c5a456a1faa.png
How to play with Viscuit
http://www.youtube.com/watch?v=l6EcDeBnbXY https://gyazo.com/7d74e1f79cd032afa0dafb1254835c49.png
Video: Viscuit
https://www.youtube.com/watch?v=kxeaeCajuFc
http://www.youtube.com/watch?v=0N8cpLHZ41I
http://www.youtube.com/watch?v=l6EcDeBnbXY https://gyazo.com/7d74e1f79cd032afa0dafb1254835c49.png
Video: Viscuit
https://www.youtube.com/watch?v=kxeaeCajuFc
http://www.youtube.com/watch?v=0N8cpLHZ41I
GRR-based programming system4
Filling the gap between operations and textual representation
https://gyazo.com/42c5d4b68b126fe6e6eb496167208781.png
Visulan
Bitmap rewrite
3D
https://gyazo.com/fe06d6e48f31568080383529d51565e1.png
https://gyazo.com/22836333d8861f8ea1a1a41b67ce5fb1.png
Visulan
https://gyazo.com/420a8925b90d75c8f224484c53c0c96d.png
https://gyazo.com/8867677d545cf15dd7d57015f02d2f91.png
GUI operations with bitmap pattern matching + scripting
http://www.youtube.com/watch?v=FxDOlhysFcM
SikuliX
https://www.amazon.co.jp/dp/4296103032 https://gyazo.com/14dff80cf6056febf48da143dac9229c
RPA = Robotic Process Automation
SikuliX
https://www.youtube.com/watch?v=VdCOg1bCmGo
Python, Ruby
SvimTunes
Music games based on firing rules
Programmig 'bugs' on the display
https://gyazo.com/94f543505e8da08895db29ac25d2f595.png
https://gyazo.com/4d7ba0cc27d429d91a6e7f53a532abfb.png
https://www.youtube.com/watch?v=hzvDeG_jc0U
MindStorms
Visual programming for LEGO robot
Visual programming environment
Easy parallel processing
Using multiple sensors
https://gyazo.com/3b55f3632dc08f29e573bf137be13b91.png
Sensor-based program
https://gyazo.com/a31f6c41ce3ef3131858044ce47ed7d1.png
Squeak-based visual programming
https://gyazo.com/f2edafb5096f86e208495aee09df42b0.png
Scratch
https://www.youtube.com/watch?v=VIpmkeqJhmQ
Logo
https://www.youtube.com/watch?v=g6kmVHfMQvY
NetLogo
https://www.youtube.com/watch?v=AJXFiO-ULv0
Visual programming by Google
https://gyazo.com/4b1c33bdfa6c049ae5d320709dbad1b4.png
Maeda block
Visual programming on EnchantMOON
https://gyazo.com/d85b3f063a447b860ae43825490eed55.png
Automator
Visual programming for Mac
https://gyazo.com/0a7f3224bcc395fc919f84a3ef4b36e2.png
Demo: Automator
Yahoo! Pipes
Web service mashup
https://gyazo.com/fc4471a1378102c027ed7de43fac4e16.png
http://shokai.org/blog/archives/4572 https://gyazo.com/1d1bf3458448056617f00b50e1fce569.png
https://gyazo.com/f6314168556e14653508c87085adbb50.png
https://gyazo.com/66afc41203c08a95680374d30ec65a4a.png
Graphical algorithm editor for Rhinoceros
https://gyazo.com/0317269567ae67b98b8c0cc28df9b8da.png
Video: Rhino/Grasshopper
http://www.youtube.com/watch?v=jIwBY6r9idM
https://gyazo.com/c01d07d30d73ebdc6dfad4cf719f84b2.png
Future issues of visual programming
Generaligation
Systems for larger programs
Automatic layout
Portable systems
Superiority to textual programs