12. User interface programming
User interface portion of software
UI-related portion is large
50-80% of the software is related to UI
Why UI programs are hard to create
Realtime
Various I/O technologies required
Application part and UI part intermingled
Parallel program
Exception handling
Why UI programs are hard to create (Cont'd)
Flexibility / robustness
Features for supporting users
Evaluation and redesign loop
Various programming techniques required
Realtime
Efficient implementation required
Fast response
Fast drawing
Various I/O technologies
Graphical drawing
Speech recognition
Computer vision
Handling sensors
Application part and UI part intermingled
UI part mixed in the code
Difficult to reuse UI part
code:sample.c
printf("Input your name: "); # UI part
scanf("%s", name); # UI part
s = calc(name); # Application part
printf("%s\n",s); # UI part
Separation of App and UI
UI can be reused
UI can be selected
GUI / CLI
For pros / beginners
Custamizable
Test module / demo module can be used
Why separation is difficult
Picking up only UI part from the code is difficult
AP part and UI part are tightly related
UI part requires AP info
AP part requires UI part
e.g. Erasing inactive portion in the menu
Menu should know AP status
Parallel processing
User and system work in parallel
User input should be accepted all the time
Various dialog structure
User answers to questions from the system
Command input
Combination of the above
Parallel processing (Cont'd)
Multiple input devices
Multiple windows
Show animation during user actions
Timeout
Exception handling
Timeout
Termination
Asynchronous events
Errors
Error handling codes
Usually complicated
Important part (calc func) hidden in many error handling codes
code:error.c
if(condition1) ....
if(condition2) ....
result = calc()
if(exception) ...
if(結果NG) ...
Flexibility / robustness
Should allow human errors
Should consider all mistakes
Clear design
Plain GUI design
Understandable state transition
Thorough tests important
User support features
Help feature
Undo
Guidance
User manual
User adaptation
Prediction
Repetitive evaluation / design
Trial and error important
Should be improved repetitively
Takes time for evaluation
Script languages
Rapid prototyping
Techniques for UI programming
Object-oriented programming
Module programming
Widgets reuse
Script languages
JavaScript, Ruby, Python, ...
Useful for rapid modification
Techniques for UI programming (Cont'd)
Visual programming
State-transition programming
Interface builder
Parallel programming
Handling simultaneous events
Event-based
Thread-based
Techniques for UI programming (Cont'd)
Programming environment
Coding, debugging, testing, documentation
Litarate programming
Special techniques for UI development
Constraint solvers
Programming by example systems
Adaptive systems
Solution1: UIMS
UIMS = User Interface Management System
Similar to DBMS
Separation of AP and UI
All complicated UI part is in UIMS
https://gyazo.com/dde7db959c2dac1cda7e3fe677ae39c1
Seeheim model
Old UIMS modeln
Separating UI and AP
https://gyazo.com/b39e0ac296c425e1d03c7d7b332655ec
Not applicable to modern complicated UI
AP info cannot be sent to UI easily
Problems of UIMS
Impossible to provide all UI patterns
Constraints to dialog
System become huge
Complete separation impossible
UI portion often depends on AP portion
Not popular these days
Solution2: UI description language
Use a special UI language
UIMSs has special languages
c.f. SQL for DBMS
Examples
HyperCard
Squeak: CSP-like language
ERL: Sassafras UIMS
Book on languages for UI
http://gyazo.com/f10ec195e6a4ac8d2ada4f99a5564dc7.png
Problems of UI languages
New language should be remembered
AP language and UI langugage not consistent
c.f. C and SQL
Huge specs
UI langs should have standard language features
Solution3: Toolkits
UI libraries
Event-driven
Main routine exists in toollkit
AP is implemented as functions
Very popular for GUI applications
Various toolkits
Cocoa (Mac)
MFC (Windows)
Flex, Air (Flash)
Swing (Java)
Qt
Tcl/Tk
Object-oriented toolkits
Widgets represented as objects
Windows, sliders, etc.
Each object has data and methods
Example: NeXTstep application
code:next.c
main(){
...
Problems of UI toolkits
UI features restricted
AP codes restricted
AP should be represented as callback functions
onKeyDown(), etc.
Problems of UI toolkits
Huge libraries
Difficult to master
Difficult to extend
Systems and languages restricted
Better than other solutions
Solution4: state programming
Describe program execution state
Simplify UI state transitions
Ex: Roma-Kana converter
https://i.gyazo.com/caaf304022fdd558037e55e20bb83b4a.png
Problems of using state transition
Number of states can get large
Difficult to draw in 1 page
Can't solve all problems
Extended state machine
Hierarchical state machine
https://gyazo.com/ab100d3b4b6c35325ae39c5b53b3933e
StateChart
https://gyazo.com/3ccc6469567662626df4ef12327edc9a
Parallel execution supported
Multiple states can be represented as one state
Programming a state machine
Use special tools
Use special programming techniques
http://gyazo.com/e9a9a3673399c33937c1ebd8e3b68f8f.png
Example of state transition programming
Using a special state machine language
code:kana.lex
ka { printf("か"); }
ki { printf("き"); }
kya { printf("きゃ"); }
Converted code
code:kana.c
c = getc();
if(c == 'k'){
c = getc();
switch(c){
case 'a': printf("か"); break;
case 'i': printf("き"); break;
case 'y':
c = getc();
if(c == 'a') ...
Solution5: Parallel programming
Parallel programming is important for UI
Using multiple devices
Mouse, KB, etc.
Simultaneous input/output
Rotate a steering wheel while using a brake pedal
Module eparation
AP and UI implemented separately
Multiple users
How toolkits handle parallel execution
All events serialized
Processed one by one
Event model (toolkit with callbacks)
Thread model (use multiple processes)
Problems of parallel programming
Special language feature required
Parallel languages sometimes don't fit AP/UI
Debugging difficult
Parallel programming is not popular
Solution6: Interface builder
Introduced on NeXT computer
Still used on Mac / iPhone / other developing environment
Interface builder
Layout toolkit widgets on screen
Automatic generation of AP code
Testable without compilation
Integrated with toolkits
Interface builder on NeXT
http://gyazo.com/26628fafb9d8cddd83ada076705b68fe.png
Xcode interface builder
http://gyazo.com/fc72e526450de45e308ef0e211a8525b.png
Problems of interface builder
Depends on systems and languages
Interaction methods restricted
Good for GUI applications only
Systems are complicated
Best solution for GUI apps at this moment
Solution7: constraint programming
Define constraints between objects
System solve constraints automatically
Applications
Constraints between graphical objects
Figure A is between B and C
When one of them is moved, others are also moved
Spreadsheets
Automatic re-calculation
Applicable to non-spreadsheet apps
Sketchpad by Ivan Sutherland
https://www.youtube.com/watch?v=USyoT_Ha_bA
Developed in 1963!
Problems of constraint programming
Applicable for small category of problems
Good for graphical layout
Sometimes slow
Not for fast interactive APs
Complicated constraints are unsolvable
Solution8: programming by example
Give examples to system
System will generalize the case
Programs are automatically generated
PBE (Programming by Example)
Programming environment for kids
Graphical Rewrite Rule
http://gyazo.com/66a451fefd3be1832a7418b017065fd9.png
Problems of PBE
Good for only simple programming
Many examples required
System cannot infer complecated intentions
What's the best solution
No perfect solution
Different systems should be used together
Examples
Object-oriented toolkits
e.g. Cocoa
Parallel programming
State transition programming
Light-weight script languages
e.g. HTML + JavaScript
Domain-specific languages/tools
For small systems
Look for useful smal systems
Use compact stand-alone tools
Processing
Flash
For large systems
Consider using large systems
Use large toollkits
Use state transition description systems
Use parallel languages
coroutines
For window systems
Use toolkits with interface builders
Xcode (Mac, iPhone)
Visual Studio (Windows)
Android Studio (Android)
Use common toolkit
Use high-level graphic libraries
For prototyping
Use Web browsers
JavaScript + HTML
With useful libraries
jQuery, React, ...
Use small programming systems
Flash
Processing
Use script language + toolkit
MacRuby / RubyMotion (Ruby + Cocoa)
CL X (X11 + CommonLisp)
Tcl/Tk (Tcl + Tk)
Qt
Example 1: Flex
Flex = state transition compiler
Converts state transition into a table
Efficient C code generated
code:roma.flex
%%
ka { printf("か"); }
ki { printf("き"); }
kya { printf("きゃ"); }
→ 状態遷移関数<code>zztrans(入力文字, 遷移機械番号)</code>と遷移表を生成
Using Flex
code:romakana.flex
%%
a { printf("あ"); }
ba { printf("ば"); }
be { printf("べ"); }
bi { printf("び"); }
bo { printf("ぼ"); }
bu { printf("ぶ"); }
bya { printf("びゃ"); }
bye { printf("びぇ"); }
byi { printf("びぃ"); }
...
zye { printf("じぇ"); }
zyo { printf("じょ");}
zyu { printf("じゅ");}
C output
code:flex.c
static short zznstate[] = {
25, 92, 2, 31, 4, 90, 13, 7, 15, 87, 11, 5, 6, 8, 81, 10,
1, 3, 18, 75, 29, 9, 22, 23, 82, 69, 99,175, 83, 65,100,174,
84, 59,101,176,177, 85, 52,102, 88, 0, 86, 63, 89, 17, 28, 12,
...
125, 0,113,151,126, 0, 0,116,152,127, 0, 0,117,153,128, 16,
134,139,119, 0,135,140,120, 0,136,141,121, 0, 0,137,142,122,
0, 0,138,143,123 } ;
zztrans(c,n) int c; int n;
{
int *state;
int index;
state = &(zzstmsn.zzstate); else{
...
}
Example2: Linda
Linda = parallel programming primitive
Communication / data sharing on a shared space
Usable from various languages
Modules can be implemented separately
https://gyazo.com/db5a5056dcc018768aa8c07eb5bea782
Linda model
Exchange 'tuples' in a shared space
4 primitives
out Write a tuple in a tuple space
in Read a tuple from the tuple space and delete it
rd Read a tuple from the tuple space, but keep it
https://gyazo.com/7a6db09a858a7c68c2d0a3688476e507
8-Queen
https://gyazo.com/2255edcc7805b037208d333976deac3d
printqueens() called at the bottom of the recursion
Difficult to write a function which prints one result every time it is called
Difficult to print the result at each mouse click
Solution by Linda
https://gyazo.com/fd294eee60be910eaa2bfe1d354c6013
Terminal-based 8-Queen program
https://gyazo.com/11b8a7d34650480499f3dc4149e83603
Window-based 8-Queen program
https://gyazo.com/74da2119decfc775337d22eaf9b8a67c
Node-Linda
Linda implementation on Web server
Running on Masui Lab.
https://www.slideshare.net/shokai/prosym-node-linda https://gyazo.com/716da53b338a9c47e40bf79176fb78a2