006 Machine Learning - Recommendation
How it works?
Input Data : created from 'movielens'. part of them are used.
20,328 users, 18,162 movies
1 to 5 (preferred) evaluation, each user evaluates 147.5 movies in average.
Quite sparse input matrix with 99.18% zero (= 1 - 147.5 / 18,162 )
----
Algorithm: Acquire Concept Matrix with Singluar Value Decomposition (SVD)
Reference video by Jure Leskovec, Info. Lab., Stanford University
Lecture 46 — Dimensionality Reduction - Introduction |
:
Lecture 50 — SVD Example and Conclusion | Stanford University
Lecture 50 : Screen Shot
Array Replisentaion of the model
https://gyazo.com/c92b4d13d602ab315e4246dee1be9df5
Query: Conversion of a user's preference to concept space (Transposed description)
https://gyazo.com/7017a3730b42beb2e5a1c67f3e9e91b6
Lecture 50 - Video
https://youtu.be/K38wVcdNuFc?t=9
Spark SVD is a single line command processed in parallel by distributed CPUs with RDD.
val svd: SingularValueDecomposition[RowMatrix, Matrix] = mat.computeSVD(k, computeU = true)
Try watching performance on SX Aurora TSUBASA desktop in the booth (Demo only)
Choose 'Recommendation - Normal Spark' and open the link in new window
Choose 'Recommendation - Spark with Frovedis and open the link in new window
https://gyazo.com/9cb31563f13e33bfc6c5aa5bd6cc3e13
https://gyazo.com/03a2d50da4c4365193a791285aa381ff
'Recommendation' consists of threeparts: Setup, Computation, Show Result.
Source code and output are shown in each pane
Contol buttons at right upper corner of each pane
Execution time is shown at left lower corner of each pane
https://gyazo.com/e553f979b46e62159db746fb8a063521
Compare Paformance of the computation
Normal Spark - 120 sec
Spark with Frovedis - 5 sec --> 24 times speedup (= 120/5 )
Spark with Frovedis Code
Adding two lines to start Frovedis daemon
Compute SVD by calling Frovedis daemon
Other part is done by normal spark
https://gyazo.com/0bc56f29a589f4a473d42782953864f2