Inspire…Educate…Transform.

Architecting Machine Learning Solutions

(a.k.a Machine Learning Practice)
Dr. Manoj Chinnakotla
Mentor, INSOFE
Senior Applied Scientist, Microsoft
Adjunct Professor, IIIT Hyderabad

The best place for students to learn Applied Engineering http://www.insofe.edu.in

 Machine Learning – Powering Data Driven
Decision Making and Analysis

The best place for students to learn Applied Engineering 2 http://www.insofe.edu.in

 The Machine Learning (ML) Process

The best place for students to learn Applied Engineering 3 http://www.insofe.edu.in

 The ML Process (Contd..)

The best place for students to learn Applied Engineering 4 http://www.insofe.edu.in

 The ML Process (Contd..)

The best place for students to learn Applied Engineering 5 http://www.insofe.edu.in

 A Typical ML Architecture

The best place for students to learn Applied Engineering 6 http://www.insofe.edu.in

 A Typical ML Architecture (Contd..)

The best place for students to learn Applied Engineering 7 http://www.insofe.edu.in

 Architecting ML Solutions
DEBUGGING LEARNING ALGORITHMS

The best place for students to learn Applied Engineering 8 http://www.insofe.edu.in

 A Motivating Example

• Anti-Spam Filtering
– You carefully chose a small set of 100 words to use as
features (instead of using all 50K+ words in English)
• You tried Logistic Regression, implemented with
gradient descent
– Gets 20% test error (which is unacceptably high)
• What to do next?

The best place for students to learn Applied Engineering 9 http://www.insofe.edu.in

 Fixing the Learning Algorithm

• Common Approach: Try improving the algorithm in different ways

– Try getting more training examples
– Try a smaller set of features
– Try a larger set of features
– Try changing the features – E-Mail body vs. E-Mail headers
– Run gradient descent for more iterations
– Try Newton’s method
– Use a different value for regularization coefficient
– Try a different algorithm (SVM)
• You may try hit-and-trail with all of the above
– Usually, it’s very time-consuming
– Relying more on luck on which one will work
The best place for students to learn Applied Engineering 10 http://www.insofe.edu.in
 A Better Approach: Diagnostic for Bias vs. Variance

The best place for students to learn Applied Engineering 11 http://www.insofe.edu.in

 Diagnostic for Bias vs. Variance (Contd..)

The best place for students to learn Applied Engineering 12 http://www.insofe.edu.in

 Diagnostic for Bias vs. Variance (Contd..)

The best place for students to learn Applied Engineering 13 http://www.insofe.edu.in

 Diagnostic for Bias vs. Variance (Contd..)

The best place for students to learn Applied Engineering 14 http://www.insofe.edu.in

 Optimization Algorithm Diagnostics

The best place for students to learn Applied Engineering 15 http://www.insofe.edu.in

 Optimization Algorithm Diagnostics (Contd..)

The best place for students to learn Applied Engineering 16 http://www.insofe.edu.in

 Optimization Algorithm Diagnostics (Contd..)

The best place for students to learn Applied Engineering 17 http://www.insofe.edu.in

 Optimization Algorithm Diagnostics (Contd..)

The best place for students to learn Applied Engineering 18 http://www.insofe.edu.in

 Optimization Algorithm Diagnostics (Contd..)

The best place for students to learn Applied Engineering 19 http://www.insofe.edu.in

 Diagnostics tell you what to try next

The best place for students to learn Applied Engineering 20 http://www.insofe.edu.in

 Architecting ML Solutions
IMPORTANCE OF ERROR ANALYSIS

The best place for students to learn Applied Engineering 21 http://www.insofe.edu.in

 Error Analysis

The best place for students to learn Applied Engineering 22 http://www.insofe.edu.in

 Ablative Analysis

The best place for students to learn Applied Engineering 23 http://www.insofe.edu.in

 Ablative Analysis (Contd..)

The best place for students to learn Applied Engineering 24 http://www.insofe.edu.in

 Architecting ML Solutions
APPROACHES TO BUILD ML SYSTEMS

The best place for students to learn Applied Engineering 25 http://www.insofe.edu.in

 How to build an ML System

The best place for students to learn Applied Engineering 26 http://www.insofe.edu.in

 Avoid Premature Statistical Optimization

The best place for students to learn Applied Engineering 27 http://www.insofe.edu.in

 Summary
• Building an end-end ML system usually involves interaction between multiple stake
holders business experts, data scientists and IT operations
• A typical data science life cycle consists of steps beyond model building such as data
preparation, transformation, evaluation etc. – sometimes these are very time-
consuming and slow
• Two approaches for designing ML solutions
– Very careful design then implement
– Building quick prototype and then diagnose, fix and refine
• Time spent coming up with diagnostics for learning algorithms is time well-spent
• It’s often up to one’s own ingenuity in coming up with the right diagnostics
• Error analysis and ablative analysis also give insight into the problem

The best place for students to learn Applied Engineering 28 http://www.insofe.edu.in

 References
• Rules of Machine Learning: Best Practices for ML
Engineering. Martin Zinkevich. Invited Talk at Workshop on
“Reliable ML in the Wild”, NIPS, 2016.
• An Architecture for Agile Machine Learning in Real-Time
Applications. Johann Schleier-Smith. KDD, 2015.
• Practical Advice for Analysis of Large, Complex Data Sets.
Patrick Riley. 2016.
http://www.unofficialgoogledatascience.com/2016/10/practical-
advice-for-analysis-of-large.html

The best place for students to learn Applied Engineering 29 http://www.insofe.edu.in

 International School of Engineering

Plot 63/A, 1st Floor, Road # 13, Film Nagar, Jubilee Hills, Hyderabad - 500 033

For Individuals: +91-9502334561/63 or 040-65743991

For Corporates: +91-9618483483
Web: http://www.insofe.edu.in
Facebook: https://www.facebook.com/insofe
Twitter: https://twitter.com/Insofeedu
YouTube: http://www.youtube.com/InsofeVideos
SlideShare: http://www.slideshare.net/INSOFE
LinkedIn: http://www.linkedin.com/company/international-school-of-
engineering

This presentation may contain references to findings of various reports available in the public domain. INSOFE makes no representation as to their accuracy or that the organization subscribes to those findings.

The best place for students to learn Applied Engineering 30 http://www.insofe.edu.in