This is the first in a series of Machine Learning posts meant to act as a gentle introduction to Machine Learning techniques and approaches for those new to the subject. The material is strongly sourced from Hands-On Machine Learning with Scikit-Learn & TensorFlow by Aurélien Géron and from the Coursera Machine Learning class by Andrew Ng. Both are excellent resources and are highly recommended.
Machine Learning is often defined as “the field of study that gives computers the ability to learn without being explicitly programmed” (Arthur Samuel, 1959).
More practically, it is a program that employs a learning algorithm or neural net architecture that once trained on an initial data set, can make predictions on new data.
Common learning algorithms:¹
Linear and polynomial regression
Support vector machines
While the above learning algorithms can be extremely effective, more complex problems -, like image classification and natural language processing (NLP) – often require a deep neural net approach.
Common Neural Net (NN) architectures:¹
Feed forward NN
Convolutional NN (CNN)
Recurrent NN (RNN)
Long short-term memory (LSTM)
We will go into further detail on the above learning algorithms and neural nets in later blog posts.
Some basic terminology:
Features – These are attributes of the data. For example, a common dataset used to introduce Machine Learning techniques is the Pima Indians Diabetes dataset, which is used to predict the onset of diabetes given additional health indicators. For this dataset, the features are pregnancies, glucose, blood pressure, skin thickness, insulin, BMI, etc.
Labels – These are the desired model predictions. In supervised training, this value is provided to the model during training so that it can learn to associate specific features with a label and increase prediction accuracy. In the Pima Indians Diabetes example, this would be a 1 (indicating diabetes onset is likely) or a 0 (indicating low likelihood of diabetes).
Supervised Learning – This is a learning task in which the training set used to build the model includes labels. Regression and classification are both supervised tasks.
Unsupervised Learning -This is a learning task in which training data is not labeled. Clustering, visualization, dimensionality reduction and association rule learning are all unsupervised tasks.
Some supervised learning algorithms:¹
Support vector machines (SVMs)
Decision trees and random forests
Unsupervised learning algorithms:¹
• Hierarchical cluster analysis (HCA)
• Expectation maximization
Visualization and Dimensionality Reduction
• Principal component analysis (PCA)
• Kernel PCA
• Locally-linear embedding (LLE)
• t-distributed Stochastic Neighbor Embedding (t-SNE)
Association Rule Learning
Dimensionality Reduction: This is the act of simplifying data without losing important information. An example of this is feature extraction, where correlated features are merged into a single feature that conveys the importance of both. For example, if you are predicting housing prices, you may be able to combine square footage with number of bedrooms to create a single feature representing living space
Batch Learning: This is a system that is incapable of learning incrementally and must be trained using all available data at once1. To learn new data, it must be retrained from scratch.
Online Learning: This is a system that is trained incrementally by feeding it data instances sequentially. This system can learn new data as it arrives.
Underfitting: This is what happens when you creating a model that generalizes too broadly. It does not perform well on the training or test set.
Overfitting: This is what occurs when you creating a model that performs well on the training set, but has become too specialized and no longer performs well on new data.
m: The total number of instances in the dataset
X: A matrix containing all of the feature values of every instance of the dataset
x(i): A vector containing all of the feature values of a single instance of the dataset, the ith instance.
y: A vector containing the labels of the dataset. This is the value the model should predict
- Géron, Aurélien (2017). Hands-On Machine Learning with Scikit-Learn & TensorFlow. Sebastopol, CA: O’Reilly.