Introduction to Artificial Intelligence and Data Analytics 笔记。
课件引用于香港理工大学comp1004课程
Global Datasphere is a measure of all new data that is captured, created, and replicated in any given year across the globe.
Data: Any piece of information stored and/or processed by a computer or mobile
device.
Data Analytics refers to the technologies and processes that turn raw data into
insight for making decisions and facilitates drawing conclusion from data
What has happened?
It is estimated that 80% of generated analytics results are descriptive in nature.
Descriptive analytics are often carried out via ad hoc reporting or dashboards
Examples
Diagnostic analytics aim to determine the cause of a phenomenon that occurred in the past using questions that focus on the reason behind the event.
Sample questions
Generate future predictions based upon past events.
Sample questions
What should I do if “x” happens?
Prescriptive analytics provide specific (prescriptive) recommendations to the user.
Various outcomes are calculated, and the best course of action for each outcome is suggested.
Examples
4V of Big Data
Structured data
Data conforms to a data model or schema and is often stored in tabular form.
Unstructured data
Data that does not conform to a data model or data schema is known as unstructured data.
Estimated to makes up 80% of the data within any given enterprise.
Semi structured data
Non tabular structure, but conform to some level of structure.
Collecting the raw data such as transactions, logs, and mobile devices.
Permits developers to ingest a wide variety of data.
Requires a secure, scalable, and durable repository to store data before or after the processing tasks.
Data is transformed from its raw state into a consumable format.
Usually by means of sorting, aggregating, joining, and performing more advanced functions and algorithms.
The resulting datasets are then stored for further processing or made available for consumption with business intelligence and data visualization tools.
Data is made available to stakeholders through self service business intelligence and data visualization tools to allow fast and easy exploration of datasets.
Users might also consume the resulting data in the form of statistical predictions (in the case of predictive analytics) or recommended actions (in the case of prescriptive analytics)
Designed to store and handle transaction data (live, real time data)
Relational databases (e.g. Mysql store data in tables with fixed rows and columns.
Non relational databases (NoSQL) store data in a variety of data models (e.g. JSON)
More flexible schema (how the data is organized)
Data warehouse is a giant database storing highly structured information that is optimized for analytics
Typically store current and historical data from one or more systems and disparate data sources
May not reflect the most up to date state of the data.
Business analysts and data scientists can connect data warehouses to explore the data, look for insights, and generate reports for business stakeholders.
Examples
Google BigQuery, Amazon
The ETL processes move data from its original source (e.g. database or other sources) to the data warehouse on a regular schedule (e.g., hourly or daily)
Extract : Extract data from homogeneous/heterogeneous
Transform: Clean the data and transform the data into appropriate format
Load: Insert data into the target data warehouse
Scaling out (Horizontal scaling)[A BETTER WAY]
The challenges of Big Data cannot be handled easily by traditional storage technology, e.g. databases
Hadoop
A framework that allows for storing a large amount of data and the distributed processing of
large data sets across clusters of computers
MapReduce
a programming paradigm that enables massive scalability across hundreds or thousands of
servers in a Hadoop cluster.
Apache Spark
An open source unified analytics engine for large scale data processing
A cluster is a tightly coupled collection of servers, or nodes.
A distributed file system can allow us to store large files which spread across the nodes of a cluster
E.g. Hadoop Distributed File System (HDFS).
Split large dataset into smaller data blocks and stored in different nodes.
In Hadoop, each block contains 128 MB of data and replicated three times by default.
Replication Factor: The number of times Hadoop framework replicate each and every data block.
Moving huge amount data to the processing unit is costly.
The processing unit becomes the bottleneck.
Instead of moving data to the processing unit, we are moving the processing unit to the data
MapReduce consists of two distinct tasks Map and Reduce.
Map: process data to create key value pairs in parallel
MapReduce consists of two distinct tasks Map and Reduce.
Map: process data by workers based on where data is stored
Reduce: Aggregate results by the “reduce workers”
Creation and study of the visual representation of data
One of the most important tools for data analytics/science.
Dashboard is a read only snapshot of an analysis that you can share with other users for reporting purposes.
Self driving vehicles or “driverless” cars
Combine sensors and software to control,
navigate, and drive the vehicle.
Drivers are NOT required to take control to safely operate the vehicle.
Classify and detect the objects in the image.
Assign a class to each object and draw a bounding box around it.
AI is concerned with developing machines with the ability that are usually done by us humans with our natural intelligence
Computer Vision: Enabling computers to derive information from images and videos
Natural Language Processing (NLP): Giving computers the ability to understand text and spoken words
Speech Recognition
Machine Learning
Deep Learning
Image classification models take an image as input and return a prediction about which class the image belongs to.
Images are expected to have only one class for each image.
Takes an image as input and output the images with bounding boxes and labels on detected objects.
Face detection: Detect if there is a face in images/videos.
Face classification: Determine the kind of face
E.g. the Age, Gender and emotion of a person from the face
Face verification: One to one
Is it the same face (e.g. unlock your mobile phone)?
Face identification: One to many
E.g. Police search
The branch of artificial intelligence (AI) concerned with giving computers the ability to understand text and spoken words in much the same way human beings can.
Extract subjective qualities (e.g. attitude, emotion) from text.
Predict whether a movie review is positive or negative, based on the words in the movie
review.
Identify specific entities in a text, such as dates, individuals and places
Software application built to simulate a human like conversation.
Involve speech recognition, natural language processing and speech synthesis
Text to Speech (TTS) is the task of generating natural sounding speech given
text input.
May generates speech for multiple speakers and multiple languages.
Convert voice to text
Example: Recognizing a digit
Let’s say that we want to teach a computer to recognize the number 7
Rules for distinguishing 7 from other characters
7s have a mostly horizontal line near the top of the figure
they have a mostly northeast southwest diagonal line
Those two lines meet in the upper right.
Finding a good and complete set of rules is frequently an overwhelmingly
difficult task.
The rules human experts follow are often not explicit
Easy to overlook exceptions and special cases
The technology, laws, and social conventions around human activities are
constantly changing
Constantly monitor, update, and repair this tangled web of interconnecting rules.
Provide many examples of each class of image
The computer looks at these examples and learn about the visual appearance and
features of each type of image
Learning the rules instead of coding the rule
In ML, features are any property or characteristic of the data that the
model can use to make predictions
Spam : junk or unwanted email, such as chain letters, promotions, etc
Ham: non spam emails.
A large visual database designed for use in visual object recognition software research
More than 14 million images have been hand annotated by the project to indicate what objects are pictured, covering 100,000 classes
ImageNet contains more than 20,000 categories
E.g. “balloon” or “strawberry”, each consisting of several hundred images
ML Model
A representation of reality using a set of rules that mimic the existing data as closely as possible
Training
Giving examples to a model so it can learn.
Split the dataset into two parts
Training set: Used to train the model
Test set: Used to test the accuracy of the model on data the model has never seen before during training
Algorithm
A procedure, or a set of steps, used to solve a problem or perform a computation
The goal of machine learning algorithms is to build a model for prediction
The nearest point to this new observation is malignant and located at the coordinates (2.1, 3.6).
If a point is close to another in the scatter plot, then the perimeter and concavity values
are similar.
We may expect that they would have the same diagnosis.
Classifying unlabelled examples by assigning them the class of similar labeled examples
“k”is a parameter that specifies the number of neighbors to consider when making
the classification.
Applications
Recommendation systems that predict whether a person will enjoy a movie or song
Identifying patterns in genetic data to detect specific proteins or diseases
Computer vision applications, including optical character recognition and facial recognition in
both still images and video.
To improve the prediction we can consider several
neighboring points
Among those 3 closest points, we use the majority class as our prediction for the new observation
MNIST handwritten digit database
Ground truth is information that is known to be real or true.
The number of epochs is a hyperparameter that defines the number of times that the learning algorithm will work through the entire training dataset.
In each epoch, each sample in the training dataset has had an opportunity to update the internal model parameters.
• In the first epoch, AI may make large prediction errors
• Feed the training data to AI multiple times to learn from the mistakes and reduce the prediction errors
Due to computational and memory limits, we generally don’t feed the entire training set to the AI model
• Break down the training data into smaller batches which are fed to the model individually
• The batch size is a hyperparameter that defines the number of samples to work through before
Any quantity that the model creates or modifies during the training process is a parameter
• We can twist many other knobs before training a model
• E.g. the number of epochs, batch size, the “k” value in k nearest neighbor, learning rate (more about it later), etc
• Any quantity that you set before the training process is a hyperparameter
The word overfitting refers to a model that models the training data well but it fails to generalize
Use attributes (1,2,….) to predict a categorical variable () yes/no, rain/no rain
To validate which model to use: Cross Validation
In the case of small data sets (say, less than 1,000 observations), a very popular scheme is cross-validation
• The data is split into K folds (e.g., 5). A model is trained on K − 1 training folds and tested on the remaining validation fold.
• This is repeated for all possible validation folds resulting in K performance estimates that can then be averaged
Simple Linear Regression
y = f ( x 1 , x 2 , x 3 , . . . ) y = f(x_1, x_2, x_3, ...) y=f(x1,x2,x3,...)
Use attributes (1,2,….) to predict a numerical variable ().
The output of a regression model is a number, e.g. prices, sizes, or weights.
Unsupervised learning is also a common type of machine learning.
Extract information from a dataset that has no labels, or targets to predict.
• Clustering algorithms
• Group data into clusters based on similarity
• Example algorithm: K-means
• Dimensionality reduction algorithms
• Simplify our data and faithfully describe it with fewer features
• Example algorithm: Principal component analysis (PCA)
• Association rule mining
• Generative algorithms
• generate new data points that resemble the existing data
Unlike supervised learning, no labelled data is given
• Reinforcement learning (RL) aim to build a history of experiences for the AI and learn through trial and error.
• An agent attempts various allowed actions in an environment over multiple cycles, observes the outcome of those actions based on the environment state
• The agent is learnt to perform the desired task by taking actions with good outcomes and avoiding actions with bad outcomes.
Simple linear regression only considers one independent variable (X) and dependent variable (Y).
Y = b + m X Y = b + mX Y=b+mX
A good linear regression model is one where the line is close to the points.
= − ( + ) _ = _ −(+) ei=yi−(mxi+b)
Absolute Error: A metric that tells us how good our model is by adding distances between predicted and actual values of the dependent variable
Square Error: The square error is a metric that tells us how good our model is by adding squares of residuals
A positive slope tells us that we should take a step to the left to get to the lowest SSE
A negative slope tells us that we should take a step to the right to get to the lowest
SSE
To find the local minima
Logistic Function:
f ( x ) = 1 / ( 1 + e − x ) f(x) = 1 / (1+e^{-x}) f(x)=1/(1+e−x)
z f r o m ∞ t o − ∞ z from {\infin}to-{\infin} zfrom∞to−∞
Artificial Neural Network (ANN)
Inspired by the structure of the human brain, with a network of many cells called “neurons”.
A single-layer perception is the basic unit of a neural network
A binary classifier which can decide whether or not an input below class
Activation: the output of the neuron
Activation function: calculates the artificial neuron’s output
Sometimes we won’t be able to fit a linear classifier to this data, we use two lines by combining linear classifiers this way, is the basis for neural networks.
The arrangement of nodes and layers: the architecture of the neural network
Input Layers -> Hidden Layers -> Output Layers
The depth of the neural network: the number of layers (excluding the input layer)
The input layer is not counted as layer
A neural network with depth of 3
• An input layer of size 4
• A hidden layer of size 3 (first hidden layer is formed by linear classifiers )
• A hidden layer of size 2 (classifiers in each successive layer are slightly more complex than those in the previous ones)
• An output layer of size 3
• To build a neural network, we use the outputs of two perceptrons and a bias node (represented by a classifier that always outputs a value of 1) to a third perceptron.
• The boundary of the resulting classifier is a combination of the boundaries of the input classifiers.
• An activation function takes a real number as input and returns a new floating-point number as output
• We can apply a different activation function to every neuron in our network
• In practice, we usually assign the same activation function to all the neurons in each layer.
Sigmoid return 0,1
Tanh return 1,-1
ReLU 0 to positive and negative infinite
Regression - Linear Activation Function (for the amount, it is numeric, regression need no activation function)
Binary Classification—Sigmoid/Logistic Activation Function
Multiclass Classification—Softmax
Multilabel Classification—Sigmoid
The activation function used in hidden layers is typically chosen based on the type of neural network architecture.
Convolutional Neural Network (CNN): ReLU activation function.
Recurrent Neural Network: Tanh and/or Sigmoid activation function.
•turn the raw numbers that come out of a classification network into class probabilities
No activiation function is needed
Remove the final sigmoid function from the neural network
• The role of this function is to turn the input into a number between 0 and 1
• If we remove it, the neural network will be able to return any number.
Local minima
Remove the final sigmoid function from the neural network
• The role of this function is to turn the input into a number between 0 and 1
• If we remove it, the neural network will be able to return any number.
In each epoch, update the weights and bias using gradient descent
• Forward Propagation
: Take the data point one by one and perform a forward pass to calculate the prediction
• Backward Propagation
: Based on the answer and the prediction error, compute how much we should adjust each weights and biases best in order decrease the errors
Shallow learning algorithms are ML algorithms that do not gain in accuracy beyond a certain amount of training data.
Results get better with more data + bigger models + more computation
• CNN is a neural network which utilizes a special type of layer (convolutional layers) to learn from image and image-like data.
• A convolution is a filter that passes over an image, processes it, and extracts the important features (and blur the inessential features). • Excels at handling image/image-like data and computer vision tasks
LeNet-5 was one of the earliest convolutional neural networks and promoted the development of deep learning
ResNet is one of the most powerful CNN winning the ImageNet challenge in 2015
The use of artificial intelligence (AI) to create a fake event
• in photo, video, or audio format
Training on pairs of images and then attempts to generate the corresponding output image from any input image you give it
OpenAI
Build a history of experiences for the AI and learn through trial and error.
Each black circle is some game state and each arrow is a transition
• Take the two games we won and slightly encourage every single action we made in that episode.
• Take the two games we lost and slightly discourage every single action we made in that episode.
Barge-in (problems with ending)
• Allowing the user to interrupt
End-pointing (problems with starting)
• The task for a speech system of deciding whether the user has stopped talking.
• Very hard, since people often pause in the middle of turns
[Example]
Directive: “Turn up the music”, ‘What day in May do you want to travel?’
Constative: ‘I need to travel in May’
Acknowledgement: ‘Thanks’
Principle of closure. Agents performing an action require evidence, sufficient for current purposes, that they have succeeded in performing it (Clark 1996, after Norman 1988)
Some conversations are controlled by one person • A reporter interviewing a chef asks questions, and the chef responds. • This reporter has the conversational initiative
(Walker and Whittaker 1990)
Most human conversations have mixed initiatives
: • I lead, then you lead, then I lead. • Mixed initiative is very hard for NLP systems, which often default to
Another chatbot with a clinical psychology focus patients usually interpret
A sense or “concept” is the meaning component of a word
Suppose you see these sentences:
• Ong choi is delicious sautéed with garlic.
• Ong choi is superb over rice
• Ong choi leaves with salty sauces
And you’ve also seen these:
• …spinach sautéed with garlic over rice
• Chard stems and leaves are delicious
• Collard greens and other salty leafy greens
Conclusion:
• Ong choi is a leafy green like spinach, chard, or collard greens
A vector is an ordered list of numbers, such as
Elements or entries, e.g., the 3rd entry is 3.6
Count of entries, dimension
Vectors of dimension n: n-vector
Numbers are called scalars
Embeddings example:
Vectors are the basis of information retrieval (comparing the similarity between documents such as google)
The dot product between two vectors is a scalar:
Summary: Text Classification
• Sentiment analysis
• Spam detection
• Authorship identification
• Language Identification
• Assigning subject categories, topics, or genre
Any kind of classifier
• Naïve Bayes
• LogisXc regression
• Neural networks
• k-Nearest Neighbors
Bag of words (the count of words in a document)
It is based on Bayes’ rules and jointly considers likelihood (probability of observing the words in labeled with the class) and prior (probability of observing a class)
• Television
• Newspaper
• Motivation 1: diverse needs
• Motivation 2: too many choices
• Television → Video website
• Newspaper → Social media
Usenet Communication System
Ringo: Social Information Filtering
Baseline method for comparison
• For each artist in the target set
• The mean score received by an artist in the source set is used as the predicted score for that artist.
• A social information filtering algorithm is neither personalized nor accurate unless it is a significant improvement over this base case (by simply averaging all scores) algorithm.
• Idea: recommendation based on similarity between users
• Notation: two users profiles U x U_x Ux and U y U_y Uy are N-dimensional vectors (N-vectors):
The lower the mean squared difference, the greater the similarity
This coefficient ranges from -1 indicating a negative correlation, via 0, indicating no correlation, to +1 indicating a positive correlation between two users.
We modifed the Pearson r scheme so that only when there is an instance where both people have rated an artist positively, above 4, or both negatively, below 4, will the correlation coefficient increase.
Use 4 as the average instead of, since there may be some constraints and considerations
Explicit feedback
Feature Matrix Demonstration
Similarity: dot production in binary case; We assume this feature matrix is binary: a non-zero value (e.g. 1)means the app has that feature.
• In this case dot product is the number of features that are active in both vectors simultaneously. As we will see, most metrics for similarity between vectors are based on the dot product.
For example, a user selects ”Education apps" in their profile. Other features can be implicit, based on the apps they have previously installed. For example, the user installed another app published by Science R Us.
To do so, you must first pick a similarity metric (for example, dot product). Then, you must set up the system to score each candidate item according to this similarity metric.
Pros
• The model doesn’t need any data about other users, since the recommendations are specific to this user. This makes it easier to scale to a large number of users. • The model can capture the specific interests of a user, and can recommend niche items that very few other users are interested in.
Cons
• Since the feature representation of the items are hand-engineered to some extent, this technique requires a lot of domain knowledge. Therefore, the model can only be as good as the hand-engineered features.
• The model can only make recommendations based on existing interests of the user
. In other words, the model has limited ability to expand on the users’ existing interests.
Preference can ve explicit or implicit
• Collaborative filtering models can recommend an item to user A based on the interests of a similar user B.
• Furthermore, the embeddings can be learned automatically, without relying on hand-engineering of features.
Pros
• We don’t need domain knowledge because the embeddings are automatically learned.
• The model can help users discover new interests. In isolation, the machine learning system may not know the user is interested in a given item, but the model might still recommend it because similar users are interested in that item.
Cons
• Cold start problem: cannot handle fresh items
• Based on the user similarity or neighborhood
In user-based collaborative filtering, we have an active user for whom the recommendation is aimed.
• The collaborative filtering engine first looks for users who are similar. That is users who share the active users rating patterns.
• Collaborative filtering basis this similarity on things like history, preference, and choices that users make when buying, watching, or enjoying something.
• For instance, if two users are similar or are neighbors in terms of their interested movies, we can recommend a movie to the active user that her neighbor has already seen.
• Based on similarity among items calculated using people’s rating of those items
What is the difference between content-based filtering and item-based collaborative filtering?
• In the item-based collaborative filtering, similar items build neighborhoods on the behavior of users. • However, it is not based on their contents.
• For example, Item 1 and Item 3 are considered neighbors as they were positively rated by both User 1 and User 2. So, Item 1 can be recommended to User 3 as he has already shown interest in Item 3. But all these can be done without knowing the content of items.
Healthcare, news, product, building information, tourism, etc.
Behind each such system there is an intricate wiring diagram, a network, that defines the interactions between the components
Distance (shortest path, geodesic) between a pair of nodes is defined as the number of edges along the shortest path connecting the nodes
• If the two nodes are disconnected, the distance is usually
defined as infinite
• In directed graphs paths need to follow the direction
of the arrows
• Consequence: Distance is not symmetric, e.g., h A , C ≠ h C , A h_{A,C} ≠ h_{C,A} hA,C=hC,A
A->B; A->C, so B AND C got 1
A B C D E F G H
B
C
D
E
F
G
H
[ 0 1 1 0 0 0 0 0 0 0 1 0 1 0 0 0 0 1 0 0 1 1 0 1 0 0 0 0 0 0 0 0 1 0 1 1 0 0 0 0 0 0 0 1 1 1 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 0 1 0 0 1 1 0 0 1 0 0 0 0 1 0 1 0 ] \begin{bmatrix} 0 & 1 & 1 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\ 1 & 0 & 1 & 0 & 0 & 0 & 0 & 1 & 0 & 0 \\ 1 & 1 & 0 & 1 & 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 1 & 0 & 1 & 1 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 1 & 1 & 1 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 1 & 1 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 1 & 0 & 0 & 1 & 1 & 0 \\ 0 & 1 & 0 & 0 & 0 & 0 & 1 & 0 & 1 & 0 \end{bmatrix} ⎣⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎡01100000101000011101000000101100000111100001100000000001010000100000001100000000⎦⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎤
MSN: Degree Distribution
MSN: Log-Log Degree Distribution
Key Properties: Degree distribution: heavily skewed, avg degree = 14.4, Path length:6.6
Most of the path lengths are small
A community is a group of nodes with greater ties internally than to the rest of the network Strictest: Clique
A clique is a subset of vertices of an undirected graph such that every two distinct vertices in the clique are adjacent.
For example, ABC, EDF, HGI
Edge betweenness: number of shortest paths (among all pair od vertices) passing over the edge.
The small-world experiment comprised several experiments conducted by Stanley Milgram and other researchers examining the average path length for social networks of people in the United States. The research was groundbreaking in that it suggested that human society is a small-world-type network characterized by short path-lengths. The experiments are often associated with the phrase “six degrees of separation”, although Milgram did not use this term himself.
The degree distributions of most real-life networks follow a power law, become straight line
Degree distribution: high skewed
In statistics, a power law is a functional relationship between two quantities, where a relative change in one quantity results in a proportional relative change in the other quantity, independent of the initial size of those quantities: one quantity varies as a power of another. for instance, considering the area of a square in terms of the length of its side, if the length is doubled, the area is multiplied by a factor of four.
Many events of interest to scientists in natural and social life tend to have a typical scale, and the scale of individuals varies little around this characteristic scale. For example, the height of human beings, the vast majority of adult Chinese men are around the average height of 1.70m. Of course, this value varies somewhat by region, but in any case, we have never seen a “dwarf” less than 10cm tall or a “giant” taller than 10m in the street. If we take the height as the horizontal coordinate, the number or probability of obtaining this height as the vertical coordinate, we can draw a bell-shaped distribution curve, which decays very quickly on both sides of the curve; similar to this to a mean can characterize the entire group characteristics of the distribution, we call Poisson distribution.
Social Network
• media = content of twitter, tag, videos, photos …
• The networks formed by individuals
• Social Media
• social network + media
Input and output of AIDA are multidisciplinary
NLP: Read and write
Speech Processing: Listen and speak
CV: See
Impacts:
Drone: Learn policy form user demonstrations with RL
Ethics change with technological progress
Industrial Revolution -> Right to Internet access -> Birth control, surrogate pregnancy, embryo selection, artificial womb -> Lab-grown meat
Data privacy is the claim of individuals, groups and institutions to determine for themselves, when, how, and to what extent information about them is communicated to others.
• Personal privacy . Protecting a person against undue interference (such as physical searches) and information that violates his/her moral sense.
• Territorial privacy. Protecting a physical area surrounding a person may not be violated without the person’s acquiescence.
• Safeguards : laws referring to trespassers’ search warrants.
• Informational privacy. Deals with the gathering, compilation, and selective dissemination of information
• Data Collection
e.g., fingerprints, health data, etc.
• Data Storage and Transportation
Online data may be accessible to anyone.
• Data Analytics.
Personal data is used for analysis, e.g., recommender systems.
• National Safety (e.g., Defense construction, infrastructure, and classified systems)
• Social Safety (e.g., unemployment) • Network Safety (e.g., illegal data access)
• Personal Safety (e.g., accidents caused by mechanical failure)
Replacing human labor
Interesting spinL AI controlling human labour
Hiring Problems: against women since the annotated data
Word Embeddings
Word2Vec introduced vector math on word embeddings
• Reveal harmful biases encoded in our language corpora
• Potential solution: de-bias at training time, but at least make users aware
Solutions
Make computers understand visual content (e.g., images and video).
Vision is an amazing feat of natural intelligence
• Visual cortex occupies about 50% of the Macaque brain
• More human brain devoted to vision than anything else
In 1966, Marvin Minsk
• 1966: Minsky assigns computer vision as an undergrad summer project • 1960’s: interpretation of synthetic worlds • 1970’s: some progress in interpreting selected images • 1980’s: artificial neural networks (ANNs) come and go; shift toward
geometry and increased mathematical rigor • 1990’s: face recognition; statistical analysis in vogue • 2000’s: broader recognition; large annotated datasets available; video
processing starts • 2010’s to present: deep learning
Images consist of pixels
• The smallest discrete component of an image on the screen
Image resolution
• The number of pixels in a digital image
Standard images
• Illustrate algorithms and compare the performance
• Lena: for gray‐level images generally 256*256
• Baboon: for color images generally 512*512
Convolutional neural networks (CNNs)
• Promoted the development of deep learning
pre-training a neural network refers to first training a model on one task or dataset. Then using the parameters or model from this training to train another model on a different task or dataset. This gives the model a head-start instead of starting from scratch
• Object detection is the field of computer vision that deals with the localization and classification of objects contained in an image or video.
• Object detection comes down to drawing bounding boxes around detected objects which allow us to locate them in a given scene (or how they move through it).
• Single-stage object detectors
• e.g., YOLO (You Only Look Once): uses a single neural network trained end to end to take in a photograph as input and predicts bounding boxes and class labels directly
• Two-stage object detectors
• First extract ROIs (Region of interest), then classify and regress the ROIs
• e.g., R-CNN, Fast-RCNN, Faster-RCNN, Mask-RCNN
Image segmentation is a sub-domain of computer vision and digital image processing which aims at grouping similar regions or segments of an image under their respective class labels.
• Refers to the classification of pixels in an image into semantic classes
• Pixels belonging to a particular class are classified to that class with no other information or context considered.
• Models classify pixels based on “instances” rather than classes
The combination of semantic segmentation and instance segmentation
• Speech Recognition: enabling a computer to understand spoken language
and convert speech signals into text (speech-to-text transcription)
• Speech Synthesis: enabling a computer to generate spoken language and generate speech signals from the text (text-to-speech transcription)
in voice assistants, the paradigm steps come as speech recognition (speech-to-text translation) → natural language understanding → natural language generation → speech synthesis (text-to-speech translation).
Articulation produces sound waves which the ear conveys to the brain for processing.
• Digitization
• Acoustic analysis of the speech signal
• Linguistic interpretation
• Digitization
Converting analogue signals into a digital representation
• Signal processing
• Separating speech from background noise distinctions (similar phonemes)
• Phonetics
Variability in human speech
• Phonology
Recognizing individual sound
• Lexicology and syntax
Disambiguating homophones
Features of continuous speech
• Syntax and pragmatics
Interpreting prosodic features
• Pragmatics
Filtering of performance errors
(disfluencies)
• Computer Vision (CV) is for “seeing” the world (like our eyes), and Speech Processing (SP) is for “hearing” and “speaking” (like our ears and mouth). Both CV and SP can be based on machine learning and, recently, deep learning.
• CV takes visual signals as input (e.g., images), usually represented as pixels accordingly to the resolution and colors.
• Several essential CV tasks, such as image classification, object detection, and semantic/instance segmentation (what are they?)
• SP involves speech recognition (speech2text) and speech synthesis (text2speech). Its input is speech signals (sound waves)