[Module 6] Deep Learning: Deep Neural Networks

Part 1. Introduction to Deep Neural Networks

Deep Learning : Deep learning refers to artificial neural networks that are composed of many layers.

심층신경망(DNN)을 이용해서 고수준의 인공지능 task를 수행할 수 있도록 한다.

 

Perceptron and Neural Networks

Perceptron : one kind of neural network, similar with structure of a neuron

 

Decision Boundary in Perceptron

→ 입력으로 주어지는 좌표 공간(input feature space)은 decision boundary 에 의해서 양분되고, 그 중 한 쪽은 최종 output이 1이고 나머지 한 쪽은 최종 output이 0인 결과가 나온다.

 

하지만, it is impossible to solve a XOR problem using a single layer perceptron. Single layer perceptron can only solve linear problem, XOR problem is non-linear.

 

Multi-layer Perceptron

But if we use two-layer perceptron, we can solve XOR problem → this model is called multi-layer perceptron.

활성화 함수를 통한 mapping 기능 추가

 

각 layer별로 많은 뉴런들을 구성해서 복잡한 패턴을 가진 task도 잘 학습시킬 수 있도록 한다.

 

tensorflow playground

Forward Propagation

𝑎 : activation function (*아래첨자: layer, 위첨자: layer 내의 몇 번째 node 인지?)

 

forward propagation

 

Softmax Layer

Because of sigmoid outputs, Prediction ∈ (0,1) & Target ∈ (0,1) → Upper limits exist on loss and gradient magnitude with MSE Loss

In addition, a better output would be a sum-to-one probability vector over multiple possible classes. (확률값의 총합이 1이 되는 output vector를 얻는 것이 더 바람직할 것!) → Softmax Layer for multi-class classification

logit - softmax layer 처리를 해서 class 별 최종 output이 확률값으로 출력된다.

 

Logistic regression

Softmax classifier whose logit for a negative class is set as a constant value of 0.

  -  Logistic regression is used for a binary classification.

  -  The softmax classifier can also be used for two classes by using the matrix 𝑊 with two columns, i.e., using the twice the number of parameters of a logistic regression.

 

Binary cross-entropy (BCE) loss for logistic regression is defined as,

Also called as sigmoid cross entropy