DeepLizard學習筆記

DeepLizard：油管位址、對應部落格筆記

1. 建立tensor

建立tensor的四種方法即不同：具體解釋

2. Tensor Operation Types

• Reshaping operations
• Element-wise operations
• Reduction operations
• Access operations

2.1 Reshaping operations

1. 常見reshape的操作 連結

   

   

2. CNN最後一層的flatten操作（）

   1）建構

   

   2）Flatten()

   這裡我們希望保留第一層：指定start_dim參數

   

2.2 Element-wise operations

原文連結

Broadcasting And Element-Wise Operations

逐元素計算要求不同tensor具有相同shape，broadcasting解決了tensor間不同shape的問題。

Broadcasting is the concept whose implementation allows us to add scalars to higher dimensional tensors.

Comparison Operations Are Element-Wise

Element-Wise Operations Using Functions

2.3 Tensor Reduction Operations

原文連結

A reduction operation on a tensor is an operation that reduces the number of elements contained within the tensor.

Reduction operations in general allow us to compute aggregate (total) values across data structures. In our case, our structures are tensors.

Reducing Tensors By Axes

注意圖中右邊的解釋

Argmax Tensor Reduction Operation

Argmax returns the index location of the maximum value inside a tensor.

2.4 Accessing Elements Inside Tensors

原文連結

With NumPy ndarray objects, we have a pretty robust set of operations for indexing and slicing, and PyTorch tensor objects support most of these operations as well.

2.5 總結

3. CNN Examples

There are four general steps that we’ll be following as we move through this project:

• Prepare the data
• Build the model
• Train the model
• Analyze the model’s results

3.1 CNN Image Preparation

原文

To prepare our data, we’ll be following an ETL process：

• Extract data from a data source.
• Transform data into a desirable format.
• Load data into a suitable structure.

Once we have completed the ETL process, we are ready to begin building and training our deep learning model.

Preparing Our Data Using PyTorch

Our ultimate goal when preparing our data is to do the following (ETL):

• Extract – Get the Fashion-MNIST image data from the source.
• Transform – Put our data into tensor form.
• Load – Put our data into an object to make it easily accessible.

For these purposes, PyTorch provides us with two classes:

但是，這裡使用torchvision包，其中對fashion-MNIST dataset的這些操作已經進行了封裝，但是其背後的原理就是上面的Dataset和DataLoader.

PyTorch Torchvision Package

The torchvision package, gives us access to the following resources:

• Datasets (like MNIST and Fashion-MNIST)
• Models (like VGG16)
• Transforms
• Utils

3.2 DataSet和DataLoader

原文

PyTorch Dataset: Working With The Training Set

PyTorch DataLoader: Working With Batches Of Data

3.3 Build the model

3.3.1 了解nn.module

原文

從python類的角度考慮pytorch中的Module類。

PyTorch’s nn.Module Class

下面是對nn.Module很好的了解：

deep neural networks are built using multiple layers. This is what makes the network deep.

Each layer in a neural network has two primary components:

• A transformation (code)
• A collection of weights (data)

Like many things in life, this fact makes layers great candidates to be represented as objects using OOP(object oriented programming)

Within the nn package, there is a class called Module, and it is the base class for all of neural network modules which includes layers.

This means that all of the layers in PyTorch extend the nn.Module class and inherit all of PyTorch’s built-in functionality within the nn.Module class. In OOP this concept is known as inheritance.

Even neural networks extend the nn.Module class. This makes sense because neural networks themselves can be thought of as one big layer (if needed, let that sink in over time).

forward() Method：

Each layer has its own transformation (code) and the tensor passes forward through each layer. The composition of all the individual layer forward passes defines the overall forward pass transformation for the network.

The goal of the overall transformation is to transform or map the input to the correct prediction output class, and during the training process, the layer weights (data) are updated in such a way that cause the mapping to adjust to make the output closer to the correct prediction.

What this all means is that, every PyTorch nn.Module has a forward() method, and so when we are building layers and networks, we must provide an implementation of the forward() method. The forward method is the actual transformation.

PyTorch’s nn.functional Package:

When we implement the forward() method of our nn.Module subclass, we will typically use functions from the nn.functional package. This package provides us with many neural network operations that we can use for building layers. In fact, many of the nn.Module layer classes use nn.functional functions to perform their operations.

The nn.functional package contains methods that subclasses of nn.Module use for implementing their forward() functions. Later, we see an example of this by looking at the PyTorch source code of the nn.Conv2d convolutional layer class.

Building A Neural Network In PyTorch:

Short version:

• Extend the nn.Module base class.
• Define layers as class attributes.
• Implement the forward() method.

More detailed version:

• Create a neural network class that extends the nn.Module base class.
• In the class constructor, define the network’s layers as class attributes using pre-built layers from torch.nn.
• Use the network’s layer attributes as well as operations from the nn.functional API to define the network’s forward pass.

3.3.2 CNN Layers

原文

Two Types Of Parameters：

• Hyperparameters：values are chosen manually and arbitrarily

  例：

  

• Data dependent hyperparameters：whose values are dependent on data

CNN Layers參數意義：

3.3.3 CNN Weights - Learnable Parameters In Neural Networks

原文

Hyperparameter values are chosen arbitrarily.

learnable parameters are the weights inside our network, and they live inside each layer.

In fact, when we say that a network is learning, we specifically mean that the network is learning the appropriate values for the learnable parameters. Appropriate values are values that minimize the loss function.

PyTorch Parameter Class：

To keep track of all the weight tensors inside the network, PyTorch has a special class called Parameter. The Parameter class extends the tensor class, and so the weight tensor inside every layer is an instance of this Parameter class.

3.3.4 Callable Neural Networks

原文

How Linear Layers Work

Callable Layers And Neural Networks

> fc(in_features)
tensor([30.0261, 40.1404, 49.7643], grad_fn=<AddBackward0>)
           

Instead of calling the forward() method directly, we call the object instance. After the object instance is called, the __ call__() method is invoked under the hood, and the __ call__() in turn invokes the forward() method. This applies to all PyTorch neural network modules, namely, networks and layers.

any time we want to invoke our forward() method, we call the object instance.

3.3.5 CNN Forward Pass Implementation

原文

一些非常好的解釋：

Each of these layers is comprised of a collection of weights (data) and a collection operations (code). The weights are encapsulated inside the nn.Conv2d() class instance. The relu() and the max_pool2d() calls are just pure operations. Neither of these have weights, and this is why we call them directly from the nn.functional API.

Sometimes we may see pooling operations referred to as pooling layers. Sometimes we may even hear activation operations called activation layers.

However, what makes a layer distinct from an operation is that layers have weights. Since pooling operations and activation functions do not have weights, we will refer to them as operations and view them as being added to the collection of layer operations.

For example, we’ll say that the second layer in our network is a convolutional layer that contains a collection of weights, and preforms three operations, a convolution operation, the relu activation operation, and the max pooling operation.

Note that the rules and terminology here are not strict. This is just one way to describe a network. There are other ways to express these ideas. The main thing we need to be aware of is which operations are defined using weights and which ones don’t use any weights.

Historically, the operations that are defined using weights are what we call layers. Later, other operations were added to the mix like activation functions and pooling operations, and this caused some confusion in terminology.

Mathematically, the entire network is just a composition of functions, and a composition of functions is a function itself. So a network is just a function. All the terms like layers, activation functions, and weights, are just used to help describe the different parts.

Don’t let these terms confuse the fact that the whole network is simply a composition of functions, and what we are doing now is defining this composition inside our forward() method.

完整的forward方法：

3.3.6 Forward Propagation Explained

原文

1. 關閉計算圖自動計算梯度

   在單純觀察前向傳播時，不需要pytorch自動計算梯度。

   Before we being, we are going to turn off PyTorch’s gradient calculation feature. This will stop PyTorch from automatically building a computation graph as our tensor flows through the network.

> torch.set_grad_enabled(False) 
<torch.autograd.grad_mode.set_grad_enabled at 0x17c4867dcc0>
           

1. CNN輸入與結果預測

   需要輸入CNN的資料格式是：(batch_size, in_channels, height, width)

   如果是單張圖像，沒有batch_size，通過unsqueeze填補：

   

2. batch Processing

   

3. CNN Output Size Formula

   

3.3.7 CNN Training

原文

CNN training loop:

3.3.8 CNN Confusion Matrix

原文

This confusion matrix will allow us to see which categories our network is confusing with one another.

這個視訊解決了當時困惑很久的不使用grad的問題，視訊中對于梯度的解釋和confusion matrix的建構更加詳細具體，建議觀看整個視訊。

Locally Disabling PyTorch Gradient Tracking

1. 使用裝飾器@torch.no_grad()

   @torch.no_grad()：omit gradient tracking.

   

   This is because gradient tracking uses memory, and during inference (getting predictions while not training) there is no need to keep track of the computational graph. The decoration is one way of locally turning off the gradient tracking feature while executing specific functions.
2. with torch.no_grad():

with torch.no_grad():
    prediction_loader = torch.utils.data.DataLoader(train_set, batch_size=10000)
    train_preds = get_all_preds(network, prediction_loader)
           

Building The Confusion Matrix

1. 自己寫
2. sklearn.metrics中的confusion_matrix()

3.3.9 Concatenate Vs Stack

原文

• Concatenating joins a sequence of tensors along an existing axis
• Stacking joins a sequence of tensors along a new axis.

1. Add Or Insert An Axis Into A Tensor：

2. Stack Vs Cat In PyTorch：

3. 執行個體分析：

1）Joining Images Into A Single Batch：

2）Joining Batches Into A Single Batch：

3）Joining Images With An Existing Batch

3.4 Analyze the model’s results

3.4.1 TensorBoard: TensorFlow’s Visualization Toolkit

原文

後面30-36暫且跳過。

4. 其他的補充

4.1 PyTorch Sequential Models

原文

The Sequential class make it much easier to rapidly build networks and allows us to skip over the step where we implement the forward() method. When we use the sequential way of building a PyTorch network, we construct the forward() method implicitly by defining our network’s architecture sequentially.

This means that we can

• compose layers to make networks
• since networks are also nn.Module instances, we can also compose networks with one another.
• since the Sequential class is also a nn.Module itself, we can even compose Sequential modules with one another.

1.Building PyTorch Sequential Networks

2.Class Definition Vs Sequential

We said that these networks are the same. But what do we mean? In this case, we mean that the networks have the same architecture. From a programming standpoint, the two networks are different types under the hood.