04 Tensor the Most Fundamental Computing Unit in Py Torch

04 Tensor The Most Fundamental Computing Unit in PyTorch #

In the previous lesson, we learned about the main usage methods and techniques of NumPy. With NumPy, we can handle various types of data well. In deep learning, the organization of data goes a step further, and from the organization of data to the parameters inside the model, they are all represented and processed using a data structure called tensor.

Today, let’s learn about tensors together and explore some common operations on tensors.

What is Tensor #

Tensor is a fundamental concept in deep learning frameworks and one of the most important topics in PyTorch and TensorFlow. It is a data storage and processing structure.

Let’s recall a few types of data representation we currently know:

  1. Scalar, also known as a scalar quantity, is a quantity with only magnitude and no direction, such as 1.8, e, 10, etc.
  2. Vector is a quantity with both magnitude and direction, such as (1, 2, 3, 4), etc.
  3. Matrix is a quantity obtained by combining multiple vectors together, such as [(1, 2, 3), (4, 5, 6)], etc.

To help you better understand scalar, vector, and matrix, I have prepared an illustrative image that you can refer to.-

It is not difficult to see that these types of data representation are actually related. Scalars can be combined to form vectors, and vectors can be combined to form matrices. So, can we consider them as a form of data?

The answer is yes. In PyTorch, we call this unified form of data tensor. From the relationship between scalar, vector, and matrix, you may think of them as tensors of different “dimensions”, which is partially correct.

I say partially correct because in the concept of tensor, we prefer to use rank to represent these “dimensions”, for example, a scalar is a tensor of rank 0; a vector is a tensor of rank 1; a matrix is a tensor of rank 2. There are also tensors of rank greater than 2. Of course, if you say “dimension,” it’s not completely wrong either. Many people also refer to them that way in everyday conversation.

Now that we’ve discussed the meaning of tensors, let’s take a look at the types of tensors and how to create them.

Types, Creation, and Transformation of Tensors #

The characteristics of tensors are generally similar across different deep learning frameworks, and the methods for using them are also quite similar. In this lesson, we will use PyTorch as an example to learn about tensor usage.

Types of Tensors #

In PyTorch, tensors support many different data types. Here are some commonly used formats:

Image

Typically, torch.float32, torch.float64, torch.uint8, and torch.int64 are used more often, but it’s not absolute, and the choice should be based on specific circumstances. Just keep these types in mind, and I will provide further explanations in later lessons.

Creating Tensors #

PyTorch provides various methods to create tensors of any shape, and each method is quite straightforward. Let’s take a look at them.

Direct Creation #

First, let’s look at the direct creation method, which is the simplest way to create a tensor. We can use the torch.tensor function to create a tensor directly.

torch.tensor(data, dtype=None, device=None,requires_grad=False)

Let’s break down the parameters from left to right. The first parameter is data, which is the data we want to pass to the model. PyTorch supports various types, such as lists, tuples, NumPy arrays, and scalars, and converts them to tensors.

Next is dtype, which specifies what type of tensor you want to return. You can refer to the table mentioned above for the specific types.

Then, device specifies the device to which the data should be sent. For now, you don’t need to worry about it and can leave it as the default.

The last parameter is requires_grad, which indicates whether the tensor needs to retain its gradient information during computation. In PyTorch, only tensors with requires_grad set to True will have their gradients calculated and stored in the grad attribute, allowing the optimizer to update the parameters.

Therefore, you need to be flexible in setting requires_grad to true or false. If it is during the training process, set it to true to facilitate gradient calculation and parameter updating. However, during the validation or testing process, our goal is to evaluate the model’s generalization ability, so requires_grad should be set to false to avoid automatic updates based on the loss.

Creating from NumPy #

You have learned about NumPy in previous lessons. In practical applications, we often use NumPy for data processing, and when we want to pass the processed data to a PyTorch deep learning model, we need to convert it to a tensor. PyTorch provides a statement to convert NumPy arrays to tensors:

torch.from_numpy(ndarry)

Sometimes, when developing models, we may need tensors with specific matrix forms, such as matrices filled with zeros or ones. In this case, we can create a NumPy array filled with zeros and then convert it to a tensor. However, this can be cumbersome because it involves importing more packages (NumPy) and writing more code, which increases the possibility of errors. But don’t worry, PyTorch provides a more convenient method internally. Let’s keep reading.

Creating Tensors with Special Forms #

Now let’s take a look at several commonly used functions used internally in PyTorch models.

  • Creating a tensor filled with zeros: As the name suggests, a zero tensor is a matrix where all elements are zeros.
torch.zeros(*size, dtype=None...)

Among the parameters, size and dtype are commonly used. size defines an integer sequence that specifies the shape of the output tensor. You may have noticed the ellipsis (…) in the function parameter list, which means that torch.zeros actually has many parameters. However, we are currently focusing on the concept of a zero tensor, so the shape is relatively more important. Other parameters (such as requires_grad mentioned earlier) are irrelevant for now, and we won’t focus on them at this stage.

  • Creating an identity matrix tensor: An identity matrix has elements equal to 1 on its main diagonal.
torch.eye(size, dtype=None...)
  • Creating a tensor filled with ones: As the name suggests, an all-ones tensor is a matrix where all elements are ones.
torch.ones(size, dtype=None...)
  • Creating a random matrix tensor: PyTorch provides several commonly used methods for creating random matrices:
torch.rand(size)
torch.randn(size)
torch.normal(mean, std, size)
torch.randint(low, high, size）

Each method has different usage scenarios, and you can use them flexibly according to your needs.

  • torch.rand is used to generate a random tensor of floating-point type with specified dimensions. The generated floating-point data is uniformly distributed in the range from 0 to 1.
  • torch.randn is used to generate a random tensor of floating-point type with specified dimensions. The generated floating-point numbers follow a standard normal distribution with a mean of 0 and a variance of 1.
  • torch.normal is used to generate a random tensor of floating-point type with specified dimensions and a specified mean and standard deviation.
  • torch.randint is used to generate random integers within a specified range for the tensor. The numbers generated are uniformly distributed between low (inclusive) and high (exclusive).

Transforming Tensors #

In real projects, we encounter many different data types, such as integers, lists, and NumPy arrays. It is important to be able to convert between different data types and tensors. Let’s take a look at how to convert integers, lists, and NumPy arrays to tensors, and vice versa.

  • Converting between integers and tensors:
a = torch.tensor(1)
b = a.item()

To convert a number (or scalar) to a tensor, we use torch.Tensor, and to convert a tensor back to a number, we use the item() function, which converts the tensor to a Python number.

  • Converting between lists and tensors:
a = [1, 2, 3]
b = torch.tensor(a)
c = b.numpy().tolist()

For a list a, we can simply use torch.Tensor to convert it to a tensor. To convert it back, we need one more step: first convert the tensor to a NumPy structure, and then use the tolist() function to get a list.

  • Converting between NumPy arrays and tensors:

Do you remember how to convert NumPy arrays to tensors? Yes, we can still use torch.Tensor to do that. It’s very convenient.

  • Converting between tensors on CPU and GPU:
CPU->GPU: data.cuda()
GPU->CPU: data.cpu()

Common Operations on Tensors #

Alright, we have just learned about the types of tensors, how to create tensors, and how to convert tensors to and from common data types. However, tensors have some other useful functionalities, such as obtaining shapes, dimension conversions, shape transformations, and dimension manipulation. Let’s explore these functionalities together.

Obtaining Shapes #

In the design of deep learning networks, it is essential to have a thorough understanding of tensors, including their forms and shapes.

To obtain the shape of a tensor, we can use shape or size(). The difference between the two is that shape is an attribute of torch.tensor, while size() is a method that torch.tensor possesses.

>>> a = torch.zeros(2, 3, 5)
>>> a.shape
torch.Size([2, 3, 5])
>>> a.size()
torch.Size([2, 3, 5])

By knowing the shape of a tensor, we can determine the number of elements it contains. To calculate this, we multiply the sizes of all dimensions together. For example, the tensor a above contains 2 * 3 * 5 = 30 elements. To simplify this calculation, PyTorch provides the numel() function, which counts the number of elements directly.

>>> a.numel()
30

Matrix Transpose (Dimension Conversion) #

In PyTorch, there are two functions, permute() and transpose(), that can be used for matrix transposition or exchanging data between different dimensions. We often use them when adjusting the dimensions of convolutional layers, changing the order of channels, or modifying the size of fully connected layers.

The permute function allows transposition of any high-dimensional matrix. However, it can only be called as tensor.permute(). Let’s take a look at the code:

>>> x = torch.rand(2, 3, 5)
>>> x.shape
torch.Size([2, 3, 5])
>>> x = x.permute(2, 1, 0)
>>> x.shape
torch.Size([5, 3, 2])

As you can see, the shape of the original tensor was [2, 3, 5]. In permute, we specify the new positions of the original indices. For example, x.permute(2, 1, 0) means that the second dimension is now in the zeroth position, the first dimension remains in the first position, and the zeroth dimension is now in the second position. This results in a new shape of [5, 3, 2].

On the other hand, the transpose function, unlike permute, can only swap two dimensions at a time. Let’s see an example:

>>> x.shape
torch.Size([2, 3, 4])
>>> x = x.transpose(1, 0)
>>> x.shape
torch.Size([3, 2, 4])

Note that after applying transpose or permute, the data becomes non-contiguous. What does this mean?

Let’s continue with the previous example. When we use torch.rand(2, 3, 4) to create a tensor, the tensor is contiguous in memory. But after applying transpose or permute, for example transpose(1, 0), the memory layout remains the same, but our data “appears” as if the data from dimensions 0 and 1 have been swapped. Now, the zeroth dimension is the originally second dimension, so all tensors become non-contiguous.

You might wonder, what’s the problem with non-contiguous tensors? Let’s move on to learn about shape transformations, and after that, you’ll understand the consequences of non-contiguous tensors.

Shape Transformations #

There are two commonly used functions in PyTorch to change the shape of a tensor, namely view and reshape. Let’s first take a look at view.

    >>> x = torch.randn(4, 4)
    >>> x.shape
    torch.Size([4, 4])
    >>> x = x.view(2,8)
    >>> x.shape
    torch.Size([2, 8])

Here, we first define a tensor with a size of [4, 4]. Then, using the view function, we modify it to a tensor with a shape of [2, 8]. Let’s continue with the previous variable x and perform one more operation:

    >>> x = x.permute(1,0)
    >>> x.shape
    torch.Size([8, 2])
    >>> x.view(4, 4)
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    RuntimeError: view size is not compatible with input tensor's size and stride (at least one dimension spans across two contiguous subspaces). Use .reshape(...) instead.

From the code, we can see that by using permute, we have transformed the data in dimensions 0 and 1 and obtained a tensor with a shape of [8, 2]. However, when we perform the view operation on this new tensor, an error occurs. This is because view cannot handle the structure of non-contiguous tensors. - What should we do in this case? We can use another function, reshape:

    >>> x = x.reshape(4, 4)
    >>> x.shape
    torch.Size([4, 4])

This solves the problem. In fact, reshape is equivalent to performing two steps: first, making the tensor contiguous in memory, and then performing the view operation.

Changing the Number of Dimensions #

Sometimes, we need to add or remove certain dimensions from a tensor, such as removing or adding channels of an image. PyTorch provides the squeeze() and unsqueeze() functions to solve this problem.

Let’s start with squeeze(). If the value of the specified dimension is 1, then that dimension is removed. If the value of the specified dimension is not 1, then the original tensor is returned. To help you understand, let me explain with an example:

    >>> x = torch.rand(2,1,3)
    >>> x.shape
    torch.Size([2, 1, 3])
    >>> y = x.squeeze(1)
    >>> y.shape
    torch.Size([2, 3])
    >>> z = y.squeeze(1)
    >>> z.shape
    torch.Size([2, 3])

From the code, we can see that we create a tensor with dimensions [2, 1, 3], and then remove the data in dimension 1 to obtain y. The squeeze operation is successful because the size of dimension 1 is 1 in y. However, when we try to further remove dimension 1 from y, we encounter an error. This is because the size of dimension 1 in y is 3, and squeeze cannot remove it. - unsqueeze(): This function mainly expands the dimensions of the data. It adds a dimension with a size of 1 at the specified position. Let’s again look at an example using code:

    >>> x = torch.rand(2,1,3)
    >>> y = x.unsqueeze(2)
    >>> y.shape
    torch.Size([2, 1, 1, 3])

Here, we create a tensor with dimensions [2, 1, 3], and then insert a dimension at position 2. This gives us a tensor with a size of [2, 1, 1, 3].

Summary #

In the previous lessons, we learned about the operations related to NumPy. If we compare NumPy and Tensor, it is not difficult to find that they have many similarities. The common point is that both of them are representations of data and can be regarded as general tools for scientific calculations. However, NumPy cannot be used for GPU acceleration, while Tensor can.

In this lesson, we have learned about the common functions of Tensor, such as creating, types, conversions, and transformations. With these functions, we can perform the most basic and commonly used operations on Tensor. These are the content that must be firmly remembered.

In addition, in actual projects, there are many other types of operations, among which the most important is mathematical operations, such as addition, subtraction, multiplication, division, merging, and connecting. However, if we list all these operations one by one, the number will be extremely large, and you may feel bored. Therefore, in the subsequent lessons, we will learn these mathematical operations in specific practical exercises.

In the next lesson, we will learn about advanced operations such as reshaping and splitting of Tensor. This is a very fun content, so please stay tuned.

Practice for Each Lesson #

In PyTorch, there are two functions: torch.Tensor() and torch.tensor(). What are the differences between them?

Feel free to leave a comment below to discuss with me, and I encourage you to share today’s content with more colleagues and friends.