How can I reduce the memory processing large dataset(Array, Tensor, etc.) in Pytorch?

The introduction: When I was processing the very large data set, I wanted to reduce that stored in my memory to run through my MacBook(), I know that is poor for deep learning.

Start: When we process the data we will copy the tensor or list etc. So let’s read the code detail of Python.

b = 1
c = 1
before = id(b)
b += c
print(b)
print(id(b) == before)
d = 1
e = 1
d = d + e
print(d)
print(id(d) == before)

In that case, I thought you should have a huge ‘Question Mark’ and “So??? Is it a new land in the world?” Let’s go next step.

b = [1]
c = [1]
before = id(b)
b += c
print(b)
print(id(b) == before)
d = [1]
e = [1]
d = d + e
print(d)
print(id(d) == before)

Incredible, Amazing!! What’s that? why do I add the [ ] around the 1 integer the result will be different? I want to answer you in the last chapter. So Let’s go!

import torch
b = torch.zeros(
    (3, 4, 5)
)
c = torch.ones(
    (3, 4, 5)
)
before = id(b)
b += c
print(b)
print(id(b) == before)
d = torch.zeros(
    (3, 4, 5)
)
e = torch.ones(
    (3, 4, 5)
)
d = d + e
print(d)
print(id(d) == before)

Let’s reveal this mysterious veil！

1. The KEY WORD is what the Immutable element and mutable element are.

1. The immutable elements: Integers, floats, strings, tuples, etc.

2. The mutable elements: Lists, Dictionaries, Tensors, etc.

2. What is the function of “=”?

Binding an object reference to an object in memory.

3. How does the “=” work in the changing of the immutable elements?

Directly create a new integer value in memory and then bind the variable reference to it. So, The type of an immutable element, “A += B” is the same as "A = A + B”..

4. What about the mutable elements?

As you see, using the “A += B” Python will add value to the same memory area. By contrast, using the “A = A + B” Python creates a new variable area to the memory and computes the value then to be bound by the variable reference “A”.

Conclusion

I should get into the habit of considering the difference between “A+=B” and “A = A+ B”. So do you.