nvidia-smi command to check the GPU usage. If you find that the program has already terminated but there is still GPU memory occupied, it indicates that a residual process is holding the memory. You can release it as follows:
From the screenshot, it can be seen that a program is occupying 4388MiB of GPU memory. To release the memory, first identify the process ID:
Use the ps -ef command.
You can see three important columns: PID, PPID, and CMD, which represent the process ID, parent process ID, and the command used to start the process, respectively.
By examining the command, you can determine which processes were started by your program. For example, python train.py in the screenshot is a process I started, while others are system processes or unrelated to GPU memory usage.
- Next, terminate the process:
- From the screenshot, the process IDs for
python train.pyare594and797. You can use thekill -9 594 797command to end these processes. However, when many processes occupy GPU memory, especially in multi-GPU parallel scenarios, this method can be cumbersome.
- From the screenshot, the process IDs for
- Here is a more powerful way to terminate processes:
- Using ps
-ef, you can see that all my processes contain the keywordtrain(and other unrelated system processes do not, to avoid accidental termination). You can filter your processes using thegrepcommand, for example:
- Using ps
Next, obtain the process IDs using the awk command. The awk command is complex, but you only need to remember the following command:
Finally, use the kill command to terminate the processes. The complete command is: ps -ef | grep train | awk '{print $2}' | xargs kill -9
The output may include an error message like “No such process”, which can be ignored. This occurs because the grep train command itself generates a process that gets filtered out.
More Explanation:
In Linux commands, the | symbol is called a pipe. Its function is to use the output of one command as the input for the next command (usually stdout; stderr requires separate handling). Pipes are very useful in many scenarios. For example, if a directory contains tens of thousands of files, but only one is a .txt file while the others are images, manually searching through the list generated by ls would be very cumbersome. Instead, you can use: ls | grep "\.txt$"