Repository navigation
Memory remains on device after ArrayFire array's are cleaned up #1754
Description
In my codes, it calls af::fft3 and cleans up all arrayfire arrays but memory allocated on GPU remains. For example, there is 711.16MB allocated on GPU at the beginning but it is increased to 1063.04MB at the end.
AF Memory @start
Allocated [ Bytes | Buffers ] = [ 0 | 0 ]
In Use [ Bytes | Buffers ] = [ 0 | 0 ]
CUDA Memory @start [0] = {711.16MB/6144.00MB}
AF Memory @EnD of program
Allocated [ Bytes | Buffers ] = [ 0 | 0 ]
In Use [ Bytes | Buffers ] = [ 0 | 0 ]
CUDA Memory @EnD of program [0] = {1063.04MB/6144.00MB}
In the real application with big data size, it causes the memory to run out from the second execution because some of the allocated memory at the first iteration is not freed up.
#include <iostream>
#include <arrayfire.h>
#include <af/data.h>
#include <nvml.h>
#define TC_MEM_INFO(msg) do { \
size_t abytes = 0, abuffs = 0, lbytes = 0, lbuffs = 0; \
af_err err = af_device_mem_info(&abytes, &abuffs, &lbytes, &lbuffs); \
if(err == AF_SUCCESS) { \
printf("AF Memory %s\n", msg); \
printf(" Allocated [ Bytes | Buffers ] = [ %lld | %lld ]\n", abytes, abuffs); \
printf(" In Use [ Bytes | Buffers ] = [ %lld | %lld ]\n", lbytes, lbuffs); \
} else { \
fprintf(stderr, "AF Memory at %s:%d: " msg "\nAF Error %d\n", \
__FILE__, __LINE__, err); \
} \
} while(0)
#define TC_PRINT(tag, idx, device) do { \
nvmlMemory_t mem; \
TC_MEM_INFO(tag); \
nvmlDeviceGetMemoryInfo(device, &mem); \
printf("CUDA Memory %s [%d] = {%.2fMB/%.2fMB}\n", tag, idx, mem.used/1024.0/1024.0, mem.total/1024.0/1024.0); \
} while(0)
using namespace std;
int main(int argc, char* argv[])
{
nvmlDevice_t device;
nvmlInit();
nvmlDeviceGetHandleByIndex(0, &device);
TC_PRINT("@start", 0, device);
const int d0 = 444;
const int d1 = d0;
const int d2 = 206;
for (int i = 0; i < 2; i++)
{
af::array a0 = af::constant(0, d0, d1, d2, f32);
a0.eval();
af::array a1 = af::fft3(a0);
a1.eval();
}
af::sync();
af::deviceGC();
TC_PRINT("@end of program", 0, device);
nvmlShutdown();
return 0;
}
Activity
pavanky commentedon Apr 10, 2017
This is possibly coming from cufft plans. They are not killed until the program goes out of scope. The 700+ MB memory used upfront is not memory used by arrayfire. It is the memory being used by the GPU if it is also being used for display or other programs using the GPU. The 300MB differential could be because of the fft3. You can try using a different (non fft) function to verify this.
dodobing commentedon Apr 10, 2017
@pavanky Yes, it is related to af::fft3. Is it possible to clean up them without finishing process?
pavanky commentedon Apr 10, 2017
@dodobing are you running fft3 on data of different sizes ?
dodobing commentedon Apr 10, 2017
@pavanky Yes, I am running it on data of different size.
pavanky commentedon Apr 10, 2017
@dodobing There is a function to limit the number of plans being used at any time. But I just looked the code and it seems to be broken..
If you can build arrayfire from source, can you try changing the following function from this file:
https://github.com/arrayfire/arrayfire/blob/2528d711f371b18c0f51d5f545737d5c62ea284f/src/backend/cuda/fft.cpp
You can then call
af_set_fft_plan_cache_size(0)from user code to destroy all previous plans.dodobing commentedon Apr 12, 2017
@pavanky I tested codes you suggested above but there was no changes unfortunately.
pavanky commentedon Apr 12, 2017
@dodobing let me test this a bit.
dodobing commentedon Apr 19, 2017
@pavanky is there any update?
pavanky commentedon Apr 21, 2017
@dodobing I need some more time to test this.
dodobing commentedon May 5, 2017
@pavanky If there is any change or idea, I want to test it on my side. Thx.