How to config gpu memory


#1

hi …I have a 4GB memory GPU card, the \memory_summay shows the server gpu memory : max is 3500MB, allocated is 2048MB, how to enlarge the memory to 30000MB, thx.


#2

Hi @giverking,

Do you mean 3000mb? You can’t configure a bigger Gpu memory than physical installed on your hardware, but you can reserve some gpu/cpu memory for other purposes.


#3

yes. i mean 3000mb, how to do ?


#4

you can limit the memory used by the mapd_server with those two parameters

GPU-buffer-mem-bytes and CPU-buffer-mem-bytes

if you place this on mapd.conf file, then you restart the server

gpu-buffer-mem-bytes=3221225472

the memory used by the mapd server will be limited to 3GB for every GPU installed into the system, leaving the rest of memory available to other software

you can reserve some memory with another parameter called res-gpu-mem; this one will not define the max memory used by the GPUs but it will reserve some memory.

so if you use res-gpu-mem=3221225472 on an 4GB cards you will use just 1G on every card

so when you login into the system and use the memory summary command, you will get something like that

MapD Server GPU Memory Summary:
[GPU]            MAX            USE      ALLOCATED           FREE
  [0]     3072.00 MB        0.00 MB        0.00 MB        0.00 MB

You can do the same with CPU memory.

If you want to know and experiment with more parameters you can run the mapd_server command with --help-advanced switch to get a list of all parameter currently supported by the server, or you can refer to this page on docs

http://docs.mapd.com/latest/4_configuration.html


#5

i have set the gpu-buffer-mem-bytes=3221225472, and i do some query, it shows:
MapD Server GPU Memory Summary:
[GPU] MAX USE ALLOCATED FREE
[0] 3072.00 MB 1684.62 MB 2048.00 MB 363.38 MB

why the ALLOCATED still is 2048mb


#6

because the queries you have runs needs only 1684.62MB so the system has allocated just a 2GB slab

you can check with \memory_gpu switch.

the parameters i described before limit the memory mapd server can allocate WHEN needed by your queries.
Do you want something to pre-allocated the memory? I think there isn’t any parameter to do thta, but as a workaround you can use the --db-query-list paramter that’s is primarly used to warm up caches, but doing that it will be force the allocation of slabs on GPU and CPU memory.


#7

when i run a query, there is two table join, every table have 7000m records.
it shows Thrift error: No more data to read.
Thrift connection error: No more data to read.
Retrying connection

i think the mapd server is crashed ,how to avoid it happy.


#8

without are rational is difficult to know why the server crashed, but the most likely scenario is that your quary is projecting too much columns, so the system run out of memory for the projection, and the mapd_server crashes.