Question about MapD compiling & profiling


#1

Q1) I am currently profiling MapD using TPC-H data. And we’re looking at where the botleneck comes from. But there’s not much information. Have you ever used CUPTI to apply to a java file about MapDJDBC? If you did, I would like to know what kind of CPTI metric you used. And I would like to know if there is a source code related to CUPTI and MapD.

Q2) I am trying to compile MapDServer.cpp. However, the error occurs because “MapDRelease.h” was not found. How do I create that file?

Q3) When compiling MapDServer.cpp, can I change the slab size within MapdServer.cpp code?

I’m sorry for asking too many questions. Is there a way to get in touch with MapD developers quickly other than this community?


#2

Hey thanks for trying MapD out.

A1) CUPTI might be a level higher for profiling you may first want to look into nvprof or nsight from nvidia to generally understand how the database works. For query profiling you can run many tools which may not be GPU specific. We have a CPU profile option which enables gperf tools ENABLE_PROFILER. As the bottlenecks could be on CPU as well.

A2) To compile mapd first, please carefully read the Read Me on our repository. https://github.com/omnisci/mapd-core. There we specify the tools you will need to compile mapd. From the pictures you have provided I can see your are using a very old version of gcc. We have specified the steps on how to compile mapd_server exactly on the same page. From the logs I can also see that you have not invoked cmake in build directory. Lastly since the cmake was not fired in correct directory the tools cannot find MapDRelease.h. That file is in the repository.

We also have shell scripts to build all the dependencies that you will need to compile mapd in scripts directory: https://github.com/omnisci/mapd-core/tree/master/scripts

A3) Sure. But we don’t guarantee that it will always work.

Feel free to ask more questions here.

Thanks.


#3

A2 -> I’ve read Documentation but I don’t understand it. MapDRelease.h file does not seem to be created even after following installation instructions. Can you explain yourself?

A3 -> It’s okay if it doesn’t always work. I’d like to refer to it.


#5

Attempting to recompile MapD-Core failed due to library installation issue. It was the same when I tried to run the files in the mapd-core/scripts folder.
If i still use cmake, i’m have library problems.
So when I tried to install the library, I had problems installing the library, so I installed some of them. But there are a lot of things that can’t be installed. Is there a solution?

image


#6

hi @Hyuck,

it looks like you haven’t run the script /usr/local/mapd-deps/mapd-deps.sh before invoking cmake

the script it’s created by the script used to compile follymthrifht and so on.


#8

In ubuntu is different but I think the result is the same

looking to the centos scripts, at the end I can find this command
cp mapd-deps-$SUFFIX.sh mapd-deps-$SUFFIX.modulefile $PREFIX

so you can try to run /usr/local/mapd-deps/[some date]/mapd-deps-[some date].sh before use cmake.


#9

Q1. I am recompiling to change the size of the cpu slap or gpu slap in the mapd_server. Can I change them just by using the configuration parameter when running Mapd_server? Then, I don’t think I need a recompile.

Q2. So once I tried the configuration parameter for changing the gpu map size, it was analyzed as nvprof and ended. Why does it end immediately when I don’t even run the query?

image
image.png1234x289 15.6 KB

Q3. We will try to program java using the ‘nvprof --unified-memory-profiling off mapd_server --config [your storage directory]/mapd.conf’ you taught last time. But this is only possible by pressing Ctrl+C. Do you think Java programming is possible?


#10

1-2) do you mean changing --gpu-buffer-mem-bytes or --CPU-buffer-mem-bytes parameters?

those parameters contain just the max memory (in bytes) the system can allocate for CPU (system) or GPU.

limit the max memory available for GPU to 12GB (13221225472 bytes)

mapd_server  --config /opt/mapd_storage/mapd.conf --gpu-buffer-mem-bytes=13221225472

mapdql> \memory_gpu
MapD Server Detailed GPU Memory Usage:
Maximum Bytes for one page: 512 Bytes
Maximum Bytes for node: **12608 MB**
Memory allocated: **2048 MB**
GPU[0] Slab Information:
SLAB     ST_PAGE NUM_PAGE  TOUCH         CHUNK_KEY
   0           0    62500      0 USED    1,29,1,0,
   0       62500    62500      1 USED    1,29,1,1,
   0      125000    62500      2 USED    1,29,1,2,
   0      187500    62500      3 USED    1,29,1,3,
   0      250000    62500      4 USED    1,29,1,4,
   0      312500    62500      5 USED    1,29,1,5,
   0      375000    62500      6 USED    1,29,1,6,
   0      437500    62500      7 USED    1,29,1,7,
   0      500000    62500      8 USED    1,29,1,8,
   0      562500    62500      9 USED    1,29,1,9,
   0      625000    62500     10 USED    1,29,1,10,
   0      687500    62500     11 USED    1,29,1,11,
   0      750000    62500     12 USED    1,29,1,12,
   0      812500    62500     13 USED    1,29,1,13,
   0      875000    62500     14 USED    1,29,1,14,
   0      937500    62500     15 USED    1,29,1,15,
   0     1000000     9433     16 USED    1,29,1,16,
   0     1009433  3184871     46 FREE
---------------------------------------------------------------

As you can see from memory_summary command, the maximum memory available to GPU is 12GB, the Slab 0 is of 2GB

So as I already said you have to recompile to change the Slab’s size.

The system cannot start with --gpu-buffer-mem-bytes set to 5000, because of 5000 bytes arent enough :wink:

  1. I don’t know if it’s possible to profiles without exiting from the program; I’ll check later, but you can use other switches and configure the profiler to save an output file with the -o option; in this way you can have a boinary files that you can open with nvvp graphical command

as example

nvprof --print-gpu-trace --unified-memory-profiling off -o /home/mapd/430_2.nvvp -f /opt/mapd/mapd-ce-4.3.0-20181119-b7f85d00bd-Linux-x86_64-render/bin/mapd_server --config /opt/mapd_storage/mapd.conf

after exiting from program you will be able to open the output file with nvvp program.

It’s easier to understand than the text output and you can get more ifnormations