How can I tell how much of the GPU is being utilized at any given time? Ideally I’d like to see the usage of each GPU execution unit, transfers to and from the GPU, instructions retired per second, and any other info that is available from the hardware.
At it simplest you can run
nvidia-smi to see high level details (and even deeper details with appropriate flags.
But based on what you sound like you are looking for I would point you to the nvidia profiler (http://docs.nvidia.com/cuda/profiler-users-guide/#axzz4iUQ8Ul3p), which will allow you to gather all the details you want.
I did try to use the profiler but it wasn’t clear which program I should be profiling or how. I tried profiling all apps and then restarting the mapd service, and it did show up in the process list, but I didn’t see any data collected. Do you have a how-to guide for profiling mapd?