Error When Querying: GPU runtime

So i believe we have everything installed correctly, i can do basic querying but most of them give me this error
“Exception: OmniSci GPU runtime library not found at /opt/omnisci/QueryEngine/cuda_mapd_rt.a
unable to rollback”

Sounds like something is corrupted or not installed correctly?

Any guidance would be great, also any additional information on query building would be awesome too. What parts of SQL work with Omnisci and and does Omnisci require that SQL wouldn’t.
I have experience with SQL but some of the documentation on the Omnisci site is still going over my head.
I’ve been struggling with getting this database working for weeks now, any suggestions or input is greatly appreciated

Hi @JustXtreme,

Welcome to our community, and thanks for trying out our database.

The cuda_mapd_rt.a library contains the code to do SUMs, counts, and other kinds of aggregations, so I believe you are getting this error when you run queries containing aggregates in GPU mode. You don’t get the error when you run projection query with filtering and or joins.

Anyway, the error looks to be self-explaining. Into the path where you installed the software, it seems the library is missing, or the user you used to run the server hasn’t the right to read it.

In a regular installation, until 5.2.x release, the QueryEngine directory would look like that

mapd@zion16:/opt/mapd/omnisci-ee-5.2.0-20200416-4cc64c2b32-Linux-x86_64-render$ ls -la QueryEngine/
total 3708
drwxrwxr-x  2 mapd mapd    4096 Apr 17 00:49 .
drwxrwxr-x 12 mapd mapd    4096 Jul  3  2019 ..
-rw-r--r--  1 mapd mapd 3621492 Apr 16 23:52 cuda_mapd_rt.a
-rw-r--r--  1 mapd mapd   18761 Apr 16 23:51 ExtensionFunctions.ast
-rw-r--r--  1 mapd mapd    3869 Apr 16 23:50 OmniSciTypes.h
-rw-r--r--  1 mapd mapd  134456 Apr 16 23:51 RuntimeFunctions.bc
-rw-r--r--  1 mapd mapd    1941 Apr 16 23:50 serialized_result_set.thrift

I guess those files are used to generate the shaders needed from the database at runtime the first time you run queries where the GPU is required, but I can wrong.

Anyway, to be able to help you, you would share some data with us.

  1. version of the omnisci database (looks you are using
  2. the method you used to install (tarball apt/rpm or compiling by yourself)
  3. the OS you are using (ubuntu, centos)
  4. the detailed contents of the QueryEngine directory
  5. the precise circumstances you are getting the error (e.g., after I installed the database, I loaded some data and tried out a group by query, and I got the error immediately or after some seconds )
    Maybe with this data, we will able to reproduce the error and give you a quick solution.

I almost reproduced your error in 2 ways renaming the cuda_mapd_rt.a file into QueryEngine directory, but I am not able to run any query in GPU mode, and the message is slightly different, because is missing the rollback part

Exception: OmniSci GPU runtime library not found at /opt/mapd/omnisci-ee-5.2.0-20200416-4cc64c2b32-Linux-x86_64-render/QueryEngine/cuda_mapd_rt.a

Best Regards.

Thank you for the response, It seems that my “cuda_mapd_rt.a” is actually “cuda_mapd_rt.fatbin” file. I’ve tried reinstalling cuda but it hasn’t changed it.

  1. Running 5.3.1
  2. APT
  3. Ubuntu
    4.
  4. SELECT Age_group, COUNT(*) as agecount FROM FLCovidCases GROUP BY Age_group ORDER BY Age_group DESC LIMIT 20;

I reinstalled Cuda and worked through some update errors and it seems to be working now. Thank you for your time and response, I’m sure i’ll have dozens of questions.

Also my "cuda_mapd_rt.fatbin file is still there with no .a file.
not sure if that means anything or not but it seems to be working.

Hi @JustXtreme,

I’m happy the database is working in GPU mode right now and don’t be worry with the release 5.3.1 the libcuda_rt.a has been replaced by the libcuda_rt.fatbin, so everything is as expected right now.

To make omnisci db work, you need just the nvidia driver, so probably, when you installed cuda, yo got a more recent driver and the problems are gone.

Waiting for you questions :wink: