Discussions

Expand all | Collapse all

Error launching the GPU kernel: an illegal memory access was encountered

  • 1.  Error launching the GPU kernel: an illegal memory access was encountered

    Posted 03-14-2019 07:40
    Hi everybody,

    I was testing the DB with a large dataset and few complex queries.
    While doing so I encountered the following error message:
    I0312 10:35:41.545658 71650 Catalog.cpp:3074] Time to load Dictionary 1_2 was 56ms
    I0312 10:35:41.550999 71664 Calcite.cpp:395] Time in Thrift 1 (ms), Time in Java Calcite server 53 (ms)
    I0312 10:35:41.551895 71664 RelAlgOptimizer.cpp:755] (RelProject<140184021675344> (RexInput 8 140184782011392) (RexLiteral 1) (RexInput 0 140184782011392) (RexInput 2 140184782011392)) eliminated 1 columns.
    F0312 10:35:42.319502 45530 Execute.cpp:2210] Error launching the GPU kernel: an illegal memory access was encountered
    I0312 10:35:42.320631 45530 Calcite.cpp:432] Shutting down Calcite server
    I0312 10:35:42.328037 45530 Calcite.cpp:441] shut down Calcite
    I0312 10:35:42.328122 45530 MapDServer.cpp:102] Interrupt signal (11) received.


    My setup is:
    • Tesla K80 with NVIDIA-SMI 410.79       Driver Version: 410.79       CUDA Version: 10.0;
    • Ubuntu 16.04.5 LTS xenial;

    • OmniSci Server Version: 4.5.0-20190221-e41be43ff0 OS from  the repositories.

    The disk space usage looks like this:

    124K /var/lib/omnisci/data/mapd_catalogs
    
    2.4M /var/lib/omnisci/data/mapd_log
    
    147G /var/lib/omnisci/data/mapd_data/table_1_1
    
    8.1M /var/lib/omnisci/data/mapd_data/DB_1_DICT_2
    
    8.1M /var/lib/omnisci/data/mapd_data/DB_1_DICT_4
    
    2.0G /var/lib/omnisci/data/mapd_data/DB_1_DICT_1
    
    8.1M /var/lib/omnisci/data/mapd_data/DB_1_DICT_3
    
    12K /var/lib/omnisci/data/mapd_data/table_0_0
    
    149G /var/lib/omnisci/data/mapd_data
    
    4.0K /var/lib/omnisci/data/mapd_export
    
    149G /var/lib/omnisci/data
    
    149G /var/lib/omnisci/


    Does anyone have any idea about why this is occurring? Is there a way for me to have a better understanding of why is this happening?


    Best,
    Francesco.
    #Core


  • 2.  RE: Error launching the GPU kernel: an illegal memory access was encountered

    Posted 03-14-2019 14:32
    Hi @Francesco Ferretti. ​Can you please provide the verbose logs and the query memory descriptor output?

    ------------------------------
    Eric Grant
    Director of Community
    OmniSci
    Seattle WA
    ------------------------------



  • 3.  RE: Error launching the GPU kernel: an illegal memory access was encountered

    Posted 03-15-2019 02:21
    Edited by Candido Dessanti 03-15-2019 02:22
    Ciao @Francesco Ferretti,

    to get a QMD output, you have to set to true the verbose option.

    If you can, please include the query you


    ------------------------------
    Candido Dessanti
    Dba
    consulting
    Rome
    ------------------------------



  • 4.  RE: Error launching the GPU kernel: an illegal memory access was encountered

    Posted 03-31-2019 07:38
    Hi everybody,

    thank you for your help but, after digging around a little bit, a friend of mine was able to find the root cause of the issue.
    While inserting data to a table like 
    CREATE TABLE foo (
                ...
               bar TEXT NOT NULL ENCODING DICT(8),
               ...);
    if you have enough data and you try to insert a line with empty string inside the DB it will accept it but it will break it even with simple queries like:
    SELECT bar FROM foo GROUP BY bar​
    This apparently does happen only in certain condition so how to replicate it is still uncertain but the issue completely disappeared after the removal of lines with empty string for column bar. 
    Is it maybe related to some casting of empty string to NULL?

    P.S.
    I can see only few programming languages for the code snippets highlight. Is there a way to have SQL formatting?

    ------------------------------
    Francesco Ferretti

    ------------------------------



  • 5.  RE: Error launching the GPU kernel: an illegal memory access was encountered

    Posted 04-02-2019 06:29
    Hi @Francesco Ferretti

    Would you mind filing an issue about this null string issue on https://github.com/omnisci/mapd-core/issues so that one of our backend engineers can evaluate this? I realize you say that you can't necessarily replicate the issue, but just providing whatever information you can will be helpful, especially what method you loaded the data in (via mapdql/omnisql, JDBC, StreamImporter, pymapd, however).

    As far as the code formatting goes, yes, we're aware that this new messageboard has some formatting issues. Stay tuned, we're hoping to get a lot of these rough edges sorted out in the near future.

    Best,
    Randy​

    ------------------------------
    Randy Zwitch
    Senior Director of Developer Advocacy
    OmniSci
    Philadelphia PA
    ------------------------------



  • 6.  RE: Error launching the GPU kernel: an illegal memory access was encountered

    Posted 05-05-2019 07:33
    Hi Randy,

    sorry for the delay but I finally opened the issue on GitHub.