Discussions

Expand all | Collapse all

omnisci server crashed

  • 1.  omnisci server crashed

    Posted 07-24-2019 07:30

    Hi

    Recently we have observed a strange behaviour  and OMNISCI database  got crashed . This OMNISCI database is running CPU version.

    Details are as following

    process status 
    *****************

    [#########]$ ps -ef|grep omni

    root      60209      1  8 00:59 ?        01:00:54 /opt/omnisci/bin/omnisci_server --config /opt/data/omnisci.conf

    root      60211  60209  0 00:59 ?        00:00:54 -Xmx1024m -DMAPD_LOG_DIR=/opt/data/data -jar /opt/omnisci/bin/calcite-1.0-SNAPSHOT-jar-with-dependencies.jar -e /opt/omnisci/QueryEngine/ -d /opt/data/data -p 6279 -m 6274 -T  -P

    *******************************
    from omnisci log:
    cat omnisci_server.pcrfreporting.bt.bt.root.log.FATAL.20190722-001007.162833
    Log file created at: 2019/07/22 00:10:07
    Running on machine: pcrfreporting.bt.bt
    Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
    F0722 00:10:07.050699 185229 File.cpp:117] Check failed: bytesRead == sizeof(int8_t) * size (544736 vs. 2097120)
    F0722 00:10:07.050716 185221 File.cpp:117] Check failed: bytesRead == sizeof(int8_t) * size (4064 vs. 2097120)
    *********************************
    cat omnisci_server.pcrfreporting.bt.bt.root.log.INFO.20190717-233139.162833
    I0722 00:09:56.087494 104579 BufferMgr.cpp:425] ALLOCATION failed to find 128000000B free. Forcing Eviction. Eviction start 4312500 Number pages requested 250000 Best Eviction Start Slab 10 CPU_MGR:0
    I0722 00:09:59.354581 104579 BufferMgr.cpp:425] ALLOCATION failed to find 128000000B free. Forcing Eviction. Eviction start 4625000 Number pages requested 250000 Best Eviction Start Slab 10 CPU_MGR:0
    I0722 00:10:02.777987 185193 MapDHandler.cpp:369] User wdbsreport connected to database wdbsreportdb
    F0722 00:10:07.050699 185229 File.cpp:117] Check failed: bytesRead == sizeof(int8_t) * size (544736 vs. 2097120)
    F0722 00:10:07.050716 185221 File.cpp:117] Check failed: bytesRead == sizeof(int8_t) * size (4064 vs. 2097120)
    I0722 00:10:21.219918 185193 MapDHandler.cpp:377] User wdbsreport disconnected from database wdbsreportdb
    I0722 00:10:21.247635 185221 Calcite.cpp:432] Shutting down Calcite server
    I0722 00:10:21.247555 185229 Calcite.cpp:432] Shutting down Calcite server
    I0722 00:10:22.352658 185229 Calcite.cpp:441] shut down Calcite
    I0722 00:10:22.352798 185229 MapDServer.cpp:102] Interrupt signal (11) received.
    ***********************************

    omnisci version :
    omnisql> \status
    The Server Version Number : 4.5.0-20190225-
    The Server Start Time : 2019-07-24 : 4:58:31
    The Server edition : 4.5.0-20190225-
    ****************************************

    thanks in advance.


    #Core


  • 2.  RE: omnisci server crashed

    Posted 07-24-2019 09:35
    Hi @Ishika [[Unknown]]

    I'm assuming you have run a query and after that the omnisci server crashed.
    looking at your log, it seems you have a filesystem/disk corruption.

    the meaning of the Fatal error you are getting here 
    F0722 00:10:07.050699 185229 File.cpp:117] Check failed: bytesRead == sizeof(int8_t) * size (544736 vs. 2097120)
    F0722 00:10:07.050716 185221 File.cpp:117] Check failed: bytesRead == sizeof(int8_t) * size (4064 vs. 2097120)

    is that the server tried to read some file asking for 2 megabytes and he got just 544736 and 4064 bytes, so the server stops.

    Have you got a crash during a load of data, or a power loss on the server while writing the data?