Immerse black screen after updating to Omnisci 5.8

Hello,
after updating to 5.8 Immerse shows a black screen without login or dashboard list:
omnisci_server and web_server are correctly loaded, and all is fine, without error into the logfiles.
The browser pointing to: localhost:6273/omnisci/dashboards return a back screen without error: it seems to me that omnisci is working, but not displaying anything.
I Already removed from browser all from file cache and offline data files, I’m running out of ideas.
Details:

Fri Oct 15 11:43:08 2021       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.63.01    Driver Version: 470.63.01    CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:01:00.0  On |                  N/A |
| 37%   62C    P0    47W / 180W |   1064MiB /  8085MiB |      2%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      1357      G   /usr/lib/xorg/Xorg                584MiB |
|    0   N/A  N/A      1771      G   /usr/bin/kwin_x11                 168MiB |
|    0   N/A  N/A      1858      G   /usr/bin/plasmashell              119MiB |
|    0   N/A  N/A     57584      G   /usr/lib/firefox/firefox          173MiB |
|    0   N/A  N/A     65466      G   /usr/lib/firefox/firefox            1MiB |
+-----------------------------------------------------------------------------+
ps -ef|grep omnisci
omnisci    10084       1  0 10:04 ?        00:00:06 /opt/omnisci/bin/omnisci_web_server --config /var/lib/omnisci/omnisci.conf
omnisci    94093       1 16 11:44 ?        00:00:00 /opt/omnisci/bin/omnisci_server --config /var/lib/omnisci/omnisci.conf
omnisci    94110   94093 83 11:44 ?        00:00:01 -Xmx1024m -DMAPD_LOG_DIR=/var/lib/omnisci/data -jar /opt/omnisci/bin/calcite-1.0-SNAPSHOT-jar-with-dependencies.jar -e /opt/omnisci/QueryEngine/ -d /var/lib/omnisci/data -p 6279 -m 6274 -T  -P  -Y  -Z  -c /var/lib/omnisci/omnisci.conf

This is the back screen as shown:

Ubuntu GTX1080 8GB Omnisci free GPU edition
Firefox browser

here a snippet of the current log:

2021-10-15T12:51:32.894641 I 148763 0 0 MapDServer.cpp:394 OmniSciDB starting up
2021-10-15T12:51:32.897539 I 148763 0 0 DBHandler.cpp:310 OmniSci Server 5.8.0-20211011-b40c4f681d
2021-10-15T12:51:33.124482 I 148763 0 0 CudaMgr.cpp:368 Using 1 Gpus.
2021-10-15T12:51:33.124569 I 148763 0 0 CudaMgr.cpp:67 Warming up the GPU JIT Compiler... (this may take several seconds)
2021-10-15T12:51:33.237265 I 148763 0 0 CudaMgr.cpp:70 GPU JIT Compiler initialized.
2021-10-15T12:51:33.237351 I 148763 0 0 ArrowForeignStorage.cpp:850 CSV backed temporary tables has been activated. Create table `with (storage_type='CSV:path/to/file.csv');`
2021-10-15T12:51:33.237365 I 148763 0 0 ArrowForeignStorage.cpp:1028 CSV backed temporary tables has been activated. Create table `with (storage_type='CSV:path/to/file.csv');`
2021-10-15T12:51:33.237393 I 148763 0 0 DataMgr.cpp:230 Min CPU Slab Size is 256MB
2021-10-15T12:51:33.237410 I 148763 0 0 DataMgr.cpp:231 Max CPU Slab Size is 4096MB
2021-10-15T12:51:33.237417 I 148763 0 0 DataMgr.cpp:232 Max memory pool size for CPU is 98304MB
2021-10-15T12:51:33.237425 I 148763 0 0 DataMgr.cpp:250 Reserved GPU memory is 1337.67MB includes render buffer allocation
2021-10-15T12:51:33.237437 I 148763 0 0 DataMgr.cpp:269 Min GPU Slab size for GPU 0 is 256MB
2021-10-15T12:51:33.237445 I 148763 0 0 DataMgr.cpp:271 Max GPU Slab size for GPU 0 is 4096MB
2021-10-15T12:51:33.237451 I 148763 0 0 DataMgr.cpp:273 Max memory pool size for GPU 0 is 32768MB
2021-10-15T12:51:33.237605 I 148763 0 0 FileMgr.cpp:230 Completed Reading table's file metadata, Elapsed time : 0ms Epoch: 0 files read: 0 table location: '/var/lib/omnisci/data/mapd_data/table_0_0'
2021-10-15T12:51:33.237642 I 148763 0 0 Calcite.cpp:371 Creating Calcite Handler,  Calcite Port is 6279 base data dir is /var/lib/omnisci/data
2021-10-15T12:51:33.237655 I 148763 0 0 Calcite.cpp:279 Running Calcite server as a daemon
2021-10-15T12:51:33.752701 I 148763 0 0 Calcite.cpp:319 Calcite server start took 500 ms 
2021-10-15T12:51:33.752813 I 148763 0 0 Calcite.cpp:320 ping took 9 ms 
2021-10-15T12:51:33.776755 I 148763 0 0 DBHandler.cpp:493 Started in GPU mode
2021-10-15T12:51:33.779220 I 148763 0 0 VulkanPlatform.cpp:156 Vulkan Instance API version: 1.2.162
2021-10-15T12:51:33.897649 I 148763 0 1 **MapDServer.cpp:368 Interrupt signal (11) received.**

where there is a MapDServer.cpp:368 Interrupt signal (11) received.

I can observe that the omnisci_server process disappears from memory (using ps command) , then reappears again.

Hi Gianfranco,

I suggest you to turn off the vulkan rendered adding this parameter to your omnisci.conf file

renderer-use-vulkan-driver=false

probably you are having some trouble with our new renderer.
if confirmed, that you are having troubles with that, you can try to fix installing vulkan libraries needed; please refer to the vulkan renderer documentation.

I f you are continuing to get crashes, please report back, so I will file the problem to the renderer team.

Candido

Thank you,
I already checked vulkan lib, it was part of installation procedure, and it seems correctly installed:

$ sudo apt show libvulkan1
Package: libvulkan1
Version: 1.2.162.0-1
Priority: optional
Section: libs
Source: vulkan-loader
Origin: Ubuntu
Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>
Original-Maintainer: Debian X Strike Force <debian-x@lists.debian.org>
Bugs: https://bugs.launchpad.net/ubuntu/+filebug
Installed-Size: 422 kB
Depends: libc6 (>= 2.29)
Recommends: mesa-vulkan-drivers | vulkan-icd
Breaks: libvulkan-dev (<< 1.1.70+dfsg1-2), vulkan-loader
Replaces: libvulkan-dev (<< 1.1.70+dfsg1-2), vulkan-loader
Homepage: https://github.com/KhronosGroup/Vulkan-Loader
Task: ubuntu-desktop-minimal, ubuntu-desktop, ubuntu-desktop-raspi, kubuntu-desktop, xubuntu-core, xubuntu-desktop, lubuntu-desktop, ubuntustudio-desktop-core, ubuntustudio-desktop, ubuntukylin-desktop, ubuntu-mate-core, ubuntu-mate-desktop, ubuntu-budgie-desktop, ubuntu-budgie-desktop-raspi
Download-Size: 97,2 kB
APT-Manual-Installed: yes
APT-Sources: http://it.archive.ubuntu.com/ubuntu hirsute/main amd64 Packages
Description: Vulkan loader library
 The Loader implements the main VK library. It handles layer management and
 driver management. The loader fully supports multi-gpu operation. As part of
 this, it dispatches API calls to the correct driver, and to the correct
 layers, based on the GPU object selected by the application.
 .
 This package includes the loader library.

Anyway, after disabling vulkan and restarted omnisci_server and omnisci_web_server, and cleared the browser cache, I got exactly the same behavior with the black screen.

So the server is keep crashing and restarting? I guess that’s the reason why immerse is returning a blank screen.

have you tried what’s happens if you start the omnisci_server from command line with this command?

/opt/omnisci/bin/omnisci_server --config /var/lib/omnisci/omnisci.conf

Candido.

That command returns:

/opt/omnisci/bin/omnisci_server --config /var/lib/omnisci/omnisci.conf
terminate called after throwing an instance of 'boost::wrapexcept<boost::filesystem::filesystem_error>'
  what():  Failed to open file for writing: Input/output error: "/var/lib/omnisci/data/mapd_log/omnisci_server.INFO.20211015-152846.log"
Aborted (core dumped)

omnisci_server keep crashing and restarting automatically with a different pid every few seconds, while no message is reported in /MAPD_LOG folder.

Omnisci folder and subfolder are owned by omnisci user with : -rw-r–r–

Have you enough space and/or the right permissions to write into /var/lib/omnisci/data/ directory?

it seems that errors you are getting when you run the DB as a service or from cli are different.

try to run from cli with the same user that’s running the service and add --log-severity-clog=DEBUG4
if you are running the omnsci DB as root run

sudo /opt/omnisci/bin/omnisci_server --data /var/lib/omnisci/data --log-severity-clog=DEBUG4

this way the server will be run with default parameters and should output a super detailed log on standard output.

Candido

The detailed log activated with DEBUG4 doesn’t write any info in ERROR and FATAL log files.

In addition, I have to point out that the system is plenty of space, and that many several omnisci_server.INFO logfiles are continuously created there.

Therefore, this should not be a problem of permission, considering that the owner of /var/lib/omnisci folder (omnisci) has the write permission and the logfiles are regularly created.

Lastly, I cannot see any error in initdb.info.

And now I run omnisci with the same user of the service, as you suggested:

I got the super detailed log in the standard output, here the last few lines:

2021-10-15T16:38:05.898118 I 330635 0 0 DBHandler.cpp:493 Started in GPU mode
2021-10-15T16:38:05.901606 I 330635 0 0 VulkanPlatform.cpp:156 Vulkan Instance API version: 1.2.162
2021-10-15T16:38:06.023318 1 330635 0 1 MapDServer.cpp:363 heartbeat thread exiting
2021-10-15T16:38:06.023387 I 330635 0 1 MapDServer.cpp:368 Interrupt signal (11) received.
Segmentation fault (core dumped)

HI Gianfranco,

Nothing special, just 1 line more that has the DEBUG1 levell.

Which OS are you on? the VI 1.2.162 comes with Ubuntu HIRSUTE

Candido

C

Sure, it’s a Hirsute build:

Distributor ID: Ubuntu
Description:    Ubuntu 21.04
Release:        21.04
Codename:       hirsute

Hi @Gianfranco_Campana,

I have the same crash on Ubuntu 20.04 with the 460.91.03 drivers and Vulkan render enabled.
switching off the vulkan render and then using the old opengl one is fixing the crash, so everything works.

Anyway, to fix this crash being able to use the Vulkan render, it’s necessary to remove the mesa-vulkan-drivers that in ubuntu are installed by default, with this command

sudo apt remove mesa-vulkan-drivers

or you can et this ENV while running the server if you need the mesa drivers.

export VK_ICD_FILENAMES=/usr/share/vulkan/icd.d/nvidia_icd.json

now runnig the server
bin/omnisci_server --data /mapd_storage/data_test/ --log-severity-clog=DEBUG4 --renderer-use-vulkan-driver=true

everything works.
Let me know if this is working on your install too

Regards,
Candido

1 Like

Thanks, Candido,
I confirm deactivating vulkan driver solved the problem : I just disabled it in the .conf file with renderer-use-vulkan-driver=false

Hi,

have you tried to remove the mesa-vulkan-drivers package? This make you work the vulkan-render

Candido.

1 Like

Thank you, Candido,
removing the mesa vulkan driver fixed the crash, and made everything working fine.