Mapd_server and mapd_web_server not starting

#1

Hello team,

we have getting the bellow error while starting the database after the server reboot.

[root@reportingbtpcrf mapddata]# systemctl status mapd_server -l
● mapd_server.service - MapD database server
Loaded: loaded (/usr/lib/systemd/system/mapd_server.service; enabled; vendor preset: disabled)
Active: failed (Result: start-limit) since Wed 2019-01-16 20:00:53 IST; 2s ago
Process: 22346 ExecStart=/opt/mapd/bin/mapd_server --config /opt/mapddata/mapd.conf (code=exited, status=6)
Main PID: 22346 (code=exited, status=6)

Jan 16 20:00:53 reportingbtpcrf systemd[1]: Unit mapd_server.service entered failed state.
Jan 16 20:00:53 reportingbtpcrf systemd[1]: mapd_server.service failed.
Jan 16 20:00:53 reportingbtpcrf systemd[1]: mapd_server.service holdoff time over, scheduling restart.
Jan 16 20:00:53 reportingbtpcrf systemd[1]: start request repeated too quickly for mapd_server.service
Jan 16 20:00:53 reportingbtpcrf systemd[1]: Failed to start MapD database server.
Jan 16 20:00:53 reportingbtpcrf systemd[1]: Unit mapd_server.service entered failed state.
Jan 16 20:00:53 reportingbtpcrf systemd[1]: mapd_server.service failed.

please find the logs&straces process logs.txt (6.6 KB)
mapd_not starting.txt (61.2 KB)

kindly help us on the above issue.

regards
pranav krishna.

#2

the relevant error is this one

Jan 16 20:00:52 reportingbtpcrf mapd_server[22346]: what(): Sqlite3 Error: unable to open database file
Jan 16 20:00:52 reportingbtpcrf systemd[1]: mapd_server.service: main process exited, code=exited, status=6/NOTCONFIGURED

Omnisci database uses SQLite as a base metadata repository, so the system isn’t unable to open 1 or more files.
Said that you have to check

  1. the path you provided for the data directory is correct, and a mapd catalog exists.
    an ls -la should return an output like that

    drwxr-xr-x 8 mapd mapd 4096 gen 2 21:57 .
    drwxr-xr-x 2 mapd mapd 4096 gen 15 15:54 mapd_catalogs
    drwxr-xr-x 1406 mapd mapd 57344 gen 15 16:05 mapd_data
    drwxr-xr-x 6 mapd mapd 4096 giu 10 2018 mapd_export
    drwxr-xr-x 2 mapd mapd 4096 ago 30 15:37 mapd_import
    drwxr-xr-x 2 mapd mapd 237568 gen 16 14:31 mapd_log

  2. if the user that runs the server (typically mapd) is the owner, or has the right permissions on files that compromise the mapd catalog (mapd_catalogs, mapd_log, and mapd_data)

#3

Hello aznable,

Many thanks for your quick response. I have verified the data directory and the contents also. I could see all the subdirectories in it, With right permissions. but still am not able to start the mapd server.

  1. mapd works fine before the reboot.

  2. mapd-version:mapd-4.4.0

Regards,

Pranav Krishna

#4

Could you post the mapd_server.INFO file located in the mapd_log directory?

#5

please find the logs (mapd_server.INFO)

Log file created at: 2019/01/17 11:23:00
Running on machine: reportingbtpcrf
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
I0117 11:23:00.590560 5628 MapDServer.cpp:645] MapD started with data directory at ‘/opt/mapddata/data’
I0117 11:23:00.591037 5628 MapDServer.cpp:654] Watchdog is set to 1
I0117 11:23:00.591047 5628 MapDServer.cpp:655] Dynamic Watchdog is set to 0
I0117 11:23:00.591051 5628 MapDServer.cpp:660] Enable access priv check is set to 1
I0117 11:23:00.591056 5628 MapDServer.cpp:662] Debug Timer is set to 0
I0117 11:23:00.591061 5628 MapDServer.cpp:664] Maximum Idle session duration 60
I0117 11:23:00.591066 5628 MapDServer.cpp:666] Maximum active session duration 43200
I0117 11:23:00.591074 5628 MapDServer.cpp:702] cuda block size 0
I0117 11:23:00.591079 5628 MapDServer.cpp:703] cuda grid size 0
I0117 11:23:00.591084 5628 MapDServer.cpp:704] calcite JVM max memory 1024
I0117 11:23:00.591089 5628 MapDServer.cpp:705] MapD Server Port 9091
I0117 11:23:00.591094 5628 MapDServer.cpp:706] MapD Calcite Port 9093
I0117 11:23:00.591121 5628 MapDHandler.cpp:168] MapD Server 4.4.0dev-20181123-
E0117 11:23:00.591132 5628 MapDHandler.cpp:175] This build isn’t CUDA enabled, will run on CPU
I0117 11:23:00.591605 5628 DataMgr.cpp:124] cpuSlabSize is 4096M
I0117 11:23:00.591794 5628 FileMgr.cpp:182] Completed Reading table’s file metadata, Elapsed time : 0ms Epoch: 0 files read: 0 table location: ‘/opt/mapddata/data/mapd_data/table_0_0/’
I0117 11:23:00.591847 5628 Calcite.cpp:214] Creating Calcite Handler, Calcite Port is 9093 base data dir is /opt/mapddata/data
I0117 11:23:00.591856 5628 Calcite.cpp:140] Running calcite server as a daemon
I0117 11:23:00.908422 5628 Calcite.cpp:171] Calcite server start took 300 ms
I0117 11:23:00.908500 5628 Calcite.cpp:172] ping took 14 ms

NOTE: when we are restarting the mapd server

first it will start the process . status =( Active: active (running) since Thu 2019-01-17 11:26:31 IST; 325ms ago)

then the status will change to deactivating =( Active: deactivating (stop-sigterm) (Result: exit-code) since Thu 2019-01-17 11:26:34 IST; 13ms ago)

last status will change to failed ( Active: failed (Result: start-limit) since Thu 2019-01-17 11:26:35 IST; 2s ago).

#6

there isn’t any erro in the log, but the log you posted end before the start of calcite server, and before the software exit with signal 6.
Have you this entry in you log file?
I0117 11:09:03.506695 22140 MapDServer.cpp:113] Interrupt signal (6) received.

I reproduced the errore making the $MAPD_STORAGE/data/mapd_catalog/mapd SQlite’s file read-only, so i think there is something wrong with that file.

Maybe with the reboot something happened to your catalog file; check if the file exists and has the right permissions

#7

hai aznable,

thanks for your support. the issue got resolved. thanks a lot