Running SQL Editor queries in Immerse causes server to crash (Docker) - user error :)

robinkraft · 2017-11-07 19:01:40 UTC

Hi,

I’m running through the setup tutorial for CPU-only + Docker on a Mac (I’m waiting on my P2 instance). I can get queries to work via the CLI, but when I try via Immerse things start breaking. Interestingly, it only happens when I try something that interacts directly with my table (flights_2008_10k). Seems all it’s doing when it dies is trying to inspect the table, to check whether the column I’m referencing exists.

A bad sql statement gets caught properly, as does a statement referencing a non-existent table (see screenshot). When I try to select a non-existent column from the table, the spinner in the UI spins forever, and the server crashes (see stacktrace below). Something that should definitely work also crashes the server, e.g. SELECT * FROM flights_2008_10k LIMIT 1.

FWIW a similar crash occurs when I try to create a new dashboard and select my table in the UI.

Anyone else seeing this?

Thanks for any help!
-Robin

docker run -v $HOME/mapd-docker-storage:/mapd-storage -p 9090-9092:9090-9092 c0bd09088fad
Backend TCP:  localhost:9091
Backend HTTP: localhost:9090
Frontend Web: localhost:9092
Calcite TCP: localhost:9093
- sleeping for 5s while server starts
E1107 04:52:37.755173     6 MapDHandler.cpp:158] This build isn't CUDA enabled, will run on CPU
Thrift: Tue Nov  7 04:52:37 2017 TSocket::open() connect() <Host: localhost Port: 9093>Connection refused
Thrift: Tue Nov  7 04:52:37 2017 TSocket::open() connect() <Host: localhost Port: 9093>Connection refused

[SNIP - removed like 15 lines like the ones above & below]

Thrift: Tue Nov  7 04:52:39 2017 TSocket::open() connect() <Host: localhost Port: 9093>Connection refused
E1107 04:52:39.452378     6 MapDHandler.cpp:190] No GPUs detected, falling back to CPU mode
Navigate to: http://localhost:9092
F1107 04:53:01.123003    28 StringDictionary.cpp:50] Check failed: ptr != reinterpret_cast<void*>(-1)
*** Check failure stack trace: ***
    @           0xf6425a  google::LogMessage::Fail()
    @           0xf641b1  google::LogMessage::SendToLog()
    @           0xf63b7c  google::LogMessage::Flush()
    @           0xf66b15  google::LogMessageFatal::~LogMessageFatal()
    @          0x22fc1ac  (unknown)
    @          0x23002d4  StringDictionary::StringDictionary()
    @           0xba3d15  Catalog_Namespace::Catalog::getMetadataForDict()
    @           0xb6bef3  MapDHandler::populateThriftColumnType()
    @           0xb84459  MapDHandler::get_table_details_impl()
    @           0xac1273  MapDProcessor::process_get_table_details()
    @           0xabc4c7  MapDProcessor::dispatchCall()
    @           0xab56bc  apache::thrift::TDispatchProcessor::process()
    @          0x2347057  apache::thrift::server::TConnectedClient::run()
    @          0x233d26d  apache::thrift::server::TThreadedServer::TConnectedClientRunner::run()
    @          0x234152d  apache::thrift::concurrency::PthreadThread::threadMain()
    @     0x7f8569ff06ba  start_thread
    @     0x7f8569d263dd  clone
2017/11/07 04:53:01 http: proxy error: EOF
/mapd/startmapd: line 102:     6 Aborted                 ./bin/mapd_server $MAPD_DATA $RO --port $MAPD_TCP_PORT --http-port $MAPD_HTTP_PORT --calcite-port $MAPD_CALCITE_PORT $*
/mapd/startmapd: line 1: kill: (-5) - No such process

dwayneberry · 2017-11-07 06:00:50 UTC

Hi,

Unfortunately you are getting an issue when MapD is trying to memory map a file. Our error message is lacking here as we do not print out errno to help us diagnose the issue. The problem could be any number of things, ranging from out of memory, corrupted filehandle or out of system resources. I will create an issue to improve the error message here.

Are you saying you can access and query different tables successfully?

regards

dwayneberry · 2017-11-07 06:07:20 UTC

Hi

Please try the following series of command as a test

CREATE TABLE TEST1 (t1 TEXT);
INSERT INTO TEST1 VALUES('HELLO');
SELECT t1 from TEST1;

please include output of these steps

regards

robinkraft · 2017-11-07 06:08:59 UTC

Ah ok, bummer. To answer your question, I can’t do anything in the SQL editor that actually interacts with real table/columns - the server instantly crashes.

Maybe the Docker process doesn’t have enough memory to work with? How much memory does MapD expect to have access to? I can close a bunch of browser windows that are eating up RAM.

robinkraft · 2017-11-07 06:09:34 UTC

Create works, but insert fails:

F1107 06:07:56.081635    28 StringDictionary.cpp:50] Check failed: ptr != reinterpret_cast<void*>(-1)
*** Check failure stack trace: ***
    @           0xf6425a  google::LogMessage::Fail()
    @           0xf641b1  google::LogMessage::SendToLog()
    @           0xf63b7c  google::LogMessage::Flush()
    @           0xf66b15  google::LogMessageFatal::~LogMessageFatal()
    @          0x22fc1ac  (unknown)
    @          0x23002d4  StringDictionary::StringDictionary()
    @           0xba3d15  Catalog_Namespace::Catalog::getMetadataForDict()
    @           0xc64841  Executor::executeSimpleInsert()
    @           0xcba634  Executor::execute()
    @           0xb82d20  MapDHandler::execute_root_plan()
    @           0xb85d8e  MapDHandler::sql_execute_impl()
    @           0xb88159  MapDHandler::sql_execute()
    @           0xacc84d  MapDProcessor::process_sql_execute()
    @           0xabc4c7  MapDProcessor::dispatchCall()
    @           0xab56bc  apache::thrift::TDispatchProcessor::process()
    @          0x2347057  apache::thrift::server::TConnectedClient::run()
    @          0x233d26d  apache::thrift::server::TThreadedServer::TConnectedClientRunner::run()
    @          0x234152d  apache::thrift::concurrency::PthreadThread::threadMain()
    @     0x7fd975e0d6ba  start_thread
    @     0x7fd975b433dd  clone
2017/11/07 06:07:56 http: proxy error: EOF
/mapd/startmapd: line 102:     6 Aborted                 ./bin/mapd_server $MAPD_DATA $RO --port $MAPD_TCP_PORT --http-port $MAPD_HTTP_PORT --calcite-port $MAPD_CALCITE_PORT $*
/mapd/startmapd: line 1: kill: (-5) - No such process

robinkraft · 2017-11-07 06:41:28 UTC

There should be plenty of system resources available.

Can you replicate this @dwayneberry? I’ve got the latest version of Docker running on Mac OS X Sierra 10.12.6. Just followed the instructions for CPU-only + Docker. I’ll try starting from scratch with a new docker pull and try your insert command.

robinkraft · 2017-11-07 06:59:14 UTC

Ok, gotta call it quits for now. I started from scratch, deleted all the old images/containers, deleted mapd-docker-storage, pulled a new image, and tried your commands again.

CREATE TABLE TEST1 (t1 TEXT);
INSERT INTO TEST1 VALUES('HELLO');
SELECT t1 from TEST1;

Again, creating the table worked, but the insert failed (see stack trace). N.B. this is from the mapdql CLI bin/mqpdql - not using Immerse this time.

I’m really not sure what to make of this. Since no one else seems to have run into this issue, I’m inclined to think it’s some interaction between Mac & Docker. I’ll try it out on a different platform tomorrow.

F1107 06:53:22.896517    40 StringDictionary.cpp:50] Check failed: ptr != reinterpret_cast<void*>(-1)
*** Check failure stack trace: ***
    @           0xf6425a  google::LogMessage::Fail()
    @           0xf641b1  google::LogMessage::SendToLog()
    @           0xf63b7c  google::LogMessage::Flush()
    @           0xf66b15  google::LogMessageFatal::~LogMessageFatal()
    @          0x22fc1ac  (unknown)
    @          0x23002d4  StringDictionary::StringDictionary()
    @           0xba3d15  Catalog_Namespace::Catalog::getMetadataForDict()
    @           0xc64841  Executor::executeSimpleInsert()
    @           0xcba634  Executor::execute()
    @           0xb82d20  MapDHandler::execute_root_plan()
    @           0xb85d8e  MapDHandler::sql_execute_impl()
    @           0xb88159  MapDHandler::sql_execute()
    @           0xacc84d  MapDProcessor::process_sql_execute()
    @           0xabc4c7  MapDProcessor::dispatchCall()
    @           0xab56bc  apache::thrift::TDispatchProcessor::process()
    @          0x2347057  apache::thrift::server::TConnectedClient::run()
    @          0x233d26d  apache::thrift::server::TThreadedServer::TConnectedClientRunner::run()
    @          0x234152d  apache::thrift::concurrency::PthreadThread::threadMain()
    @     0x7f5029eca6ba  start_thread
    @     0x7f5029c003dd  clone
/mapd/startmapd: line 102:     8 Aborted                 ./bin/mapd_server $MAPD_DATA $RO --port $MAPD_TCP_PORT --http-port $MAPD_HTTP_PORT --calcite-port $MAPD_CALCITE_PORT $*
/mapd/startmapd: line 1: kill: (-5) - No such process

andrew · 2017-11-07 15:25:07 UTC

Hi Robin,

I haven’t been able to reproduce, at least not on 10.13 running Docker 17.09.0-ce.

Could you exec’ing df -h and free inside the container to see if there is anything strange going on there?

docker exec <container id> df -h
docker exec <container id> free

If need be we could quickly fix up the errno reporting and send you a build of master from the open source repo.

robinkraft · 2017-11-07 18:06:08 UTC

Glad to hear that you haven’t been able to reproduce! I’m going to restart now, and also try the docker approach on an EC2 box to see what happens.

Here’s what I’m getting from those commands on my Mac:

$ docker exec f1a1fcc1c9aa df -h
Filesystem      Size  Used Avail Use% Mounted on
none             18G  1.2G   16G   7% /
tmpfs            64M     0   64M   0% /dev
tmpfs           498M     0  498M   0% /sys/fs/cgroup
Users           465G  331G  134G  72% /mapd-storage
/dev/sda1        18G  1.2G   16G   7% /etc/hosts
shm              64M     0   64M   0% /dev/shm
tmpfs           498M     0  498M   0% /sys/firmware
$ 
$ docker exec f1a1fcc1c9aa free -h
              total        used        free      shared  buff/cache   available
Mem:           995M         85M        168M        189M        742M        580M
Swap:          1.1G          0B        1.1G

dwayneberry · 2017-11-07 18:17:04 UTC

HI,

It appears you only have 1G of RAM memory available for your docker image to execute in?

can you increase that?

regards

robinkraft · 2017-11-07 18:31:46 UTC

Same issue even with 4 gigs of total memory available to docker. But changing the memory settings made me realize that I’ve got Docker for Mac as well as Docker Toolbox installed. May not have any impact on this, but I should get that cleaned up before assuming there’s something wrong with MapD. I’ve been updating Docker for Mac, but my Docker Toolbox install could be old.

Looks like I also need to upgrade to 10.13.

Here’s docker info -

$ docker info
Containers: 12
 Running: 0
 Paused: 0
 Stopped: 12
Images: 1
Server Version: 17.10.0-ce
Storage Driver: aufs
 Root Dir: /mnt/sda1/var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 34
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 06b9cb35161009dcb7123345749fef02f7cea8e0
runc version: 0351df1c5a66838d0c392b4ac4cf9450de844e2d
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.4.93-boot2docker
Operating System: Boot2Docker 17.10.0-ce (TCL 7.2); HEAD : 34fe485 - Wed Oct 18 17:16:34 UTC 2017
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 3.858GiB
Name: default
ID: E2PW:2IPD:VKAC:T4HQ:RDIO:OJJN:QSKV:YXZL:ORR3:MOEC:O6US:DRNZ
Docker Root Dir: /mnt/sda1/var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 19
 Goroutines: 27
 System Time: 2017-11-07T18:27:24.401535455Z
 EventsListeners: 0
Registry: https://index.docker.io/v1/
Labels:
 provider=virtualbox
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

robinkraft · 2017-11-07 19:01:18 UTC

Ha! Purging docker toolbox, virtual box, and docker for mac and reinstalling only Docker for Mac did the trick. My bad