Hi there! I’m struggling to get some basic functionality work with a table of ~60k columns. I’m running a CPU-only build.
I was able to create the table as expected with each of the columns, and then performed the
copy from successfully, great docs! The table looks something like this:
CREATE TABLE IF NOT EXISTS expressions ( sample TEXT ENCODING DICT, ENSG00000000003_14 DECIMAL(10), ENSG00000000005_5 DECIMAL(10), ENSG00000000419_12 DECIMAL(10), ... );
I can then use copy from to load my data and confirm they have loaded properly:
> select count(*) from expressions limit 1; EXPR$0 466
However, I have to be careful which SQL commands I execute as selecting all 60k columns, even for a single row, will lead to timeouts.
select * from expressions; just hangs indefinitely.
During execution I observed the memory summary and mapd’s resource consumption via top
mapdql> \memory_summary MapD Server Memory Usage CPU RAM IN BUFFER USE : 236.78 MB GPU VRAM USAGE (in MB's) GPU MAX ALLOC IN-USE FREE
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 14468 root 20 0 10.540g 1.641g 71628 S 100.0 0.7 51:52.15 mapd_server
I built mapd from source without GPU support. The system has a lot of resources available.
ubuntu@davidcs-mapd-aggregator-0:~$ free -m total used free shared buff/cache available Mem: 246008 14871 211033 24 20103 229818 Swap: 0 0 0
Have I set up my table properly for it to be queried? Is there a way to optimize for long rows? I might try reducing the fragment size and reloading the data. Thanks!