Build mapd-core failed


#1

hello everyone, I got problem when i installed mapd. I hope that someone can help me, thank you very much.

I have executed the commond
curl -OJ https://dependencies.mapd.com/mapd-deps/deploy.sh sudo bash deploy.sh
and then I activated my environment module
module load mapd-deps

after that,when i want to execute the file ‘mapd-core/scripts/mapd-deps-centos.sh’ i found that i could not download file from the ftp server ‘ftp.gnu.org’, so i want to install build mapd-core directly.

i ertered the folder ‘/mapd-core/build’ and executed the commond
cmake -DCMAKE_BUILD_TYPE=debug ..
after that, i got the response
– The C compiler identification is GNU 6.4.0
– The CXX compiler identification is GNU 6.4.0
– Check for working C compiler: /usr/local/mapd-deps/20180930/bin/gcc
– Check for working C compiler: /usr/local/mapd-deps/20180930/bin/gcc – works
– Detecting C compiler ABI info
– Detecting C compiler ABI info - done
– Detecting C compile features
– Detecting C compile features - done
– Check for working CXX compiler: /usr/local/mapd-deps/20180930/bin/g++
– Check for working CXX compiler: /usr/local/mapd-deps/20180930/bin/g++ – works
– Detecting CXX compiler ABI info
– Detecting CXX compiler ABI info - done
– Detecting CXX compile features
– Detecting CXX compile features - done
– Looking for pthread.h
– Looking for pthread.h - found
– Looking for pthread_create
– Looking for pthread_create - not found
– Looking for pthread_create in pthreads
– Looking for pthread_create in pthreads - not found
– Looking for pthread_create in pthread
– Looking for pthread_create in pthread - found
– Found Threads: TRUE
– Found CUDA: /usr/local/cuda (found version “8.0”)
– Found Thrift: /usr/local/mapd-deps/20180930/lib/libthrift.so
– Found Git: /usr/bin/git (found version “1.8.3.1”)
– Found Gflags: /usr/local/mapd-deps/20180930/lib/libgflags.a
– Found Glog: /usr/local/mapd-deps/20180930/lib/libglog.a
– Found ZLIB: /usr/lib64/libz.so (found version “1.2.7”)
– Found PNG: /usr/lib64/libpng.so (found version “1.5.13”)
– Found GDAL: /usr/local/mapd-deps/20180930/lib/libgdal.so
– Found Folly: /usr/local/mapd-deps/20180930/lib/libfolly.a
– Found Curses: /usr/lib64/libcurses.so
– Found Arrow: /usr/local/mapd-deps/20180930/lib64/libarrow.so
– Looking for clock_gettime in c
– Looking for clock_gettime in c - found
– Looking for clock_gettime in rt
– Looking for clock_gettime in rt - found
– Found LibAwsS3: /usr/local/mapd-deps/20180930/include
– Looking for bison++
– Looking for bison++ – /usr/local/mapd-deps/20180930/bin/bison++
– Looking for flex++
– Looking for flex++ – /usr/bin/flex++
– Found LibArchive: /usr/local/mapd-deps/20180930/lib/libarchive.a
– Found Doxygen: /usr/bin/doxygen (found version “1.8.5”) found components: doxygen dot
– Configuring done
– Generating done
– Build files have been written to: /root/chenhui21/mapd-core/build

i thought that it worked well, and then executed the commond
make -j $(nproc)
after that, i got the error
[ 0%] Generating gen-cpp/CalciteServer.cpp, gen-cpp/calciteserver_constants.cpp, gen-cpp/calciteserver_types.cpp, gen-cpp/completion_hints_constants.cpp, gen-cpp/completion_hints_types.cpp
[ 0%] Generating gen-cpp/MapD.cpp, gen-cpp/MapD.h, gen-cpp/mapd_constants.cpp, gen-cpp/mapd_types.cpp, gen-cpp/completion_hints_constants.cpp, gen-cpp/completion_hints_types.cpp
Scanning dependencies of target ParserFiles
Scanning dependencies of target rerun_cmake
Scanning dependencies of target gdal-data
[ 1%] Generating gen-cpp/MapD.cpp, gen-cpp/mapd_constants.cpp, gen-cpp/mapd_types.cpp, gen-cpp/completion_hints_constants.cpp, gen-cpp/completion_hints_types.cpp
Scanning dependencies of target ProfileUtils
Scanning dependencies of target generate_cert
Scanning dependencies of target mapd_java_components
Scanning dependencies of target ScannerFiles
Scanning dependencies of target mapd_web_server
Scanning dependencies of target geo_samples
Scanning dependencies of target linenoise
Scanning dependencies of target bcrypt
[ 1%] Generating Parser.cpp
[ 2%] Generating libProfileUtils.so
[ 2%] Generating Scanner.cpp
[ 2%] Generating …/java/thrift/src/gen/com/mapd/thrift/server/MapD.java, …/java/thrift/src/gen/com/mapd/thrift/server/TRow.java
[ 2%] Generating …/java/thrift/src/gen/com/mapd/thrift/calciteserver/CalciteServer.java
[ 2%] Building CXX object ThirdParty/linenoise/CMakeFiles/linenoise.dir/linenoise.cpp.o
/bin/sh: nvcc: command not found
make[2]: *** [Tests/libProfileUtils.so] Error 127
/root/chenhui21/mapd-core/Parser/parser.y contains 1 useless nonterminal and 2 useless rules.
make[1]: *** [Tests/CMakeFiles/ProfileUtils.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs…
/root/chenhui21/mapd-core/Parser/parser.y contains 5 shift/reduce conflicts and 4 reduce/reduce conflicts.
[ 3%] Generating bin/mapd_web_server
[ 3%] Building C object ThirdParty/bcrypt/CMakeFiles/bcrypt.dir/crypt_blowfish/crypt_blowfish.c.o
[ 3%] Building C object ThirdParty/bcrypt/CMakeFiles/bcrypt.dir/crypt_blowfish/crypt_gensalt.c.o
[ 4%] Building C object ThirdParty/bcrypt/CMakeFiles/bcrypt.dir/crypt_blowfish/wrapper.c.o
[ 4%] Built target geo_samples
[ 4%] Building C object ThirdParty/bcrypt/CMakeFiles/bcrypt.dir/bcrypt.c.o
Scanning dependencies of target gtest
[ 4%] Built target gdal-data
Scanning dependencies of target CudaMgr
[ 5%] Built target ParserFiles
[ 5%] Building CXX object ThirdParty/googletest/CMakeFiles/gtest.dir/gmock-gtest-all.cc.o
[ 6%] Building CXX object CudaMgr/CMakeFiles/CudaMgr.dir/CudaMgr.cpp.o
Scanning dependencies of target sqlite3
[ 7%] Built target ScannerFiles
[ 8%] Generating bin/mapd-1.0-SNAPSHOT-jar-with-dependencies.jar, bin/mapdjdbc-4.2.0-SNAPSHOT.jar, bin/calcite-1.0-SNAPSHOT-jar-with-dependencies.jar
[ 8%] Linking C static library libbcrypt.a
[ 8%] Linking CXX static library liblinenoise.a
[ 8%] Built target bcrypt
[ 8%] Built target linenoise
Scanning dependencies of target Shared
Scanning dependencies of target calciteserver_thrift
Scanning dependencies of target mapd_thrift
Scanning dependencies of target mapd_sample_thrift
Scanning dependencies of target rdkafka
[ 8%] Building CXX object Shared/CMakeFiles/Shared.dir/Datum.cpp.o
[ 9%] Building CXX object Shared/CMakeFiles/Shared.dir/mapd_glob.cpp.o
[ 9%] Building CXX object Calcite/CMakeFiles/calciteserver_thrift.dir/gen-cpp/CalciteServer.cpp.o
[ 10%] Building CXX object Shared/CMakeFiles/Shared.dir/timegm.cpp.o
[ 10%] Building CXX object Shared/CMakeFiles/Shared.dir/StringTransform.cpp.o
[ 11%] Building CXX object Calcite/CMakeFiles/calciteserver_thrift.dir/gen-cpp/completion_hints_constants.cpp.o
[ 11%] Building CXX object Calcite/CMakeFiles/calciteserver_thrift.dir/gen-cpp/calciteserver_constants.cpp.o
[ 11%] Building CXX object Calcite/CMakeFiles/calciteserver_thrift.dir/gen-cpp/completion_hints_types.cpp.o
[ 12%] Building CXX object Shared/CMakeFiles/Shared.dir/geo_types.cpp.o
[ 12%] Building C object ThirdParty/librdkafka/src/CMakeFiles/rdkafka.dir/rdaddr.c.o
[ 13%] Building C object ThirdParty/librdkafka/src/CMakeFiles/rdkafka.dir/rdavl.c.o
[ 14%] Building CXX object CMakeFiles/mapd_thrift.dir/gen-cpp/MapD.cpp.o
Scanning dependencies of target Utils
[ 14%] Building CXX object SampleCode/CMakeFiles/mapd_sample_thrift.dir/gen-cpp/MapD.cpp.o
Scanning dependencies of target RowToColumn
Scanning dependencies of target Fragmenter
[ 14%] Building CXX object Utils/CMakeFiles/Utils.dir/StringLike.cpp.o
[ 15%] Building CXX object Utils/CMakeFiles/Utils.dir/Regexp.cpp.o
[ 15%] Building CXX object Import/CMakeFiles/RowToColumn.dir/RowToColumnLoader.cpp.o
[ 15%] Building CXX object Fragmenter/CMakeFiles/Fragmenter.dir/InsertOrderFragmenter.cpp.o
[ 15%] Building C object ThirdParty/librdkafka/src/CMakeFiles/rdkafka.dir/rdkafka.c.o
[ 15%] Built target generate_cert
[ 16%] Building C object ThirdParty/librdkafka/src/CMakeFiles/rdkafka.dir/rdkafka_assignor.c.o
[ 16%] Building CXX object SampleCode/CMakeFiles/mapd_sample_thrift.dir/gen-cpp/mapd_constants.cpp.o
[ 16%] Building CXX object Fragmenter/CMakeFiles/Fragmenter.dir/UpdelStorage.cpp.o
[ 16%] Building CXX object CMakeFiles/mapd_thrift.dir/gen-cpp/mapd_constants.cpp.o
[ 16%] Linking CXX static library libCudaMgr.a
[ 16%] Built target CudaMgr
[ 16%] Building CXX object CMakeFiles/mapd_thrift.dir/gen-cpp/mapd_types.cpp.o
[ 16%] Building CXX object Utils/CMakeFiles/Utils.dir/ChunkIter.cpp.o
[ 16%] Building C object ThirdParty/librdkafka/src/CMakeFiles/rdkafka.dir/rdkafka_broker.c.o
[ 17%] Building CXX object SampleCode/CMakeFiles/mapd_sample_thrift.dir/gen-cpp/mapd_types.cpp.o
[ 17%] Building C object ThirdParty/librdkafka/src/CMakeFiles/rdkafka.dir/rdkafka_buf.c.o
[ 18%] Building CXX object CMakeFiles/mapd_thrift.dir/gen-cpp/completion_hints_constants.cpp.o
[ 19%] Building C object ThirdParty/librdkafka/src/CMakeFiles/rdkafka.dir/rdkafka_cgrp.c.o
[ 19%] Building C object ThirdParty/librdkafka/src/CMakeFiles/rdkafka.dir/rdkafka_conf.c.o
[ 19%] Built target mapd_web_server
[ 19%] Building C object ThirdParty/librdkafka/src/CMakeFiles/rdkafka.dir/rdkafka_event.c.o
[ 19%] Building CXX object CMakeFiles/mapd_thrift.dir/gen-cpp/completion_hints_types.cpp.o
[ 19%] Building CXX object Utils/CMakeFiles/Utils.dir/ChunkAccessorTable.cpp.o
[ 19%] Building CXX object SampleCode/CMakeFiles/mapd_sample_thrift.dir/gen-cpp/completion_hints_constants.cpp.o
[ 19%] Building CXX object SampleCode/CMakeFiles/mapd_sample_thrift.dir/gen-cpp/completion_hints_types.cpp.o
[ 20%] Building C object ThirdParty/librdkafka/src/CMakeFiles/rdkafka.dir/rdkafka_feature.c.o
[ 20%] Building C object ThirdParty/librdkafka/src/CMakeFiles/rdkafka.dir/rdkafka_metadata.c.o
[ 20%] Building C object ThirdParty/librdkafka/src/CMakeFiles/rdkafka.dir/rdkafka_metadata_cache.c.o
[ 21%] Building C object ThirdParty/librdkafka/src/CMakeFiles/rdkafka.dir/rdkafka_msg.c.o
[ 21%] Building C object ThirdParty/librdkafka/src/CMakeFiles/rdkafka.dir/rdkafka_offset.c.o
[ 21%] Building C object ThirdParty/librdkafka/src/CMakeFiles/rdkafka.dir/rdkafka_op.c.o
[ 21%] Building C object ThirdParty/librdkafka/src/CMakeFiles/rdkafka.dir/rdkafka_partition.c.o
[ 22%] Building C object ThirdParty/librdkafka/src/CMakeFiles/rdkafka.dir/rdkafka_pattern.c.o
[ 22%] Building C object ThirdParty/librdkafka/src/CMakeFiles/rdkafka.dir/rdkafka_queue.c.o
[ 22%] Linking CXX static library libcalciteserver_thrift.a
[ 22%] Building C object ThirdParty/librdkafka/src/CMakeFiles/rdkafka.dir/rdkafka_range_assignor.c.o
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-install-plugin:2.3.1:install (default-install) on project parent: Execution default-install of goal org.apache.maven.plugins:maven-install-plugin:2.3.1:install failed: A required class was missing while executing org.apache.maven.plugins:maven-install-plugin:2.3.1:install: org/codehaus/plexus/digest/DigesterException
[ERROR] -----------------------------------------------------
[ERROR] realm = plugin>org.apache.maven.plugins:maven-install-plugin:2.3.1
[ERROR] strategy = org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy
[ERROR] urls[0] = file:/root/.m2/repository/org/apache/maven/plugins/maven-install-plugin/2.3.1/maven-install-plugin-2.3.1.jar
[ERROR] urls[1] = file:/root/.m2/repository/org/codehaus/plexus/plexus-utils/1.1/plexus-utils-1.1.jar
[ERROR] Number of foreign imports: 1
[ERROR] import: Entry[import from realm ClassRealm[maven.api, parent: null]]
[ERROR]
[ERROR] -----------------------------------------------------: org.codehaus.plexus.digest.DigesterException
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginContainerException
make[2]: *** [bin/mapd-1.0-SNAPSHOT-jar-with-dependencies.jar] Error 1
make[1]: *** [CMakeFiles/mapd_java_components.dir/all] Error 2
[ 23%] Building C object ThirdParty/librdkafka/src/CMakeFiles/rdkafka.dir/rdkafka_request.c.o
[ 23%] Building C object ThirdParty/librdkafka/src/CMakeFiles/rdkafka.dir/rdkafka_roundrobin_assignor.c.o
[ 23%] Building C object ThirdParty/librdkafka/src/CMakeFiles/rdkafka.dir/rdkafka_subscription.c.o
[ 23%] Built target calciteserver_thrift
[ 24%] Building C object ThirdParty/librdkafka/src/CMakeFiles/rdkafka.dir/rdkafka_timer.c.o
[ 24%] Building C object ThirdParty/librdkafka/src/CMakeFiles/rdkafka.dir/rdkafka_transport.c.o
[ 24%] Building C object ThirdParty/librdkafka/src/CMakeFiles/rdkafka.dir/rdkafka_topic.c.o
[ 25%] Building C object ThirdParty/librdkafka/src/CMakeFiles/rdkafka.dir/rdlist.c.o
[ 25%] Building C object ThirdParty/librdkafka/src/CMakeFiles/rdkafka.dir/rdlog.c.o
[ 25%] Building C object ThirdParty/librdkafka/src/CMakeFiles/rdkafka.dir/rdports.c.o
[ 25%] Building C object ThirdParty/librdkafka/src/CMakeFiles/rdkafka.dir/rdrand.c.o
[ 26%] Building C object ThirdParty/librdkafka/src/CMakeFiles/rdkafka.dir/rdregex.c.o
[ 26%] Building C object ThirdParty/librdkafka/src/CMakeFiles/rdkafka.dir/rdstring.c.o
[ 26%] Building C object ThirdParty/librdkafka/src/CMakeFiles/rdkafka.dir/snappy.c.o
[ 27%] Building C object ThirdParty/librdkafka/src/CMakeFiles/rdkafka.dir/tinycthread.c.o
[ 27%] Building C object ThirdParty/librdkafka/src/CMakeFiles/rdkafka.dir/xxhash.c.o
[ 27%] Building C object ThirdParty/librdkafka/src/CMakeFiles/rdkafka.dir/lz4.c.o
[ 28%] Building C object ThirdParty/librdkafka/src/CMakeFiles/rdkafka.dir/lz4frame.c.o
[ 28%] Building C object ThirdParty/librdkafka/src/CMakeFiles/rdkafka.dir/lz4hc.c.o
[ 28%] Building C object ThirdParty/librdkafka/src/CMakeFiles/rdkafka.dir/rdgz.c.o
[ 28%] Linking CXX static library libShared.a
[ 28%] Built target Shared
[ 29%] Linking C static library librdkafka.a
[ 29%] Built target rdkafka
– Configuring done
[ 29%] Linking C static library libsqlite3.a
[ 29%] Built target sqlite3
– Generating done
– Build files have been written to: /root/chenhui21/mapd-core/build
[ 29%] Built target rerun_cmake
[ 29%] Linking CXX static library libgtest.a
[ 29%] Built target gtest
[ 30%] Linking CXX static library libUtils.a
[ 30%] Built target Utils
[ 31%] Linking CXX static library libFragmenter.a
[ 31%] Built target Fragmenter
[ 31%] Linking CXX static library libmapd_sample_thrift.a
[ 31%] Built target mapd_sample_thrift
[ 31%] Linking CXX static library libmapd_thrift.a
[ 31%] Built target mapd_thrift
[ 31%] Linking CXX static library libRowToColumn.a
[ 31%] Built target RowToColumn
make: *** [all] Error 2

I really do not know how to deal with the problom. Can you help me? Thank you very much.


#2

Hi @hchen

This line says it can’t find the CUDA compiler:

/bin/sh: nvcc: command not found

Are you planning on running on GPUs or will you be running CPU-only? If the latter, add -DENABLE_CUDA=off when running cmake:

mkdir build && cd build
cmake -DENABLE_CUDA=off ..
make -j 4

Not sure about the Java/maven errors, but let’s first get past the CUDA issues.

(Looks like for we’re calling nvcc directly instead of using the path to it that’s found by cmake. Will look into fixing this.)


#3

@andrew

He has cuda libraries installed so probably is targeting a GPU build.

@hchen

It’s likely you have to add in the PATH variable the entry /usr/local/cuda/bin and probably also the entry /usr/local/cuda/lib64 on LD_LIBRARY_PATH


#4

Thank you for your answer. I just want to run mapd on GPUs.
I added the path of nvcc to environment variable and eventually i could find the path to nvcc.
But a problem about bison occurs
/root/XXX/mapd-core/Parser/parser.y contains 1 useless nonterminal and 2 useless rules /root/XXX/mapd-core/Parser/parser.y contains 5 shift/reduce confilcts and 4 reduce/reduce conflicts
And the problem about maven is still exists. Thank you.


#5

Thank you for your answer, i solved the problem of nvcc with your help. Thank you very much.


#6

In addition, there is one more thing about maven to say. when the first time i wanted to build mapd-core, there was an error Failed to parse plugin descriptor for org.apache.maven.plugins:maven-clean-plugin:2.4.1(root/.m2/respository/org/apache/maven/plugins/maven-clen-plugin/2.4.1/maven-clean-plugin-2.4.1.jar):error in opening zip file. I entered the folder and found maven-clean-plugin-2.4.1.jar and maven-clean-plugin-2.4.1.pom, they had the same size and the same content. So i thought that there were errors about these two files, and i downloaded files from http://repo1.maven.org/maven2/org/apache/maven/plugins/maven-clean-plugin/2.4.1/,then i replaced the original files.
Similary, i replaced the file maven-install-plugin-2.3.1.jar and maven-install-plugin-2.3.1.pom. Then i rebuild mapd-core, and the problem above arose. I do not know the right way to solve the problem.