Curl Error on make

I just downloaded the source code from github and ran the mapd-deps-prebuilt.sh to download the dependencies folder. As far as I can tell I have downloaded all the dependencies.

my cmake command runs fine

cmake -DCMAKE_BUILD_TYPE=debug ..

but when i run

make -j 4

I get the following errors. Can anybody shed some light on why this might be the case or how I can debug this issue.
fatal: not a git repository (or any of the parent directories): .git
– Could NOT find Doxygen (missing: DOXYGEN_EXECUTABLE)
/usr/bin/ld: /usr/local/mapd-deps/lib/libarrow.so: undefined reference to curl_easy_getinfo@CURL_OPENSSL_4' /usr/bin/ld: /usr/local/mapd-deps/lib/libarrow.so: undefined reference to curl_global_init@CURL_OPENSSL_4’
/usr/bin/ld: /usr/local/mapd-deps/lib/libarrow.so: undefined reference to curl_easy_perform@CURL_OPENSSL_4' /usr/bin/ld: /usr/local/mapd-deps/lib/libarrow.so: undefined reference to curl_slist_free_all@CURL_OPENSSL_4’
/usr/bin/ld: /usr/local/mapd-deps/lib/libarrow.so: undefined reference to curl_easy_strerror@CURL_OPENSSL_4' /usr/bin/ld: /usr/local/mapd-deps/lib/libarrow.so: undefined reference to curl_easy_setopt@CURL_OPENSSL_4’
/usr/bin/ld: /usr/local/mapd-deps/lib/libarrow.so: undefined reference to curl_easy_init@CURL_OPENSSL_4' /usr/bin/ld: /usr/local/mapd-deps/lib/libarrow.so: undefined reference to curl_slist_append@CURL_OPENSSL_4’
/usr/bin/ld: /usr/local/mapd-deps/lib/libarrow.so: undefined reference to curl_easy_cleanup@CURL_OPENSSL_4' /usr/bin/ld: /usr/local/mapd-deps/lib/libarrow.so: undefined reference to curl_version_info@CURL_OPENSSL_4’
/usr/bin/ld: /usr/local/mapd-deps/lib/libarrow.so: undefined reference to curl_easy_reset@CURL_OPENSSL_4' /usr/bin/ld: /usr/local/mapd-deps/lib/libarrow.so: undefined reference to curl_global_cleanup@CURL_OPENSSL_4’
collect2: error: ld returned 1 exit status
make[2]: *** [Tests/CMakeFiles/ArrowIpcIntegrationTest.dir/build.make:137: Tests/ArrowIpcIntegrationTest] Error 1
make[1]: *** [CMakeFiles/Makefile2:2796: Tests/CMakeFiles/ArrowIpcIntegrationTest.dir/all] Error

Which environment are you building the source in? I got a similar issue (mismatch of curl version) on Ubuntu 16.04 but it’s unlikely that is the case.

If the problem is circumscribed to arrow tests only, you can use the enable_test switch in CMake to skip the generation of unit tests for the database

I am on Ubuntu 20.04.
I will try skipping unit tests as a temporary workaround.

how did you identify the curl version mismatch? when I type curl --version I get “curl 7.71.1”
but I’m not sure if the code is looking at the /usr/local/mapd-deps for curl or another previously installed version.

Do you have any hints on how I can start debugging this issue?
i.e. what part of the codebase contains the import of the library or tests

Thank you for replying to my question.

Hi @asaeed,

The problem is happening in the linking phase, so to get the command that’s failing, I suggest running the make with just 1 worker using the verbose switch turned to 1.
something like
make -j 1 VERBOSE=1

I talked about a mismatch because of the undefined reference errors, so the libarrow was compiled using a different libcurl-openssl, or the library isn’t found at all.
You can check which library the libarrow.so is looking for using the ldd command.

in the case of my ubuntu 18.04 installation

candido@zion-legion:/usr/local/mapd-deps/lib$ ldd libarrow.so | grep curl
libcurl.so.4 => /usr/lib/x86_64-linux-gnu/libcurl.s/usr/lib/x86_64-linux-gnu/libcurl.so.4o.4 (0x00007fd30877b000)
candido@zion-legion:/usr/local/mapd-deps/lib$ ldd libarrow.so | grep ssl
libssl.so.1.1 => /usr/lib/x86_64-linux-gnu/libssl.so.1.1 (0x00007ff68825a000)

checking the libcurl with nm command

candido@zion-legion:/usr/local/mapd-deps/lib$ nm -D /usr/lib/x86_64-linux-gnu/libcurl.so.4 | grep curl_easy
000000000002c450 T curl_easy_cleanup
000000000002c540 T curl_easy_duphandle
0000000000024b10 T curl_easy_escape
000000000002c4e0 T curl_easy_getinfo
000000000002c150 T curl_easy_init
000000000002c960 T curl_easy_pause
000000000002c1c0 T curl_easy_perform
000000000002cae0 T curl_easy_recv
000000000002c860 T curl_easy_reset
000000000002cb70 T curl_easy_send
0000000000057c00 T curl_easy_setopt
0000000000038330 T curl_easy_strerror
0000000000024df0 T curl_easy_unescape

Anyway I reproduced the error; it looks you are missing the libcurl.so.4 (or it’s not in the path expected by libarrow.so)

you should check and eventually install libcurl4-openssl-dev. For reference in my system the version installed is 7.58.0-2ubuntu3.12

please let me know if the problem is solved

Regards,
Candido

Thank you for your detailed instructions i really appreciated it. The problem is unfortunately still unresolved.

I tried to install libcurl4-openssl-dev and got the following message
libcurl4-openssl-dev is already the newest version (7.68.0-1ubuntu2.4).

I started again and ran CMake and whilst it did complete generation, i got lots of the following errors, now sure if i should worry about them

CMake Warning at CMakeLists.txt:670 (add_executable):
Cannot generate a safe runtime search path for target initdb because there
is a cycle in the constraint graph:

dir 0 is [/home/arif/anaconda3/lib]
  dir 1 must precede it due to runtime library [libtbb.so]
dir 1 is [/usr/local/mapd-deps/lib]
  dir 0 must precede it due to runtime library [libblosc.so.1]

Some of these libraries may not be found correctly.

I then ran make with the verbose setting, and the code failed with the following error

/usr/bin/ld: /usr/local/mapd-deps/lib/libgdal.so: undefined reference to `curl_multi_cleanup@CURL_OPENSSL_4’

So i ran the the ld command on both libgdal and arrow libraries

ldd libgdal.so | grep curl
ldd libarrow.so | grep curl

they gave the following locations respecttively.

libcurl.so.4 => /lib/x86_64-linux-gnu/libcurl.so.4 (0x00007f0ebcdad000)
libcurl.so.4 => /lib/x86_64-linux-gnu/libcurl.so.4 (0x00007ff142471000)

I then ran the following command

nm -D /usr/lib/x86_64-linux-gnu/libcurl.so.4 | grep curl_easy

I got the following

0000000000030e30 T curl_easy_cleanup
0000000000030f30 T curl_easy_duphandle
0000000000029140 T curl_easy_escape
0000000000030ed0 T curl_easy_getinfo
0000000000030b90 T curl_easy_init
00000000000313e0 T curl_easy_pause
0000000000030c00 T curl_easy_perform
00000000000315a0 T curl_easy_recv
00000000000312c0 T curl_easy_reset
0000000000031650 T curl_easy_send
000000000005d760 T curl_easy_setopt
000000000003d0f0 T curl_easy_strerror
0000000000029460 T curl_easy_unescape
0000000000031730 T curl_easy_upkeep

below is the command that was running immediately before it failed. I’m not sure if this is relevant or not but there is no mention of libcurl.so.4 in the list below, and the only mention of curl is
/home/arif/anaconda3/lib/libcurl.so
Could this be the mismatch mentioned could exist? If it is, I’m not sure what I need to do to fix the problem.

[ 60%] Linking CXX executable bin/initdb
/usr/local/mapd-deps/bin/cmake -E cmake_link_script CMakeFiles/initdb.dir/link.txt --verbose=1
/usr/bin/c++ -Wall -Wno-unused-local-typedefs -fdiagnostics-color=auto -D__STDC_LIMIT_MACROS -D__STDC_CONSTANT_MACROS -pthread -g -Wl,–export-dynamic -rdynamic CMakeFiles/initdb.dir/initdb.cpp.o -o bin/initdb -L/home/arif/anaconda3/lib -L/usr/local/mapd-deps/lib -Wl,-rpath,/usr/local/mapd-deps/lib:/home/arif/anaconda3/lib: libmapd_thrift.a DataMgr/libDataMgr.a OSDependent/Unix/libOSDependent.a Shared/libShared.a Catalog/libCatalog.a SqliteConnector/libSqliteConnector.a MigrationMgr/libMigrationMgr.a TableArchiver/libTableArchiver.a Parser/libParser.a Analyzer/libAnalyzer.a ImportExport/libImportExport.a QueryRunner/libQueryRunner.a QueryEngine/libQueryEngine.a ThriftHandler/libQueryState.a LockMgr/libLockMgr.a DataMgr/libDataMgr.a Fragmenter/libFragmenter.a Logger/libLogger.a Geospatial/libGeospatial.a Calcite/libCalcite.a /usr/local/mapd-deps/lib/libarrow_cuda.so /usr/local/mapd-deps/lib/libaws-cpp-sdk-s3.a /usr/local/mapd-deps/lib/libaws-cpp-sdk-identity-management.a /usr/local/mapd-deps/lib/libaws-cpp-sdk-cognito-identity.a /usr/local/mapd-deps/lib/libaws-cpp-sdk-core.a /usr/local/mapd-deps/lib/libaws-c-event-stream.a /usr/local/mapd-deps/lib/libaws-c-common.a /usr/local/mapd-deps/lib/libaws-checksums.a /usr/local/mapd-deps/lib/libaws-cpp-sdk-sts.a /home/arif/anaconda3/lib/libcurl.so /home/arif/anaconda3/lib/libssl.so /home/arif/anaconda3/lib/libcrypto.so /usr/local/mapd-deps/lib/libarrow.so /usr/lib/x86_64-linux-gnu/libdouble-conversion.so /home/arif/anaconda3/lib/libsnappy.so -ldl /usr/local/mapd-deps/lib/libfolly.so /usr/lib/x86_64-linux-gnu/libdouble-conversion.so /usr/local/mapd-deps/lib/libfmt.a /home/arif/anaconda3/lib/libssl.so /home/arif/anaconda3/lib/libcrypto.so -ldl /usr/lib/x86_64-linux-gnu/libunwind.so /usr/lib/x86_64-linux-gnu/libglog.so /usr/lib/x86_64-linux-gnu/libgflags.so -ltbb /usr/lib/x86_64-linux-gnu/libboost_log.so /usr/lib/x86_64-linux-gnu/libboost_log_setup.so /usr/lib/x86_64-linux-gnu/libboost_filesystem.so /usr/lib/x86_64-linux-gnu/libboost_program_options.so /usr/lib/x86_64-linux-gnu/libboost_regex.so /usr/lib/x86_64-linux-gnu/libboost_system.so /usr/lib/x86_64-linux-gnu/libboost_thread.so /usr/lib/x86_64-linux-gnu/libboost_timer.so /usr/lib/x86_64-linux-gnu/libboost_locale.so /usr/lib/x86_64-linux-gnu/libboost_iostreams.so -ldl /usr/local/cuda/lib64/libcudart_static.a -lpthread -ldl /usr/lib/x86_64-linux-gnu/librt.so /usr/lib/x86_64-linux-gnu/libcuda.so /home/arif/anaconda3/lib/libz.so TableArchiver/libTableArchiver.a Parser/libParser.a TableArchiver/libTableArchiver.a Parser/libParser.a LockMgr/libLockMgr.a Parser/libParserGenerated.a ImportExport/libImportExport.a libmapd_thrift.a DataMgr/libDataMgr.a CudaMgr/libCudaMgr.a /usr/local/cuda/lib64/libcudart_static.a -lpthread /usr/lib/x86_64-linux-gnu/librt.so /usr/lib/x86_64-linux-gnu/libcuda.so /home/arif/anaconda3/lib/libarchive.so /home/arif/anaconda3/lib/libbz2.so /home/arif/anaconda3/lib/liblzma.so /usr/local/mapd-deps/lib/libparquet.so Catalog/libCatalog.a MigrationMgr/libMigrationMgr.a Analyzer/libAnalyzer.a QueryEngine/libQueryEngine.a Calcite/libCalcite.a Catalog/libCatalog.a MigrationMgr/libMigrationMgr.a Analyzer/libAnalyzer.a QueryEngine/libQueryEngine.a Calcite/libCalcite.a SqliteConnector/libSqliteConnector.a Fragmenter/libFragmenter.a ThirdParty/bcrypt/libbcrypt.a /usr/local/mapd-deps/lib/libarrow_cuda.so /usr/local/mapd-deps/lib/libaws-cpp-sdk-s3.a /usr/local/mapd-deps/lib/libaws-cpp-sdk-identity-management.a /usr/local/mapd-deps/lib/libaws-cpp-sdk-cognito-identity.a /usr/local/mapd-deps/lib/libaws-cpp-sdk-core.a /usr/local/mapd-deps/lib/libaws-c-event-stream.a /usr/local/mapd-deps/lib/libaws-c-common.a /usr/local/mapd-deps/lib/libaws-checksums.a /usr/local/mapd-deps/lib/libaws-cpp-sdk-sts.a /usr/local/mapd-deps/lib/libarrow.so /home/arif/anaconda3/lib/libsnappy.so StringDictionary/libStringDictionary.a /usr/local/mapd-deps/lib/libfolly.so /usr/lib/x86_64-linux-gnu/libdouble-conversion.so /usr/local/mapd-deps/lib/libfmt.a /usr/lib/x86_64-linux-gnu/libunwind.so /usr/lib/x86_64-linux-gnu/libglog.so /usr/lib/x86_64-linux-gnu/libgflags.so -ltbb Shared/libThriftClient.a ThirdParty/sqlite3/libsqlite3.a -ldl Utils/libUtils.a Shared/libShared.a Logger/libLogger.a /home/arif/anaconda3/lib/libblosc.so /usr/local/mapd-deps/lib/libclang-cpp.so /usr/local/mapd-deps/lib/libLLVM.so Calcite/libcalciteserver_thrift.a /usr/local/mapd-deps/lib/libthrift.so ThriftHandler/libQueryState.a /usr/lib/x86_64-linux-gnu/libboost_log.so /usr/lib/x86_64-linux-gnu/libboost_log_setup.so /usr/lib/x86_64-linux-gnu/libboost_filesystem.so /usr/lib/x86_64-linux-gnu/libboost_program_options.so /usr/lib/x86_64-linux-gnu/libboost_regex.so /usr/lib/x86_64-linux-gnu/libboost_system.so /usr/lib/x86_64-linux-gnu/libboost_thread.so /usr/lib/x86_64-linux-gnu/libboost_timer.so /usr/lib/x86_64-linux-gnu/libboost_locale.so /usr/lib/x86_64-linux-gnu/libboost_iostreams.so OSDependent/Unix/libOSDependent.a /usr/local/mapd-deps/lib/libgdal.so /home/arif/anaconda3/lib/libpng.so /home/arif/anaconda3/lib/libcurl.so /home/arif/anaconda3/lib/libssl.so /home/arif/anaconda3/lib/libcrypto.so /home/arif/anaconda3/lib/libz.so
/usr/bin/ld: /usr/local/mapd-deps/lib/libgdal.so: undefined reference to `curl_multi_cleanup@CURL_OPENSSL_4’

hi @asaeed,

I’m sorry for the not so fast reply because someone marked our message as SPAM for whatever reason.

without the generation of TEST, the error message has changed and it looks like the paths of miniconda eviroment are interfering with the mapd build environment; so probably is better to create an ad-hoc user for the build environment (or at least unset miniconda path before run cmake)

if you look at the verbose compilation command some of libraries of the anconda environment are being used, so the errors

e.g.
gnu/libboost_locale.so /usr/lib/x86_64-linux-gnu/libboost_iostreams.so OSDependent/Unix/libOSDependent.a /usr/local/mapd-deps/lib/libgdal.so /home/arif/anaconda3/lib/libpng.so /home/arif/anaconda3/lib/libcurl.so /home/arif/anaconda3/lib/libssl.so /home/arif/anaconda3/lib/libcrypto.so /home/arif/anaconda3/lib/libz.so
/usr/bin/ld: /usr/local/mapd-deps/lib/libgdal.so: undefined reference to `curl_multi_cleanup@CURL_OPENSSL_4’

Hopes this helps,
Candido