My h/w specs are dual intel xeon 8core CPUs, 8xGTX 1080Tis as GPU, 128GB of RAM and enough HDD space.
I started doing some loads into the DB. This was a 10column data table. Initially I started with 45mil records and that went thru without any issues. The load was done in small batches (5-10mil per batch). Once the load was done, I was able to do some analysis thru the UI.
Next I consolidated all my files (has about 280mil records) and did a load. This failed at the 90Mil mark. Another try with 120Mil also failed with a partial load in the DB. The error that I noticed is ‘not enough host memory’.
Any recommendations on load parameters to be set or should I go in smaller batches ? Also any other OS level parms that needed tweaking ?