Discussions

Expand all | Collapse all

MapD Community Edition: Data Compressions

  • 1.  MapD Community Edition: Data Compressions

    Posted 11-27-2017 19:55

    What level/kind of compressions are allowed in mapD ce ?



  • 2.  RE: MapD Community Edition: Data Compressions

    Posted 11-27-2017 19:58

    Are you referring to importing compressed files?



  • 3.  RE: MapD Community Edition: Data Compressions

    Posted 11-28-2017 01:42

    You basically get a compression thru text dictionary encoding; you can save memory and disk space using fixed encodings on integer datatypes



  • 4.  RE: MapD Community Edition: Data Compressions

    Posted 11-28-2017 15:54

    Thx @aznable , Is dict encoding being implemented by mapd at the data ingestion time? Are there any settings we need to get this encoding enabled in Community edition?



  • 5.  RE: MapD Community Edition: Data Compressions

    Posted 11-28-2017 17:56

    You have to define a text column with the dictionary encoding on table creation; the encoding has to be defined accordingly the cardinality of distinct values of the column itself ( 8 bits if the dv are less than 256, 16 bit it they are are less tha 64k and 32 otherwise). At ingestion time they will be encoded and you will be able to filter, join and aggregate using the encoded column.

    Anyway don’t overestimate the compression factor because mapd is incredibly fast performing joins, so a balanced design of tables is important as usual