Export to compressed files


#1

Omnisci/MapD Core currently allows the direct import of compressed file, which is a huge benefit when dealing with semi-large datasets.

Would it be possible, as en enhancement idea, to also allow the direct compression of exported files ?

I.E., you can currently do that :

COPY tableName FROM 'dataset.csv.bz2'
COPY (SELECT * FROM tableName) TO 'dataset.csv'

It would be great to be able to do a

COPY (SELECT * FROM tableName) TO 'dataset.csv.bz2'

=> This currently “works”, ie produce a valid CSV with a bz2 extension, but it would be great that it outputs a bz2 compressed csv.

This would largely easen the backup tasks by reducing the operations needed to decrease the needed space.

Hope this is of interest to you :slight_smile:


#2

It would be cool, in the meanwhile you can copy to a pipe and compress the file on the fly with your preferred compression tool.
I suggest xz or gz that have multithread solutions; bz2 is just too slow


#3

@RCura along with the suggestion from @aznable, we think this is an interesting suggestion and we invite you to create an issue in the Github repo (or I can do it for you).


#4

Thanks, did that in this issue (#292)