Result return limit

Hi,

I am running this SQL query in omniscidb which should return 1.4 billion rows. But I am getting this error which says:
“would contain 1454616751 rows, which is more than the current system limit of 32000000”
how to resolve this ?

Thanks,
Dipanwita.

Hi @dipanwita2019,

The 32m of rows is an arbitrary number hard-coded into the database, so you can’t change this behavior.

Anyway the runtime and the amount of resources that would required to fetch such a massive number of rows would be staggering, putting in knee the entire system.

Which kind of workload are you running? I can’t imagine any that would require a such massive resultset.

I would run multiple batches of sql filtering (if possible) with the rowid logical column, but it’s going to take a very long time, because our database is tuned to process big datasets returning small dataset (after filtering aggregation) and like all columnar database. At least the one I worked with, isnt stellar on returning big datasets (we are working on it of course)

1 Like

hi,

yes the dataset has massive number of rows, it is network data collected at every 1 second and for a day.

But thank you for the clarification “because our database is tuned to process big datasets returning small dataset.” This totally make sense.

Dipanwita.

Well,
We are working in order to improve the projections of medium/big datasets, but 1.4B are a lot.

So after filtering you are getting 1.4B rows as result?

no, sorry I should have been more clear …

My dataset has 1.4 b as the number of rows , i wanted to return the whole data before filtering for some time series analysis, but i got an error saying it can return only 32m number of rows. So, now i think i should do some resampling with the timestamp to bring the data to a more manageable size …(meaning instead of 1 second i should filter the data for every 10 seconds may be).

hope I made some sense.

Dipanwita.

1 Like