Discussions

Expand all | Collapse all

I have a few doubts about the function ""check_total_bitmap_memory

  • 1.  I have a few doubts about the function ""check_total_bitmap_memory

    Posted 01-17-2019 10:25

    void check_total_bitmap_memory(const QueryMemoryDescriptor& query_mem_desc) {
    const int32_t groups_buffer_entry_count = query_mem_desc.getEntryCount();
    if (g_enable_watchdog) {
    checked_int64_t total_bytes_per_group = 0;
    const size_t num_count_distinct_descs =
    query_mem_desc.getCountDistinctDescriptorsSize();
    for (size_t i = 0; i < num_count_distinct_descs; i++) {
    const auto count_distinct_desc = query_mem_desc.getCountDistinctDescriptor(i);
    if (count_distinct_desc.impl_type_ != CountDistinctImplType::Bitmap) {
    continue;
    }
    //“Total_bytes_per_group” represents the total number of predicted rows (bits) converted to bytes
    total_bytes_per_group += count_distinct_desc.bitmapPaddedSizeBytes();
    }
    int64_t total_bytes{0};
    // Need to use OutOfHostMemory since it’s the only type of exception
    // QueryExecutionContext is supposed to throw.
    try {
    total_bytes =
    static_cast<int64_t>(total_bytes_per_group * groups_buffer_entry_count);
    } catch (…) {
    // Absurd amount of memory, merely computing the number of bits overflows int64_t.
    // Don’t bother to report the real amount, this is unlikely to ever happen.
    throw OutOfHostMemory(std::numeric_limits<int64_t>::max() / 8);
    }
    if (total_bytes >= 2 * 1000 * 1000 * 1000L) {
    throw OutOfHostMemory(total_bytes);
    }
    }
    }

    I have two questions:
    (1)

    total_bytes_per_group += count_distinct_desc.bitmapPaddedSizeBytes();

    Total_bytes_per_group” represents the total number of predicted rows (bits) converted to bytes,Why do I have to multiply by the number of rows again in the ""total_bytes "".
    (2)What is the basis of “2 * 1000 * 1000 * 1000L”?



  • 2.  RE: I have a few doubts about the function ""check_total_bitmap_memory

    Posted 01-17-2019 13:18
    1. You can have multiple groups where you are doing a count(distinct)

    e.g.

    select count(distinct field1) from table1

    obviously here you have just one group so the groups_buffer_entry_count will be 1

    select field2, count(distinct field1) from table1 group by 1

    where you will have multiple groups and the groups_buffer_entry_count will be different.

    1. maybe it’s related to the maximum slab size of GPUs (2GB); the operations actually uses a single slab, so it makes sense for the watchdog to raise an exception