Kafka Importer-- Batch Size based on Time Interval?

Hi ,

Please let us know is it possible to select the Batch based on Time Duration in place of Batch Size.

Now i am using the KafkaImporter —>[–batch <batch_size>]

For example : Load Data every 30 seconds or 120 seconds.

Regards,
sumit

Hi @Sumit,

right now there isn’t any option to load data on time intervals usiong the KafkaImporter, but looking at the code it’s not so difficult to implement , while it could lead to a very small batch inserts

Hi @candido.dessanti,

Thank you for feedback and please let us know if we can achieve it.

Let me explain you use case,

  • i would like to ensure that Available Data/Report in x seconds old (X second will configure based on need and incoming Data stream) [Note- I am aware about OmniSCI Batch DB file and issue related to EPOCH]
  • Problem in my use case Batch file is Time sensitive, in day time 10,000 Records receive in few seconds but in late evening difficult to get 1000 records in minute.