Kafka Importer - with Topic security

How to pass Kafka topic authentication details with KafkaImporter?

Hi @jshah,

I don’t see any option to specify security or encryption. I took at the code, but I am not a Kafka expert, so I will ask internally.

In the meantime, as a workaround, think you could use safka-console-consumer pipelining the messages to Streaminserter.

Something like that
kafka-console-consumer | StreamInserter table database

Every message should be inserted into the target table.

Thanks. I had already tried with the consumer, but that has two concerns

  • How error handling will work? What if consumer reads from Kafka without any issue and KafkaImporter fails?
  • KafkaImporter batch option doesn’t seem to work when used with the console consumer

Hi @jshah,

Your concerns are justified, in fact, I had to ask internally because I’ve not the right skill to give you a satisfying answer. It’s weird the batch option of Kafka importer isn’t working; what’s the issue?
Have you tried with StreamImporter? Same syntax

Thanks!

It looks like batch parameter is ignored when used with Kafka console. One of the other thread from user “Sumit” had a request to use kafka importer at certain time interval rather than batch size. We also need same feature ( if KafkaImporter supports topic security).

No I haven’t tried StreamImporter. I will look into it.

I implemented the time interval but I didn’t do a pr against our repo. I can share the binary, but without topic security yet.

Probably the time interval would be a valuable feature for all the importer’s utilities.

Anyway, when I will get an answer about t security I will report back for sure

Sadly we didn’t support security in the actual version of KafkaImporter, so the workaround I proposed is the only way to load Kafka topic with security enabled

We will take a look to the new libs to investigate how to implement new features.

Tell me if you want to try out the utility with the timer option you requested; I would Tha KS to give me the idea to implement the same on Streamimporter too

Regards,
Candido

Yeah, I would like to try out the timer option. Thanks!