How to pass Kafka topic authentication details with KafkaImporter?
I don’t see any option to specify security or encryption. I took at the code, but I am not a Kafka expert, so I will ask internally.
In the meantime, as a workaround, think you could use safka-console-consumer pipelining the messages to Streaminserter.
Something like that
kafka-console-consumer | StreamInserter table database
Every message should be inserted into the target table.
Thanks. I had already tried with the consumer, but that has two concerns
- How error handling will work? What if consumer reads from Kafka without any issue and KafkaImporter fails?
- KafkaImporter batch option doesn’t seem to work when used with the console consumer
Your concerns are justified, in fact, I had to ask internally because I’ve not the right skill to give you a satisfying answer. It’s weird the batch option of Kafka importer isn’t working; what’s the issue?
Have you tried with StreamImporter? Same syntax
It looks like batch parameter is ignored when used with Kafka console. One of the other thread from user “Sumit” had a request to use kafka importer at certain time interval rather than batch size. We also need same feature ( if KafkaImporter supports topic security).
No I haven’t tried StreamImporter. I will look into it.
I implemented the time interval but I didn’t do a pr against our repo. I can share the binary, but without topic security yet.
Probably the time interval would be a valuable feature for all the importer’s utilities.
Anyway, when I will get an answer about t security I will report back for sure
Sadly we didn’t support security in the actual version of KafkaImporter, so the workaround I proposed is the only way to load Kafka topic with security enabled
We will take a look to the new libs to investigate how to implement new features.
Tell me if you want to try out the utility with the timer option you requested; I would Tha KS to give me the idea to implement the same on Streamimporter too
Yeah, I would like to try out the timer option. Thanks!