I have created a new Apache Nifi processor that ingests data into MapD.
There are three projects in total:
- A maven project that builds a thrift java client for MapD - https://github.com/jonrmayer/mapd_thrift_maven
- A maven project that builds a Nifi Controller service that exposes the MapD client and session details - https://github.com/jonrmayer/mapdservices-bundle
- A maven project that builds 2 Nifi Processors - https://github.com/jonrmayer/mapdprocessors-bundle
- Imports data into MapD
- Exports data from MapD
You will need to build them in the above order ( please see notes on each project )
Once built, you will need to place the following files into Apache Nifi’s lib folder prior to starting:-
In Nifi’s Designer - you will need to create a mapdservice controller with your connection details.
The PutMapDProcessor requires this service as well as the MapD tablename and a buffersize ( the avro batch sizes )
The GetMapDProcessor requires this service as well as the SQL Select Statement
This is a first pass and I need some help with reviewing MapDConversionUtilities.java for ineffeciencies/ optimization.
Also I would like to create a Spatial Ingest Processor but I am having problems with column mismatch for spatial types - all help gratefully received.
Any problems let me know here or on Github.