How to read and write Map<String, Object> from/to parquet file in Java or Scala? -
looking concise example on how read , write map<string, object>
from/to parquet file in java or scala?
here expected structure, using com.fasterxml.jackson.databind.objectmapper
serializer in java (i.e. looking equivalent using parquet):
public static map<string, object> read(inputstream inputstream) throws ioexception { objectmapper objectmapper = new objectmapper(); return objectmapper.readvalue(inputstream, new typereference<map<string, object>>() { }); } public static void write(outputstream outputstream, map<string, object> map) throws ioexception { objectmapper objectmapper = new objectmapper(); objectmapper.writevalue(outputstream, map); }
i'm not quite parquet but, here:
schema schema = new schema.parser().parse(resources.getresource("map.avsc").openstream()); file tmp = file.createtempfile(getclass().getsimplename(), ".tmp"); tmp.deleteonexit(); tmp.delete(); path file = new path(tmp.getpath()); avroparquetwriter<genericrecord> writer = new avroparquetwriter<genericrecord>(file, schema); // write record empty map. immutablemap emptymap = new immutablemap.builder<string, integer>().build(); genericdata.record record = new genericrecordbuilder(schema) .set("mymap", emptymap).build(); writer.write(record); writer.close(); avroparquetreader<genericrecord> reader = new avroparquetreader<genericrecord>(file); genericrecord nextrecord = reader.read(); assertnotnull(nextrecord); assertequals(emptymap, nextrecord.get("mymap"));
in situation change immutablemap
(google collections) default map below:
schema schema = new schema.parser().parse( resources.getresource( "map.avsc" ).openstream() ); file tmp = file.createtempfile( getclass().getsimplename(), ".tmp" ); tmp.deleteonexit(); tmp.delete(); path file = new path( tmp.getpath() ); avroparquetwriter<genericrecord> writer = new avroparquetwriter<genericrecord>( file, schema ); // write record empty map. map<string,object> emptymap = new hashmap<string, object>(); // not empty more emptymap.put( "something", new something() ); genericdata.record record = new genericrecordbuilder( schema ).set( "mymap", emptymap ).build(); writer.write( record ); writer.close(); avroparquetreader<genericrecord> reader = new avroparquetreader<genericrecord>( file ); genericrecord nextrecord = reader.read(); assertnotnull( nextrecord ); assertequals( emptymap, nextrecord.get( "mymap" ) );
i didn't test code, give try..
Comments
Post a Comment