java - Create dataframe from rdd objectfile -
what method create ddf rdd saved objectfile. want load rdd don't have java object, structtype want use schema ddf.
i tried retrieving row
val myrdd = sc.objectfile[org.apache.spark.sql.row]("/home/bipin/"+name)
but get
java.lang.classcastexception: [ljava.lang.object; cannot cast org.apache.spark.sql.row
is there way this.
edit
from understand, have read rdd array of objects , convert row. if can give method this, acceptable.
if have array of object have use row apply method array of any. in code this:
val myrdd = sc.objectfile[array[object]]("/home/bipin/"+name).map(x => row(x))
edit
you rigth @user568109 create dataframe 1 field array parse whole array have this:
val myrdd = sc.objectfile[array[object]]("/home/bipin/"+name).map(x => row.fromseq(x.toseq))
as @user568109 said there other ways this:
val myrdd = sc.objectfile[array[object]]("/home/bipin/"+name).map(x => row(x:_*))
no matters 1 because both wrappers same code:
/** * method can used construct [[row]] given values. */ def apply(values: any*): row = new genericrow(values.toarray) /** * method can used construct [[row]] [[seq]] of values. */ def fromseq(values: seq[any]): row = new genericrow(values.toarray)
Comments
Post a Comment