java - Create dataframe from rdd objectfile -


what method create ddf rdd saved objectfile. want load rdd don't have java object, structtype want use schema ddf.

i tried retrieving row

    val myrdd = sc.objectfile[org.apache.spark.sql.row]("/home/bipin/"+name) 

but get

java.lang.classcastexception: [ljava.lang.object; cannot cast org.apache.spark.sql.row

is there way this.

edit

from understand, have read rdd array of objects , convert row. if can give method this, acceptable.

if have array of object have use row apply method array of any. in code this:

val myrdd = sc.objectfile[array[object]]("/home/bipin/"+name).map(x => row(x)) 

edit

you rigth @user568109 create dataframe 1 field array parse whole array have this:

val myrdd = sc.objectfile[array[object]]("/home/bipin/"+name).map(x => row.fromseq(x.toseq)) 

as @user568109 said there other ways this:

val myrdd = sc.objectfile[array[object]]("/home/bipin/"+name).map(x => row(x:_*)) 

no matters 1 because both wrappers same code:

  /**    * method can used construct [[row]] given values.    */    def apply(values: any*): row = new genericrow(values.toarray)    /**    * method can used construct [[row]] [[seq]] of values.    */    def fromseq(values: seq[any]): row = new genericrow(values.toarray) 

Comments

Popular posts from this blog

angularjs - ADAL JS Angular- WebAPI add a new role claim to the token -

php - CakePHP HttpSockets send array of paramms -

node.js - Using Node without global install -