hadoop - Spark insert to HBase slow -


i inserting hbase using spark it's slow. 60,000 records takes 2-3mins. have 10 million records save. 

object writetohbase extends serializable {     def main(args: array[string]) {         val csvrows: rdd[array[string] = ...         val dateformatter = datetimeformat.forpattern("yyyy-mm-dd hh:mm:ss")         val usersrdd = csvrows.map(row => {             new usertable(row(0), row(1), row(2), row(9), row(10), row(11))         })         processusers(sc: sparkcontext, usersrdd, dateformatter)     }) }  def processusers(sc: sparkcontext, usersrdd: rdd[usertable], dateformatter: datetimeformatter): unit = {      usersrdd.foreachpartition(part => {         val conf = hbaseconfiguration.create()         val table = new htable(conf, tablename)          part.foreach(userrow => {             val id = userrow.id             val name = userrow.name             val date1 = dateformatter.parsedatetime(userrow.date1)             val hrow = new put(bytes.tobytes(id))             hrow.add(cf, q, bytes.tobytes(date1))             hrow.add(cf, q, bytes.tobytes(name))             ...             table.put(hrow)         })         table.flushcommits()         table.close()     }) }

i using in spark-submit:

--num-executors 2 --driver-memory 2g --executor-memory 2g --executor-cores 2

it's slow because implementation doesn't leverage proximity of data; piece of spark rdd in server may transferred hbase regionserver running on server.

currently there no spark's rrd operation use hbase data store in efficient manner.


Comments

Post a Comment

Popular posts from this blog

node.js - Using Node without global install -

i'm working on few node & electron (atom shell) projects , i'm curious how them work without user having install node globally. i've seen few applications including node directory in project contains node.exe binary. how go setting node project use binary? edit: thought should add, i've searched everywhere answer this, unrelated results.
Read more

How to access a php class file from PHPFox framework into javascript code written in simple HTML file? -

i've website developed using phpfox v3.0.7 . now i'm trying implement 'sse(server sent events)' 1 of php files. reference i'm giving below necessary code part of file. file titled process.class.php , present @ location /var/www/module/notification/include/service/process.class.php present on server having ip http://34.124.40.142/ . <?php /** * [phpfox_header] */ header('content-type: text/event-stream'); header('cache-control: no-cache'); defined('phpfox') or exit('no dice!'); /** * * * @copyright [phpfox_copyright] * @author raymond benc * @package phpfox_service * @version $id: service.class.php 67 2009-01-20 11:32:45z raymond_benc $ */ class notification_service_process extends phpfox_service { /** * class constructor */ public function __construct() { $this->_stable = phpfox::gett('notification'); } public function add($...
Read more

java - Null response to php query in android, even though php works properly -

Image
table i'm trying retrieve above data using 2 php files below. want pass email parameter , return bets user = email. work fine when tested advanced rest client, problem not php. however, when try through android 2 java classes below, null pointer exception (logcat below) , json object returned null. i'm not sure error is, think i'm not passing parameter php , that's why it's producing null response. logcat 05-31 15:10:50.227 26074-26074/com.example.albert.betterapp e/androidruntime﹕ fatal exception: main process: com.example.albert.betterapp, pid: 26074 java.lang.runtimeexception: unable start activity componentinfo{com.example.albert.betterapp/com.example.albert.betterapp.displayallbets}: java.lang.nullpointerexception @ android.app.activitythread.performlaunchactivity(activitythread.java:2292) @ android.app.activitythread.handlelaunchactivity(activitythread.java:2350) @ android.app.activitythread.acce...
Read more