hadoop - Add Spark to Oozie shared lib -


by default, oozie shared lib directory provides libraries hive, pig, , map-reduce. if want run spark job on oozie, might better add spark lib jars oozie's shared lib instead of copy them app's lib directory.
how can add spark lib jars (including spark-core , dependencies) oozie's shared lib? comment / answer appreciated.

spark action scheduled released oozie 4.2.0, though doc seems bit behind. see related jira here : oozie jira - add spark action executor

cloudera's release cdh 5.4 has though, see official doc here: cdh 5.4 oozie doc - oozie spark action extension

with older version of oozie, jars shared various approaches. first approach may work best. complete listings anyway :

below various ways include jar workflow:

set oozie.libpath=/path/to/jars,another/path/to/jars in job.properties.

this useful if have many workflows need same jar; can put in 1 place in hdfs , use many workflows. jars available actions in workflow. there no need ever point @ sharelib location. (i see in lot of workflows.) oozie knows sharelib , include automatically if set oozie.use.system.libpath=true in job.properties.

create directory named “lib” next workflow.xml in hdfs , put jars in there.

this useful if have jars need 1 workflow. oozie automatically make jars available actions in workflow.

specify tag in action path single jar; can have multiple tags.

this useful if want jars specific action , not actions in workflow. downside have specify them in workflow.xml, if ever need add/remove jars, have change workflow.xml.

add jars sharelib (e.g. /user/oozie/share/lib/lib_/pig)

while work, it’s not recommended 2 reasons: additional jars included every workflow using sharelib, may unexpected workflows , users. when upgrading sharelib, you’ll have recopy additional jars new sharelib.

quoted rober kanter's blog here : how-to: use sharelib in apache oozie (cdh 5)


Comments

Post a Comment

Popular posts from this blog

angularjs - ADAL JS Angular- WebAPI add a new role claim to the token -

i have created angular spa webapi backend using adal js authentication.as there no roles in ad, need manually add role claims in order give users access different api controllers. the roles stored in database. expecting inject claim through call webapi after authentication ad. webapi code might this. identity.addclaim(new claim("role", "user")); var ticket = new authenticationticket(identity, props); var accesstoken = startup.oauthbeareroptions.accesstokenformat.protect(ticket); is possible replace adal idtoken new token? is viable solution or there other better way handle this? as initial token generated azuread, possible edit token add new claim? appreciated. graph api supporting group claims. see here: http://justazure.com/azure-active-directory-part-4-group-claims/ if @ examples on page, users assigned groups , app can check group in claims. in current version of portal, need app manifest , modify it.
Read more

php - CakePHP HttpSockets send array of paramms -

i problem when make request (i can request). $link = 'http://uri/path/?param1=1&sid=2&sid=3&sid=4' $http = $httpsocket = new httpsocket(); $res = $httpsocket->get($link); if check request pr($http); i see : [line] => /url/path/?param1=1&sid%5b0%5d=2&sid%5b1%5d=3&sid%5b2%5d=4 http/1.1 and i'm getting empty response because server doesn't know parameters sid%5b0%5d=2 when try send parameters array : $data = array('param1' => '1', 'sid' => array('2','3', '4')); i see same changes %5b0%5d - additional indexes. how fix it? unfortunately , can not change server side, if send request in browser, i'll normal response in json format. when make request, can parameters this: $param1 = $_get['param1']; $sid= $_get['sid']; tip: use diferente names parameters. have multiples sid. use sid1, sid2, sid3... update: can parameters this:
Read more

node.js - Using Node without global install -

i'm working on few node & electron (atom shell) projects , i'm curious how them work without user having install node globally. i've seen few applications including node directory in project contains node.exe binary. how go setting node project use binary? edit: thought should add, i've searched everywhere answer this, unrelated results.
Read more