mapreduce - matrix multiplication hadoop code does not work for large matrices - Spill failed -


i trying large matrix multiplication. code below works fine, when try large matrices, below error. note there absolutely nothing wrong input file (no weird characters etc).

also, mention after running time, crashes following error, , prompt on ubuntu system saying file system root has around 360 mb. because of space issue crash happening? 

import java.io.ioexception; import java.util.*;  import org.apache.hadoop.fs.filesystem; import org.apache.hadoop.fs.path; import org.apache.hadoop.conf.*; import org.apache.hadoop.io.*; import org.apache.hadoop.mapreduce.*; import org.apache.hadoop.mapreduce.lib.input.fileinputformat; import org.apache.hadoop.mapreduce.lib.input.textinputformat; import org.apache.hadoop.mapreduce.lib.output.fileoutputformat; import org.apache.hadoop.mapreduce.lib.output.textoutputformat;  public class onestepmatrixmultiplication {      public static class map extends mapper<longwritable, text, text, text> {         public void map(longwritable key, text value, context context) throws ioexception, interruptedexception {             configuration conf = context.getconfiguration();             int m = integer.parseint(conf.get("m"));             int p = integer.parseint(conf.get("p"));             string line = value.tostring();             string[] indicesandvalue = line.split(",");             text outputkey = new text();             text outputvalue = new text();             if (indicesandvalue[0].equals("a")) {                 (int k = 0; k < p; k++) {                     outputkey.set(indicesandvalue[1] + "," + k);                     outputvalue.set("a," + indicesandvalue[2] + "," + indicesandvalue[3]);                     context.write(outputkey, outputvalue);                 }             } else {                 (int = 0; < m; i++) {                     outputkey.set(i + "," + indicesandvalue[2]);                     outputvalue.set("b," + indicesandvalue[1] + "," + indicesandvalue[3]);                     context.write(outputkey, outputvalue);                 }             }         }     }      public static class reduce extends reducer<text, text, text, text>      {         public void reduce(text key, iterable<text> values, context context) throws ioexception, interruptedexception {             string[] value;             hashmap<integer, float> hasha = new hashmap<integer, float>();             hashmap<integer, float> hashb = new hashmap<integer, float>();              (text val : values)              {                 value = val.tostring().split(",");                  if (value[0].equals("a"))                  {                     hasha.put(integer.parseint(value[1]), float.parsefloat(value[2]));                 } else                  {                     hashb.put(integer.parseint(value[1]), float.parsefloat(value[2]));                 }             }              int n = integer.parseint(context.getconfiguration().get("n"));             float result = 0.0f;             float a_ij;             float b_jk;             (int j = 0; j < n; j++)              {                 a_ij = hasha.containskey(j) ? hasha.get(j) : 0.0f;                 b_jk = hashb.containskey(j) ? hashb.get(j) : 0.0f;                 result += a_ij * b_jk;             }             if (result != 0.0f)              {                 context.write(null, new text(key.tostring() + "," + float.tostring(result)));             }         }     }      public static void main(string[] args) throws exception {         configuration conf = new configuration();         // m-by-n matrix; b n-by-p matrix.          conf.set("m", "10000");         conf.set("n", "3");         conf.set("p", "10000");          job job = new job(conf, "matrixmatrixmultiplicationonestep");         job.setjarbyclass(onestepmatrixmultiplication.class);         job.setoutputkeyclass(text.class);         job.setoutputvalueclass(text.class);          job.setmapperclass(map.class);         job.setreducerclass(reduce.class);          job.setinputformatclass(textinputformat.class);         job.setoutputformatclass(textoutputformat.class);          fileinputformat.addinputpath(job, new path(args[0]));         fileoutputformat.setoutputpath(job, new path(args[1]));          job.waitforcompletion(true);     } }

error

5/05/31 18:45:35 warn mapred.localjobrunner: job_local1019087739_0001 java.lang.exception: java.io.ioexception: spill failed     @ org.apache.hadoop.mapred.localjobrunner$job.run(localjobrunner.java:354)

try change 

hadoop.tmp.dir

to location sufficient storage.


Comments

Post a Comment

Popular posts from this blog

angularjs - ADAL JS Angular- WebAPI add a new role claim to the token -

i have created angular spa webapi backend using adal js authentication.as there no roles in ad, need manually add role claims in order give users access different api controllers. the roles stored in database. expecting inject claim through call webapi after authentication ad. webapi code might this. identity.addclaim(new claim("role", "user")); var ticket = new authenticationticket(identity, props); var accesstoken = startup.oauthbeareroptions.accesstokenformat.protect(ticket); is possible replace adal idtoken new token? is viable solution or there other better way handle this? as initial token generated azuread, possible edit token add new claim? appreciated. graph api supporting group claims. see here: http://justazure.com/azure-active-directory-part-4-group-claims/ if @ examples on page, users assigned groups , app can check group in claims. in current version of portal, need app manifest , modify it.
Read more

php - CakePHP HttpSockets send array of paramms -

i problem when make request (i can request). $link = 'http://uri/path/?param1=1&sid=2&sid=3&sid=4' $http = $httpsocket = new httpsocket(); $res = $httpsocket->get($link); if check request pr($http); i see : [line] => /url/path/?param1=1&sid%5b0%5d=2&sid%5b1%5d=3&sid%5b2%5d=4 http/1.1 and i'm getting empty response because server doesn't know parameters sid%5b0%5d=2 when try send parameters array : $data = array('param1' => '1', 'sid' => array('2','3', '4')); i see same changes %5b0%5d - additional indexes. how fix it? unfortunately , can not change server side, if send request in browser, i'll normal response in json format. when make request, can parameters this: $param1 = $_get['param1']; $sid= $_get['sid']; tip: use diferente names parameters. have multiples sid. use sid1, sid2, sid3... update: can parameters this:
Read more

node.js - Using Node without global install -

i'm working on few node & electron (atom shell) projects , i'm curious how them work without user having install node globally. i've seen few applications including node directory in project contains node.exe binary. how go setting node project use binary? edit: thought should add, i've searched everywhere answer this, unrelated results.
Read more