java - Fast and scalable IO passthrough -


original

i have batch job passes through contents downloads various urls s3 storage. using blocking io , have reached point job io bound blocking of time because of io. in order speed whole process thinking using non-blocking io.

unfortunately wasn't able find utility code passing through content set of channels set of channels. since read writing correct non-blocking code not easy, prefer use existing utility/framework write code myself.

the transfermanager seems possible option higher throughput when using aws sdk, offers usage of streams , seems use io threads in background. apparently there no out of box option non-blocking uploads s3.

what recommend? right can imagine following solutions.

  1. stay blocking io , use own io thread pool
  2. use non-blocking io download files local filesystem, upload transfermanager
  3. use non-blocking io pass through

option 1 not scale , 2 work while, keep ios on ebs low i'd rather use 3.

to implement option 3 guess have implement lot myself final question is, whether think it's worth , if so, tools use make work.

edit 1

clarifying io bound meant job waiting io. here can see bandwidth not saturated, if possible. monitoring view

if job i/o bound you've finished. you're bound speed of network, not of code. using nio won't make faster.

clarifying io bound meant job waiting io.

yes, that's 'i/o bound' means. nothing has changed.

here can see bandwidth not saturated, if possible.

you try using larger buffers, said seems me you've finished.


Comments

Post a Comment

Popular posts from this blog

angularjs - ADAL JS Angular- WebAPI add a new role claim to the token -

i have created angular spa webapi backend using adal js authentication.as there no roles in ad, need manually add role claims in order give users access different api controllers. the roles stored in database. expecting inject claim through call webapi after authentication ad. webapi code might this. identity.addclaim(new claim("role", "user")); var ticket = new authenticationticket(identity, props); var accesstoken = startup.oauthbeareroptions.accesstokenformat.protect(ticket); is possible replace adal idtoken new token? is viable solution or there other better way handle this? as initial token generated azuread, possible edit token add new claim? appreciated. graph api supporting group claims. see here: http://justazure.com/azure-active-directory-part-4-group-claims/ if @ examples on page, users assigned groups , app can check group in claims. in current version of portal, need app manifest , modify it.
Read more

php - CakePHP HttpSockets send array of paramms -

i problem when make request (i can request). $link = 'http://uri/path/?param1=1&sid=2&sid=3&sid=4' $http = $httpsocket = new httpsocket(); $res = $httpsocket->get($link); if check request pr($http); i see : [line] => /url/path/?param1=1&sid%5b0%5d=2&sid%5b1%5d=3&sid%5b2%5d=4 http/1.1 and i'm getting empty response because server doesn't know parameters sid%5b0%5d=2 when try send parameters array : $data = array('param1' => '1', 'sid' => array('2','3', '4')); i see same changes %5b0%5d - additional indexes. how fix it? unfortunately , can not change server side, if send request in browser, i'll normal response in json format. when make request, can parameters this: $param1 = $_get['param1']; $sid= $_get['sid']; tip: use diferente names parameters. have multiples sid. use sid1, sid2, sid3... update: can parameters this:
Read more

node.js - Using Node without global install -

i'm working on few node & electron (atom shell) projects , i'm curious how them work without user having install node globally. i've seen few applications including node directory in project contains node.exe binary. how go setting node project use binary? edit: thought should add, i've searched everywhere answer this, unrelated results.
Read more