Run scrapy on a set of hundred plus urls -


i need download cpu , gpu data of set of phones fro gsmarena. step one, downloaded urls of phones running scrapy , deleted unnecessary items.

code same below.

# -*- coding: utf-8 -*-  scrapy.selector import selector scrapy import spider scrapy.contrib.linkextractors.sgml import sgmllinkextractor gsmarena_data.items import gsmarenadataitem   class mobileinfospider(spider): name = "mobile_info" allowed_domains = ["gsmarena.com"] start_urls = ( # 'http://www.gsmarena.com/samsung-phones-f-9-10.php', # 'http://www.gsmarena.com/apple-phones-48.php', # 'http://www.gsmarena.com/microsoft-phones-64.php', # 'http://www.gsmarena.com/nokia-phones-1.php', # 'http://www.gsmarena.com/sony-phones-7.php', # 'http://www.gsmarena.com/lg-phones-20.php', # 'http://www.gsmarena.com/htc-phones-45.php', # 'http://www.gsmarena.com/motorola-phones-4.php', # 'http://www.gsmarena.com/huawei-phones-58.php', # 'http://www.gsmarena.com/lenovo-phones-73.php', # 'http://www.gsmarena.com/xiaomi-phones-80.php', # 'http://www.gsmarena.com/acer-phones-59.php', # 'http://www.gsmarena.com/asus-phones-46.php', # 'http://www.gsmarena.com/oppo-phones-82.php', # 'http://www.gsmarena.com/blackberry-phones-36.php', # 'http://www.gsmarena.com/alcatel-phones-5.php', # 'http://www.gsmarena.com/xolo-phones-85.php', # 'http://www.gsmarena.com/lava-phones-94.php', # 'http://www.gsmarena.com/micromax-phones-66.php', # 'http://www.gsmarena.com/spice-phones-68.php', 'http://www.gsmarena.com/gionee-phones-92.php', )  def parse(self, response): phone = gsmarenadataitem() hxs = selector(response) phone_listings = hxs.css('.makers')  phone_listings in phone_listings: phone['model'] = phone_listings.xpath("ul/li/a/strong/text()").extract() phone['link'] = phone_listings.xpath("ul/li/a/@href").extract() yield phone

now, need run scrapy on set of urls cpu , gpu data. info comes css selector = ".ttl".

kindly guide how loop scrapy on set of urls , output data in single csv or json. i'm aware creating items , using css selectors. need how loop on hundred plus pages.

i have list of urls like:  www.gsmarena.com/samsung_galaxy_s5_cdma-6338.php www.gsmarena.com/samsung_galaxy_s5-6033.php www.gsmarena.com/samsung_galaxy_core_lte_g386w-6846.php www.gsmarena.com/samsung_galaxy_core_lte-6099.php www.gsmarena.com/acer_iconia_one_8_b1_820-7217.php www.gsmarena.com/acer_iconia_tab_a3_a20-7136.php www.gsmarena.com/microsoft_lumia_640_dual_sim-7082.php www.gsmarena.com/microsoft_lumia_532_dual_sim-6951.php  links phone descriptions on gsm arena.  need download cpu , gpu info of 100 models have.      extracted urls of 100 models data required.      spider written same is,       scrapy.selector import selector     scrapy import spider     gsmarena_data.items import gsmarenadataitem      class mobileinfospider(spider):     name = "cpu_gpu_info"     allowed_domains = ["gsmarena.com"]     start_urls = (     "http://www.gsmarena.com/microsoft_lumia_435_dual_sim-6949.php",     "http://www.gsmarena.com/microsoft_lumia_435-6942.php",     "http://www.gsmarena.com/microsoft_lumia_535_dual_sim-6792.php",     "http://www.gsmarena.com/microsoft_lumia_535-6791.php",     )     def parse(self, response):     phone = gsmarenadataitem()     hxs = selector(response)     cpu_gpu = hxs.css('.ttl')     phone_listings in phone_listings:     phone['cpu'] = cpu_gpu.xpath("ul/li/a/strong/text()").extract()     phone['gpu'] = cpu_gpu.xpath("ul/li/a/@href").extract()     yield phone

if somehow run on urls want extract data, required data in single csv file.

i think need information every vendors. if don't have put hundreds of urls in start-url, alternatively can use link start-url after in parse() extract urls programatically , process want.

this answer so.


Comments

Post a Comment

Popular posts from this blog

angularjs - ADAL JS Angular- WebAPI add a new role claim to the token -

i have created angular spa webapi backend using adal js authentication.as there no roles in ad, need manually add role claims in order give users access different api controllers. the roles stored in database. expecting inject claim through call webapi after authentication ad. webapi code might this. identity.addclaim(new claim("role", "user")); var ticket = new authenticationticket(identity, props); var accesstoken = startup.oauthbeareroptions.accesstokenformat.protect(ticket); is possible replace adal idtoken new token? is viable solution or there other better way handle this? as initial token generated azuread, possible edit token add new claim? appreciated. graph api supporting group claims. see here: http://justazure.com/azure-active-directory-part-4-group-claims/ if @ examples on page, users assigned groups , app can check group in claims. in current version of portal, need app manifest , modify it.
Read more

node.js - Using Node without global install -

i'm working on few node & electron (atom shell) projects , i'm curious how them work without user having install node globally. i've seen few applications including node directory in project contains node.exe binary. how go setting node project use binary? edit: thought should add, i've searched everywhere answer this, unrelated results.
Read more

php - CakePHP HttpSockets send array of paramms -

i problem when make request (i can request). $link = 'http://uri/path/?param1=1&sid=2&sid=3&sid=4' $http = $httpsocket = new httpsocket(); $res = $httpsocket->get($link); if check request pr($http); i see : [line] => /url/path/?param1=1&sid%5b0%5d=2&sid%5b1%5d=3&sid%5b2%5d=4 http/1.1 and i'm getting empty response because server doesn't know parameters sid%5b0%5d=2 when try send parameters array : $data = array('param1' => '1', 'sid' => array('2','3', '4')); i see same changes %5b0%5d - additional indexes. how fix it? unfortunately , can not change server side, if send request in browser, i'll normal response in json format. when make request, can parameters this: $param1 = $_get['param1']; $sid= $_get['sid']; tip: use diferente names parameters. have multiples sid. use sid1, sid2, sid3... update: can parameters this:...
Read more