javascript - Python, scrapy: scrape links then iterate over those links to scrape further links -


i trying create scrapy spider focuses on site called weedmaps.com. weedmaps uses googlemaps api generate information on dispensaries based on state regional location information. want spider start top layer, diver states, scrape regional links within states, go regional links 1 @ time, scrape dispensary links, , go dispensary links 1 @ time , scrape specific information regarding individual dispensaries. given site dynamic, have been using selenium account javascript. site, have been able scrape regional links, , dispensary links, separately. when try combine them, spider collects first regional link, , goes directly regional link collect dispensary information , ends. somehow create list of regional links, , dispensary links populated spider runs.

any insights on how accomplish fantastic. below code have far, , can't seem figure thing out. thank in advance!

import scrapy import time selenium import webdriver selenium.webdriver.support.wait import webdriverwait selenium.webdriver.common.by import selenium.webdriver.support import expected_conditions ec scrapybot import __init__  class scrapybot_spider(scrapy.spider):     name = "scrapybot_spider"     allowed_domains = ['https://weedmaps.com']     regionlinks = []     dispensarylinks = []     start_urls = ["https://weedmaps.com/dispensaries/in/united-states/colorado"]      #initialize selenium webdriver via firefox     def __init__(self):         self.browser = webdriver.firefox()      #scraping regional links in states     def parse(self, response):         self.browser.get('https://weedmaps.com/dispensaries/in/united-states/colorado')         wait = webdriverwait(self.browser, 10)         wait.until(ec.visibility_of_element_located((by.css_selector, "div.subregion a.region-ajax-link")))         region in self.browser.find_elements_by_css_selector("div.subregion a.region-ajax-link"):             time.sleep(5)             region = region.get_attribute("href")             self.regionlinks.append(region)             print regionlinks       #scraping dispensary links within regional links above     def dispensaryparse(self, response):         global region         self.browser.get(region)         wait = webdriverwait(self.browser, 10)         wait.until(ec.visibility_of_element_located((by.css_selector, "div.dispensary div.name a")))         dispensary in self.browser.find_elements_by_css_selector("div.dispensary div.name a"):             dispensary = dispensary.get_attribute("href")             self.dispensarylinks.append(dispensary)             print dispensary             return dispensary


Comments

Post a Comment

Popular posts from this blog

angularjs - ADAL JS Angular- WebAPI add a new role claim to the token -

i have created angular spa webapi backend using adal js authentication.as there no roles in ad, need manually add role claims in order give users access different api controllers. the roles stored in database. expecting inject claim through call webapi after authentication ad. webapi code might this. identity.addclaim(new claim("role", "user")); var ticket = new authenticationticket(identity, props); var accesstoken = startup.oauthbeareroptions.accesstokenformat.protect(ticket); is possible replace adal idtoken new token? is viable solution or there other better way handle this? as initial token generated azuread, possible edit token add new claim? appreciated. graph api supporting group claims. see here: http://justazure.com/azure-active-directory-part-4-group-claims/ if @ examples on page, users assigned groups , app can check group in claims. in current version of portal, need app manifest , modify it.
Read more

node.js - Using Node without global install -

i'm working on few node & electron (atom shell) projects , i'm curious how them work without user having install node globally. i've seen few applications including node directory in project contains node.exe binary. how go setting node project use binary? edit: thought should add, i've searched everywhere answer this, unrelated results.
Read more

php - CakePHP HttpSockets send array of paramms -

i problem when make request (i can request). $link = 'http://uri/path/?param1=1&sid=2&sid=3&sid=4' $http = $httpsocket = new httpsocket(); $res = $httpsocket->get($link); if check request pr($http); i see : [line] => /url/path/?param1=1&sid%5b0%5d=2&sid%5b1%5d=3&sid%5b2%5d=4 http/1.1 and i'm getting empty response because server doesn't know parameters sid%5b0%5d=2 when try send parameters array : $data = array('param1' => '1', 'sid' => array('2','3', '4')); i see same changes %5b0%5d - additional indexes. how fix it? unfortunately , can not change server side, if send request in browser, i'll normal response in json format. when make request, can parameters this: $param1 = $_get['param1']; $sid= $_get['sid']; tip: use diferente names parameters. have multiples sid. use sid1, sid2, sid3... update: can parameters this:
Read more