python - BeautifulSoup adds content when I run find_all -


i trying scrape listings yp.com, , in building code, able isolate section names (div class="search-results organic"), when run find_all() on content, returns listings outside section.

the url http://www.yellowpages.com/search?search_terms=septic&geo_location_terms=80521

here running:

from bs4 import beautifulsoup import urllib import re import xml import requests urlparse import urlparse  filename = "webspyorganictag.html" term = "septic" zipcode = "80521" url = "http://www.yellowpages.com/search?search_terms="+ term +"&geo_location_terms="+ zipcode  open(filename, "w") myfile:     myfile.write("information organic<br>")  r = requests.get(url) soup = beautifulsoup(r.content, "xml") organic = soup.find("div", {"class": "search-results organic"})  open(filename, "a") myfile:     myfile.write(str(organic))

and returns content in organic listings section. there 30 listings.

then, add:

listings = organic.find_all("div", {"class": "info"}) = 1 open(filename, "a") myfile:     listing in listings:         myfile.write("this listing " + str(i) + "<br>")         myfile.write(str(listing) + "<br>")         += 1

and returns original 30 listings plus 10 more listings (aside id="main-aside") not included in variable 'organic'.

shouldn't calling organic.find_all() limit scope data in variable 'organic'?

using "xml" finding 41 class="info"> soup.find("div", {"class": "search-results organic"}) not surprising 41 returned find_all. getting other elements returned can seen looking @ organic returns i.e href="/wray-co/mip/ritcheys-redi-mix-precast-inc-10367117?lid=1000575822573", href="/longmont-co/mip/rays-backhoe-service-6327932?lid=216924340" , every other listing ten featured.

if @ line 41 of html write contains:

href="/wray-co/mip/ritcheys-redi-mix-precast-inc-10367117?lid=1000575822573" last of featured listings.

the problem parser, if change parser "lxml":

soup = beautifulsoup(r.content,"lxml")  organic = soup.find("div", {"class": "search-results organic"})  print(len(organic.find_all("h3",{"class":"info"}))) 30

or use html.parser: 

soup = beautifulsoup(r.content,"html.parser")   organic = soup.find("div", {"class": "search-results organic"})  print(len(organic.find_all("div",{"class":"info"}))) 30

you correct result.


Comments

Post a Comment

Popular posts from this blog

node.js - Using Node without global install -

i'm working on few node & electron (atom shell) projects , i'm curious how them work without user having install node globally. i've seen few applications including node directory in project contains node.exe binary. how go setting node project use binary? edit: thought should add, i've searched everywhere answer this, unrelated results.
Read more

How to access a php class file from PHPFox framework into javascript code written in simple HTML file? -

i've website developed using phpfox v3.0.7 . now i'm trying implement 'sse(server sent events)' 1 of php files. reference i'm giving below necessary code part of file. file titled process.class.php , present @ location /var/www/module/notification/include/service/process.class.php present on server having ip http://34.124.40.142/ . <?php /** * [phpfox_header] */ header('content-type: text/event-stream'); header('cache-control: no-cache'); defined('phpfox') or exit('no dice!'); /** * * * @copyright [phpfox_copyright] * @author raymond benc * @package phpfox_service * @version $id: service.class.php 67 2009-01-20 11:32:45z raymond_benc $ */ class notification_service_process extends phpfox_service { /** * class constructor */ public function __construct() { $this->_stable = phpfox::gett('notification'); } public function add($...
Read more

java - Null response to php query in android, even though php works properly -

Image
table i'm trying retrieve above data using 2 php files below. want pass email parameter , return bets user = email. work fine when tested advanced rest client, problem not php. however, when try through android 2 java classes below, null pointer exception (logcat below) , json object returned null. i'm not sure error is, think i'm not passing parameter php , that's why it's producing null response. logcat 05-31 15:10:50.227 26074-26074/com.example.albert.betterapp e/androidruntime﹕ fatal exception: main process: com.example.albert.betterapp, pid: 26074 java.lang.runtimeexception: unable start activity componentinfo{com.example.albert.betterapp/com.example.albert.betterapp.displayallbets}: java.lang.nullpointerexception @ android.app.activitythread.performlaunchactivity(activitythread.java:2292) @ android.app.activitythread.handlelaunchactivity(activitythread.java:2350) @ android.app.activitythread.acce...
Read more