python - Scrapy, Parse items from 1st page then a follow link to get additional items -

-

update: able moving, doesn't return subpage , iterate sequence again. data trying extract in table this:

table date_1 | source_1 | link article_1 | date_2 | source_2 | link article_2 | etc....

and need first collect date_1, source_1 , go link article , repeat...

any appreciated. :)

from scrapy.spiders import basespider, rule scrapy.selector import htmlxpathselector scrapy.contrib.linkextractors import linkextractor dirbot.items import websiteloader scrapy.http import request scrapy.http import htmlresponse    class dindexspider(basespider): name = "dindex" allowed_domains = ["newslookup.com"] start_urls = [       "http://www.newslookup.com/business/" ]  def parse_subpage(self, response):     self.log("scraping: " + response.url)     il = response.meta['il']     time = response.xpath('//div[@id="update_data"]//td[@class="stime3"]//text()').extract()     il.add_value('publish_date', time)     yield il.load_item()   def parse(self, response):     self.log("scraping: " + response.url)     hxs = htmlxpathselector(response)     sites = hxs.select('//td[@class="article"]')      site in sites:         il = websiteloader(response=response, selector=site)         il.add_xpath('name', 'a/text()')         il.add_xpath('url', 'a/@href')         yield request("http://www.newslookup.com/business/", meta={'il': il}, callback=self.parse_subpage)

that's because need use crawlspider class instead of basespider:

from scrapy.spiders import crawlspider  class dindexspider(crawlspider):     # ...

Comments

Post a Comment

Popular posts from this blog

Delphi XE2 Indy10 udp client-server interchange using SendBuffer-ReceiveBuffer -

-
i use delphi xe2 , indy10 udp protocol. can't receive server echo on client side if use receivebuffer method. got "socket error # 10040" although send small echo message server client. console application illustrate problem below. in advance. program project1; {$apptype console} {$r *.res} uses system.sysutils, idglobal, idbasecomponent, idcomponent, idsockethandle, idudpclient, idudpserver, idudpbase, idstack; type tudp_serv = class(tidudpserver) procedure udpsvudpread(athread: tidudplistenerthread; adata: tidbytes; abinding: tidsockethandle); end; var udpserver: tudp_serv; udpcl: tidudpclient; bsnd, brcv: tbytes; s: string; k: integer; //============================================================================== procedure tudp_serv.udpsvudpread(athread: tidudplistenerthread; adata: tidbytes; abinding: tidsockethandle); begin writeln(' server read: ' + tohex(adata, length(adata))); abinding sendto(peerip, peerport, adat...
Read more

Qt ActiveX WMI QAxBase::dynamicCallHelper: ItemIndex(int): No such property in -

-
i new windows programming. here goes code objiwbemlocator = new qaxobject("wbemscripting.swbemlocator"); objwmiservice = objiwbemlocator->querysubobject("connectserver(qstring&,qstring&)",qstring("."),qstring("root\\cimv2")); qaxobject* returnlist = objwmiservice->querysubobject("execquery(qstring&)", qstring("select * %1").arg(domain)); qaxobject* result = returnlist->querysubobject("itemindex(int)", 0); i getting error on runtime qt activex wmi qaxbase::dynamiccallhelper: itemindex(int): no such property in but itemindex method exists msdn says the itemindex method not work collections not contain swbemobjects, such swbemmethodset, swbemnamedvalueset, swbemprivilegeset, swbempropertyset, , swbemqualifierset. what selecting ? have tried iterating on collection ? plus need minimum windows vista itemindex method. hope helps
Read more

Enable autocomplete or intellisense in Atom editor for PHP -

-
i'm first time user of atom editor php code. atom can autocomplete built-in functions of php. possible make editor autocomplete custom function written in separate file? function.php <?php function printsomething() { echo "hello world"; } index.php <?php require_once "function.php"; printsomething(); // autocompleted atom editor while typing thanks in advance. there text file in %userprofile%.atom called snippets.cson. in there can add snippet. more this: http://flight-manual.atom.io/using-atom/sections/snippets/ but need think can use later on other files well. otherwise need copy , paste.
Read more