{"id":1247,"date":"2016-11-05T10:53:05","date_gmt":"2016-11-05T14:53:05","guid":{"rendered":"http:\/\/www.craigperler.com\/blog\/?p=1247"},"modified":"2024-06-06T23:25:25","modified_gmt":"2024-06-07T03:25:25","slug":"web-scraping-part-4-scrapy","status":"publish","type":"post","link":"https:\/\/www.craigperler.com\/blog\/2016\/11\/05\/web-scraping-part-4-scrapy\/","title":{"rendered":"Web Scraping, Part 4 – Scrapy"},"content":{"rendered":"
\n
\"\"<\/figure>\n<\/div>\n\n\n

While it’s all well and good to know how to retrieve pages, parse them, and build a crude web crawler, there’s no reason to reinvent the wheel given there are many open sources libraries that package this up for you.  The most prolific Python library for scraping is Scrapy<\/a>.<\/p>\n\n\n\n

In this post, I’ll use Scrapy to scrape the American Gem Society<\/a>, the same site we scraped in the last post<\/a>.<\/p>\n\n\n\n

What is Scrapy?<\/h2>\n\n\n\n

Scrapy is a modular framework for crawling and parsing the web.  “Modular” here means that each component of Scrapy is not just customizable, but can be re-written on its own and plugged back in.  By far, the best way to learn about the library is via the official documentation<\/a>.  (What we cover here is but a fraction of the feature set offered by Scrapy.)<\/p>\n\n\n\n

We develop with Scrapy by building “spiders” – bits of code that crawl through the web.  A spider defines what we want to scrape and in some cases how we want to scrape it.  As spiders traverse the web, they hand back “items” – the data we have extracted from the web.  Of course, this is a gross simplification.  Scrapy has support for cookie management, downloading different types of files, multi-server clustering, throttling, server proxies, etc…<\/p>\n\n\n\n

Installing Scrapy<\/h2>\n\n\n\n

In theory, installing Scrapy is easy as pip install Scrapy<\/code> – but there are all sorts of things that can go wrong.  I defer to the official install guide<\/a>.  Once Scrapy is installed, then you should be able to run scrapy<\/em> at your command line.  If you run scrapy createproject ags<\/code>, that will create the necessary folder structure and basic files you’ll need to create a working spider.  In this case, ags<\/em> will be the root folder.  Head on over to \/ags\/spiders and create a new file, ags_spider.py.<\/p>\n\n\n

\nfrom scrapy.spiders.crawl import CrawlSpider\n\nclass AmericanGemSpider(CrawlSpider):\n    name = 'ags'\n    allowed_domains = ['www.americangemsociety.org']\n    start_urls = ['https:\/\/www.americangemsociety.org\/en\/find-a-jeweler']\n<\/pre><\/div>\n\n\n
Let’s break this down. We have defined a new spider, AmericanGemSpider, that extends from Scrapy’s CrawlSpider. The CrawlSpider class provides some basic functionality for crawling web pages – if you want to crawl the web, use a CrawlSpider. We’ve given our spider a name, ags<\/em> – every spider needs a name.  We’ve then told the spider the domains it’s allowed to crawl, and the page from which to start crawling.<\/p>\n\n\n\n
The allowed_domains<\/em> field restricts what domains our spider is allowed to crawl.  Many sites have pages on subdomains, so we need to account for that.  Also, you might just want a spider that can crawl over different sites entirely.<\/p>\n\n\n\n
The start_urls<\/em> is where our spider will start.  We’re specifying just one page, but this can be a long list of pages.  Our first step is pulling out the link for each state; we could just specify those links explicitly here, but it’s way less typing to give just the main URL.<\/p>\n\n\n\n
Defining Spider Rules<\/h2>\n\n\n\n
The next step is to define some rules for how we find links to traverse, and what we do when we find those links.  Remember, with AGS, we want to find the state-specific link for each state on the main page<\/a>; on each state’s page<\/a>, we want to find links for each jeweler’s page<\/a>; and then when we get to the jeweler pages, we want to extract some data.  We accomplished identifying links with XPath in the last post, and we can re-use the same XPath here:<\/p>\n\n\n\n
state_link_xpath = '\/\/area'
\njeweler_link_xpath = '\/\/a[@class=\"jeweler__link\"]'<\/code><\/p>\n\n\n\n
With Scrapy, we need to wrap this XPath inside “link extractors” which provide the instrumentation for converting XPath into traversable links in Scrapy.  Then each of those link extractors is wrapped in a rule, which tells Scrapy what to do with the pages retrieved for each traversed link.<\/p>\n\n\n
\nfrom scrapy.linkextractors.lxmlhtml import LxmlLinkExtractor\nfrom scrapy.spiders.crawl import Rule, CrawlSpider\n\nclass AmericanGemSpider(CrawlSpider):\n    name = 'ags'\n    allowed_domains = ['www.americangemsociety.org']\n    start_urls = ['https:\/\/www.americangemsociety.org\/en\/find-a-jeweler']\n\n    rules = [\n        Rule(LxmlLinkExtractor(restrict_xpaths='\/\/area')),\n        Rule(LxmlLinkExtractor(restrict_xpaths='\/\/a[@class=\"jeweler__link\"]'), callback='parse_jeweler_page')\n    ]\n<\/pre><\/div>\n\n\n
We’ve assigned a list of Rule objects to our spider’s rules<\/em> field.  Whenever our spider downloads a page, it will use our rules to find links.  On the starting page, it will find each of the state links with the first rule.  Scrapy will download each of those pages, and then run the rules over those newly downloaded pages.  The XPath for the first rule doesn’t identify any links on those state pages, as expected; the XPath for the second rule does – it finds all of the jeweler-specific links.<\/p>\n\n\n\n
With this second rule, we’ve specified a callback<\/em> function.  When Scrapy downloads the pages for each of the jeweler-specific links, it will hand the response to a new function we’ve defined, parse_jeweler_page<\/em>, where we can analyze the HTML and extract the data.  If we didn’t specify a callback here, then the default behavior would be to use the rules to find links on the page, and continue traversing.  Neither of our rules would find any links on the jeweler pages, so that would effectively terminate the crawl.  Given we’ve specified a callback, we get access to the HTML responses, and can use XPath to extract the data.<\/p>\n\n\n\n
Parsing the Page<\/h2>\n\n\n\n
Here is the parse_jeweler_page<\/em> method where we convert the HTML into a dictionary of extracted data:<\/p>\n\n\n
\n    def parse_jeweler_page(self, response):\n        yield {\n            f: response.xpath(s).extract_first() for f, s in {\n                'name': '\/\/h1[@class=\"page__heading\"]\/text()',\n                'grading': '\/\/p[@class=\"appraiser__grading\"]\/strong\/text()',\n                'certified': '\/\/p[@class=\"appraiser__certified\"]\/following-sibling::ul\/\/li\/text()',\n                'address': '\/\/p[@class=\"appraiser__hours\"]\/text()',\n                'phone': '\/\/p[@class=\"appraiser__phone\"]\/text()',\n                'website': '\/\/p[@class=\"appraiser__website\"]\/a\/@href'\n            }.items()\n        }\n<\/pre><\/div>\n\n\n
Rather than returning data, this method uses yield<\/em> to hand execution back to the caller.  In this example, we are returning one object per response, so there’s no real distinction between using yield<\/em> versus return<\/em>.  In other cases, when there might be multiple objects returned, or if the callback is both extracting data and generating new links to crawl, yield<\/em> makes more sense – it allows the Scrapy engine to work on the returned information while the spider continues executing.<\/p>\n\n\n\n
Here we are yielding a dictionary comprehension.  The dictionary returned is a transformation of the dictionary in lines 4-9, which is made up of field names and the corresponding XPath needed to extract that information.  The parameter passed to our callback is a wrapped response that exposes an xpath<\/em> method, so we can simply call that on each of our XPaths and get the data we want.  This method returns data that looks like:<\/p>\n\n\n\n
{'website': u'http:\/\/www.jcjewelers.com', 'name': u'J.C. Jewelers', 'grading': u'Offers AGS Labs Diamond Grading Reports', 'phone': u' (307) 733-5933', 'address': u' 132 N Cache St, Jackson, WY 83001-8681', 'certified': u'Jan Case, CGA'}<\/code><\/p>\n\n\n\n
The Complete Example<\/h2>\n\n\n
\nfrom scrapy.linkextractors.lxmlhtml import LxmlLinkExtractor\nfrom scrapy.spiders.crawl import Rule, CrawlSpider\n\nclass AmericanGemSpider(CrawlSpider):\n    name = 'ags'\n    allowed_domains = ['www.americangemsociety.org']\n    start_urls = ['https:\/\/www.americangemsociety.org\/en\/find-a-jeweler']\n\n    rules = [\n        Rule(LxmlLinkExtractor(restrict_xpaths='\/\/area')),\n        Rule(LxmlLinkExtractor(restrict_xpaths='\/\/a[@class=\"jeweler__link\"]'), callback='parse_jeweler_page')\n    ]\n\n    def parse_jeweler_page(self, response):\n        yield {\n            f: response.xpath(s).extract_first() for f, s in {\n                'name': '\/\/h1[@class=\"page__heading\"]\/text()',\n                'grading': '\/\/p[@class=\"appraiser__grading\"]\/strong\/text()',\n                'certified': '\/\/p[@class=\"appraiser__certified\"]\/following-sibling::ul\/\/li\/text()',\n                'address': '\/\/p[@class=\"appraiser__hours\"]\/text()',\n                'phone': '\/\/p[@class=\"appraiser__phone\"]\/text()',\n                'website': '\/\/p[@class=\"appraiser__website\"]\/a\/@href'\n            }.items()\n        }\n<\/pre><\/div>\n\n\n
That’s a pretty small amount of code to extract a whole lot of data.  You can run the spider by executing scrapy crawl ags<\/code> – here’s sample output:<\/p>\n\n\n
\n(scraper)Craigs-MBP:scraper craigperler$ scrapy crawl americangem2\n2016-11-04 14:10:09 [scrapy] INFO: Scrapy 1.0.5 started (bot: boot)\n2016-11-04 14:10:09 [scrapy] INFO: Optional features available: ssl, http11\n2016-11-04 14:10:09 [scrapy] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'boot.spiders', 'CONCURRENT_REQUESTS_PER_DOMAIN': 1, 'CONCURRENT_REQUESTS': 1, 'SPIDER_MODULES': ['thuzio.spider', 'crunchbase.spider', 'leafly.spider', 'shopify.spider', 'florida_bar.spider', 'indiegogo.spider', 'uktariff.spider', 'giaalumni.spider', 'americangem.spider', 'australiantpb.spider', 'walmart.spider', 'aidn.spider', 'reddit.spider', 'capropertysearch.spider', 'pokemongomap.spider', 'selfstoragefinders.spider', 'yelp.spider', 'catholicdirectory.spider', 'americangem.flat_spider'], 'BOT_NAME': 'boot', 'USER_AGENT': 'Mozilla\/5.0 (X11; Linux x86_64) AppleWebKit\/53 (KHTML, like Gecko) Chrome\/15.0.87', 'DOWNLOAD_DELAY': 0.5}\n2016-11-04 14:10:09 [scrapy] INFO: Enabled extensions: CloseSpider, TelnetConsole, LogStats, CoreStats, SpiderState\n2016-11-04 14:10:09 [scrapy] INFO: Enabled downloader middlewares: CrackDistilMiddleware, HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMiddleware, CookiesMiddleware, ChunkedTransferMiddleware, DownloaderStats\n2016-11-04 14:10:09 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware\n2016-11-04 14:10:09 [scrapy] INFO: Enabled item pipelines:\n2016-11-04 14:10:09 [scrapy] INFO: Spider opened\n2016-11-04 14:10:09 [scrapy] INFO: Crawled 0 pages (at 0 pages\/min), scraped 0 items (at 0 items\/min)\n2016-11-04 14:10:09 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023\n2016-11-04 14:10:10 [scrapy] DEBUG: Crawled (200) &lt;GET https:\/\/www.americangemsociety.org\/en\/find-a-jeweler&gt; (referer: None)\n2016-11-04 14:10:12 [scrapy] DEBUG: Crawled (200) &lt;GET https:\/\/www.americangemsociety.org\/en\/nevada-jewelers&gt; (referer: https:\/\/www.americangemsociety.org\/en\/find-a-jeweler)\n2016-11-04 14:10:14 [scrapy] DEBUG: Crawled (200) &lt;GET https:\/\/www.americangemsociety.org\/en\/wyoming-jewelers&gt; (referer: https:\/\/www.americangemsociety.org\/en\/find-a-jeweler)\n2016-11-04 14:10:15 [scrapy] DEBUG: Crawled (200) &lt;GET https:\/\/www.americangemsociety.org\/en\/500194&gt; (referer: https:\/\/www.americangemsociety.org\/en\/nevada-jewelers)\n2016-11-04 14:10:15 [scrapy] DEBUG: Scraped from &lt;200 https:\/\/www.americangemsociety.org\/en\/500194&gt;\n{'website': u'http:\/\/www.tbirdjewels.com', 'name': u'T-Bird Jewels', 'grading': u'Offers AGS Labs Diamond Grading Reports', 'phone': u' (702) 256-3900', 'address': u' 1990 Village Center CirSte P6, Las Vegas, NV 89134-6242', 'certified': u'Jenny O Calleri, CGA'}\n2016-11-04 14:10:19 [scrapy] DEBUG: Crawled (200) &lt;GET https:\/\/www.americangemsociety.org\/en\/j-c-jewelers&gt; (referer: https:\/\/www.americangemsociety.org\/en\/wyoming-jewelers)\n2016-11-04 14:10:19 [scrapy] DEBUG: Scraped from &lt;200 https:\/\/www.americangemsociety.org\/en\/j-c-jewelers&gt;\n{'website': u'http:\/\/www.jcjewelers.com', 'name': u'J.C. Jewelers', 'grading': u'Offers AGS Labs Diamond Grading Reports', 'phone': u' (307) 733-5933', 'address': u' 132 N Cache St, Jackson, WY 83001-8681', 'certified': u'Jan Case, CGA'}\n...\n<\/pre><\/div>\n\n\n
Once the scraper is running, we can see our spider crawl the start_url, start traversing the state-specific pages, and as it finds the jeweler links, it grabs the responses, parses them, and spits out the data.  In this example, the data goes nowhere; in practice, we could use Scrapy’s item pipeline<\/a> to persist the objects, or do whatever we want them.<\/p>\n\n\n\n
There’s no reason not to use Scrapy.  Whether you’re working on a simple project (such as scraping AGS), or something massive, Scrapy provides a set of tools that you don’t need to write yourself.  If you need something very customized or specific, you can still start with Scrapy and just replace the components that need it.<\/p>\n","protected":false},"excerpt":{"rendered":"
While it’s all well and good to know how to retrieve pages, parse them, and build a crude web crawler, there’s no reason to reinvent the wheel given there are…<\/p>\n","protected":false},"author":1,"featured_media":1589,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[64],"tags":[],"powerkit_post_featured":[],"class_list":{"0":"post-1247","1":"post","2":"type-post","3":"status-publish","4":"format-standard","5":"has-post-thumbnail","7":"category-application-development"},"jetpack_featured_media_url":"https:\/\/i0.wp.com\/www.craigperler.com\/blog\/wp-content\/uploads\/2016\/11\/scrapy.webp?fit=300%2C300&ssl=1","jetpack_shortlink":"https:\/\/wp.me\/p1SwZ6-k7","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/www.craigperler.com\/blog\/wp-json\/wp\/v2\/posts\/1247","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.craigperler.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.craigperler.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.craigperler.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.craigperler.com\/blog\/wp-json\/wp\/v2\/comments?post=1247"}],"version-history":[{"count":5,"href":"https:\/\/www.craigperler.com\/blog\/wp-json\/wp\/v2\/posts\/1247\/revisions"}],"predecessor-version":[{"id":1590,"href":"https:\/\/www.craigperler.com\/blog\/wp-json\/wp\/v2\/posts\/1247\/revisions\/1590"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.craigperler.com\/blog\/wp-json\/wp\/v2\/media\/1589"}],"wp:attachment":[{"href":"https:\/\/www.craigperler.com\/blog\/wp-json\/wp\/v2\/media?parent=1247"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.craigperler.com\/blog\/wp-json\/wp\/v2\/categories?post=1247"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.craigperler.com\/blog\/wp-json\/wp\/v2\/tags?post=1247"},{"taxonomy":"powerkit_post_featured","embeddable":true,"href":"https:\/\/www.craigperler.com\/blog\/wp-json\/wp\/v2\/powerkit_post_featured?post=1247"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}