forked from netkiller/netkiller.github.io
-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathshell.html
More file actions
256 lines (202 loc) · 14.6 KB
/
Copy pathshell.html
File metadata and controls
256 lines (202 loc) · 14.6 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"><html xmlns="http://www.w3.org/1999/xhtml"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /><title>11.3. Scrapy Shell</title><link rel="stylesheet" type="text/css" href="..//docbook.css" /><meta name="generator" content="DocBook XSL Stylesheets V1.79.1" /><meta name="keywords" content="php, pear, pecl, phar" /><link rel="home" href="../index.html" title="Netkiller Python 手札" /><link rel="up" href="index.html" title="第 11 章 Scrapy - Python web scraping and crawling framework" /><link rel="prev" href="scrapy.html" title="11.2. scrapy 命令" /><link rel="next" href="tutorial.html" title="11.4. 爬虫项目" /></head><body><a xmlns="" href="//www.netkiller.cn/">Home</a> |
<a xmlns="" href="//netkiller.github.io/">简体中文</a> |
<a xmlns="" href="http://netkiller.sourceforge.net/">繁体中文</a> |
<a xmlns="" href="/journal/index.html">杂文</a> |
<a xmlns="" href="//www.netkiller.cn/home/donations.html">打赏(Donations)</a> |
<a xmlns="" href="http://netkiller-github-com.iteye.com/">ITEYE 博客</a> |
<a xmlns="" href="http://my.oschina.net/neochen/">OSChina 博客</a> |
<a xmlns="" href="https://www.facebook.com/bg7nyt">Facebook</a> |
<a xmlns="" href="http://cn.linkedin.com/in/netkiller/">Linkedin</a> |
<a xmlns="" href="https://zhuanlan.zhihu.com/netkiller">知乎专栏</a> |
<a xmlns="" href="/search.html">Search</a> |
<a xmlns="" href="mailto:netkiller@msn.com">Email</a><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">11.3. Scrapy Shell</th></tr><tr><td width="20%" align="left"><a accesskey="p" href="scrapy.html">上一页</a> </td><th width="60%" align="center">第 11 章 Scrapy - Python web scraping and crawling framework</th><td width="20%" align="right"> <a accesskey="n" href="tutorial.html">下一页</a></td></tr></table><hr /></div><table xmlns=""><tr><td><iframe src="//ghbtns.com/github-btn.html?user=netkiller&repo=netkiller.github.io&type=watch&count=true&size=large" height="30" width="170" frameborder="0" scrolling="0" style="width:170px; height: 30px;" allowTransparency="true"></iframe></td><td><iframe src="//ghbtns.com/github-btn.html?user=netkiller&repo=netkiller.github.io&type=fork&count=true&size=large" height="30" width="170" frameborder="0" scrolling="0" style="width:170px; height: 30px;" allowTransparency="true"></iframe></td><td><iframe src="//ghbtns.com/github-btn.html?user=netkiller&type=follow&count=true&size=large" height="30" width="240" frameborder="0" scrolling="0" style="width:240px; height: 30px;" allowTransparency="true"></iframe></td></tr></table><div class="section"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a id="shell"></a>11.3. Scrapy Shell</h2></div></div></div>
<p>Scrapy Shell 是一个爬虫命令行交互界面调试工具，可以使用它分析被爬的页面</p>
<pre class="screen">
neo@MacBook-Pro /tmp % scrapy shell http://www.netkiller.cn
2017-09-01 15:23:05 [scrapy.utils.log] INFO: Scrapy 1.4.0 started (bot: scrapybot)
2017-09-01 15:23:05 [scrapy.utils.log] INFO: Overridden settings: {'DUPEFILTER_CLASS': 'scrapy.dupefilters.BaseDupeFilter', 'LOGSTATS_INTERVAL': 0}
2017-09-01 15:23:05 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
'scrapy.extensions.telnet.TelnetConsole',
'scrapy.extensions.memusage.MemoryUsage']
2017-09-01 15:23:05 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
'scrapy.downloadermiddlewares.retry.RetryMiddleware',
'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
'scrapy.downloadermiddlewares.stats.DownloaderStats']
2017-09-01 15:23:05 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
'scrapy.spidermiddlewares.referer.RefererMiddleware',
'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
'scrapy.spidermiddlewares.depth.DepthMiddleware']
2017-09-01 15:23:05 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2017-09-01 15:23:05 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023
2017-09-01 15:23:05 [scrapy.core.engine] INFO: Spider opened
2017-09-01 15:23:05 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.netkiller.cn> (referer: None)
[s] Available Scrapy objects:
[s] scrapy scrapy module (contains scrapy.Request, scrapy.Selector, etc)
[s] crawler <scrapy.crawler.Crawler object at 0x103b2afd0>
[s] item {}
[s] request <GET http://www.netkiller.cn>
[s] response <200 http://www.netkiller.cn>
[s] settings <scrapy.settings.Settings object at 0x1049019e8>
[s] spider <DefaultSpider 'default' at 0x104be2a90>
[s] Useful shortcuts:
[s] fetch(url[, redirect=True]) Fetch URL and update local objects (by default, redirects are followed)
[s] fetch(req) Fetch a scrapy.Request and update local objects
[s] shelp() Shell help (print this help)
[s] view(response) View response in a browser
>>>
</pre>
<div class="section"><div class="titlepage"><div><div><h3 class="title"><a id="response"></a>11.3.1. response</h3></div></div></div>
<p>response 是爬虫返回的页面，可以通过 css(), xpath() 等方法取出你需要的内容。</p>
<div class="section"><div class="titlepage"><div><div><h4 class="title"><a id="response.url"></a>11.3.1.1. 当前URL地址</h4></div></div></div>
<pre class="screen">
>>> response.url
'https://netkiller.cn/linux/index.html'
</pre>
</div>
<div class="section"><div class="titlepage"><div><div><h4 class="title"><a id="status"></a>11.3.1.2. status HTTP 状态</h4></div></div></div>
<pre class="screen">
>>> response.status
200
</pre>
</div>
<div class="section"><div class="titlepage"><div><div><h4 class="title"><a id="text"></a>11.3.1.3. text 正文</h4></div></div></div>
<p>返回 HTML 页面正文</p>
<pre class="screen">
response.text
</pre>
</div>
<div class="section"><div class="titlepage"><div><div><h4 class="title"><a id="response.css"></a>11.3.1.4. css</h4></div></div></div>
<p>css() 这个方法可以用来选择html和css</p>
<pre class="screen">
>>> response.css('title')
[<Selector xpath='descendant-or-self::title' data='<title>Netkiller ebook - Linux ebook</ti'>]
>>> response.css('title').extract()
['<title>Netkiller ebook - Linux ebook</title>']
>>> response.css('title::text').extract()
['Netkiller ebook - Linux ebook']
</pre>
<p>基于 class 选择</p>
<pre class="screen">
>>> response.css('a.ulink')[1].extract()
'<a class="ulink" href="http://netkiller.github.io/" target="_top">http://netkiller.github.io</a>'
>>> response.css('a.ulink::text')[3].extract()
'http://netkiller.sourceforge.net'
</pre>
<p>数组的处理</p>
<pre class="screen">
>>> response.css('a::text').extract_first()
'简体中文'
>>> response.css('a::text')[1].extract()
'繁体中文'
>>> response.css('div.blockquote')[1].css('a.ulink::text').extract()
['Netkiller Architect 手札', 'Netkiller Developer 手札', 'Netkiller PHP 手札', 'Netkiller Python 手札', 'Netkiller Testing 手札', 'Netkiller Java 手札', 'Netkiller Cryptography 手札', 'Netkiller Linux 手札', 'Netkiller FreeBSD 手札', 'Netkiller Shell 手札', 'Netkiller Security 手札', 'Netkiller Web 手札', 'Netkiller Monitoring 手札', 'Netkiller Storage 手札', 'Netkiller Mail 手札', 'Netkiller Docbook 手札', 'Netkiller Project 手札', 'Netkiller Database 手札', 'Netkiller PostgreSQL 手札', 'Netkiller MySQL 手札', 'Netkiller NoSQL 手札', 'Netkiller LDAP 手札', 'Netkiller Network 手札', 'Netkiller Cisco IOS 手札', 'Netkiller H3C 手札', 'Netkiller Multimedia 手札', 'Netkiller Perl 手札', 'Netkiller Amateur Radio 手札']
</pre>
<p>正则表达式</p>
<pre class="screen">
>>> response.css('title::text').re(r'Netkiller.*')
['Netkiller ebook - Linux ebook']
>>> response.css('title::text').re(r'N\w+')
['Netkiller']
>>> response.css('title::text').re(r'(\w+) (\w+)')
['Netkiller', 'ebook', 'Linux', 'ebook']
</pre>
<div class="section"><div class="titlepage"><div><div><h5 class="title"><a id="idp52"></a>11.3.1.4.1. 获取 html 属性</h5></div></div></div>
<p>通过 a::attr() 可以获取 html 标记的属性值</p>
<pre class="screen">
>>> response.css('td a::attr(href)').extract_first()
'http://netkiller.github.io/'
</pre>
</div>
</div>
<div class="section"><div class="titlepage"><div><div><h4 class="title"><a id="response.xpath"></a>11.3.1.5. xpath</h4></div></div></div>
<pre class="screen">
>>> response.xpath('//title')
[<Selector xpath='//title' data='<title>Netkiller ebook - Linux ebook</ti'>]
>>> response.xpath('//title/text()').extract_first()
'Netkiller ebook - Linux ebook'
</pre>
<p>xpath 也可以使用 re() 方法做正则处理</p>
<pre class="screen">
>>> response.xpath('//title/text()').re(r'(\w+)')
['Netkiller', 'ebook', 'Linux', 'ebook']
>>> response.xpath('//div[@class="time"]/text()').re('[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}')
['2017-09-21 02:01:38']
</pre>
<p>抽取HTML属性值，如图片URL。</p>
<pre class="screen">
>>> response.xpath('//img/@src').extract()
['graphics/spacer.gif', 'graphics/note.gif', 'graphics/by-nc-sa.png', '/images/weixin.jpg', 'images/neo.jpg', '/images/weixin.jpg']
</pre>
<p>筛选 class</p>
<pre class="screen">
>>> response.xpath('//a/@href')[0].extract()
'http://netkiller.github.io/'
>>> response.xpath('//a/text()')[0].extract()
'简体中文'
>>> response.xpath('//div[@class="blockquote"]')[1].css('a.ulink::text').extract()
['Netkiller Architect 手札', 'Netkiller Developer 手札', 'Netkiller PHP 手札', 'Netkiller Python 手札', 'Netkiller Testing 手札', 'Netkiller Java 手札', 'Netkiller Cryptography 手札', 'Netkiller Linux 手札', 'Netkiller FreeBSD 手札', 'Netkiller Shell 手札', 'Netkiller Security 手札', 'Netkiller Web 手札', 'Netkiller Monitoring 手札', 'Netkiller Storage 手札', 'Netkiller Mail 手札', 'Netkiller Docbook 手札', 'Netkiller Project 手札', 'Netkiller Database 手札', 'Netkiller PostgreSQL 手札', 'Netkiller MySQL 手札', 'Netkiller NoSQL 手札', 'Netkiller LDAP 手札', 'Netkiller Network 手札', 'Netkiller Cisco IOS 手札', 'Netkiller H3C 手札', 'Netkiller Multimedia 手札', 'Netkiller Perl 手札', 'Netkiller Amateur Radio 手札']
</pre>
<p>使用 | 匹配多组规则</p>
<pre class="screen">
>>> response.xpath('//ul[@class="topnews_nlist"]/li/h2/a/@href|//ul[@class="topnews_nlist"]/li/a/@href').extract()
</pre>
</div>
<div class="section"><div class="titlepage"><div><div><h4 class="title"><a id="headers"></a>11.3.1.6. headers</h4></div></div></div>
<pre class="screen">
response.headers.getlist('Set-Cookie')
</pre>
</div>
</div>
</div><div xmlns="" id="disqus_thread"></div><script xmlns="">
var disqus_config = function () {
this.page.url = "http://www.netkiller.cn"; // Replace PAGE_URL with your page's canonical URL variable
this.page.identifier = 'netkiller'; // Replace PAGE_IDENTIFIER with your page's unique identifier variable
};
(function() { // DON'T EDIT BELOW THIS LINE
var d = document, s = d.createElement('script');
s.src = '//netkiller.disqus.com/embed.js';
s.setAttribute('data-timestamp', +new Date());
(d.head || d.body).appendChild(s);
})();
</script><noscript xmlns="">Please enable JavaScript to view the <a href="https://disqus.com/?ref_noscript">comments powered by Disqus.</a></noscript><br xmlns="" /><script xmlns="" type="text/javascript" id="clustrmaps" src="//cdn.clustrmaps.com/map_v2.js?u=r5HG&d=9mi5r_kkDC8uxG8HuY3p4-2qgeeVypAK9vMD-2P6BYM"></script><div class="navfooter"><hr /><table width="100%" summary="Navigation footer"><tr><td width="40%" align="left"><a accesskey="p" href="scrapy.html">上一页</a> </td><td width="20%" align="center"><a accesskey="u" href="index.html">上一级</a></td><td width="40%" align="right"> <a accesskey="n" href="tutorial.html">下一页</a></td></tr><tr><td width="40%" align="left" valign="top">11.2. scrapy 命令 </td><td width="20%" align="center"><a accesskey="h" href="../index.html">起始页</a></td><td width="40%" align="right" valign="top"> 11.4. 爬虫项目</td></tr></table></div><script xmlns="">
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
})(window,document,'script','//www.google-analytics.com/analytics.js','ga');
ga('create', 'UA-11694057-1', 'auto');
ga('send', 'pageview');
</script><script xmlns="" async="async">
var _hmt = _hmt || [];
(function() {
var hm = document.createElement("script");
hm.src = "https://hm.baidu.com/hm.js?93967759a51cda79e49bf4e34d0b0f2c";
var s = document.getElementsByTagName("script")[0];
s.parentNode.insertBefore(hm, s);
})();
</script><script xmlns="" async="async">
(function(){
var bp = document.createElement('script');
var curProtocol = window.location.protocol.split(':')[0];
if (curProtocol === 'https') {
bp.src = 'https://zz.bdstatic.com/linksubmit/push.js';
}
else {
bp.src = 'http://push.zhanzhang.baidu.com/push.js';
}
var s = document.getElementsByTagName("script")[0];
s.parentNode.insertBefore(bp, s);
})();
</script><script xmlns="" type="text/javascript" src="/js/q.js" async="async"></script></body></html>