我个人更喜欢在单独的容器中使用scrapy和硒，并进行docker化。通过这种方式，您可以轻松地安装和抓取几乎所有都以一种形式或另一种形式包含javascript的现代网站。这是一个例子：
使用scrapy startproject来创建刮板并编写蜘蛛，其骨架可以像这样简单：
import scrapy
class MySpider(scrapy.Spider):
name = 'my_spider'
start_urls = ['https://somewhere.com']
def start_requests(self):
yield scrapy.Request(url=self.start_urls[0])
def parse(self, response):
# do stuff with results, scrape items etc.
# now were just checking everything worked
print(response.body)
真正的魔力发生在middlewares.py中。通过以下方式覆盖下载器中间件中的两个方法 ：__init__和 process_request：
# import some additional modules that we need
import os
from copy import deepcopy
from time import sleep
from scrapy import signals
from scrapy.http import HtmlResponse
from selenium import webdriver
class SampleProjectDownloaderMiddleware(object):
def __init__(self):
SELENIUM_LOCATION = os.environ.get('SELENIUM_LOCATION', 'NOT_HERE')
SELENIUM_URL = f'http://{SELENIUM_LOCATION}:4444/wd/hub'
chrome_options = webdriver.ChromeOptions()
# chrome_options.add_experimental_option("mobileEmulation", mobile_emulation)
self.driver = webdriver.Remote(command_executor=SELENIUM_URL,
desired_capabilities=chrome_options.to_capabilities())
def process_request(self, request, spider):
self.driver.get(request.url)
# sleep a bit so the page has time to load
# or monitor items on page to continue as soon as page ready
sleep(4)
# if you need to manipulate the page content like clicking and scrolling, you do it here
# self.driver.find_element_by_css_selector('.my-class').click()
# you only need the now properly and completely rendered html from your page to get results
body = deepcopy(self.driver.page_source)
# copy the current url in case of redirects
url = deepcopy(self.driver.current_url)
return HtmlResponse(url, body=body, encoding='utf-8', request=request)
不要忘记通过取消注释settings.py文件中的以下行来启用此中间件软件：
DOWNLOADER_MIDDLEWARES = {
'sample_project.middlewares.SampleProjectDownloaderMiddleware': 543,}
接下来进行dockerization。建立你的Dockerfile从轻量级映像的映像（我在这里使用python Alpine），将您的项目目录复制到其中，安装要求：
# Use an official Python runtime as a parent image
FROM python:3.6-alpine
# install some packages necessary to scrapy and then curl because it's handy for debugging
RUN apk --update add linux-headers libffi-dev openssl-dev build-base libxslt-dev libxml2-dev curl python-dev
WORKDIR /my_scraper
ADD requirements.txt /my_scraper/
RUN pip install -r requirements.txt
ADD . /scrapers
最后将它们整合在一起 docker-compose.yaml：
version: '2'
services:
selenium:
image: selenium/standalone-chrome
ports:
- "4444:4444"
shm_size: 1G
my_scraper:
build: .
depends_on:
- "selenium"
environment:
- SELENIUM_LOCATION=samplecrawler_selenium_1
volumes:
- .:/my_scraper
# use this command to keep the container running
command: tail -f /dev/null
运行docker-compose up -d。如果您是第一次进行此操作，则需要一段时间才能获取最新的硒/独立铬以及构建刮板图像。
完成后，您可以检查您的容器是否正在运行 docker ps还可以检查硒容器的名称是否与我们传递给我们的scraper容器的环境变量（此处为SELENIUM_LOCATION=samplecrawler_selenium_1）相匹配。
输入您的刮板容器docker exec -ti YOUR_CONTAINER_NAME sh，对我来说，命令是docker exec -ti samplecrawler_my_scraper_1 sh，cd进入正确的目录，然后使用以下命令运行刮板scrapy crawl my_spider。
整个内容都在我的github页面上，您可以从这里获取