当服务器在做数据迁移的时候，老是要安装相关的scrapy软件，今天来整理一下。
安装依赖
yum update -y
yum groupinstall 'Development Tools'
yum install gcc gcc-devel -y
yum install libxml2 libxml2-devel -y
yum install libxslt libxslt-devel -y
yum install openssl openssl-devel -y
yum install libffi libffi-devel -y
yum install sqlite sqlite-devel -y
安装python3
下载地址：https://www.python.org/ftp/python/3.6.5/Python-3.6.5.tgz
tar -zxvf Python-3.6.5.tgz
cd Python-3.6.5
./configure --prefix=/usr/local/python3
make && make install
添加软连接
ln -s /usr/local/python3/bin/python3 python3
ln -s /usr/local/python3/bin/pip3 pip3这里笔者习惯使用pip3而不是pip
检查安装
python3
Python 3.6.0 (default, Jun 1 2018, 23:15:53)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-11)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 安装Twisted
wget https://twistedmatrix.com/Releases/Twisted/18.4/Twisted-18.4.0.tar.bz2
tar jxvf Twisted-18.4.0.tar.bz2
cd Twisted-18.4.0
python setup.py install安装scrapy
pip3 install lxml
pip3 install scrapy
# 创建软连接
ln -s /usr/local/python3/bin/scrapy /usr/bin/scrapy检查安装
scrapy -v
Scrapy 1.5.0 - no active project
Usage:
scrapy <command> [options] [args]
Available commands:
bench Run quick benchmark test
fetch Fetch a URL using the Scrapy downloader
genspider Generate new spider using pre-defined templates
runspider Run a self-contained spider (without creating a project)
settings Get settings values
shell Interactive scraping console
startproject Create new project
version Print Scrapy version
view Open URL in browser, as seen by Scrapy
[ more ] More commands available when run from project directory常用库安装
pip3 install requests
pip3 install selenium
pip3 install beautifulsoup4
pip3 install pyquery
pip3 install pymongo
pip3 install redis
pip3 install flask
pip3 install django
pip3 install jupyter
