CentOS安装python3及scrapy

当服务器在做数据迁移的时候,老是要安装相关的scrapy软件,今天来整理一下。

安装依赖

yum update -y
yum groupinstall 'Development Tools'
yum install gcc gcc-devel -y
yum install libxml2 libxml2-devel -y
yum install libxslt libxslt-devel -y
yum install openssl openssl-devel -y
yum install libffi libffi-devel -y
yum install sqlite sqlite-devel -y

安装python3

下载地址:https://www.python.org/ftp/python/3.6.5/Python-3.6.5.tgz

tar -zxvf Python-3.6.5.tgz
cd Python-3.6.5
./configure --prefix=/usr/local/python3
make && make install

添加软连接

ln -s /usr/local/python3/bin/python3 python3
ln -s /usr/local/python3/bin/pip3 pip3

这里笔者习惯使用pip3而不是pip

检查安装

python3

Python 3.6.0 (default, Jun  1 2018, 23:15:53)                                                                                                             
[GCC 4.8.5 20150623 (Red Hat 4.8.5-11)] on linux                                                                                                          
Type "help", "copyright", "credits" or "license" for more information.                                                                                    
>>>  

安装Twisted

wget https://twistedmatrix.com/Releases/Twisted/18.4/Twisted-18.4.0.tar.bz2
tar jxvf Twisted-18.4.0.tar.bz2
cd Twisted-18.4.0
python setup.py install

安装scrapy

pip3 install lxml
pip3 install scrapy
# 创建软连接
ln -s /usr/local/python3/bin/scrapy /usr/bin/scrapy

检查安装

scrapy -v

Scrapy 1.5.0 - no active project                                                                                                                          
                                                                                                                                                          
Usage:                                                                                                                                                    
  scrapy <command> [options] [args]                                                                                                                       
                                                                                                                                                          
Available commands:                                                                                                                                       
  bench         Run quick benchmark test                                                                                                                  
  fetch         Fetch a URL using the Scrapy downloader                                                                                                   
  genspider     Generate new spider using pre-defined templates                                                                                           
  runspider     Run a self-contained spider (without creating a project)                                                                                  
  settings      Get settings values                                                                                                                       
  shell         Interactive scraping console                                                                                                              
  startproject  Create new project                                                                                                                        
  version       Print Scrapy version                                                                                                                      
  view          Open URL in browser, as seen by Scrapy                                                                                                    
                                                                                                                                                          
  [ more ]      More commands available when run from project directory

常用库安装

pip3 install requests
pip3 install selenium
pip3 install beautifulsoup4
pip3 install pyquery
pip3 install pymongo
pip3 install redis
pip3 install flask
pip3 install django
pip3 install jupyter