|
|
相关文章推荐
-
All AWS Services Root
-
Compute Docs
-
AWS EC2 Root
- AWS EC2 Overview
- Launch and SSH to EC2
- Use IAM Role or Instance Profile in EC2
- Amazon Machine Image (AMI) Management
- EC2 Instance Type
- Jump box vs Bastion Host
- Setup A Jump Box EC2
- Setup a Bastion Host Cluster
- AMI Backup Strategy
- Auto Scale Group (ASG)
- Elastic Load Balancer and Auto Scaling
- Deploy Flask App to EC2
- VM Virtualization 虚拟机技术
- Virtual Machine Virtualization Type
-
AWS Lambda Docs
- AWS Elastic Beanstalk (EB) Docs
-
AWS Batch Root
- What is AWS Batch
- AWS Batch Example Project
- Enterprise Level AWS Batch Project
- AWS Batch Job Definition Revision
- Array Job
-
Storage Docs
-
Amazon Simple Storage Service (S3) Root
- Hosting a static website using Amazon S3
- Summary
- 使用 S3 host static website 的关键步骤
- 使用自己的 Domain 的关键步骤
- Managing your storage lifecycle
- S3 Data Consistency Model
- Server Access Logging and Object Level Logging
- Using Object Lock
-
Using versioning in S3 buckets
- Using Versioning in S3 Bucket
-
Network and cContent Delivery
-
AWS VPC Root
- VPC Endpoint, Private Link
- VPN Endpoint
- VPC Peering
- VPC Transit Gateway
- VPC Security Group Best Practice
- AWS VPC Nat Devices, Nat Gateways and Nat Instances
- Nat Gateway vs VPC Endpoint
- Multi VPC Multi Region Multi Account Global Networking Architecture
- Route 53
- Direct Connect
- App Mesh
- VPN Split Tunnel
- VPC Connectivity Options
- Networking in AWS
-
Security Identity Compliance (SIC)
-
Identity Authentication Management (IAM) Root
- Setup IAM For Build Lab
- IAM Permission Boundary
- Delegate access across AWS accounts using IAM roles
- Congnito
- AWs Key Management Service (KMS) Docs
- AWS Secret Manager Docs
- AWS Cloud HSM Docs
- Web Application Firewall (WAF) and Shield Docs
-
Management and Governance
-
AWS Account Management Root
- AWS Organization Accounts Management
- AWS Account Management Overview
- Why User Assume Role for Multiple AWS Account
- Implement Assume Role Correctly
- Track Original Caller When Assume Role
- Use MFA Protection
- Remove Unused Resource
- Organizing Your AWS Environment Using Multiple Accounts
- AWS CLI Tutorial
- AWS Organization Accounts Management Runbook
- AWS Organization Policy
- AWS CloudWatch Docs
- AWS CloudTrail Docs
-
AWS System Manager Root
- Run Remote Command on EC2 via SSM
- AWS Parameter Store
- Run Remote Command on Ec2 Ultimate Solution
-
AWS CloudFormation Root
- CloudFormation Quick Start
- How to Automate Test for your Infrastructure as Code
- Nested Stack vs Cross Stack Pattern
- AWS CloudFormation StackSets
- 一个简单, 却又包含了大部分设计模式的 CloudFormation 项目
-
AWS CDK Root
- Getting Started with AWS CDK
-
CDK Examples
- Learn AWS CDK Pipeline
-
AWS Database Service Docs
-
AWS DynamoDB Root
- Dynamodb Cheatsheet
- What is Dynamodb
- Dynamodb Table and Item
- Dynamodb Index
- DynamoDB Data Modeling
- DynamoDB Pricing
- Dynamodb Transaction
- Dynamodb Backups
- Dynamodb Best Practice
-
Dynamodb Tracker
- Dynamodb Tracker
- Dynamodb Architect
- PynamoDB Python Library
-
Dynamodb Blogs
- How to determine if Amazon DynamoDB is appropriate for your needs, and then plan your migration
- AWS Elastic Cache Docs
-
AWS Redshift Root
- Authentication and Access a Redshift Cluster
- AWS Redshift User Management
-
AWS Redshift Best Practice
- Choose the best sort key
- Choose the best distribution style
- Use datetime types for date columns
- Best practice for loading data
- Working with data distribution style
- Working with sort keys
- Working with column compression
- Unloading Data
- AWS Redshift Quotas and Limits
- AWS Redshift Knowledge Graph
- Redshift Serverless
-
AWS Redshift Spectrum Root
- Backup and Snapshot
- AWS MemoryDB
-
AWS Timestream
- AWS TimeStream DB Summary
- AWS Timestream Quick Start
- TimeStream Data Modeling
-
AWS TimeStream DB - Code Example
-
AWS TimeStream DB - Code Example - Quick Start
- TimeStream Quick Start
- Data Generator
-
Containers
-
AWS Elastic Container Registry (ECR) Root
- AWS ECR Core Concepts
- AWS ECR Push and Pull Private Image
- AWS Elastic Container Service (ECS) Docs
- Elastic k8s Service
- AWS Fargate Root
- AWS App Runner Root
-
Analytics
-
AWS Athena Root
- AWS Athena Basic
- AWS Athena Declare a Table - DDL
- AWS Athena with Python
- Partition
- AWS Athena SQL Query Reference
- Optimize Performance and Cost
-
Kinesis Root
- Kinesis Basic
- Kinesis Advance
- Kinesis Producer
- Kinesis Consumer
- Kinesis Boto3 API
- AWS Kinesis Firehose Delivery Stream Basic
- Kinesis Firehose Delivery Stream Transformation
- Kinesis - Resharding a Stream
- Kinesis Consumer - Handle Slow Consumer
- Kinesis Analytics vs Redshift Pricing Compare
- Kinesis - Path to Master
- Elastic Map Reduce (EMR)
- AWS Glue Root
- Amazon ElasticSearch (AES)
-
AWS MSK (Managed Streaming for Kafka) Root
- MSK Serverless
- MSK Network Connection
- MSK Authentication Authorization
-
MSK Producer Consumer Best Practice
-
Kafka with Python
- Kafka Python
-
AWS - Developer Tools Root
-
AWS CodeCommit Root
- CodeCommit Basic
- AWS CodeCommit and CodeBuild CI/CD Example
- AWS CodeCommit Authentication
- AWS CodeCommit Event
- Sync CodeCommit Repo
-
AWS Code Build
- AWS CodeBuild Basic
- AWS CodeBuild BuildSpec File
- Trigger Build Job with boto3
- AWS CodeBuild Runtimes
- Trunkbased CICD with CodeCommit and CodeBuild
- Monorepo CICD with CodeBuild
- Large Project DevOps and CI/CD
- CodeBuild Artifacts
-
AWS CodeArtifact Root
- CodeArtifact Basic
- CodeArtifact Advance
- AWS CodeArtifact with Python
-
AWS Code Pipeline Root
- CI/CD Product Overview
- CodePipeline Basic
- Single Account Single Region Multi Environment CICD
- Multi Environment Multi AWS Account Lambda App CI/CD
-
AWS Cloud9 Root
- What is AWS Cloud9
- Set up cloud9 development environment
- AWS Cloud9 Best Practice
- SSH To AWS Cloud9
-
Application Integration Services
-
Simple Queue Service (SQS) Docs
- Understand SQS
- Standard Queue
- FIFO Queues
- Dead Letter Queues
- Avoid Double Consume
- Short and Long Polling
- Pub Sub Pattern with SQS (用 SQS 实现发布订阅模式)
-
Simple Queue Service (SQS) POC
- Simple Notification Service (SNS) Docs
-
AWS EventBridge Root
- What is Amazon EventBridge
- Learn EventBridge by Doing
-
AWS Machine Learning Services
-
AWS Sagemaker Root
- What is Sagemaker
- Sagemaker Domain, Studio and Canvas
- Catalog Models with Model Registry
- MLOps foundation roadmap for enterprises with Amazon SageMaker
- Sagemaker Notebook Instance
- Multi-account model deployment with Amazon SageMaker Pipelines
- Sagemaker Script Mode
-
Amazon Comprehend Root
- Amazon Comprehend Overview
- Entities Detection
- Key Phrases
- Dominant Language
- Sentiment
- Target Sentiment
- Entities Detection
-
Custom Document Classification
- Amazon Comprehend - Custom Document Classification Example
- 一些准备工作
- Unzip Dataset
- Split Train and Test
- 将数据整理成 Comprehend Manifest CSV file 的格式
- Create Document Classifier
- Start Document Classification Job
- Evaluate Prediction Result
- Custom Entities detection
-
What is AWS Textract
- Amazon Textract Overview
-
amazon-textract-textractor
- Explore amazon-textract-textractor
- Set AWS Credential
-
Use Textract with PDF and Image
-
AWS Augmented AI Root
- AWS Augmented AI Root
-
A2I Learning Lab
- Amazon Augmented AI Learning Lab
- Custom Task Template Example
- Text Insight Solution
- AWS Kendra Root
-
Migration and Transfer
- Linux Server Migration to AWS
- Database Migration Service
-
AWS Transfer Family
- Setup AWS Managed SFTP Server
- AWS Transfer Family managed workflows
- SFTP To S3 Ingestion Best Practice
-
Linux Development Tool Docs
- fabric2 - Run Command Line on Remote Server via SSH
- Identify Current OS In Python
- doitlive - Presentation with Command Line
- jq Cheat Sheet
-
Software Development Engineer (SDE)
-
Programming Language Docs
-
Shell Scripting (Script) in Bash Docs
- Command Line (CLI) and Shell Script
- Shell Script - Special Character (特殊符号)
- Shell Script - String
- Shell Script - Redirection
- Shell Script - List
-
Shell Script - IF Else Condition Test
-
Function
- return vs exit in function
- Get Where the Bash Script Locate
- Set Environment Variable for Subsequent Script
-
Python: Beautiful is better than ugly
-
Type Hint in Python
- Typing Extensions
- Boto3 Stubs
- Type Hint With TypeVar and Generic
- Nested Parent and Child
- Depth First Search vs Width First Search in Python
- Pagination in Python
- 如何在 GitHub 上开发并维护一个 Python 开源项目?
-
Python: Testing Strategy
-
The Python Community Standard - pytest framework
- Pytest Mark
- Pytest Fixture Mechanism Deep Dive
- ``pytest`` ``tmp_path`` fixture
- Python: Mixin Pattern
- Manipulate PDF in Python
-
Awesome Python Library for Software Development
- PyPubSub - Publisher Subscriber Pattern (Pub Sub)
- Record and Replay your terminal Session
- Atomic Write to Files
-
attrs
- PyCharm Support
- Immutable 和 Mutable
- Converter
- bi-directional mapping dictionary
- ``cookiecutter``
- decorator
- enum - Enumeration
- Python Fire
- ijson - Interactive Json
- Create Interactive CLI tool
- Json dump any complex object
-
Locust Root - Python Load Test (Testing) Framework
- Locust - Summary
- Loguru 的设计
- 有几点特别注意事项
- Python Library - mock / pytest-mock
- MPIRE (MultiProcessing Is Really Easy)
-
Python Poetry
- Pygtrie, prefix tree data structure
- box - dot notation pattern for dictionary
-
Python Standard Library - ``subprocess``
-
subprocessExamples - Basic
- Advance
- Choose-Dataclass-and-Validation-Library
- Determinative Dependencies
- Distribute Python Package
- Python Requirements.txt Syntax
-
Sphinx Doc Root
- Host Your Docs on ReadTheDocs For OpenSource Project
- Sphinx Doc - Include Code File
- Sphinx Doc - Include Jupyter Notebook
- This is Title
-
Software Engineering Docs
-
Distribution System (分布式系统) Docs
- CAP Theorem (一致性, 可用性, 分区容错性 定理)
- Consistency (一致性)
-
Availability (可用性)
- Implement High Availability (实现高可用性)
- Node Cluster Patterns (节点集群模式)
- Partition Tolerance (分区容错)
-
Distribution System Algorithms
- Distributive System Middleware
- Distribution System Software
- Two Phase Commit
-
Microservices Architect (微服务架构)
- Design Build Deploy Microservices (设计, 构建, 部署微服务)
- Service Discovery 服务发现
- 负载均衡 与 代理
- Publisher Subscriber System
- Web Server 网站服务器
-
Documentation Skill for Software Engineer
-
RestructuredText RST Root
- RestructuredText RST with DrawIO
-
Docker Root
- Container Technique History
-
Docker Register Service
- Everyday Docker Command Cheatsheet
- Restrict CPU, Memory, GPU usage of Docker Container (限制容器所使用的系统资源)
- Build Your Own Docker Image (构建容器镜像)
- Deploy with Docker (使用 Docker 部署你的应用)
- Build Python Container Image Best Practice
- Choose Base Container Image For Python App
- Dockerfile Commands
- Minimize Container Image Size
-
Misc
- TinyPng
-
DevOps (运维) Docs
-
DevOps Practice in Testing
- Software Test Strategy and Category
-
DevOps Practice in Deployment
-
DevOps Best Practice in Chaos Engineering
-
Continuous Integration / Continuous Delivery (CI/CD or CICD) Docs
- CICD Solutions
-
CircleCI Docs
- Jenkins Docs
- Consistent Behavior in CI/CD
- Compare Solutions for Checkout Additional Repo in CI/CD
- Infrastructure as Code (IAC) - Terraform Docs
- Ansible
- Environment Variables
-
Cloud Migration (迁徙到云) Docs
-
Database Root
- ElasticSearch
- RDBMS ORM 中的常见问题
-
Developer
- 开发者专用字体
-
Git
- Understand Git Concept (理解 Git 的 概念)
- Understand Git Workflow (理解 Git 工作流)
- Git Authentication
- Semantic Versioning (SemVer)
- Conventional Commits
- Keep A ChangeLog
-
GitLab
-
GitHub
- Git and Github Desktop Basic
- GitHub Desktop
-
IDLE
-
PyCharm
- PyCharm Keymap / Shortcut Key (快捷键大全)
- PyCharm Settings
- Plugins
- Execute
- PyCharm Remote Development
-
Operation System (OS) Docs
-
Linux OS Docs
- Linux Command Learning Methodology
- Linux Command Cheat Sheet
- Linux Command For System Admin
- Linux Command Grep Sed Awk
- Linux Command ``zip``
- Create Command Line Tool in Python
-
Environment Variable (env var)
- Linux Daemon
- Linux System Directory Structure (Linux 系统文件目录结构)
- ODP Container Audit Log Best Practice
-
Linux Distribution
- Amazon Linux
- Chmod in Python
- Keep-Long-Live-Session-Using-GNU-Screen
-
MacOS Root
- Makefile介绍和使用说明
- Shell
- Embrace Windows 10
-
Open Source Software
-
Apache Airflow
-
Airflow Document
- What is Airflow
- UI Screenshots
-
Amazon Managed Workflows for Apache Airflow (MWAA) - Root
- Amazon MWAA Quick Start
- Airflow Common Patterns
- Amazon MWAA - Manage Environment
- ElasticSearch
-
draw.io
- Embed draw.io diagram in restructuredtext
-
如果你想使用洋葱浏览器进行匿名上网, 那么你要到 https://www.torproject.org/download/download-easy.html.en 下载洋葱浏览器软件. 洋葱浏览器会自动连上洋葱网络, 当浏览器关闭时会自动断开. 只有通过洋葱浏览器浏览时才是通过洋葱网络进行的匿名访问. 开着洋葱浏览器而用chrome浏览网页是没有效果的.
-
如果你想要在机器上运行洋葱网络的服务, 然后通过本机的一个端口去和外网通信, (这通常应用于你想用编程来对洋葱网络进行访问的情况). 那么请参考这个教程 https://www.torproject.org/docs/tor-doc-osx.html.en , 使用包管理软件安装tor, 然后使用以下三个命令
启动,重启,关闭对洋葱网络进行操作. -
brew services start tor -
brew services restart tor -
brew services stop tor Running Tor on Mac OS X: https://2019.www.torproject.org/docs/tor-doc-osx.html
Mac 上的安装命令是
brew install tor, Ubuntu 择有sudo apt install -y tor privoxy. 在安装前记得运行sudo apt update更新索引.Mac 上的配置文件的位置是
/usr/local/etc/tor/torrc和/usr/local/etc/privoxy/config, Ubuntu 则是/etc/tor/torrc和/etc/privoxy/config.Mac 上的启动服务的命令是
brew services start tor,brew services start privoxy, 而 Ubuntu 则是sudo service tor start,sudo service privoxy start(可以用 restart 命令替换 start 是的应用最新更改的配置文件).下载源代码:
下载页面 (下载链接在这个页面上找): https://www.torproject.org/download/tor/
安装 Build 依赖.
洋葱网络的作用
洋葱网络本身并不是一个网络, 而是一个通讯协议, 任何在互联网上同意洋葱协议, 并安装并运行着洋葱客户端的电脑互相之间连接在一起, 就组成了洋葱网络.
简单来说, 如果我想
tor是一个在 Windows, Mac, Linux (Windows 上的配置很复杂, 不推荐) 上的后台服务. 将你的机器连入洋葱网络, 给你分配一个临时的链路 (也就是从你的 IP, 到中间的跳转机器, 到最终发出真正 HTTP 请求的机器的链路), 在此期间, 你如果查询你的公网 IP, 会得到最终发出真正请求的机器的 IP. 这个链路会根据设置保持一段时间, 并自动更换. 你也可以手动更新. 然后你的 Http 请求不直接走默认 HTTP/HTTPS 端口, 而是通过 tor proxy 代理端口发出.在爬虫中, 你希望你能够随时切换 IP. 由于
tor服务不支持自动更换 IP, 你只能通过手动向其发送 SIGNAL 要求他重新连接, 来获得新的链路.下面我们来介绍如何实现自动在 Python 爬虫中自动切换 IP.
你需要安装
tor(洋葱网络主程序),privoxy(无缓存代理) 两个系统包, 在 MacOS 和 Linux 中的安装略有不同. 以及在 Python 中需要安装requests(进行 HTTP 请求),pysocks(通过 socks 端口通信),stem(跟tor互动, 发送 reconnect 信号)Mac 系统 ¶
# 安装 tor, privoxy brew install tor brew install privoxy修改 tor 配置文件:
选择一个密码, 用于防止外部程序使用 tor 端口. 例如 “mypassword”, 然后使用
tor --hash-password mypassword(请自行替换你的密码) 计算出 hash 值 (你不会把密码明文保存吧!), 例如 mypassword 的哈希值是16:81A060A2BC85DA2A60D4D81BFD3A8701E99B275CC6C1E7E711B9F1363F. 然后修改 tor 的配置文件/usr/local/etc/tor/torrc(如果没有, 备份/usr/local/etc/tor/torrc.sample并改名即可). 在其中加入以下三个配置 (请自行检查是否有重复,#后面的都是注释):ControlPort 9051 HashedControlPassword 16:81A060A2BC85DA2A60D4D81BFD3A8701E99B275CC6C1E7E711B9F1363F CookieAuthentication 1修改 privoxy 配置文件:
打开 privoxy 的配置文件
/usr/local/etc/privoxy/config. 在其中加入以下配置:forward-socks5t / 127.0.0.1:9050 . # the dot at the end is important然后在后台启动 tor, privoxy. 每次重启电脑后你都需要使用这个命令来启动洋葱网络:
brew services start tor brew services start privoxy使用 Python 通过洋葱网络访问网络并切换 IP:
首先安装依赖包:
pip install requests pip install pysocks pip install stem然后使用下面的代码, 你会看到当重新连接之后, ip 会改变:
# -*- coding: utf-8 -*- Install these packages first: .. code-block:: python pip install requests pip install pysocks pip install stem import time import requests from stem import Signal from stem.control import Controller RENEW_CONNECTION_WAIT_TIME = 5 def renew_connection(): Renew tor network connection (change route). with Controller.from_port(port=9051) as controller: controller.authenticate(password="mypassword") controller.signal(Signal.NEWNYM) controller.close() time.sleep(RENEW_CONNECTION_WAIT_TIME) def make_session(): Make a session to keep connect session = requests.session() session.proxies = {} session.proxies["http"] = "socks5h://localhost:9050" session.proxies["https"] = "socks5h://localhost:9050" return session def check_ip(): Use Amazon Web Service to check your public IP. session = make_session() url = "http://checkip.amazonaws.com/" return session.get(url).text print(check_ip()) renew_connection() print(check_ip())参考资料:
Linux 系统¶
Linux 有许多主流的发行版, 比如 CentOS, Redhat, Debian, Ubuntu, 以及 Amazon 为云定制的 Amazon Linux. 而 Tor 的安装在 Debian 和 Ubuntu 上最为简单, 可以使用系统自带的包管理工具 apt 安装. 而其他的 Linux 发行版则要么需要配置安装源, 要么需要编译源码安装, 步骤非常繁琐. 所以这里我们以 Ubuntu 为例进行安装.
Ubuntu 系统¶
在 Ubuntu 上的步骤跟在 Mac 上的步骤基本一致, 只有以下几个区别:
在 Ubuntu 上的 Python3 是系统所用的 Python, 不建议将一大堆库安装到系统中, 建议使用 virtualenv. 而自带的 Python3 没有 pip, 而
python3 get-pip.py会因为缺少distutils而导致失败, 所以要先运行sudo apt install python3-distutils, 再运行python3 get-pip.py.Amazon Linux (TODO)¶
Amazon Linux 默认使用 yum 包管理工具, 但是 Amazon Linux 的 repo 中并没有 tor, 所以你只能从源代码 Build. Build 过程参考官方文档 (本文内容也是根据这篇文档总结的) http://www.torproject.org/docs/tor-doc-unix.html.
为了避免从非官方渠道下载到带病毒的软件, 请严格根据官方文档中的指引从官方下载.
wget https://dist.torproject.org/tor-0.4.2.5.tar.gz
-
Airflow Document
-
Apache Airflow
-
Linux OS Docs
-
PyCharm
-
GitHub
-
Database Root
-
Continuous Integration / Continuous Delivery (CI/CD or CICD) Docs
-
DevOps Best Practice in Chaos Engineering
-
DevOps Practice in Testing
-
RestructuredText RST Root
-
Distribution System (分布式系统) Docs
-
-
The Python Community Standard - pytest framework
-
Type Hint in Python
-
Function
-
Shell Scripting (Script) in Bash Docs
-
Programming Language Docs
-
AWS Augmented AI Root
-
AWS Sagemaker Root
-
Simple Queue Service (SQS) Docs
-
AWS CodeCommit Root
-
Kafka with Python
-
AWS Athena Root
-
AWS Elastic Container Registry (ECR) Root
-
AWS TimeStream DB - Code Example - Quick Start
-
AWS DynamoDB Root
-
AWS Account Management Root
-
Identity Authentication Management (IAM) Root
-
AWS VPC Root
-
Amazon Simple Storage Service (S3) Root
-
AWS EC2 Root
-
Compute Docs