AWS (Amazon Web Services) 云计算
推荐资源
| AWS 官网 | https://aws.amazon.com/ |
| AWS Management Console | https://aws.amazon.com/console/ |
| AWS Doc |
https://docs.aws.amazon.com/ |
| AWS CLI (Lastest 2.13.20 Command Reference) | https://awscli.amazonaws.com/v2/documentation/api/latest/reference/index.html |
| AWS SDK | https://aws.amazon.com/developer/tools/ Python: https://aws.amazon.com/sdk-for-python/ Python API 1.28.53: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/index.html Java: https://docs.aws.amazon.com/sdk-for-java/ Java API 2.20.153: https://sdk.amazonaws.com/java/api/latest/ |
| AWS Github | https://github.com/aws |
| AWS Book | 《Amazon Web Services in Action, Third Edition》By Michael Wittig and Andreas Wittig (中文版叫《AWS云计算实战》) |
| AWS Video | https://www.udemy.com/course/aws-solutions-architect-professional/ https://www.udemy.com/course/aws-certified-solutions-architect-professional-training/ https://learning.oreilly.com/videos/amazon-web-services/9780137928521/ |
| AWS Glossary | https://docs.aws.amazon.com/glossary/latest/reference/glos-chap.html |
| AWS Training | https://aws.amazon.com/training/learn-about/architect/ |
AWS Certification 认证 |
https://aws.amazon.com/certification |
| AWS Certification Practice Test | Examtopic：（注意甄别答案） https://www.examtopics.com/exams/amazon/ https://www.examtopics.com/exams/amazon/aws-certified-solutions-architect-professional-sap-c02/
https://www.udemy.com/course/practice-exam-aws-certified-solutions-architect-professional/ |
附：如有需要，以上链接的网页右上角都可以切换成中文版。
Management Console （管理控制台）
CLI (Command Line Interface)
SDK (Software Development Kit)
Please note that everything in AWS is 100% API driven.
其他推荐阅读：
| 谷歌云GCP | https://blog.csdn.net/Beth_Chan/article/details/113461721 |
| 阿里云（包含阿里云计算、存储、数据处理、Java 微服务等案例） | https://blog.csdn.net/Beth_Chan/article/details/111176779 |
| MOOC网站访问日志分析（阿里云案例） | https://blog.csdn.net/Beth_Chan/article/details/113727493 |
| Infrastructure as Code - Terraform | https://blog.csdn.net/Beth_Chan/article/details/133276479（待整理） |
| Python | https://blog.csdn.net/Beth_Chan/article/details/133421056（待整理） |
安全
IAM (Identity and Access Management)
IAM 身份和访问管理
Identity & Federation: MFA 多重身份认证
Account 账号
Key Concepts:
IAM Users 用户
IAM Roles 角色
IAM Permission 权限
IAM Groups 组
IAM Policies: Service Control Policies (SCP)
STS: Security Token Service
允许这两者共存
IAM Access Analyzer
AWS Organizations
(Organizational Units, OU; Management Account, Member Account)
AWS SSO
AWS Directory Service
AWS Resource Access Manager
Amazon Cognito
Detection and Incident Response
-
Security Hub
-
GuardDuty
-
Amazon Inspector
-
AWS Cloudtrail
-
Amazon Detective
-
AWS Config
-
AWS IoT Device Defender
-
CloudEndure Disaster Recovery
Infrastructure Protection
-
WAF：Web Application Firewall
-
AWS Shield
-
AWS Firewall Manager
Data Protection
-
KMS (Key Management Service)
Create Customer Master Key (CMK) for S3 Encryption
-
ASM( Secrets Manager)
Parameter Store
-
ACM (AWS Certificate Manager)
Create TLS certificate
-
Amazon Macie
-
CloudHSM (Hardware Security Module)
Compliance
-
AWS Artifact
-
AWS Audit Manager
网络
Overview
Amazon Virtual Private Cloud is a commercial cloud computing service that provides users a virtual private cloud, by "provision[ing] a logically isolated section of Amazon Web Services Cloud". Enterprise customers are able to access the Amazon Elastic Compute Cloud over an IPsec based virtual private network.
You need to provide IPv4 or IPv6 CIDR range while creating the VPC.
AWS Direct Connect and Site to Site VPN are the services which provide the connectivity between AWS and on-premises networks. AWS Direct connect provides the private connectivity via the dedicated network while Site to Site VPN provides the secure (IPSec) connectivity over the internet.
AWS Direct Connect and VPN both provides the private connectivity between AWS and your corporate network. However VPN traffic flows over the internet and hence can not be considered as consistent whereas Direct Connect connection is over the dedicated physical connection and is more consistent and stable.
AWS Lambda requires NAT to connect to the Internet. Public IP addresses cannot be assigned to an AWS Lambda function.
Amazon CloudFront is a content delivery network operated by Amazon Web Services. Content delivery networks provide a globally-distributed network of proxy servers that cache content, such as web videos or other bulky media, more locally to consumers, thus improving access speed for downloading the content.
AWS Shield is a managed Distributed Denial of Service (DDoS) protection service that safeguards applications running on AWS. AWS Shield provides always-on detection and automatic inline mitigations that minimize application downtime and latency, so there is no need to engage AWS Support to benefit from DDoS protection.
VPC (Virtual Private Cloud)
虚拟私有云
基本概念:
- Region, Availability Zone, VPC
- CIDR (无类别域间路由）
- 子网 Subnet：Public/Private/Hybrid
- 路由表 Route Table
- IP(Internet Protocol，网络协议）v4 / v6 (Private/Public/Elastic IP)
- (Elastic) Network Interfaces 网络接口
-
Security Group 安全组
-
Network Access Control List (NACL)，Network ACLs (Access Control Lists，访问控制列表）
-
NAT Gateway, NAT Instance (Setup up NAT on EC2)
-
Ingress/Inbound；Egress/Outbound
-
Firewall
- Resource Group 资源组
CIDR
Subnet, Route Table, IP, IGW (Internet Gateway，网关)
Private, Public vs Elastic IP
Elastic Network Interfaces (ENI)
Firewalls - Security group, Network ACLs
Internetwork traffic privacy in Amazon VPC - Amazon Virtual Private Cloud
NAT Gateway
如果Web架构是单台EC2的话还算好处理，把EC2设定好固定IP即可；但如果是比较大型或是高流量的架构通常都会用 Auto Scaling，这时候EC2的数量跟IP变成不固定，随时都可能变动，因此我们需要把所有EC2内对外的请求连线，做出一些处理，让这些请求到达第三方服务的时候，IP永远是固定的。
NAT (Network Address Translation， 网络地址转换），字面上的意思就是它可以转换IP。
由AWS官方提供的架构图可知，DB Server是设定在Private Subnet里面，并不能直接连到外网，但透过Route table的设定可以连到Public Subnet的NAT Gateway，再通过NAT Gateway转换IP，并通过Public Subnet的Internet Gateway连到Internet。
https://docs.aws.amazon.com/vpc/latest/userguide/amazon-vpc-limits.html
Amazon Route 53（DNS）
DNS: Domain Name System
Route 53 Scenairos: EC2 instance; EC2 DNS name; ALB; CloudFront distribution; API Gateway; RDS DB instance; S3 bucket; VPC interface endpoint
VPC Flow Logs
VPC Flow Logs Monitoring
Network monitoring
VPC Endpoint (Gateway, Interface) & VPC Endpoint Service
PrivateLink （VPC Interface Endpoint）
What is PrivateLink?
https://docs.aws.amazon.com/vpc/latest/privatelink/what-is-privatelink.html
Target Group & Application Load Balancer & Network Load Balancer & VPC Endpoint Service & VPC Endpoint：
https://aws.amazon.com/blogs/networking-and-content-delivery/application-load-balancer-type-target-group-for-network-load-balancer/
直接将ALB注册为NLB目标，无需主动管理不断变化的ALB IP地址。这是通过使用引入的应用程序负载均衡器类型的目标组来实现的。由此可以将NLB的优势（包括PrivateLink和区域静态IP地址）与ALB提供的高级路由结合起来，对应用程序的流量进行负载平衡。
AWS Transit Gateway (TGW)
Transit Gateway Route Table
Attachment
Transit Gateway Peering
Hybrid
AWS Direct Connect (DX)
What is Direct Connect?
Direct Connect uses private, public, and transit virtual interfaces (VIF).
Direct Connect Gateway
- Global network device – Accessible in all regions
- Direct Connect integrates via a private VIF or a transit VIF
- The Private VIF or Transit VIF and Direct Connect gateway must be owned by same AWS account however VPCs (VGWs) or Transit Gateways can be from same or different AWS accounts
Border Gateway Protocol (BGP)
Bidirectional Forwarding Detection (BFD)
Link Aggregation Group (LAG)
Direct Connect Monitoring
Metrics
VPC Peering
AWS VPN
- Site-to-Sute VPN
- AWS Cilent VPN
Load Balancing
Elastic Load Balancers (ELB)
- 1. Application Load Balancer (ALB)
2. Network Load Balancer
3. Gateway Load Balancer
Auto Scaling
Auto Scaling Groups（ASGs，自动扩展组/自动伸缩组）AZ: Availability Zone 可用区
计算
EC2 (Elastic Compute Cloud )
AMI (Amazon Machine Image, Amazon系统映像）：操作系统和预安装软件的组合。不包括操作系统内核。操作系统内核从Amazon Kernel Image (AKI) 加载。
HVM (Hardware Virtual Machine)：最新也是最快的虚拟化类型。
Container Service
ECS
Amazon Elastic Container Service (Amazon ECS)
EKS
Amazon Elastic Kubernetes Service (Amazon EKS)
Kubernetes Architecture
Pod to Pod communication
Security Groups in EKS
- Cluster security groups
- Pod security groups
Exposing services
EKS Summary
• EKS control plane is launched in AWS managed VPC and EKS data plane (worker nodes) is launched in customer VPC.
• EKS provisions ENIs into customer VPC to enable communication between EKS control plane and data plane
• EKS cluster API endpoint is publicly accessible by default but can be configured as a private in which case it can be accessed from customer VPC via the EKS owned ENI
• EKS uses Amazon VPC Container Network Interface (CNI) plugin for Pod networking.
• CNI allocates IPs to each Pod from available Secondary IPs
• Maximum number of Pods per node depends on number of ENIs and number of IP addresses per ENI
• For supported Nitro based instance types, Pod per node limit can be increased using Prefix delegation (/28 for IPv4 and /80 for IPv6)
• Custom Networking enables associating secondary VPC CIDR (100.64.0.0/16) and when combined with SNAT enables much larger IPv4 private IPs for Pods.
• CNI allows Nodes to enable/disable SNAT to allow outbound internet access to Pods through the Internet gateway or NAT gateway respectively.
• By default, ENI security group is assigned to all the Pods which have been allocated secondary IPs for that ENI
• Pods specific security group can be assigned using Trunk & Branch ENI feature for selected Nitro system based instances.
• Pod services can be configured using ClusterIP, NodePort, LoadBalancer and Ingress resources.
• ClusterIP allows accessing services from within the cluster only.• NodePort allows accessing services externally using Node IP and static port
• LoadBalancer service can be configured to use CLB or NLB in instance mode.
• Ingress service can be configured to use ALB in instance or IP mode.
• AWS Load Balancer Controller can be used for LoadBalancer (with NLB IP mode) and Ingress service (with ALB) configurations.
• externalTrafficPolicy=Local allows NLB in instance mode to preserve client IP address by disabling kube-proxy to send traffic to other nodes.
ECR
Amazon Elastic Container Registry (Amazon ECR)
AWS Fargate
ECS Fargate (Serverless Docker)
EKS Fargate (Serverless Kubernetes)
AWS Lambda
No need to provison and manage server.
Implement a function in Python, Java, JavaScript（Node.js）, Go, C# or Ruby, etc.
在调用AWS lambda函数时，输入可提供一个事件（event）和一个上下文（context）对象。event是函数获得输入参数的一种方法，通常采用JSON格式。
Python的print和JavaScript的console.log都默认会被重定向到CloudWatch Logs。
Python JSON dumps & load
json.dumps(): from JSON object to string
json.load(): from string to JSON object
JavaScript（Node.js） JSON stringify & parse
JSON.stringify(): from JSON object to string
JSON.parse(): parse string to JSON object
用code inline，zip或者contain image部署都可以。
例子：
Python:
def handler_name(event, context):
//...
return some_valueContext:
import time
def lambda_handler(event, context):
print("Lambda function ARN:", context.invoked_function_arn)
print("CloudWatch log stream name:", context.log_stream_name)
print("CloudWatch log group name:", context.log_group_name)
print("Lambda Request ID:", context.aws_request_id)
print("Lambda function memory limits in MB:", context.memory_limit_in_mb)
# We have added a 1 second delay so you can see the time remaining in get_remaining_time_in_millis.
time.sleep(1)
print("Lambda time remaining in MS:", context.get_remaining_time_in_millis())
Node.js:
Context:
exports.handler = async function(event, context) {
console.log('Remaining time: ', context.getRemainingTimeInMillis())
console.log('Function name: ', context.functionName)
return context.logStreamName
}AWS Step Functions
Elastic Beanstalk
AWS Batch
AWS LightSail
AWS Outposts
AWS App Runner
存储 Storage
S3 (Simple Storage Service)
对象存储服务 Amazon S3
存储桶 bucket：可提供访问控制，不同存储桶可以有不同的可访问性
数据对象 data object：由内容和元数据组成。元数据：最后修改日期、内容类型、用户自定义。每个对象由键来确定。存储桶位于一个区域内。上传静态文件后得到的是一个URL （https://bucket-name.s3.amazonaws.com/sample+key/name.jpg)
单个文件最大5T。
设计为99.999999999%的可靠性。
EBS (Elastic Block Store)
Amazon Glacier
备份和归档的存储服务
NAS (Network Attached Storage，网络附加存储）
NFS (Network File System, 解决多个EC2实例之间共享块存储的问题）
Amazon EBS
Amazon EFS
Elastic File System, 基于NFSv4协议
GlusterFS
AWS Transfer Family
数据库
RDS
PostgreSQL/MySQL/Oracle/Microsoft Server SQL/MariaDB
Amazon Aurora
Amazon Aurora PostgreSQL
Amazon Aurora MySQL DB
Amazon Aurora Serverless
DynamoDB (Key-value DB)
Amazon DocumentDB (MongoDB)
Amazon OpenSearch Service
Amazon Redshift
AWS Neptune
缓存 Caching
CloudFront （CDN)
ElasticCache and MemoryDB
Analytics Services / Data Engineering
Amazon Kinesis
- Data Streams
- Data Firehose
- Data Analytics
Amazon Athena
EMR (Elastic MapReduce)
AWS Glue
Data Pipeline
Amazon MSK
Amazon MQ
Managed ActiveMQ, RabbitMQ
Amazon Timestream
Amazon QuickSight
机器学习/人工智能（ML/AI）
SageMaker
Amazon Bedrock
Amazon Rekognition
Amazon Translate
Detect and translate text
Amazon Transcribe
Amazon Polly
Perform speech-to-text and vice versa
Amazon Comprehend
Extract information from text
Amazon Lex
build voice and text chatbox
Amazon Forecast
AWS DeepRacer
TensorFlow on AWS
PyTorch on AWS
Monitoring and Automation Services
Foundations of Monitoring
What's Monitoring & 360 Degree View:
1. Monitoring End-to-End
2. External Monitoring (End User Experience)
External monitoring is anything that happens out HERE, from the USER perspective. Not from AWS centric perspective, like EC2, ALB, CloudFront, etc.
Passive Response
- Alert with Emails, eg: SNS to Email (Individuals and group/distribution list)
- Create help desk tickets, eg: SNS to HTTPS
- Highlight a metric on a dashboard，eg: CloudWatch Dashboard (Turn yellow or red)
Active Response
- Reboot the instance, or stop/start, eg: EC2 Rescue
- Scale horizontally, eg: Auto Scaling
- Custom actions requiring code, eg: Lambda Function
Concept Overview
- Metric & Log collection, aggregation, persistence
- Dashboards
- Alarms
- Actions
- Rules and filters
- Cross-service permissions
Differenct type of Monitoring
- Performance Monitoring
- Availability Monitoring
- Log Monitoring
- Compliance Monitoring
AWS Services
- CloudWatch (Performance metrics, Dashboards, Alarms)
- AWS Trusted Advisor (Canned reports, Recommendations, Limits)
- CloudTrail
- Amazon Macie
- Amazon GuardDuty
- AWS Config
- Amazon Systems Manager
Monitoring Permissions
- Identity-based permissions
IAM Role allowing CloudTrail to write to CloudWatch
Logs (when services access other service APIs)
- Resource-based permissions
Bucket Policy to allow ELB to write access logs to S3
Bucket
- Access Control Lists
Primarily used for S3 bucket access
Primitive - Pre-dates IAM
Permission Combination with Multiple Policy
Amazon CloudWatch 监控
CloudWatch Agent
Events
Alarms
Logs
Metrics
Dashboards
Performance Monitoring
Logging Monitoring
- Access logs
- Execution logs
- Event
- Flow logs
VPC Flow Logs
Record is great for monitoring, troubleshooting and root cause analysis.
重要字段有：srcaddr, dstport, bytes, action等。
VPC Traffic Mirroring
Amazon CloudTrail 日志
Amazon EventBridge
AWS Distro for OpenTelemetry
Amazon Managed Service for Prometheus (AMP)
Amazon Managed Grafana (AMG)
ELK with Amazon OpenSearch
CloudFormation
Create stack from 蓝图 (blueprint)，配置管理服务 AWS CloudFormation 使用的蓝图被称为模版 (template)。
AWS Config
Management Tools
Organization and SSO
AWS Systems Manager (SSM)
Parameter Store
Run Command
Session Manager
Change Manager
Inventory
Billing and Cost Management
Cost Explorer
Cost Allocation Tags
Billing Alarms
AWS Budgets
Migration
AWS Database Migration Service (DMS)
Schema Tool Service
其他服务
AWS Marketplace
Amazon SNS
Amazon Simple Notification Service
Amazon SQS
Amazon Simple Queue Service
Amazon MQ
Amazon SES
Simple Email Service
AWS Cloud Map
AWS X-Ray
distribute tracing
AWS OpsWorks
Automate operations with Chef and Puppet
Amazon DevOps Guru
AWS DataSync
AWS App Mesh
Amazon FSx
- for NetApp ONTAP
- for Windows File Server
AWS FIS
Fault Injection Simulator
部分术语
JSON基础架构标记语言（JSON Infrastructure Markup Language，JIML）。
JMESPath (JSON Matching Expression paths)
ICMP ( Internet Control Message Protocol, 因特网控制报文协议）
RTO (Recovery Time Objective 恢复时间目标)
RPO (Recovery Point Objective 恢复点目标)
TTL （Time to Live 生存时间）
DSL (Domain-Specific Language)
CICD
(可用Jenkins)
AWS自带的服务有：
CodeCommit / Github / Bitbucket
CodeBuild / Jenkins
CodePipeline
CodeDeploy / Jenkins
Elastic Beanstalk
实验
AWS Credentials 一个小时自动过期！
Cost Saving
Using S3 Lifecycle Policies to Reduce Storage Costs
- Create an S3 Lifecycle rule to transition objects to the S3 IA storage class after a time period of 30 days
- Apply the lifecycle policy to your S3 bucket
$ ls
book_cover.png lifecycle-rule.json README.md
$ RANDOM_STRING=$(aws secretsmanager get-random-password \
> --exclude-punctuation --exclude-uppercase \
> --password-length 6 --require-each-included-type \
> --output text \
> --query RandomPassword)
$ aws s3api create-bucket --bucket awscookbook301-$RANDOM_STRING
{
"Location": "/awscookbook301-4e5oom"
}
$ cat lifecycle-rule.json
{
"Rules": [
{
"ID": "Move all objects to Standard Infrequently Access",
"Prefix": "",
"Status": "Enabled",
"Transitions": [
{
"Days": 30,
"StorageClass": "STANDARD_IA"
}
]
}
]
}
$ aws s3api put-bucket-lifecycle-configuration \
> --bucket awscookbook301-$RANDOM_STRING \
> --lifecycle-configuration file://lifecycle-rule.json
$ aws s3api get-bucket-lifecycle-configuration \
> --bucket awscookbook301-$RANDOM_STRING
{
"Rules": [
{
"ID": "Move all objects to Standard Infrequently Access",
"Prefix": "",
"Status": "Enabled",
"Transitions": [
{
"Days": 30,
"StorageClass": "STANDARD_IA"
}
]
}
]
}
$ aws s3 cp book_cover.png s3://awscookbook301-$RANDOM_STRING
upload: ./book_cover.png to s3://awscookbook301-4e5oom/book_cover.png
$ aws s3api list-objects-v2 --bucket awscookbook301-$RANDOM_STRING
{
"Contents": [
{
"Key": "book_cover.png",
"LastModified": "2023-11-18T07:37:43+00:00",
"ETag": "\"d38461283ddc63b80044e2af6a7afd0d\"",
"Size": 255549,
"StorageClass": "STANDARD"
}
],
"RequestCharged": null
}