AWS助理架构师认证培训 | AWS Storage Extras
视频来源：B站《AWS 认证解决方案架构师 助理级 SAA-C03》
一边学习一边整理老师的课程内容及试验笔记，并与大家分享，侵权即删，谢谢支持！
附上汇总贴：AWS助理架构师认证培训 | 汇总_热爱编程的通信人的博客-CSDN博客
AWS Snow Family Overview
AWS Snow Family
- Highly-secure, portable devices to collect and process data at the edge, and migrate data into and out of AWS
- Data migration:
- Edge computing:
Data Migrations with AWS Snow Family
Challenges：
- Limited connectivity
- Limited bandwidth
- High network cost
- Shared bandwidth (can't maximize the line)
- Connection stability
AWS Snow Family: offline devices to perform data migrations
lf it takes more than a week to transfer over the network, use Snowball devices!
Diagrams
- Direct upload to S3:
With Snow Family:
Snowball Edge (for data transfers)
- Physical data transport solution: move TBs or PBs of data in or out of AWS
- Alternative to moving data over the network (and paying network fees)
- Pay per data transfer job
- Provide block storage and Amazon S3-compatible object storage
- Snowball Edge Storage Optimized80TB of HDD capacity for block volume and S3 compatible object storage
- Snowball Edge Compute Optimized42TB of HDD capacity for block volume and S3 compatible object storage
- Use cases: large data cloud migrations, DC decommission, disaster recovery
AWS Snowcone
- Small, portable computing, anywhere, rugged & secure, withstands harsh environments
- Light (4.5 pounds, 2.1kg)
- Device used for edge computing, storage, and data transfer
- 8TBs of usable storage
- Use Snowcone where Snowball does not fit (space-constrained environment)
- Must provide your own battery / cables
- Can be sent back to AWS offline, or connect it to internet and use AWS DataSync to send data
AWS Snowmobile
- Transfer exabytes of data (1 EB = 1,000 PB = 1,000,000 TBs)
- Each Snowmobile has 100PB of capacity (use multiple in parallel)
- High security: temperature controlled, GPS, 24/7 video surveillance
- Better than Snowball if you transfer more than 10 PB
AWS Snow Family for Data Migrations
Snow Family - Usage Process
- Request Snowball devices from the AWS console for delivery
- Install the snowball client / AWS OpsHub on your servers
- Connect the snowball to your servers and copy files using the client
- Ship back the device when you're done (goes to the right AWS facility)
- Data will be loaded into an S3 bucket
- Snowball is completely wiped
What is Edge Computing?
- Process data while it's being created on an edge location A truck on the road, a ship on the sea, a mining station underground...
- These locations may have Limited / no internet access Limited / no easy access to computing power
- We setup a Snowball Edge / Snowcone device to do edge computing
- Use cases of Edge Computing:Preprocess dataMachine learning at the edge Transcoding media streams
- Eventually (if need be) we can ship back the device to AWS (for transferring data for example)
Snow Family - Edge Computing
- Snowcone (smaller)2CPUs, 4GB of memory, wired or wireless access USB-C power using a cord or the optional battery
- Snowball Edge - Compute Optimized52 vCPUs, 208 GiB of RAMOptional GPU (useful for video processing or machine learning)42TB usable storage
- Snowball Edge - Storage OptimizedUp to 40 vCPUs, 80 GiB of RAMObject storage clustering available
- All: Can run EC2 Instances & AWS Lambda functions (using AWS loT Greengrass)
- Long-term deployment options: 1 and 3 years discounted pricing
AWS OpsHub
- Historically, to use Snow Family devices, you needed a CLI (Command Line Interface tool)
- Today, you can use AWS OpsHub (a software you install on your computer / laptop) to manage your Snow Family DeviceUnlocking and configuring single or clustered devices Transferring filesLaunching and managing instances running on Snow Family DevicesMonitor device metrics (storage capacity, active instances on your device)Launch compatible AWS services on your devices (ex: Amazon EC2 instances, AWS Data Sync, Network File System (NFS)
Architecture Snowball into Glacier
Solution Architecture: Snowball into Glacier
- Snowball cannot import to Glacier directly
- You must use Amazon S3 first, in combination with an S3 lifecycle policy
Amazon FSx
Amazon FSx - Overview
- Launch 3rd party high-performance file systems on AWS
- Fully managed service
Amazon FSx for Windows (File Server)
- FSx for Windows is a fully managed Windows fle system share drive
- Supports SMB protocol & Windows NTFS
- Microsoft Active Directory integration, ACLs, user quotas
- Can be mounted on Linux EC2 instances
- Supports Microsoft's Distributed File System (DFS) Namespaces (group files across multiple FS)
- Scale up to 10s of GB/s, millions of IOPS, 100s PB of data
- Storage Options:SSD - latency sensitive workloads (databases, media processing, data analytics, ...)HDD - broad spectrum of workloads (home directory, CMS, ...)
- Can be accessed from your on-premises infrastructure (VPN or Direct Connect)
- Can be configured to be Multi-AZ (high availability)
- Data is backed-up daily to S3
Amazon FSx for Lustre
- Lustre is a type of parallel distributed fle system, for large-scale computing
- The name Lustre is derived from "Linux" and "cluster"
- Machine Learning, High Performance Computing (HPC)
- Video Processing, Financial Modeling, Electronic Design Automation
- Scales up to 100s GB/s, millions of IOPS, sub-ms latencies
- Storage Options:SSD - low-latency, IOPS intensive workloads, small & random fe operationsHDD - throughput-intensive workloads, large & sequential fle operations
- Seamless integration with S3Can "read S3" as a file system (through FSx)Can write the output of the computations back to S3 (through FSx)
- Can be used from on-premises servers (VPN or Direct Connect)
FSx File System Deployment Options
- Scratch File SystemTemporary storageData is not replicated (doesn't persist if file server fails)High burst (6x faster, 200MBps per TiB)Usage: short-term processing, optimize costs
- Persistent File SystemLong-term storageData is replicated within same AZ Replace failed files within minutes Usage: long-term processing, sensitive data
Amazon FSx for NetApp ONTAP
- Managed NetApp ONTAP on AWS
- File System compatible with NFS, SMB, iSCSI protocol
- Move workloads running on ONTAP or NAS to AWS
- Works with:LinuxWindows MacOS VMware Cloud on AWS Amazon Workspaces & AppStream 2.0Amazon EC2, ECS and EKS
- Storage shrinks or grows automatically
- Snapshots, replication, low-cost, compression and data de-duplication
- Point-in-time instantaneous cloning (helpful for testing new workloads)
Amazon FSx for OpenZFS
- Managed OpenZFS file system on AWS
- File System compatible with NFS (v3, v4, v4.1, v4.2)
- Move workloads running on ZFS to AWS
- Works with:LinuxWindowsMacOSVMware Cloud on AWSAmazon Workspaces & AppStream 2.0Amazon EC2, ECS and EKS
- Up to 1,000,000 IOPS with < 0.5ms latency
- Snapshots, compression and low-cost
- Point-in-time instantaneous cloning (helpful for testing new workloads)
Storage Gateway Overview
Hybrid Cloud for Storage
- AWS is pushing for "hybrid cloud"Part of your infrastructure is on the cloud Part of your infrastructure is on-premises
- This can be due toLong cloud migrationsSecurity requirementsCompliance requirementsIT strategy
- S3 is a proprietary storage technology (unlike EFS / NFS), so how do you expose the S3 data on-premises?
- AWS Storage Gateway!
AWS Storage Cloud Native Options
AWS Storage Gateway
- Bridge between on-premises data and cloud data
- Use cases:disaster recovery backup & restore tiered storageon-premises cache & low-latency files access
- Types of Storage Gateway:S3 File Gateway FSx File Gateway Volume Gateway Tape Gateway
Amazon S3 File Gateway
- Configured S3 buckets are accessible using the NFS and SMB protocol
- Most recently used data is cached in the file gateway
- Supports S3 Standard, S3 Standard IA, S3 One Zone A, S3 Intelligent Tiering
- Transition to S3 Glacier using a Lifecycle Policy
- Bucket access using IAM roles for each File Gateway
- SMB Protocol has integration with Active Directory (AD) for user authentication
Amazon FSx File Gateway
- Native access to Amazon FSx for Windows File Server
- Local cache for frequently accessed data
- Windows native compatibility (SMB, NTFS, Active Directory...)
- Useful for group file shares and home directories
Volume Gateway
- Block storage using iSCSl protocol backed by S3
- Backed by EBS snapshots which can help restore on-premises volumes!
- Cached volumes: low latency access to most recent data
- Stored volumes: entire dataset is on premise, scheduled backups to S3
Tape Gateway
- Some companies have backup processes using physical tapes (!)
- With Tape Gateway, companies use the same processes but, in the cloud
- Virtual Tape Library (VTL) backed by Amazon S3 and Glacier
- Backup data using existing tape-based processes (and iSCSI interface)
- Works with leading backup software vendors
Storage Gateway - Hardware appliance
- Using Storage Gateway means you need on-premises virtualization
- Otherwise, you can use a Storage Gateway Hardware Appliance
- You can buy it on http://amazon.com
- Works with File Gateway, Volume Gateway, Tape Gateway
- Has the required CPU, memory, network, SSD cache resources
- Helpful for daily NFS backups in small data centers
AWS Storage Gateway
AWS Transfer Family
AWS Transfer Family
- A fully-managed service for file transfers into and out of Amazon S3 or Amazon EFS using the FTP protocol
- Supported ProtocolsAWS Transfer for FTP (File Transfer Protocol (FTP))AWS Transfer for FTPS (File Transfer Protocol over SSL (FTPS))AWS Transfer for SFTP (Secure File Transfer Protocol (SFTP))
- Managed infrastructure, Scalable, Reliable, Highly Available (multi-AZ)
- Pay per provisioned endpoint per hour + data transfers in GB
- Store and manage user's credentials within the service
- Integrate with existing authentication systems (Microsoft Active Directory, LDAP, Okta, Amazon Cognito, custom)
- Usage: sharing files, public datasets, CRM, ERP, ...
DataSync - Overview
AWS DataSync
- Move large amount of data to and fromOn-premises / other cloud to AWS (NFS, SMB, HDFS, S3 API...) - needs agent AWS to AWS (different storage services) - no agent needed
- Can synchronize to:Amazon S3 (any storage classes - including Glacier)Amazon EFSAmazon FSx (Windows, Lustre, NetApp, Open ZFS...)
- Replication tasks can be scheduled hourly, daily, weekly
- File permissions and metadata are preserved (NFS POSIX, SMB...)
- One agent task can use 10Gbps, can setup a bandwidth limit
AWS DataSync NFS / SMB to AWS (S3, EFS, FSx...)
AWS DataSync Transfer between AWS storage services
All AWS Storage Options Compared
Storage Comparison
- S3: Object Storage
- S3 Glacier: Object Archival
- EBS volumes: Network storage for one EC2 instance at a time
- Instance Storage: Physical storage for your EC2 instance (high IOPS)
- EFS: Network File System for Linux instances, POSIX filesystem
- FSx for Windows: Network File System for Windows servers
- FSx for Lustre: High Performance Computing Linux file system
- FSx for NetApp ONTAP: High OS Compatibility
- FSx for Open ZFS: Managed ZFS fle system
- Storage Gateway: S3 & FSx File Gateway, Volume Gateway (cache & stored), Tape Gateway
- Transfer Family: FTP, FTPS, SFTP interface on top of Amazon S3 or Amazon EFS
- DataSync: Schedule data sync from on-premises to AWS, or AWS to AWS
- Snowcone / Snowball / Snowmobile: to move large amount of data to the cloud, physically
- Database: for specific workloads, usually with indexing and querying