Week 5 - Day 3: S3 Storage | DevOps Curriculum

⏱ Duration: 5 Hours

📚 Learning Objectives

Understand S3 storage concepts and use cases
Create and manage S3 buckets
Upload, download, and manage objects
Configure bucket policies and permissions
Host a static website on S3

📖 Core Concepts (2 Hours)

What is Amazon S3?

Amazon Simple Storage Service (S3) is object storage built to store and retrieve any amount of data. It offers 99.999999999% (11 9's) durability.

Buckets: Containers for storing objects
Objects: Files stored in buckets (up to 5TB each)
Keys: Unique identifier for objects in a bucket
Versioning: Keep multiple versions of objects
Storage Classes: Different tiers for cost optimization

S3 Storage Classes

                    
Storage Classes (by access frequency):

┌────────────────────┬──────────────┬─────────────────────┐
│ Class              │ Use Case     │ Retrieval           │
├────────────────────┼──────────────┼─────────────────────┤
│ S3 Standard        │ Frequent     │ Immediate           │
│ S3 Intelligent-    │ Unknown      │ Auto-tiering        │
│   Tiering          │   patterns   │                     │
│ S3 Standard-IA     │ Infrequent   │ Immediate           │
│ S3 One Zone-IA     │ Infrequent   │ Immediate (1 AZ)    │
│ S3 Glacier Instant │ Archive      │ Milliseconds        │
│ S3 Glacier Flexible│ Archive      │ Minutes to hours    │
│ S3 Glacier Deep    │ Long-term    │ 12-48 hours         │
│   Archive          │   archive    │                     │
└────────────────────┴──────────────┴─────────────────────┘

Free Tier: 5GB Standard, 20,000 GET, 2,000 PUT/month
                    
                

S3 Bucket Naming Rules

                    
Bucket Name Requirements:
✓ Globally unique across ALL AWS accounts
✓ 3-63 characters long
✓ Lowercase letters, numbers, hyphens
✓ Start with letter or number
✓ Cannot be formatted as IP address

Examples:
✓ my-app-bucket-2024
✓ devops-training-files
✗ My_Bucket (uppercase, underscore)
✗ 192.168.1.1 (IP format)
                    
                

S3 Security

                    
S3 Security Layers:

1. Bucket Policies (JSON)
   - Resource-based permissions
   - Apply to entire bucket

2. Access Control Lists (ACLs)
   - Legacy, prefer bucket policies
   - Object-level permissions

3. IAM Policies
   - User/role-based permissions
   - Cross-service access

4. Block Public Access
   - Account or bucket level
   - Prevents accidental exposure

5. Encryption
   - SSE-S3 (AWS managed keys)
   - SSE-KMS (Customer managed)
   - SSE-C (Customer provided)
                    
                

🔬 Hands-on Lab (2.5 Hours)

Lab 1: Create S3 Bucket

Create a bucket via CLI
Verify bucket creation
Understand bucket settings

                    
# Create unique bucket name
BUCKET_NAME="devops-training-$(date +%s)"

# Create bucket (us-east-1 doesn't need LocationConstraint)
aws s3api create-bucket \
  --bucket $BUCKET_NAME \
  --region us-east-1

# For other regions, specify location constraint
aws s3api create-bucket \
  --bucket $BUCKET_NAME \
  --region us-west-2 \
  --create-bucket-configuration LocationConstraint=us-west-2

# List all buckets
aws s3 ls

# Get bucket location
aws s3api get-bucket-location --bucket $BUCKET_NAME
                    
                

Lab 2: Upload and Manage Objects

Upload files to S3
List and download objects
Copy and delete objects

                    
# Create test files
echo "Hello from S3!" > hello.txt
echo "<html><body><h1>My Website</h1></body></html>" > index.html

# Upload single file
aws s3 cp hello.txt s3://$BUCKET_NAME/

# Upload with specific path
aws s3 cp index.html s3://$BUCKET_NAME/website/

# Upload entire directory
mkdir -p myfiles
echo "file1" > myfiles/file1.txt
echo "file2" > myfiles/file2.txt
aws s3 sync myfiles/ s3://$BUCKET_NAME/myfiles/

# List objects
aws s3 ls s3://$BUCKET_NAME/
aws s3 ls s3://$BUCKET_NAME/ --recursive

# Download file
aws s3 cp s3://$BUCKET_NAME/hello.txt ./downloaded.txt

# Copy between locations
aws s3 cp s3://$BUCKET_NAME/hello.txt s3://$BUCKET_NAME/backup/hello.txt

# Delete object
aws s3 rm s3://$BUCKET_NAME/hello.txt

# Delete multiple objects (with prefix)
aws s3 rm s3://$BUCKET_NAME/myfiles/ --recursive
                    
                

Lab 3: Configure Bucket Policy

Create a bucket policy for public read
Apply policy to bucket
Test public access

                    
# First, disable "Block Public Access" for this bucket
aws s3api put-public-access-block \
  --bucket $BUCKET_NAME \
  --public-access-block-configuration \
  "BlockPublicAcls=false,IgnorePublicAcls=false,BlockPublicPolicy=false,RestrictPublicBuckets=false"

# Create bucket policy file
cat > bucket-policy.json <<EOF
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "PublicReadGetObject",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::$BUCKET_NAME/public/*"
        }
    ]
}
EOF

# Apply bucket policy
aws s3api put-bucket-policy \
  --bucket $BUCKET_NAME \
  --policy file://bucket-policy.json

# Upload file to public folder
echo "This is public content" > public.txt
aws s3 cp public.txt s3://$BUCKET_NAME/public/

# Access publicly
curl https://$BUCKET_NAME.s3.amazonaws.com/public/public.txt
                    
                

Lab 4: Host Static Website

Enable static website hosting
Upload website files
Access website via S3 URL

                    
# Create website files
cat > index.html <<EOF
<!DOCTYPE html>
<html>
<head>
    <title>My S3 Website</title>
    <style>
        body { font-family: Arial; text-align: center; padding: 50px; }
        h1 { color: #ff9900; }
    </style>
</head>
<body>
    <h1>Welcome to My S3 Static Website!</h1>
    <p>Hosted on Amazon S3</p>
</body>
</html>
EOF

cat > error.html <<EOF
<!DOCTYPE html>
<html>
<head><title>Error</title></head>
<body><h1>404 - Page Not Found</h1></body>
</html>
EOF

# Enable static website hosting
aws s3 website s3://$BUCKET_NAME/ \
  --index-document index.html \
  --error-document error.html

# Update bucket policy for website access
cat > website-policy.json <<EOF
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "PublicWebsite",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::$BUCKET_NAME/*"
        }
    ]
}
EOF

aws s3api put-bucket-policy \
  --bucket $BUCKET_NAME \
  --policy file://website-policy.json

# Upload website files
aws s3 cp index.html s3://$BUCKET_NAME/
aws s3 cp error.html s3://$BUCKET_NAME/

# Get website URL
echo "Website URL: http://$BUCKET_NAME.s3-website-us-east-1.amazonaws.com"
                    
                

Lab 5: Clean Up

                    
# Delete all objects in bucket
aws s3 rm s3://$BUCKET_NAME --recursive

# Delete bucket
aws s3api delete-bucket --bucket $BUCKET_NAME

# Verify deletion
aws s3 ls | grep $BUCKET_NAME

# Clean up local files
rm -f hello.txt index.html error.html public.txt bucket-policy.json website-policy.json downloaded.txt
rm -rf myfiles/
                    
                

✅ Day 3 Checklist

Understand S3 storage classes
Can create and delete buckets
Can upload, download, and manage objects
Can configure bucket policies
Successfully hosted static website on S3
Understand S3 security best practices

← Day 2: EC2 Fundamentals Day 4: IAM Security →