Week 8 - Day 4: Integration | DevOps Curriculum

⏱ Duration: 5 Hours

📚 Learning Objectives

Connect all project components together
Test the complete CI/CD pipeline end-to-end
Verify application deployment and functionality
Troubleshoot and fix integration issues
Implement monitoring and health checks

📖 Core Concepts (2 Hours)

Integration Architecture

End-to-End Flow:
┌─────────────────────────────────────────────────────────────┐
│ DEVELOPER                                                    │
│ ┌─────────────┐                                             │
│ │ Code Change │                                             │
│ └──────┬──────┘                                             │
│        │ git push                                           │
│        ▼                                                    │
├─────────────────────────────────────────────────────────────┤
│ GITHUB                                                       │
│ ┌─────────────┐  ┌─────────────┐  ┌─────────────────────┐  │
│ │ Repository  │→ │   Actions   │→ │ Trigger Pipeline    │  │
│ └─────────────┘  └─────────────┘  └──────────┬──────────┘  │
│                                              │              │
├──────────────────────────────────────────────┼──────────────┤
│ CI/CD PIPELINE                               │              │
│ ┌─────────┐  ┌─────────┐  ┌─────────┐      │              │
│ │  Test   │→ │  Build  │→ │  Push   │      │              │
│ │ pytest  │  │ Docker  │  │ to Hub  │      │              │
│ └────┬────┘  └────┬────┘  └────┬────┘      │              │
│      │            │            │            │              │
│      │ Pass       │ Image      │ Tagged     │              │
│      ▼            ▼            ▼            ▼              │
├─────────────────────────────────────────────────────────────┤
│ DEPLOYMENT                                                   │
│ ┌─────────────────────────────────────────────────────────┐│
│ │                 SSH to EC2                               ││
│ │  docker pull → docker stop → docker run → health check  ││
│ └─────────────────────────────────────────────────────────┘│
│                            │                                │
├────────────────────────────┼────────────────────────────────┤
│ AWS EC2                    ▼                                │
│ ┌─────────────────────────────────────────────────────────┐│
│ │  Docker Container Running                                ││
│ │  ┌─────────────────────────────────────────────────┐   ││
│ │  │ Python Flask App                                 │   ││
│ │  │ • GET /          → Welcome message              │   ││
│ │  │ • GET /health    → Health status                │   ││
│ │  │ • GET /api/info  → App information              │   ││
│ │  └─────────────────────────────────────────────────┘   ││
│ └─────────────────────────────────────────────────────────┘│
└─────────────────────────────────────────────────────────────┘

Integration Verification Script

#!/bin/bash
# scripts/verify-deployment.sh
# Comprehensive deployment verification script

set -e

# Configuration
APP_URL="${APP_URL:-http://localhost:8000}"
MAX_RETRIES=30
RETRY_INTERVAL=10

echo "=========================================="
echo "  Deployment Verification Script"
echo "=========================================="
echo "Target URL: $APP_URL"
echo ""

# Function to check endpoint
check_endpoint() {
    local endpoint=$1
    local expected_status=$2
    local description=$3
    
    echo -n "Testing $description... "
    
    response=$(curl -s -w "\n%{http_code}" "$APP_URL$endpoint" 2>/dev/null)
    status_code=$(echo "$response" | tail -n1)
    body=$(echo "$response" | sed '$d')
    
    if [ "$status_code" -eq "$expected_status" ]; then
        echo "✅ PASS (HTTP $status_code)"
        return 0
    else
        echo "❌ FAIL (Expected $expected_status, got $status_code)"
        return 1
    fi
}

# Function to wait for app to be ready
wait_for_app() {
    echo "Waiting for application to be ready..."
    for i in $(seq 1 $MAX_RETRIES); do
        if curl -s -f "$APP_URL/health" > /dev/null 2>&1; then
            echo "✅ Application is ready!"
            return 0
        fi
        echo "Attempt $i/$MAX_RETRIES - Waiting ${RETRY_INTERVAL}s..."
        sleep $RETRY_INTERVAL
    done
    echo "❌ Application did not become ready"
    return 1
}

# Main verification
echo "Step 1: Checking application availability"
echo "----------------------------------------"
wait_for_app

echo ""
echo "Step 2: Testing API endpoints"
echo "----------------------------------------"
check_endpoint "/" 200 "Root endpoint"
check_endpoint "/health" 200 "Health check"
check_endpoint "/api/info" 200 "Info endpoint"
check_endpoint "/api/status" 200 "Status endpoint"
check_endpoint "/nonexistent" 404 "404 handling"

echo ""
echo "Step 3: Validating response content"
echo "----------------------------------------"

# Check health response content
health_response=$(curl -s "$APP_URL/health")
if echo "$health_response" | grep -q '"status":"healthy"'; then
    echo "✅ Health check returns healthy status"
else
    echo "❌ Health check response invalid"
    exit 1
fi

# Check info response content
info_response=$(curl -s "$APP_URL/api/info")
if echo "$info_response" | grep -q '"app_name"'; then
    echo "✅ Info endpoint returns app details"
else
    echo "❌ Info endpoint response invalid"
    exit 1
fi

echo ""
echo "=========================================="
echo "  All Verification Tests Passed! ✅"
echo "=========================================="

Troubleshooting Common Issues

Common Integration Issues and Solutions:
┌────────────────────────────────────────────────────────────┐
│ Issue: Pipeline fails at Docker push                        │
│ Symptoms: Authentication error to Docker Hub               │
│ Solution:                                                   │
│   1. Verify DOCKERHUB_USERNAME secret is correct           │
│   2. Verify DOCKERHUB_TOKEN is an access token (not pwd)   │
│   3. Regenerate token if expired                           │
├────────────────────────────────────────────────────────────┤
│ Issue: SSH deployment fails                                 │
│ Symptoms: Connection refused or permission denied          │
│ Solution:                                                   │
│   1. Verify EC2_HOST is the correct public IP              │
│   2. Verify EC2_SSH_KEY format (full key with headers)     │
│   3. Check security group allows port 22 from GitHub IPs   │
│   4. Verify key permissions on EC2                         │
├────────────────────────────────────────────────────────────┤
│ Issue: Container won't start on EC2                         │
│ Symptoms: Docker run fails or container exits immediately  │
│ Solution:                                                   │
│   1. SSH into EC2 and check: docker logs capstone-app      │
│   2. Verify image was pulled: docker images                │
│   3. Check port conflicts: netstat -tlnp                   │
│   4. Check system resources: free -h, df -h                │
├────────────────────────────────────────────────────────────┤
│ Issue: Application unreachable from internet                │
│ Symptoms: Connection timeout when accessing app URL        │
│ Solution:                                                   │
│   1. Verify security group allows port 8000/80             │
│   2. Verify EC2 has public IP/Elastic IP                   │
│   3. Check container is running: docker ps                 │
│   4. Check container port binding: docker port capstone-app│
├────────────────────────────────────────────────────────────┤
│ Issue: Health check fails in CI/CD                          │
│ Symptoms: Deployment job fails at verification step        │
│ Solution:                                                   │
│   1. Increase wait time before health check                │
│   2. Check application logs: docker logs capstone-app      │
│   3. Verify health endpoint path: /health                  │
│   4. Test locally first: curl http://localhost:8000/health │
└────────────────────────────────────────────────────────────┘

EC2 Debugging Commands

# SSH into EC2 instance
ssh -i ~/.ssh/your-key.pem ec2-user@EC2_PUBLIC_IP

# Check Docker status
sudo systemctl status docker

# List running containers
docker ps -a

# Check container logs
docker logs capstone-app
docker logs --tail 50 -f capstone-app

# Check if app is listening
netstat -tlnp | grep 8000

# Test locally on EC2
curl http://localhost:8000/health

# Check system resources
free -h           # Memory
df -h             # Disk space
top -bn1 | head   # CPU usage

# Check user-data script execution
cat /var/log/user-data-complete.log
cat /var/log/cloud-init-output.log

# Restart container manually
docker stop capstone-app
docker rm capstone-app
docker run -d --name capstone-app -p 80:8000 -p 8000:8000 your-image:latest

# Check security group rules (from AWS CLI)
aws ec2 describe-security-groups --group-ids sg-xxx

🔬 Hands-on Lab (2.5 Hours)

Lab 1: Trigger Full Pipeline

# Make a code change to trigger CI/CD
cd capstone-project

# Update app version
sed -i 's/APP_VERSION = "1.0.0"/APP_VERSION = "1.1.0"/' app/main.py

# Or add a new endpoint
cat >> app/main.py << 'EOF'

@app.route("/api/version")
def version():
    """Version endpoint"""
    return jsonify({
        "version": APP_VERSION,
        "updated": "Day 4 Integration"
    })
EOF

# Commit and push
git add .
git commit -m "Add version endpoint - trigger CI/CD"
git push origin main

# Monitor pipeline
echo "Watch pipeline at: https://github.com/YOUR_USERNAME/capstone-project/actions"

Lab 2: Verify Deployment End-to-End

# Get EC2 public IP from Terraform
cd terraform
APP_IP=$(terraform output -raw instance_public_ip)
echo "Application IP: $APP_IP"

# Wait for pipeline to complete (check GitHub Actions)

# Test all endpoints
echo "Testing endpoints..."
curl -s "http://$APP_IP:8000/" | jq .
curl -s "http://$APP_IP:8000/health" | jq .
curl -s "http://$APP_IP:8000/api/info" | jq .
curl -s "http://$APP_IP:8000/api/version" | jq .

# Verify version was updated
VERSION=$(curl -s "http://$APP_IP:8000/api/info" | jq -r '.version')
echo "Deployed version: $VERSION"

# Test response times
echo "Testing response times..."
for i in {1..5}; do
    time curl -s "http://$APP_IP:8000/health" > /dev/null
done

Lab 3: Implement Basic Monitoring

# Create monitoring script
cat > scripts/monitor.sh << 'EOF'
#!/bin/bash
# Simple monitoring script

APP_URL="${1:-http://localhost:8000}"
LOG_FILE="monitoring.log"

while true; do
    timestamp=$(date '+%Y-%m-%d %H:%M:%S')
    
    # Check health endpoint
    response=$(curl -s -w "%{http_code}:%{time_total}" "$APP_URL/health" 2>/dev/null)
    status_code=$(echo "$response" | grep -oE '[0-9]{3}:' | tr -d ':')
    response_time=$(echo "$response" | grep -oE ':[0-9.]+$' | tr -d ':')
    
    if [ "$status_code" = "200" ]; then
        echo "$timestamp | Status: UP | Response: ${response_time}s" | tee -a $LOG_FILE
    else
        echo "$timestamp | Status: DOWN | HTTP: $status_code" | tee -a $LOG_FILE
        # Could add alerting here (email, Slack, etc.)
    fi
    
    sleep 60
done
EOF

chmod +x scripts/monitor.sh

# Run monitoring (in background)
nohup ./scripts/monitor.sh "http://$APP_IP:8000" > /dev/null 2>&1 &
echo "Monitoring started in background"

# View monitoring log
tail -f monitoring.log

Lab 4: Load Testing

# Simple load test with curl
echo "Running simple load test..."

APP_URL="http://$APP_IP:8000"
REQUESTS=100
CONCURRENT=10

# Sequential test
echo "Sequential requests:"
time for i in $(seq 1 $REQUESTS); do
    curl -s "$APP_URL/health" > /dev/null
done

# Parallel test (if xargs available)
echo "Parallel requests ($CONCURRENT concurrent):"
time seq 1 $REQUESTS | xargs -P $CONCURRENT -I {} curl -s "$APP_URL/health" > /dev/null

# Using Apache Bench (if installed)
# ab -n 1000 -c 10 "$APP_URL/health"

# Using hey (if installed)
# hey -n 1000 -c 10 "$APP_URL/health"

echo "Load test complete!"

Lab 5: Create Operations Runbook

# Create runbook documentation
cat > docs/runbook.md << 'EOF'
# Operations Runbook

## Quick Reference

| Task | Command |
|------|---------|
| Check app status | `curl http://APP_IP:8000/health` |
| SSH to EC2 | `ssh -i key.pem ec2-user@APP_IP` |
| View logs | `docker logs capstone-app` |
| Restart app | `docker restart capstone-app` |

## Deployment

### Manual Deployment
```bash
ssh -i key.pem ec2-user@APP_IP
docker pull your-user/capstone-app:latest
docker stop capstone-app && docker rm capstone-app
docker run -d --name capstone-app -p 80:8000 your-user/capstone-app:latest
```

### Rollback
```bash
# Deploy previous version
docker run -d --name capstone-app -p 80:8000 your-user/capstone-app:v1.0.0
```

## Troubleshooting

### App Not Responding
1. Check if container is running: `docker ps`
2. Check container logs: `docker logs capstone-app`
3. Restart container: `docker restart capstone-app`

### High Memory Usage
1. Check memory: `free -h`
2. Check container stats: `docker stats capstone-app`
3. Restart if needed: `docker restart capstone-app`

### Deployment Failed
1. Check GitHub Actions logs
2. Verify secrets are configured
3. Test SSH connection manually

## Contacts
- On-call: DevOps Team
- Escalation: Platform Team
EOF

git add docs/runbook.md
git commit -m "Add operations runbook"
git push origin main

✅ Day 4 Checklist

Triggered full CI/CD pipeline with code change
Verified all API endpoints are accessible
Tested deployment automation end-to-end
Debugged and fixed any integration issues
Implemented basic health monitoring
Performed load testing
Created operations runbook

← Day 3: Infrastructure Day 5: Presentation →