Logging: CloudWatch (AWS) & Cloud Logging (GCP)

Logging is the foundation of observability. Every function call generates logs—capturing errors, performance data, and user actions. AWS and GCP provide managed logging services that automatically collect and store logs from your serverless functions.

---

Simple Explanation

What it is

Logging is the written record of what your function did. It is the first place you look when something goes wrong.

Why we need it

In serverless you cannot SSH into a machine. Logs are the only way to understand what happened inside a function.

Benefits

• Fast troubleshooting with clear error messages.
• Audit trail of requests and actions.
• Performance clues when requests slow down.

Tradeoffs

• Costs can grow if logs are too verbose.
• Sensitive data risk if you log carelessly.

Real-world examples (architecture only)

• Checkout fails -> Log shows validation error.
• Slow API -> Log shows long database query.

---

Part 1: AWS CloudWatch Logs

Lambda Logs

Every print() or logger output goes to CloudWatch Logs automatically.

Basic Logging

import json
import logging
from datetime import datetime

logger = logging.getLogger()
logger.setLevel(logging.INFO)


def handler(event, context):
    logger.info("Event: %s", json.dumps(event))
    logger.info("Processing started at %s", datetime.utcnow().isoformat())

    try:
        result = process(event)
        logger.info("Success: %s", result)
        return result
    except Exception as exc:
        logger.exception("Error: %s", exc)
        raise

View Logs in Console

1. Go to Lambda Console
2. Select function
3. Click Monitor
4. Click View CloudWatch Logs

CloudWatch Log Groups

Lambda creates a log group: /aws/lambda/function-name

Each invocation adds to log stream: 2026/02/08/[$LATEST]abc123...

Structured Logging

Use JSON for machine-readable logs:

import json
from datetime import datetime

print(json.dumps({
    "level": "INFO",
    "message": "Processing item",
    "itemId": "123",
    "timestamp": datetime.utcnow().isoformat(),
    "duration": 45,
}))

Better than:

print("Processing item 123, took 45ms")

Benefit

Parse and filter logs:

aws logs filter-log-events \
  --log-group-name /aws/lambda/myfunction \
  --filter-pattern '{ $.level = "ERROR" }'

Log Levels

Organize by severity:

import json
from datetime import datetime


def log(level, message, **data):
    print(json.dumps({
        "level": level,
        "message": message,
        "timestamp": datetime.utcnow().isoformat(),
        **data,
    }))


log("DEBUG", "Starting handler", event="...")
log("INFO", "Item retrieved", itemId=123)
log("WARN", "Retrying failed request", attempt=2)
log("ERROR", "Failed to save", error="Timeout")

Log Retention

By default, CloudWatch keeps logs forever (costs money).

Set Retention Policy

aws logs put-retention-policy \
  --log-group-name /aws/lambda/myfunction \
  --retention-in-days 30

Or in SAM:

HelloWorldFunction:
  Type: AWS::Serverless::Function
  Properties:
    LogRetentionInDays: 30

CloudWatch Logs Insights

Query logs with SQL-like syntax:

fields @timestamp, @message, @duration
| filter @duration > 100
| stats avg(@duration) by bin(5m)

1. Go to CloudWatch Logs
2. Select log group
3. Click Insights
4. Enter query
5. Click Run query

Examples

All errors in last hour

fields @timestamp, @message, @logStream
| filter @message like /ERROR/
| stats count() by @logStream

Slowest requests

fields @timestamp, @duration, @message
| sort @duration desc
| limit 10

Error rate over time

fields @message
| stats count(1) as total, count(@message like /ERROR/) as errors by bin(1m)
| fields bin as time, (errors / total * 100) as error_rate

Centralized Logging

Send logs from multiple functions to one place:

DefaultFunction:
  Type: AWS::Serverless::Function
  Properties:
    LogRetentionInDays: 7
    
SpecialFunction:
  Type: AWS::Serverless::Function
  Properties:
    LogRetentionInDays: 30  # Keep longer for audit

Privacy & Security

Never log sensitive data:

# ❌ Bad
print("User password:", password)
print("API key:", api_key)

# ✅ Good
print("Authentication attempt for user:", user_id)
print("API call to service:", service_name)

Custom Metrics from Logs

Extract metrics from logs:

aws logs put-metric-filter \
  --log-group-name /aws/lambda/myfunction \
  --filter-name ErrorCount \
  --filter-pattern '[time, request_id, level = ERROR, ...]' \
  --metric-transformations \
    metricName=ErrorCount,metricNamespace=MyApp,metricValue=1

Log Aggregation (Advanced)

Use ELK Stack, Datadog, or Cloudflare for cross-account logging.

Best Practices

1. Use context logging — Include request IDs
2. Log at appropriate levels — DEBUG for dev, INFO for prod
3. Avoid excessive logging — Impacts performance
4. Set retention — Don't keep logs forever
5. Structure logs as JSON — Easier to parse

---

Part 2: Google Cloud Logging

Cloud Logging Basics

Google Cloud Logging automatically captures all function output, plus structured log entries you send programmatically.

Basic Logging in Cloud Functions

import json
from datetime import datetime

import functions_framework


@functions_framework.http
def hello_world(request):
    print(json.dumps({
        "message": "Request received",
        "method": request.method,
        "path": request.path,
        "timestamp": datetime.utcnow().isoformat(),
    }))
    return ("Hello World!", 200)

Structured Logging (Recommended)

Use the @google-cloud/logging library for fine-grained control:

from datetime import datetime

import functions_framework
from google.cloud import logging as cloud_logging

logging_client = cloud_logging.Client()
log = logging_client.logger("my-app-logs")


@functions_framework.http
def process_order(request):
    payload = request.get_json(silent=True) or {}
    order_id = payload.get("orderId")

    log.log_struct({
        "message": "Order processing started",
        "orderId": order_id,
        "timestamp": datetime.utcnow().isoformat(),
        "userId": payload.get("userId"),
        "function": "process_order",
        "environment": "production",
    }, severity="INFO")

    try:
        result = process_order_logic(order_id)
        log.log_struct({
            "message": "Order processed successfully",
            "orderId": order_id,
            "result": result,
        }, severity="INFO")
        return ({"success": True, "result": result}, 200)
    except Exception as exc:
        log.log_struct({
            "message": "Order processing failed",
            "orderId": order_id,
            "error": str(exc),
        }, severity="ERROR")
        return ({"error": str(exc)}, 500)

Querying Cloud Logs

Use Google Cloud Console's Log Explorer or CLI:

# View logs for a specific function
gcloud functions logs read myfunction --limit 50 --runtime python312

# Filter by severity
gcloud functions logs read myfunction --limit 50 \
  --execution-id=abc123 \
  --region=us-central1

In Cloud Console, use filter syntax:

resource.type="cloud_function"
resource.labels.function_name="myfunction"
severity="ERROR"

Log Retention

Set retention policy on log types:

# Keep ERROR and above for 30 days
gcloud logging sinks update _Default \
  --log-filter='severity >= ERROR'

Or via code:

from google.cloud import logging as cloud_logging

logging_client = cloud_logging.Client()
sink = logging_client.sink(
    "my-error-sink",
    destination="storage.googleapis.com/my-bucket",
)

sink.create(filter_="severity >= ERROR")

JSON Structured Logging Example

Cloud Logging automatically parses JSON formatted logs:

import json
import random
from datetime import datetime


def my_function(request):
    print(json.dumps({
        "timestamp": datetime.utcnow().isoformat(),
        "severity": "INFO",
        "message": "Processing request",
        "requestId": request.headers.get("x-request-id"),
        "userId": (request.get_json(silent=True) or {}).get("userId"),
        "duration_ms": random.random() * 1000,
    }))

    return ("Done", 200)

Performance Monitoring via Logs

Extract duration and error patterns:

import json
import time


def handler(request):
    start_time = time.time()
    request_id = request.headers.get("x-request-id")

    print(json.dumps({
        "severity": "INFO",
        "message": "Function started",
        "requestId": request_id,
    }))

    try:
        result = heavy_computation()
        duration_ms = int((time.time() - start_time) * 1000)

        print(json.dumps({
            "severity": "INFO",
            "message": "Function completed",
            "requestId": request_id,
            "duration_ms": duration_ms,
            "success": True,
        }))

        return (result, 200)
    except Exception as exc:
        duration_ms = int((time.time() - start_time) * 1000)

        print(json.dumps({
            "severity": "ERROR",
            "message": "Function failed",
            "requestId": request_id,
            "duration_ms": duration_ms,
            "error": str(exc),
        }))

        return ({"error": str(exc)}, 500)

Advanced Filtering

Find slowest operations:

resource.type="cloud_function"
jsonPayload.duration_ms > 5000
severity!="DEBUG"

Find errors from specific user:

resource.type="cloud_function"
severity="ERROR"
jsonPayload.userId="user-123"

---

CloudWatch (AWS) vs. Cloud Logging (GCP)

Feature	CloudWatch Logs	Cloud Logging
Auto-capture	All stdout/stderr from Lambda	All function output automatically
Log format	Plain text by default	JSON recommended, auto-parsed
Retention default	7 days	Indefinite (configurable)
Query language	CloudWatch Logs Insights (custom syntax)	Cloud Logging filter syntax (simpler)
Pricing model	$0.50/GB ingested, $0.03/GB scanned	Free up to 50GB/month, $0.50/GB after
Export to	S3, CloudWatch, Data Firehose, Lambda	Cloud Storage, BigQuery, Pub/Sub, Cloud Storage
Structured logging	Manual (print JSON strings)	Built-in with google-cloud-logging
Log groups	Created per function automatically	Same resources for all functions
Real-time streaming	CloudWatch Logs Insights, third-party	Cloud Logging, Pub/Sub, third-party
Integration	CloudWatch alarms, SNS, Lambda	Cloud Monitoring, Cloud Alerting, Cloud Tasks

Key Differences

• Retention: CloudWatch deletes old logs; Cloud Logging keeps indefinitely (you control deletion)
• Query syntax: CloudWatch Insights uses keywords like stats, fields, filter; Cloud Logging uses simpler field-based filtering
• Pricing: CloudWatch charges for ingestion + queries; Cloud Logging has a generous free tier
• Structured logging: Cloud Logging has better native support via the SDK

---

Privacy & Security

Never log sensitive data on either platform:

# ❌ Bad
print("User password:", password)
print("API key:", api_key)
print("Credit card:", credit_card)

# ✅ Good
print("Authentication attempt for user:", user_id)
print("API call to service:", service_name)
print("Payment processed for user:", user_id)

Both platforms allow you to redact logs after the fact using filter-based log exclusion or log sink filtering.

---

Best Practices (Both Platforms)

1. Use context logging — Include request/trace IDs for request tracking across functions
2. Log at appropriate levels — DEBUG for dev/staging, INFO/WARN/ERROR for production
3. Avoid excessive logging — High log volume increases costs and degrades performance
4. Structured logging — Use JSON format for easier querying and parsing
5. Set retention policies — Delete old logs to manage costs
6. Use trace IDs — Add x-trace-id or x-request-id to correlate logs across services
7. Log errors with context — Include stack traces, input parameters, and state at failure time
8. Monitor log volume — Track ingestion to catch runaway logging

---

Hands-On: Multi-Cloud Logging

AWS CloudWatch

1. Deploy a Lambda that logs structured JSON:

aws lambda create-function \
    --function-name log-demo \
    --runtime python3.12 \
    --role arn:aws:iam::ACCOUNT:role/lambda-role \
    --handler lambda_function.handler \
    --zip-file fileb://function.zip

2. Trigger it and view logs:

aws logs tail /aws/lambda/log-demo --follow

3. Query errors in CloudWatch Logs Insights:

fields @timestamp, @message
| filter @message like /ERROR/
| stats count() by @message

Google Cloud

1. Deploy a Cloud Function with structured logging:

gcloud functions deploy log-demo \
    --runtime python312 \
    --trigger-http \
    --allow-unauthenticated

2. View logs:

gcloud functions logs read log-demo --limit 50

3. Query in Cloud Console Log Explorer:

resource.type="cloud_function"
severity="ERROR"
jsonPayload.userId="user-123"

---

Key Takeaway

Logging is your primary window into production behavior. Both CloudWatch and Cloud Logging automatically capture function output, but structured logging with JSON enables powerful queries, faster debugging, and better cost management. Choose the query syntax and pricing model that fits your team's observability needs.