Google Cloud Pub/Sub

Google Cloud Pub/Sub

by Google Cloud

A fully-managed, real-time messaging service for asynchronous event delivery and streaming.

See https://cloud.google.com/pubsub

Overview

• Google Cloud Pub/Sub is a managed publish/subscribe messaging service that decouples senders (publishers) from receivers (subscribers).
• Designed for real-time data streaming, event-driven architectures, and asynchronous workflows with typical latencies on the order of 100ms.
• Two product variants historically: Standard Pub/Sub (global, fully-managed) and Pub/Sub Lite (zonal, lower-cost/throughput model). Note: Pub/Sub Lite deprecation timelines should be checked in Google’s docs.

Features

• Global, fully-managed messaging with automatic scaling and high availability across regions.
• At-least-once delivery with configurable acknowledgment deadlines and retry policies.
• Message ordering support (ordering keys) to preserve per-key ordering guarantees.
• Dead-letter topics for handling messages that repeatedly fail processing.
• Message filtering on attributes to reduce subscriber-side filtering.
• Push and pull subscription delivery models.
• Exactly-once delivery (in beta/GA historically - check current docs for availability in your region) via message deduplication and publish IDs.
• Persistent message retention and configurable acknowledgement deadlines.
• Strong integration with Google Cloud ecosystem: Dataflow, BigQuery, Cloud Functions, Cloud Run, Eventarc, Logging, IAM, and more.
• End-to-end encryption and IAM-based access control; support for CMEK (customer-managed encryption keys).

Superpowers

• Minimal operational overhead: no brokers to manage, automatic scaling for spikes in throughput.
• Deep GCP integrations enable serverless event-driven pipelines (e.g., trigger Cloud Functions or Cloud Run jobs directly from messages).
• Flexible consumption models: push to HTTPS endpoints or pull with client libraries across many languages.
• Ordering keys let you keep per-key ordering while still using parallel consumers for throughput.
• Cost/throughput options: standard Pub/Sub for most workloads and Lite historically for cheaper, provisioned workloads (review current availability and deprecation notices).

Pricing (summary)

• Pricing is usage-based. Typical fee components:
  • Ingress (publish) and egress (delivery) data volumes (first 10 GB/month free in many regions).
  • Retention and snapshot storage when retained beyond default windows.
  • Regional or cross-region data transfer costs if messages cross locations.
• Pub/Sub Lite used a capacity-reservation pricing model (zonal), but customers should verify current pricing and the deprecation status on Google’s pricing pages.
• Always consult the official pricing pages for up-to-date numbers: https://cloud.google.com/pubsub/pricing

Common use cases

• Event-driven microservices and decoupled application architectures.
• Real-time analytics pipelines (Pub/Sub → Dataflow → BigQuery).
• Log and telemetry ingestion from distributed services and IoT devices.
• Reliable job dispatch and background task queues.
• Fan-out to multiple subscribers for notifications and caching.

Best practices & considerations

• Choose ordering keys only when per-key ordering is required; they reduce parallelism for a single key but allow parallelism across keys.
• Use dead-letter topics to isolate and analyze poison messages without blocking healthy message flow.
• Prefer pull subscriptions for large-scale consumers that can autoscale; use push for simple HTTP endpoints or serverless triggers.
• Monitor subscription backlog and configure appropriate acknowledgement deadlines and retry policies.
• Secure topics with IAM roles and consider CMEK if you have strict encryption requirements.
• Evaluate cost trade-offs between standard Pub/Sub vs any lower-cost alternatives; consider message batching and compression to save bandwidth costs.

Useful links

• Official docs: https://cloud.google.com/pubsub/docs

• Pricing: https://cloud.google.com/pubsub/pricing

• Pub/Sub vs Pub/Sub Lite comparison: https://cloud.google.com/pubsub/lite

• Integration guide (Dataflow, BigQuery): https://cloud.google.com/solutions/real-time-analytics

Quick example (conceptual)

Publisher publishes JSON events to a topic. A Dataflow pipeline reads from the topic, transforms data, and writes aggregated results to BigQuery in near real-time. Cloud Functions subscribed to a different subscription send notifications when certain events occur.