Amazon MSK: Managed Kafka on AWS

Apache Kafka is the standard tool for event streaming, and it is not easy to run. A production cluster means brokers spread across failure zones, a metadata quorum, disks that fill, rolling upgrades and security configuration. Amazon Managed Streaming for Apache Kafka, MSK, takes that operational work on. AWS describes it as a fully managed service that enables you to build and run applications that use Apache Kafka to process streaming data (MSK docs: what is Amazon MSK).

The important property is in the same paragraph of the guide: MSK runs open-source versions of Apache Kafka, so existing applications, tooling and plugins work without changes to application code. It is Kafka, not a Kafka-like service. Everything in Apache Kafka explained about topics, partitions, offsets and consumer groups applies unchanged.

What AWS manages, and what it does not

Managed for you:

• Provisioning brokers across Availability Zones. You say how many per zone.
• The metadata layer. MSK creates the ZooKeeper nodes, or the KRaft controllers that replace ZooKeeper in newer Kafka; the guide notes KRaft controllers come at no additional cost and need no management.
• Failure recovery. MSK detects a failed broker and mitigates or replaces it, reusing its storage where possible, and clients keep the same broker addresses afterwards.
• Patching and version upgrades, on request.

Still yours:

• Topic design: partition counts, replication factor, retention.
• Keys and ordering. A skewed key causes a hot partition on MSK exactly as it does anywhere.
• Producers and consumers: delivery settings, retries, idempotence, consumer lag.
• Schemas and compatibility rules.
• Capacity, on Provisioned clusters.
• Who may read and write what.

Managed Kafka removes the server work. It does not remove the need to understand Kafka.

Cluster types

There are two cluster types, and Provisioned has two broker types.

Serverless runs Kafka without managing or scaling cluster capacity, with throughput-based pricing, and remains compatible with Kafka clients. It requires IAM access control and does not support Kafka ACLs (MSK docs: MSK Serverless). It is also offered in a limited list of Regions, which the documentation enumerates.

Express brokers remove storage management from Provisioned clusters: storage is pay-as-you-go and scales automatically, and the brokers come pre-configured with AWS's recommended defaults and enforce client throughput quotas (MSK docs: Express brokers).

Standard brokers give the most control, and with it the classic capacity questions. Tiered storage helps with one of them: recent data stays on primary storage, and older data moves automatically to a low-cost tier, so long retention does not require ever larger broker disks (MSK docs: tiered storage).

A reasonable default: Serverless to start or for spiky workloads, Provisioned when throughput is steady and high enough that capacity planning pays for itself, or when you need configuration Serverless does not expose.

Getting data in and out

MSK Connect runs Kafka Connect connectors as a managed service. Connectors move data between Kafka and other systems: sink connectors write topics to stores such as S3 and OpenSearch, and source connectors bring data in, including third-party connectors such as Debezium for streaming database change logs (MSK docs: MSK Connect). For a data engineer this is often the main reason to use Kafka at all: change data capture from operational databases, landed in the lake, with no custom consumer code. CDC explained covers the pattern.

Other consumers on AWS include Lambda functions, Managed Service for Apache Flink for stream processing, and Spark Structured Streaming on EMR or Glue; building a streaming pipeline with Kafka and Spark shows the Spark side.

MSK Replicator copies data between MSK clusters, in the same Region or across Regions, and from self-managed Kafka into MSK, which serves both disaster recovery and migration (MSK docs: MSK Replicator).

Security

A cluster lives inside your VPC, in subnets you choose, and clients must have network access to it. On top of that, the guide lists the authentication options (MSK docs: authentication and authorization):

• IAM access control, which handles both authentication and authorisation with IAM policies. Convenient when clients are AWS services with roles.
• Mutual TLS or SASL/SCRAM for authentication, with Apache Kafka ACLs for authorisation. Closer to how Kafka is secured outside AWS, and friendlier to non-AWS clients.

Decide early. The choice affects every client's configuration and is tedious to change later.

MSK or Kinesis?

Both are managed streaming services on AWS. They differ in what they ask of you and what they give.

Choose MSK when you already use Kafka, need its connectors or stream-processing libraries, want portability, or exchange events with systems outside AWS. Choose Kinesis when the workload is AWS-only and you want the least to operate; Amazon Kinesis explained covers it. For Kafka against other brokers, see Kafka vs RabbitMQ vs Pulsar.

Cost, in outline

Provisioned clusters are billed for brokers and storage while they exist, whether or not traffic flows; Serverless is billed on throughput and usage. Data transfer and connectors are separate. Two habits keep MSK bills reasonable: set topic retention deliberately, since stored data is the cost that grows unnoticed, and do not leave a Provisioned development cluster running. Check the pricing page for current rates in your Region.

Mistakes to avoid

• Treating MSK as "Kafka solved". Partitioning, keys and consumer behaviour are still design work.
• Over-partitioning topics on a small cluster.
• Unlimited retention by default.
• No monitoring of consumer lag, so a stalled consumer goes unnoticed.
• Choosing an authentication method casually.
• Using Kafka where a queue or Firehose would do. A simple delivery of events to S3 does not need a cluster.

Apache Kafka fundamentals and real-time streaming on AWS are covered in the Data Engineering course.

Sources

• Welcome to the Amazon MSK Developer Guide (Amazon MSK Developer Guide) — AWS Documentation
• What is MSK Serverless? (Amazon MSK Developer Guide) — AWS Documentation
• Amazon MSK Express brokers (Amazon MSK Developer Guide) — AWS Documentation
• Understand MSK Connect (Amazon MSK Developer Guide) — AWS Documentation
• Authentication and authorization for Apache Kafka APIs (Amazon MSK Developer Guide) — AWS Documentation
• Tiered storage for Standard brokers (Amazon MSK Developer Guide) — AWS Documentation
• Amazon MSK Replicator (Amazon MSK Developer Guide) — AWS Documentation