Amazon MSK resources

Why MSK?

Apache Kafka is an open-source platform for building real-time streaming data pipelines and applications. However, Apache Kafka is difficult to architect, operate, and manage on your own. Amazon MSK is a fully managed service that makes it easy for you to build and run applications that use Apache Kafka to process streaming data without needing Apache Kafka infrastructure management expertise.

The resources on this page help you create an Amazon MSK cluster, use best practices when operating and monitoring your cluster, migrate Apache Kafka workloads to Amazon MSK, and use Amazon Managed Service for Apache Flink to analyze data streams.

Amazon MSK Labs

Gain hands-on experience with Amazon MSK:

  • Set up an Amazon MSK cluster using both the Amazon MSK Console and AWS CLI
  • Add brokers and rebalance the cluster to adapt to additional producers, consumers, and growing workloads, both on the Console and using the CLI
  • Scale your Amazon MSK cluster to add more storage capacity
  • Gather all of the JMX metrics and work with the data in the popular open source monitoring tool Prometheus
  • Enable encryption-in-transit and authentication using various security features

Getting started Lab

This workshop teaches you how to analyze the performance of various products in an e-commerce site by ingesting, transforming, and analyzing real-time clickstream data.

  • Set up your data producer to deliver clickstreams to Apache Kafka topics in Amazon MSK
  • Configure and start an Amazon Managed Service for Apache Flink application to process and aggregate the clickstream data, and send the results to Amazon MSK and Amazon Elasticsearch Service
  • Create Kibana visualizations and a Kibana dashboard to visualize the real-time clickstream analytics

Start the Clickstream Analytics Lab

The Migration Lab teaches you different ways of migrating a self-managed Apache Kafka cluster, whether on Amazon EC2 or on premises, to Amazon MSK. Gain experience with different tools to complete a migration to Amazon MSK.

  • Migrate a self-managed cluster on Amazon EC2 or on-premises to Amazon MSK
  • Setup monitoring and your pre-migration environment
  • Use tools like MirrorMaker 2.0
  • Migrate your client and perform the final cutover

Start the Migration Lab

Featured Amazon MSK Partners

lenses.io

Lenses is a simple, powerful, and secure self service DataOps platform. Operate with confidence on Apache Kafka & Amazon MSK. Lenses delivers broad access to data with fine-grained controls while empowering data gurus with SQL capabilities for data flows.

Visit partner
lenses.io logo

Datadog

Datadog is a SaaS-based monitoring and analytics platform for large-scale applications and infrastructure. Combining real-time logs, metrics from servers, containers, databases, and applications with end-to-end tracing, Datadog delivers actionable alerts and powerful visualizations to provide full-stack observability.

Visit partner
datadog logo

New Relic

With complete visibility from the customer to code to containers, New Relic gives you an easy way to manage the complexities of your AWS environment. Monitor Amazon MSK using the New Relic Prometheus OpenMetrics integration.

Visit partner
New Relic logo

Sumo Logic

The Sumo Logic cloud-native analytics platform helps you collect, correlate, and analyze all types of machine data (logs, metrics, events) to reduce the time to identify, troubleshoot, and resolve performance, security, and compliance issues. Gather information on your Amazon MSK cluster for use with Telegraf.

Visit partner
Sumo Logic logo

Videos

Poshmark accelerates growth via real-time analytics & personalization (49:28)
How Riot Games processes 20 TB of analytics data daily on AWS (44:07)
How Sony Orchard accelerated innovation with Amazon MSK (36:32)