Invoke an AWS Lambda function from cross-account Amazon Kinesis Data Streams

AWS’s multi-account architecture is essential for improving security, compliance, and resource management by separating workloads, enabling granular cost allocation, and facilitating collaboration across different environments. It also reduces risk, increases scalability, and enables advanced network configurations.

In a streaming architecture, event producers, stream storage, and event consumers can be in one account or distributed across different accounts, depending on your business and IT requirements. For example, your company may want to centralize clickstream and log data from several different producers across different accounts. Data consumers in marketing, product engineering, or analytics need access to the same streaming data across accounts, which requires the ability to provide a multi-account streaming architecture.

To build a multi-account streaming architecture, you can use Amazon Kinesis Data Streams as your stream storage and AWS Lambda as your event consumer. Amazon Kinesis Data Streams enables real-time processing of streaming data at scale. Integrating with Lambda enables serverless data processing, allowing you to analyze and act on data streams in real-time without managing infrastructure. This integration supports a variety of use cases including real-time analytics, log processing, Internet of Things (IoT) data ingestion, and more, making it valuable for enterprises that need timely insights from streaming data. This post shows you how to process data ingested into a stream from one account using a Lambda function from another account.

With the recently released support for resource-based policies in Kinesis Data Streams, you can now invoke Lambda from another account. Resource-based policies allow you to specify the AWS account, AWS Identity and Access Management (IAM) user, or IAM role, and the exact Kinesis Data Streams actions to grant access. Once access is granted, you can configure his Lambda function in another account to start processing data streams belonging to your account. This eliminates the need to use Lambda functions to copy streaming data in both accounts, reducing costs and simplifying data processing pipelines. There are no additional charges to your account for sharing data streams or access to registered consumers. Cross-account usage of Kinesis Data Streams resources continues to be billed to the resource owner.

In this post, we use Kinesis Data Streams with enhanced fan-out capabilities to provide consumers with dedicated read throughput tailored to their applications. By default, Kinesis Data Streams provides shared read throughput of 2 MB/s per shard between consumers, but with enhanced fan-out each consumer can enjoy a dedicated throughput of 2 MB/s per shard. Masu. This flexibility allows you to seamlessly adapt Kinesis Data Streams to your specific requirements, choosing between dedicated or shared throughput scalable fan-outs depending on your needs.

Solution overview

This solution deploys Kinesis Data Streams in Account 1 and deploys Lambda as a consumer in Account 2 to receive data from the data stream. The following diagram shows the high-level architecture.

Setup requires the following key elements:

• Kinesis data stream in account 1 and Lambda function in account 2
• Kinesis Data Streams resource policy for Account 1. Enables cross-account Lambda execution roles to perform operations on Kinesis data streams.
• The Lambda execution role in Account 2 and the enhanced fan-out consumer resource policy in Account 1 enable the cross-account Lambda execution role to perform operations on Kinesis data streams.

The setup uses three AWS CloudFormation templates to create key resources.

• CloudFormation Template 1 creates the following major resources in Account 1:
  • Kinesis data stream
  • Enhanced fan-out consumer for Kinesis data streams
• CloudFormation Template 2 creates the following major resources in Account 2:
  • consumer lambda function
  • Consumer Lambda function execution role
• CloudFormation template 3 creates the following resources in account 2.
  • Event source mapping for consumer Lambda functions

This solution supports single-region deployment, and CloudFormation templates must be deployed to the same region across different AWS accounts. This solution uses enhanced fanout for Kinesis Data Streams. This is a best practice for deploying architectures that require high throughput across multiple consumers. To deploy this solution, complete the steps in the following section.

Prerequisites

You need two AWS accounts and the necessary permissions to run the CloudFormation template and create the services described in the solution architecture. Additionally, you must have AWS Command Line Interface (AWS CLI) version 2.15 or later installed.

Launch CloudFormation template 1

To launch your first CloudFormation template, follow these steps:

1. Sign in to the AWS Management Console as Account 1 and choose the appropriate AWS Region.
2. Download and launch CloudFormation template 1 that deploys Kinesis data streams.
3. for LambdaConsumerAccountIdenter your Lambda consumer account ID, and[送信]Click. It takes a few minutes for the CloudFormation template deployment to complete.
4. Once the stack is complete, navigate to the stack in the AWS CloudFormation console. output Click the tab and copy the values ​​for the following parameters.
   • KinesisStreamArn
   • KinesisStreamEFOConsumerArn
   • KMSKeyArn

You will need these values ​​in a later step.

Launch CloudFormation template 2

To launch the second CloudFormation template, follow these steps:

1. Sign in to the console as Account 2 and select the appropriate region.
2. Download and launch CloudFormation template 2 that hosts the Lambda consumer.
3. Specify the following input parameters obtained in the previous step.
   • KinesisStreamArn
   • KinesisStreamEFOConsumerArn
   • KMSKeyArn

The CloudFormation template creates the following major resources:

• lambda consumer
• Lambda execution role

A Lambda function’s execution role is an IAM role that grants your function permissions to access AWS services and resources. Now, create a Lambda execution role with the necessary Kinesis Data Streams and Lambda invocation permissions.

It takes a few minutes for the CloudFormation template deployment to complete.

1. Once the stack is complete, navigate to the stack in the AWS CloudFormation console. output Click the tab and copy the values ​​for the following parameters.
   • KinesisStreamCreateResourcePolicyCommand
   • KinesisStreamEFOConsumerCreateResourcePolicyCommand
2. Run the following AWS CLI command in Account 1 using AWS CloudShell. We recommend using CloudShell because it includes the latest version of the AWS CLI and avoids all types of failures.
   • KinesisStreamCreateResourcePolicyCommand – This creates a resource policy for Kinesis Data Stream in Account 1. Below is a sample resource policy.

     {
     "Version": "2012-10-17",
     "Statement": [
     {
     "Sid": "StreamEFOReadStatementID",
     "Effect": "Allow",
     "Principal": {
     "AWS": [
     "arn:aws:iam::<AWS Lambda - Consumer account id>:role/kds-cross-account-stream-consumer-lambda-execution-role"
     ]
     },
     "Action": [
     "kinesis:DescribeStreamSummary",
     "kinesis:ListShards",
     "kinesis:DescribeStream",
     "kinesis:GetRecords",
     "kinesis:GetShardIterator"
     ],
     "Resource": "arn:aws:kinesis:<region id>:<Account 1 - Amazon KDS account id>:stream/kds-cross-account-stream"
     }
     ]
     }

   • KinesisStreamEFOConsumerCreateResourcePolicyCommand – This creates a resource policy for the enhanced fan-out consumer for Kinesis data streams in Account 1. Below is a sample resource policy.

     {
     "Version": "2012-10-17",
     "Statement": [
     {
     "Sid": "ConsumerEFOReadStatementID",
     "Effect": "Allow",
     "Principal": {
     "AWS": [
     " arn:aws:iam::<AWS Lambda - Consumer account id>:role/kds-cross-account-stream-consumer-lambda-execution-role"
     ]
     },
     "Action": [
     "kinesis:DescribeStreamConsumer",
     "kinesis:SubscribeToShard"
     ],
     "Resource": "arn:aws:kinesis:<region id>:<Account 1 - Amazon KDS account id>:stream/kds-cross-account-stream/consumer/kds-cross-account-stream-efo-consumer:1706616477"
     }
     ]
     }

You can also access this policy from the following locations in the Kinesis Data Streams console: Enhanced fan-out, consumer nameand Consumer shared resource-based policy.

Launch CloudFormation template 3

Now that you have created a resource policy for your Kinesis data stream and its enhanced fanout consumer in Account 1, you can create a Lambda event source mapping for your consumer Lambda function in Account 2. Follow these steps:

1. Sign in to the console as Account 2 and select the appropriate region.
2. Download and launch CloudFormation Template 3 to update the stack that you created using CloudFormation Template 2.

A CloudFormation template creates a Lambda event source mapping.

Validate the solution

At this point, the deployment is complete. Kinesis data streams can be used to consume messages, and Lambda functions receive these messages in the destination account. To send a sample message to the data stream in Account 1, run the following AWS CLI command using CloudShell.

aws kinesis put-record --stream-name kds-cross-account-stream --data sampledatarecord --partition-key samplepartitionkey3 --region <region id>

The Lambda function in account 2 should be able to receive messages and you should be able to verify the same using Amazon CloudWatch logs.

1. In the CloudWatch console, log group in the navigation pane.
2. Locate the log group /aws/lambda/kds-cross-account-stream-efo-consumer.
3. choose Search for log groups View related log messages. Below is an example message.

   "Records": [
   {
   "kinesis": {
   "kinesisSchemaVersion": "1.0",
   "partitionKey": "samplepartitionkey3",
   "sequenceNumber": "49648798411111169765201534322676841348246990356337393698",
   "data": "sampledatarecord",
   "approximateArrivalTimestamp": 1706623274.658
   },

cleaning

It is always recommended to clean up all resources created as part of this post to avoid additional costs.

To clean up your resources, remove the respective CloudFormation stacks from accounts 1 and 2 to stop the producers from pushing events to your Kinesis data stream. This will prevent you from being charged unnecessary fees.

summary

In this post, you learned how to configure cross-account Lambda and Kinesis Data Streams integration using AWS resource-based policies. This allows data ingested into a stream in one AWS account to be processed through a Lambda function in another account. To support customers who use Kinesis data streams in a central account and have multiple consumers reading data from them, we used the enhanced fan-out capabilities of Kinesis Data Streams.

To get started, open the Kinesis Data Streams console or use the new API PutResourcePolicy to attach a resource policy to your data stream or consumer.

About the author

Pratik Patel Senior Technical Account Manager and Streaming Analytics Specialist. He works with his AWS customers to help plan and build solutions using best practices, and provides ongoing support and technical guidance to keep customers’ AWS environments proactively and operationally healthy. It offers.

Amar I am a Senior Solutions Architect at Amazon AWS in the UK. He works on strategic implementations across power, utilities, manufacturing, and automotive customers, and he specializes in using AWS streaming and advanced data analytics solutions to drive optimal business outcomes.