A managed service that leverages AWS Lambda as first-class Kafka consumers.
The main documentation is available in the Wiki pages of this repository. Here are most important topics:
- Deploying Malka in your AWS infrastructure
- AWS Deployment Infrastructure Overview
- Docker Container Architecture Overview
Malka is containerised and publicaly available for download as a Docker image. You can use this image to deploy it on your favourite container orchestration platform (e.g. ECS, BeanStalk, K8s, etc).
The easiest (but a bit opinionated) method to deploy it would be using Terraform.
module "malka" {
source = "miere/malka/aws"
version = "0.2.0"
# AWS network configuration
subnet_ids = data.aws_subnet_ids.default.ids
vpc_id = data.aws_vpc.default.id
# Kafka Brokers URLs.
kafka_brokers = ["${var.my_kafka_broker_url}"]
# Malka Configuration
configuration = [
# Each entry means a consumer group
{
# Topic to subscribe
topic_name = "topic",
# Number of parallel consumers for this topic.
# It implies in the number of Lambda functions that will be invoked simultaneously.
topic_number_of_consumers = 2,
# Custom Kafka configuration for this consumer group.
# For further details, please check the options available at the librdkafka documentation
# https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md
consumer_configuration = {},
# The AWS functions that will receive the notification
target_functions = ["my_lambda_function"]
}
]
}For a complete example, please check the example folder. For more details, please check the documentation page that covers this topic.
Copyright 2021 - Malka maintainers and contributors
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.