Skip to content

Commit

Permalink
Added data processing example
Browse files Browse the repository at this point in the history
Rename instance.yaml to instance-template.yaml

Added scripts for processing

Fixed scripts path

Update README.md
  • Loading branch information
giedri authored and a-hilaly committed Nov 26, 2024
1 parent 0583ea9 commit 45cac11
Show file tree
Hide file tree
Showing 7 changed files with 1,032 additions and 0 deletions.
107 changes: 107 additions & 0 deletions examples/data-processor/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,107 @@
# Event Driven Architectures with Amazon EKS and AWS Controllers for Kubernetes

This data processing example uses event-driven approach for data ingestion and process orchestration, along with Amazon EMR on EKS for data processing implementation. This example uses [New York City taxi data](https://www.nyc.gov/site/tlc/about/tlc-trip-record-data.page).
## Architecture

Following diagram illustrates flow of the example and services used:

![Architecture diagram](./assets/eks-eda.png)
1. Once data object lands into input bucket on Amazon S3, it sends event to the Amazon EventBridge
2. EventBridge, based on a rule, starts AWS Step Functions workflow execution that orchestrates data processing
3. Workflow creates EMR on EKS virtual cluster and starts Apache Spark job, specifying script in an S3 bucket to be used and data object in the S3 to be processed
4. Spark job reads the newly arrived data from S3, runs the processing, and saves output data to a S3 bucket.
5. In parallel to the data processing by Spark, Step Functions workflow copies incoming data file to a data lake (another S3 bucket)
6. Once Spark finishes data processing, workflow reads results from the S3 bucket and puts them to an Amazon DynamoDB database
7. Workflow sends event to the EventBridge custom bus, notifying all subscribers that data processing task finished
8. Amazon Simple Notification Service (SNS) receives event from the event bus and sends e-mail message to the subscribers

Following diagram shows Step Functions workflow:
![StepFunctions workflow](./assets/stepfunctions_graph.png)

## Prerequisites
EKS cluster with EMR on EKS deployed and IRSA configured. Following steps are based on [AWS ACK tutorial instructions](https://aws-controllers-k8s.github.io/community/docs/tutorials/emr-on-eks-example/)

Install kro in the cluster created in the previous step following [instructions](https://kro.run/docs/getting-started/Installation)

## Create instance

Create kro ResourceGroup for the data processor:
```shell
kubectl apply -f eda-eks-data-processor.yaml
```

Create instance of the data processor

Set workload name (it will be used as a prefix for the stack components):
```shell
export WORKLOAD_NAME="eda-eks-demo"
```

Following steps assume that input, scripts and data lake buckets are the same one. Resource group creates a new input bucket only. If you use the same name for scripts and lake, it will use input bucket for all purposes. Specify different bucket name for the scripts library and data lake if necessary.

```shell
export INPUT_BUCKET_NAME="${WORKLOAD_NAME}-bucket"
export SCRIPTS_BUCKET_NAME="${WORKLOAD_NAME}-bucket"
export LAKE_BUCKET_NAME="${WORKLOAD_NAME}-bucket"
```

```shell
envsubst < "instance-template.yaml" > "instance.yaml"
```
Check that the instance definition populated with values. Update prefix, API name or description values in the definition if desired.
```shell
cat instance.yaml
```

## Deploy instance

Apply the instance definition:
```shell
kubectl apply -f instance.yaml
```


## Post-deployment steps
### S3 event notification configuration update
Check newly created bucket name:
```shell
export BUCKET_NAME=$(kubectl get bucket.s3.services.k8s.aws -o jsonpath='{.items..metadata.name}' --namespace $WORKLOAD_NAME | grep $WORKLOAD_NAME)
```
This step is required until ACK missing feature is implemented.
```bash
# Enable EventBridge notifications as ACK does not support it at this time and they are not enabled by default
aws s3api put-bucket-notification-configuration --bucket $BUCKET_NAME --notification-configuration='{ "EventBridgeConfiguration": {} }'
```
### Spark data processing script upload
```bash
aws s3 cp ./scripts s3://$BUCKET_NAME/scripts --recursive
```

## Test

List all resources in the stack namespace (it will take some time to get all results):
```shell
kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 kubectl get --show-kind --ignore-not-found -n $WORKLOAD_NAME
```
Copy sample data for processing (for example `yellow_tripdata_2024-05.parquet`):
```shell
aws s3 cp <your-sample-data-file-folder>/yellow_tripdata_2024-05.parquet s3://$BUCKET_NAME/input/yellow_tripdata_2024-05.parquet
```

## Clean up

Delete S3 bucket content:
```shell
aws s3 rm --recursive s3://$BUCKET_NAME
```
Delete instance and resource group:
```shell
kubectl delete -f instance.yaml
kubectl delete -f eda-eks-data-processor.yaml
```

Note: You may need to patch resource finalizer in case deletion of the resource hangs. For example, following command patches SNS subscription in `eda-eks-demo` namespace (unconfirmed subscriptions cannot be deleted (they are cleaned up automatically after 48hrs) and prevent resource from deletion):
```shell
kubectl patch subscription.sns.services.k8s.aws/eda-eks-demo-notifications-subscription -p '{"metadata":{"finalizers":[]}}' --type=merge --namespace eda-eks-demo
```

128 changes: 128 additions & 0 deletions examples/data-processor/assets/eks-eda.drawio
Original file line number Diff line number Diff line change
@@ -0,0 +1,128 @@
<mxfile host="drawio.corp.amazon.com" modified="2024-09-04T19:22:20.354Z" agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:129.0) Gecko/20100101 Firefox/129.0" etag="LsC5K9HA4w_bQBm9TeA2" version="21.7.4" type="device">
<diagram name="Page-1" id="sDRVptemRm56i0psgaTi">
<mxGraphModel dx="1056" dy="973" grid="1" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1" pageWidth="850" pageHeight="1100" math="0" shadow="0">
<root>
<mxCell id="0" />
<mxCell id="1" parent="0" />
<mxCell id="cbFFLlK-bVltba2RXdJL-4" value="" style="outlineConnect=0;fontColor=#232F3E;gradientColor=none;fillColor=#8C4FFF;strokeColor=none;dashed=0;verticalLabelPosition=bottom;verticalAlign=top;align=center;html=1;fontSize=12;fontStyle=0;aspect=fixed;pointerEvents=1;shape=mxgraph.aws4.data_lake_resource_icon;" parent="1" vertex="1">
<mxGeometry x="530" y="328" width="78" height="78" as="geometry" />
</mxCell>
<mxCell id="cbFFLlK-bVltba2RXdJL-28" value="" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;" parent="1" source="cbFFLlK-bVltba2RXdJL-9" target="cbFFLlK-bVltba2RXdJL-27" edge="1">
<mxGeometry relative="1" as="geometry" />
</mxCell>
<mxCell id="cbFFLlK-bVltba2RXdJL-11" value="" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;" parent="1" source="cbFFLlK-bVltba2RXdJL-6" target="cbFFLlK-bVltba2RXdJL-8" edge="1">
<mxGeometry relative="1" as="geometry" />
</mxCell>
<mxCell id="cbFFLlK-bVltba2RXdJL-6" value="" style="points=[[0,0,0],[0.25,0,0],[0.5,0,0],[0.75,0,0],[1,0,0],[0,1,0],[0.25,1,0],[0.5,1,0],[0.75,1,0],[1,1,0],[0,0.25,0],[0,0.5,0],[0,0.75,0],[1,0.25,0],[1,0.5,0],[1,0.75,0]];outlineConnect=0;fontColor=#232F3E;fillColor=#7AA116;strokeColor=#ffffff;dashed=0;verticalLabelPosition=bottom;verticalAlign=top;align=center;html=1;fontSize=12;fontStyle=0;aspect=fixed;shape=mxgraph.aws4.resourceIcon;resIcon=mxgraph.aws4.s3;" parent="1" vertex="1">
<mxGeometry x="330" y="60" width="78" height="78" as="geometry" />
</mxCell>
<mxCell id="cbFFLlK-bVltba2RXdJL-15" value="" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;entryX=0.5;entryY=0;entryDx=0;entryDy=0;" parent="1" source="cbFFLlK-bVltba2RXdJL-8" target="cbFFLlK-bVltba2RXdJL-13" edge="1">
<mxGeometry relative="1" as="geometry">
<mxPoint x="369" y="388" as="targetPoint" />
</mxGeometry>
</mxCell>
<mxCell id="cbFFLlK-bVltba2RXdJL-8" value="" style="points=[[0,0,0],[0.25,0,0],[0.5,0,0],[0.75,0,0],[1,0,0],[0,1,0],[0.25,1,0],[0.5,1,0],[0.75,1,0],[1,1,0],[0,0.25,0],[0,0.5,0],[0,0.75,0],[1,0.25,0],[1,0.5,0],[1,0.75,0]];outlineConnect=0;fontColor=#232F3E;fillColor=#E7157B;strokeColor=#ffffff;dashed=0;verticalLabelPosition=bottom;verticalAlign=top;align=center;html=1;fontSize=12;fontStyle=0;aspect=fixed;shape=mxgraph.aws4.resourceIcon;resIcon=mxgraph.aws4.eventbridge;" parent="1" vertex="1">
<mxGeometry x="330" y="230" width="78" height="78" as="geometry" />
</mxCell>
<mxCell id="cbFFLlK-bVltba2RXdJL-9" value="" style="outlineConnect=0;fontColor=#232F3E;gradientColor=none;fillColor=#E7157B;strokeColor=none;dashed=0;verticalLabelPosition=bottom;verticalAlign=top;align=center;html=1;fontSize=12;fontStyle=0;aspect=fixed;pointerEvents=1;shape=mxgraph.aws4.eventbridge_custom_event_bus_resource;" parent="1" vertex="1">
<mxGeometry x="330" y="590" width="78" height="69" as="geometry" />
</mxCell>
<mxCell id="cbFFLlK-bVltba2RXdJL-10" value="" style="outlineConnect=0;fontColor=#232F3E;gradientColor=none;fillColor=#E7157B;strokeColor=none;dashed=0;verticalLabelPosition=bottom;verticalAlign=top;align=center;html=1;fontSize=12;fontStyle=0;aspect=fixed;pointerEvents=1;shape=mxgraph.aws4.event;" parent="1" vertex="1">
<mxGeometry x="330" y="170" width="30" height="30" as="geometry" />
</mxCell>
<mxCell id="cbFFLlK-bVltba2RXdJL-21" value="" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;" parent="1" source="cbFFLlK-bVltba2RXdJL-13" target="cbFFLlK-bVltba2RXdJL-4" edge="1">
<mxGeometry relative="1" as="geometry" />
</mxCell>
<mxCell id="cbFFLlK-bVltba2RXdJL-52" value="" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;" parent="1" source="cbFFLlK-bVltba2RXdJL-13" target="cbFFLlK-bVltba2RXdJL-9" edge="1">
<mxGeometry relative="1" as="geometry" />
</mxCell>
<mxCell id="cbFFLlK-bVltba2RXdJL-13" value="AWS Step Functions workflow" style="points=[[0,0],[0.25,0],[0.5,0],[0.75,0],[1,0],[1,0.25],[1,0.5],[1,0.75],[1,1],[0.75,1],[0.5,1],[0.25,1],[0,1],[0,0.75],[0,0.5],[0,0.25]];outlineConnect=0;gradientColor=none;html=1;whiteSpace=wrap;fontSize=12;fontStyle=0;container=1;pointerEvents=0;collapsible=0;recursiveResize=0;shape=mxgraph.aws4.group;grIcon=mxgraph.aws4.group_aws_step_functions_workflow;strokeColor=#CD2264;fillColor=none;verticalAlign=top;align=left;spacingLeft=30;fontColor=#CD2264;dashed=0;" parent="1" vertex="1">
<mxGeometry x="304" y="390" width="130" height="130" as="geometry" />
</mxCell>
<mxCell id="cbFFLlK-bVltba2RXdJL-18" value="" style="group;" parent="1" vertex="1" connectable="0">
<mxGeometry x="30" y="120" width="250" height="256" as="geometry" />
</mxCell>
<mxCell id="cbFFLlK-bVltba2RXdJL-17" value="" style="fillColor=#EFF0F3;strokeColor=none;dashed=0;verticalAlign=top;fontStyle=0;fontColor=#232F3D;whiteSpace=wrap;html=1;" parent="cbFFLlK-bVltba2RXdJL-18" vertex="1">
<mxGeometry width="250" height="256" as="geometry" />
</mxCell>
<mxCell id="cbFFLlK-bVltba2RXdJL-1" value="" style="points=[[0,0,0],[0.25,0,0],[0.5,0,0],[0.75,0,0],[1,0,0],[0,1,0],[0.25,1,0],[0.5,1,0],[0.75,1,0],[1,1,0],[0,0.25,0],[0,0.5,0],[0,0.75,0],[1,0.25,0],[1,0.5,0],[1,0.75,0]];outlineConnect=0;fontColor=#232F3E;fillColor=#ED7100;strokeColor=#ffffff;dashed=0;verticalLabelPosition=bottom;verticalAlign=top;align=center;html=1;fontSize=12;fontStyle=0;aspect=fixed;shape=mxgraph.aws4.resourceIcon;resIcon=mxgraph.aws4.eks;" parent="cbFFLlK-bVltba2RXdJL-18" vertex="1">
<mxGeometry width="78" height="78" as="geometry" />
</mxCell>
<mxCell id="RE9Nt39qmt5__te_f2S3-1" value="Amazon EKS" style="text;whiteSpace=wrap;html=1;" vertex="1" parent="cbFFLlK-bVltba2RXdJL-18">
<mxGeometry y="78" width="80" height="10" as="geometry" />
</mxCell>
<mxCell id="RE9Nt39qmt5__te_f2S3-2" value="Amazon EMR on EKS" style="text;whiteSpace=wrap;html=1;" vertex="1" parent="cbFFLlK-bVltba2RXdJL-18">
<mxGeometry x="80" y="190" width="130" height="20" as="geometry" />
</mxCell>
<mxCell id="cbFFLlK-bVltba2RXdJL-44" value="3" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;strokeWidth=2;fontFamily=Tahoma;spacingBottom=4;spacingRight=2;strokeColor=#d3d3d3;" parent="cbFFLlK-bVltba2RXdJL-18" vertex="1">
<mxGeometry x="150" y="220" width="20" height="20" as="geometry" />
</mxCell>
<mxCell id="cbFFLlK-bVltba2RXdJL-2" value="" style="points=[[0,0,0],[0.25,0,0],[0.5,0,0],[0.75,0,0],[1,0,0],[0,1,0],[0.25,1,0],[0.5,1,0],[0.75,1,0],[1,1,0],[0,0.25,0],[0,0.5,0],[0,0.75,0],[1,0.25,0],[1,0.5,0],[1,0.75,0]];outlineConnect=0;fontColor=#232F3E;fillColor=#8C4FFF;strokeColor=#ffffff;dashed=0;verticalLabelPosition=bottom;verticalAlign=top;align=center;html=1;fontSize=12;fontStyle=0;aspect=fixed;shape=mxgraph.aws4.resourceIcon;resIcon=mxgraph.aws4.emr;" parent="1" vertex="1">
<mxGeometry x="136" y="230" width="78" height="78" as="geometry" />
</mxCell>
<mxCell id="cbFFLlK-bVltba2RXdJL-19" value="" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=0;exitY=0.5;exitDx=0;exitDy=0;" parent="1" source="cbFFLlK-bVltba2RXdJL-13" target="cbFFLlK-bVltba2RXdJL-2" edge="1">
<mxGeometry relative="1" as="geometry">
<mxPoint x="300" y="460" as="sourcePoint" />
</mxGeometry>
</mxCell>
<mxCell id="cbFFLlK-bVltba2RXdJL-20" value="" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;startArrow=classic;startFill=1;" parent="1" source="cbFFLlK-bVltba2RXdJL-6" target="cbFFLlK-bVltba2RXdJL-2" edge="1">
<mxGeometry relative="1" as="geometry" />
</mxCell>
<mxCell id="cbFFLlK-bVltba2RXdJL-25" value="" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=0;exitY=0.5;exitDx=0;exitDy=0;" parent="1" source="cbFFLlK-bVltba2RXdJL-13" target="cbFFLlK-bVltba2RXdJL-24" edge="1">
<mxGeometry relative="1" as="geometry">
<mxPoint x="304" y="455" as="sourcePoint" />
<mxPoint x="175" y="308" as="targetPoint" />
</mxGeometry>
</mxCell>
<mxCell id="cbFFLlK-bVltba2RXdJL-24" value="" style="points=[[0,0,0],[0.25,0,0],[0.5,0,0],[0.75,0,0],[1,0,0],[0,1,0],[0.25,1,0],[0.5,1,0],[0.75,1,0],[1,1,0],[0,0.25,0],[0,0.5,0],[0,0.75,0],[1,0.25,0],[1,0.5,0],[1,0.75,0]];outlineConnect=0;fontColor=#232F3E;fillColor=#C925D1;strokeColor=#ffffff;dashed=0;verticalLabelPosition=bottom;verticalAlign=top;align=center;html=1;fontSize=12;fontStyle=0;aspect=fixed;shape=mxgraph.aws4.resourceIcon;resIcon=mxgraph.aws4.dynamodb;" parent="1" vertex="1">
<mxGeometry x="136" y="520" width="78" height="78" as="geometry" />
</mxCell>
<mxCell id="cbFFLlK-bVltba2RXdJL-27" value="" style="points=[[0,0,0],[0.25,0,0],[0.5,0,0],[0.75,0,0],[1,0,0],[0,1,0],[0.25,1,0],[0.5,1,0],[0.75,1,0],[1,1,0],[0,0.25,0],[0,0.5,0],[0,0.75,0],[1,0.25,0],[1,0.5,0],[1,0.75,0]];outlineConnect=0;fontColor=#232F3E;fillColor=#E7157B;strokeColor=#ffffff;dashed=0;verticalLabelPosition=bottom;verticalAlign=top;align=center;html=1;fontSize=12;fontStyle=0;aspect=fixed;shape=mxgraph.aws4.resourceIcon;resIcon=mxgraph.aws4.sns;" parent="1" vertex="1">
<mxGeometry x="330" y="740" width="78" height="78" as="geometry" />
</mxCell>
<mxCell id="cbFFLlK-bVltba2RXdJL-38" value="" style="outlineConnect=0;fontColor=#232F3E;gradientColor=none;fillColor=#E7157B;strokeColor=none;dashed=0;verticalLabelPosition=bottom;verticalAlign=top;align=center;html=1;fontSize=12;fontStyle=0;aspect=fixed;pointerEvents=1;shape=mxgraph.aws4.event;" parent="1" vertex="1">
<mxGeometry x="340" y="525" width="30" height="30" as="geometry" />
</mxCell>
<mxCell id="cbFFLlK-bVltba2RXdJL-41" value="1" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;strokeWidth=2;fontFamily=Tahoma;spacingBottom=4;spacingRight=2;strokeColor=#d3d3d3;" parent="1" vertex="1">
<mxGeometry x="380" y="200" width="20" height="20" as="geometry" />
</mxCell>
<mxCell id="cbFFLlK-bVltba2RXdJL-43" value="2" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;strokeWidth=2;fontFamily=Tahoma;spacingBottom=4;spacingRight=2;strokeColor=#d3d3d3;" parent="1" vertex="1">
<mxGeometry x="380" y="356" width="20" height="20" as="geometry" />
</mxCell>
<mxCell id="cbFFLlK-bVltba2RXdJL-45" value="4" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;strokeWidth=2;fontFamily=Tahoma;spacingBottom=4;spacingRight=2;strokeColor=#d3d3d3;" parent="1" vertex="1">
<mxGeometry x="280" y="70" width="20" height="20" as="geometry" />
</mxCell>
<mxCell id="cbFFLlK-bVltba2RXdJL-46" value="5" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;strokeWidth=2;fontFamily=Tahoma;spacingBottom=4;spacingRight=2;strokeColor=#d3d3d3;" parent="1" vertex="1">
<mxGeometry x="490" y="376" width="20" height="20" as="geometry" />
</mxCell>
<mxCell id="cbFFLlK-bVltba2RXdJL-47" value="6" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;strokeWidth=2;fontFamily=Tahoma;spacingBottom=4;spacingRight=2;strokeColor=#d3d3d3;" parent="1" vertex="1">
<mxGeometry x="234" y="535" width="20" height="20" as="geometry" />
</mxCell>
<mxCell id="cbFFLlK-bVltba2RXdJL-49" value="7" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;strokeWidth=2;fontFamily=Tahoma;spacingBottom=4;spacingRight=2;strokeColor=#d3d3d3;" parent="1" vertex="1">
<mxGeometry x="380" y="570" width="20" height="20" as="geometry" />
</mxCell>
<mxCell id="cbFFLlK-bVltba2RXdJL-51" value="8" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;strokeWidth=2;fontFamily=Tahoma;spacingBottom=4;spacingRight=2;strokeColor=#d3d3d3;" parent="1" vertex="1">
<mxGeometry x="380" y="710" width="20" height="20" as="geometry" />
</mxCell>
<mxCell id="RE9Nt39qmt5__te_f2S3-3" value="Amazon S3" style="text;whiteSpace=wrap;html=1;" vertex="1" parent="1">
<mxGeometry x="338" y="138" width="70" height="20" as="geometry" />
</mxCell>
<mxCell id="RE9Nt39qmt5__te_f2S3-4" value="Amazon EventBridge" style="text;whiteSpace=wrap;html=1;" vertex="1" parent="1">
<mxGeometry x="310" y="308" width="130" height="20" as="geometry" />
</mxCell>
<mxCell id="RE9Nt39qmt5__te_f2S3-8" value="Amazon DynamoDB" style="text;whiteSpace=wrap;html=1;" vertex="1" parent="1">
<mxGeometry x="115" y="598" width="120" height="20" as="geometry" />
</mxCell>
<mxCell id="RE9Nt39qmt5__te_f2S3-9" value="Amazon SNS" style="text;whiteSpace=wrap;html=1;" vertex="1" parent="1">
<mxGeometry x="330" y="818" width="80" height="20" as="geometry" />
</mxCell>
<mxCell id="RE9Nt39qmt5__te_f2S3-10" value="Amazon EventBridge&lt;br&gt;Custom Bus" style="text;whiteSpace=wrap;html=1;align=center;" vertex="1" parent="1">
<mxGeometry x="310" y="650" width="120" height="40" as="geometry" />
</mxCell>
<mxCell id="RE9Nt39qmt5__te_f2S3-12" value="Amazon S3&lt;br&gt;(Data Lake)" style="text;whiteSpace=wrap;html=1;" vertex="1" parent="1">
<mxGeometry x="530" y="406" width="70" height="20" as="geometry" />
</mxCell>
</root>
</mxGraphModel>
</diagram>
</mxfile>
Binary file added examples/data-processor/assets/eks-eda.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading

0 comments on commit 45cac11

Please sign in to comment.