-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathindex.html
146 lines (139 loc) · 6.46 KB
/
index.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Genie-do</title>
<link rel="stylesheet" href="style.css" />
<link rel="shortcut icon" href="logo.jpeg" type="image/x-icon">
</head>
<body>
<div id="home">
<div id="nav">Genie-do</div>
<button class="btn"><a target="_blank" href="https://github.com/radumarias/genie-do">View on Github</a></button>
</div>
<div class="container">
<div class="giphy">
<img src="robotic.gif" alt="">
</div>
<div class="info">
<div class="heading">Motivation</div>
<div class="content">
<li>
The idea is to relieve developers of the
<span class="highlight">burden of managing the cluster</span> and
offer a REST API to interact, similar to tools like Airflow but
without Python-specific constraints.
</li>
<li>
In the future, <span class="highlight">genie-do</span> will support
libraries in various languages, enabling custom local actions similar
to GitHub Actions, where the community can build and contribute their
own actions.
</li>
<li>
This platform aims to bring the functionality of GitHub Actions + AWS
Lambda + Apache Spark together, enabling seamless orchestration and automation.
</li>
<li>
<span class="highlight">Genie-do</span> is designed as a distributed
executor in Kubernetes, orchestrated with Apache Airflow, and is
deployable on any cloud provider or on-premises environment. If you
can represent your job as a Docker image or containerized application,
<span class="highlight">we will run it, retry it, distribute it, and much more</span>.
You’ll also have access to Kafka, S3, Apache Spark & Flink,
PostgreSQL, NoSQL, and other essential technologies, supporting
real-time streaming, data pipelines, and event-driven architecture.
</li>
<li>
Supporting gRPC services and REST API management, with libraries in
multiple languages, <span class="highlight">genie-do</span> makes it
easy for you to run and manage your code effortlessly, with built-in
scalability, fault tolerance, and secure integration across distributed
systems.
</li>
</div>
</div>
</div>
<div class="container2">
<div class="information">
<div class="heading">Features</div>
<div class="content">
<ul class="features-list">
<li>Retries, failure tolerance, recovery, resilience, suspendable jobs</li>
<li>Sequential and parallel execution</li>
<li>Jobs graph</li>
<li>Job groups</li>
<li>Replicated jobs</li>
<li>Inter-process communication with dbus, zbus, gRPC, Apache Arrow Flight, Spark, Flink</li>
<li>Persistent job states in PostgreSQL and NoSQL solutions</li>
<li>Access file storage like S3</li>
<li>Concurrent execution and exclusive job execution</li>
<li>Remote scheduling, manual, automatic, cron</li>
<li>Can execute any code which can be represented as a container image</li>
</ul>
</div>
</div>
<div class="features">
<img src="features.gif" alt="">
</div>
</div>
<div class="stack-heading">Stack</div>
<div class="stack-container">
<div class="stack-item" data-title="Containerization tool">
<a href="https://kubernetes.io" target="_blank">
<img src="icons/Kubernetes.png" alt="Kubernetes" />
<div class="name">Kubernetes</div>
</a>
</div>
<div class="stack-item" data-title="Workflow orchestration tool">
<a href="https://airflow.apache.org" target="_blank">
<img src="icons/airflow.jpeg" alt="Apache Airflow" />
<div class="name">Apache Airflow</div>
</a>
</div>
<div class="stack-item" data-title="Event bus for inter-service communication">
<a href="https://kafka.apache.org" target="_blank">
<img src="icons/kafka.png" alt="Apache Kafka" />
<div class="name">Apache Kafka</div>
</a>
</div>
<div class="stack-item" data-title="Event bus for inter-service communication">
<a href="https://pulsar.apache.org" target="_blank">
<img src="icons/pulsar.svg" alt="Apache Pulsar" />
<div class="name">Apache Pulsar</div>
</a>
</div>
<div class="stack-item" data-title="Data engineering and machine learning execution">
<a href="https://spark.apache.org" target="_blank">
<img src="icons/spark.png" alt="Apache Spark" />
<div class="name">Apache Spark</div>
</a>
</div>
<div class="stack-item" data-title="Real-time event processing">
<a href="https://flink.apache.org" target="_blank">
<img src="icons/flink.jpeg" alt="Apache Flink" />
<div class="name">Apache Flink</div>
</a>
</div>
<div class="stack-item" data-title="Scalable cloud storage service">
<a href="https://aws.amazon.com/s3" target="_blank">
<img src="icons/amazon.png" alt="Amazon S3" />
<div class="name">Amazon S3</div>
</a>
</div>
<div class="stack-item" data-title="Monitoring and observability tool">
<a href="https://grafana.com" target="_blank">
<img src="icons/Grafana.jpeg" alt="Grafana" />
<div class="name">Grafana</div>
</a>
</div>
<div class="stack-item" data-title="Monitoring and alerting toolkit">
<a href="https://prometheus.io" target="_blank">
<img src="icons/Prometheus.png" alt="Prometheus" />
<div class="name">Prometheus</div>
</a>
</div>
</div>
</body>
</html>