I am using GitLab CI now for more than a year and I really love the features in GitLab. GitLab provides a complete and powerful tool for day to day development. And of course, there are always feature that you miss. Until now there is no support for periodic builds, in the coming release the feature will be shipped as experimental feature and in the next one is should be general available. But in case you must deal for some reason with an older version, a work around is described below as I used last year.
+
You could argue why you should need a feature as a periodic build. Ideally a build should be immutable and only trigger by a change in GIT, a commit. But the world is not always perfect, project build are sometime not of the quality that you are expect or tools are not that reliable as you hope. For example, dependencies resolving could break over the time, to avoid you find the problem once your boss is watching you when fixing a critical bug, a periodic build can alert you earlier. Another and much better reason to argue for the feature is that the GitLab build are so powerful that is handy to use them for a scenario based health check.
+
The build trigger
+
To setup a periodic build you first need to be able to trigger a build in some way. GitLab provides an API to trigger a build. Setting up the trigger is simple and complete guided in GitLab, just execute the steps below:
+
+
Go to your GitLab projects
+
Navigate to Settings -> CI/CD Pipelines
+
Scroll down to the trigger section and create a trigger
+
Make a note of the TOKEN and trigger URL. An curl example for trigger is shown as well.
The next step is to execute the trigger periodic. A standard way of running a job periodic is by defining crontab. We will use a docker container to execute the crontab so we can deploy it anywhere.
+
I have createe a base docker image containing a script to trigger GitLab by executing a curl command as shown above. The crontab will be copied to the image ONBUILD. In the crontab we point to the script that can be executed as follow:
Now all peace’s are ready we only have to build our docker image and run it to trigger the build. First, we create a Dockerfile and only extend the base image npalm/gitlab-periodic-trigger. Create a new directory, and add a Dockerfile with only the line:
+
FROM npalm/gitlab-periodic-trigger:1.0.0
+
This image expects a file gitlabcrontab containing the crontab and will copy it to the image once build. Create a file named gitlabcrontab, and add one ore more triggers, an example is below.
+
# Example of job definition:
+# .---------------- minute (0 - 59)
+# | .------------- hour (0 - 23)
+# | | .---------- day of month (1 - 31)
+# | | | .------- month (1 - 12) OR jan,feb,mar,apr ...
+# | | | | .---- day of week (0 - 6) (Sunday=0 or 7)
+# | | | | |
+22 11 * * * trigger-gitlab.sh -t <token> -r <ref> -u <gitlab_trigger_url>
+
Now we build the container docker build -t periodic-trigger . and finally start the container docker run -d periodic-trigger
+
To enable the feature for your teams you could set up a build to build the docker image as described below and push it automated to an environment.
\ No newline at end of file
diff --git a/2017/05/07/docker-multi-stage-builds/index.html b/2017/05/07/docker-multi-stage-builds/index.html
new file mode 100644
index 00000000..6cf8493e
--- /dev/null
+++ b/2017/05/07/docker-multi-stage-builds/index.html
@@ -0,0 +1,214 @@
+Docker Multi Stage Builds | 040code
Building an application requires typically multiple steps. First compiling the application, next testing it and finally shipping the artifact in a Docker image for execution. All these steps can be combined in one Dockerfile, but typically you end up with a Docker image that contains all kinds of tools that are not necessary for running the application.
+
We could try to get rid of all unneeded files in the image after build, but a more useful approach is to separate Docker images for building and running the application. However, this approach requires two separate Dockerfiles that needs to be maintained.
+
Let’s have a look on a example for a simple Java micro service. The micro services require the JDK and Gradle to build. The JDK needs to be installed before the build and Gradle will be download by the build itself. We can build the micro service in the following way.
Now the artifact is available, we can create a Docker runtime image. The application only needs the Java Runtime (JRE) to run. We create a Dockerfile based JRE alpine image and add our application.
Now we have created a clean Docker images that only containes the needed libraries to run the application. But would it not be nice if we can do this more straightforward? For example, in one Docker build.
+
Moving on to a multi stage build
+
Docker 17.05.x brings a new feature; it is now possible to define a multi stage builds in one Dockerfile. The Dockerfile can contain multiple FROM lines and each stage starts with a new FROM line and a fresh context. You can copy artifacts from stage to stage.
Now it is time to have a second look on the same example. Instead of passing the build arguments and mounting our sources, we will build our software using a single Dockerfile.
Nothing new here, we are now able to build our software in a container and ship an image, but the image contains all kinds of build stuff that we do not want in a runtime image. So, the next step is to solve this problem by getting rid of all the unwanted build tools.
+
We update the Dockerfile and make a few modifications.
In the Dockerfile above we marked the first stage (1) as build to refer later, if not specifying a name you can refer to the stage as 0. Next (2), we add the second stage which specifies the base image that we need to run our application. Finally (3), we copy the results from the build stage to the image for executing.
+
docker build -t notes-service:02 .
+
And now we can run via the following docker run command.
+
docker run -it --rm -p 8080 notes-service:2
+
Inspection of the tree images build will show that version 0 and 2 are equal in size, version 1 that contains all the build files and the JDK is almost three times bigger.
+
Conclusion
+
Docker’s new multi layer build feature can be quite useful when build pipeline without parallel builds will suite your needs. In that case, you can reduce your build to a single image. For projects that requires a more complex multi stage build, including parallel builds, this new Docker feature will not provide an added value.
\ No newline at end of file
diff --git a/2017/05/20/graphql/index.html b/2017/05/20/graphql/index.html
new file mode 100644
index 00000000..d95712e9
--- /dev/null
+++ b/2017/05/20/graphql/index.html
@@ -0,0 +1,195 @@
+GraphQL | 040code
During the talk, I demonstrated GraphQL via graphcool, which is cloud service for creating a flexible GraphGL backend. I modelled a basic model of a Talk and Person with relation speakers and visitors on graphcool. Furthermore, I modelled two basic sample implementations of a GraphQL backend in Java and JavaScript that are available via GitHub. Those implementation are more limited than the online one, but both serve the following minimal queries.
+
Query for the name and the title of the talks given by all persons:
+
{
+ persons {
+ name
+ talks {
+ title
+ }
+ }
+}
+
And secondly, a query for all titles of talks with their speaker’s names.
+
{
+ talks {
+ title
+ speakers {
+ name
+ }
+ }
+}
Instead of yarn, you can use npm install && npm start. Once started, browse to http://localhost:8080/graphiql and play around with the GraphQL web interface. Implementing the JavaScript version was straightforward due to the many examples and good tutorials around.
+
Spring Boot Java sample
+
The Spring Boot Java sample is based on the Spring Boot Starter available for GraphQL. This starter uses a basic GraphQL java library as well the Java annotation library for GraphQL.
Alternatively, you can build and run via gradle. Once started, browse to http://localhost:8080/ to play around with the GraphQL web interface. The Java version implementation was not as easy as expected. The GraphQL Spring Boot starter dependencies requires the GrahpQL java annotation library, which seems to be broken after an update in the Java GraphQL base library. I am not sure if the annotation library is still supported. To get the GraphQL Spring Boot library working I have to add a mandatory mutation to the scheme. And downgrade the Java GraphQL, see gradle/depedencies.gradle to get the annotation library to work.
+
Many languages already have support, so feel free to try out your favourite language. Currently the support on JavaScript looks like by far the best.
+
Updates
+
Refactored Java implementation (June 20, 2017)
+
I have updated the Spring Boot implmentation, upgrade all libraries and removed the annotations for graphql resolving. The annoation are replaced java-graphql-tools. Which looks a better way to ingrate graphql in a Java service. You will see the implementation will have a lot of similarities with the javascript one.
+
Developers.nl meetup Rotterdam (June 28, 2017)
+
On June, the 28th I have presented the GraphQL talk at the developers.nl Meetup in Rotterdom. I have used a slightly minimal version for the slides, which can be found here. Comparing the to the presentaton on the nextbuild I have also walked through the code examples mentioned above.
\ No newline at end of file
diff --git a/2017/06/18/coding-a-vpc-in-terraform/index.html b/2017/06/18/coding-a-vpc-in-terraform/index.html
new file mode 100644
index 00000000..a028276c
--- /dev/null
+++ b/2017/06/18/coding-a-vpc-in-terraform/index.html
@@ -0,0 +1,319 @@
+Coding a VPC in Terraform | 040code
One of the common uses network setups in AWS is called Scenario 2: VPC with Public and Private Subnets. This is a that defines a Virtual Private Cloud (VPC), public subnets and private subnets. Setting up this infrastructure can be done via the AWS console or via cloud formation scripting. However, I prefer the tool Terraform in which you can manage your infrastructure as code with a declarative language that supports building, changing and versioning your cloud in a modular way.
+
In this article I will describe how you can create a VPC with a public and private subnet on AWS using terraform. I will describe the setup step by step and I will show how to encapsulate all logic to one re-usable module.
+
+
Why we need subnets. When you create a VPC, it spans all the Availability Zones in the region. After creating a VPC, you can add one or more subnets in each Availability Zone. When you create a subnet, you specify the CIDR block for the subnet, which is a subset of the VPC CIDR block. Each subnet must reside entirely within one Availability Zone and cannot span zones. Availability Zones are distinct locations that are engineered to be isolated from failures in other Availability Zones. By launching instances in separate Availability Zones, you can protect your applications from the failure of a single location.
+
+
Before you can start, you need an AWS account with sufficient rights (admin), and you should create a AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY to be able to access your account programmatically.
+
Setting up your system.
+
The easiest way to use terraform is via docker. In that case you do not need any locally isntalled tools. We simply create a file called .aws where we put the AWS key and secret to athentication API calls, as shown below:
By default, terraform expects the configuration in a file called main.tf. Create the file and add the terraform provide for Amazon:
+
# main.tf
+provider "aws" {
+ region = "eu-west-1"
+}
+
We are now ready to, step-by-step, add the terraform resources to create the VPC setup. First, have a look on the VPC setup as shown in the picture below.
+
+
+
+
+
+
+
+
+
+
+
We will start by creating the VPC itself: add the following snippet to your main.tf
Before we make the change effective we run a terraform plan to inspect the planned changes. You can install terraform locally or run the commands in a docker container. The docker command will export the AWS credentials to the container, mount the current directory to /data in the container, and set /data as working directory to ensure the container will execute the terraform plan command on the directory that contains the terraform files.
+
docker run -it --env-file ./.aws -v $(pwd):/data -w /data \
+ hashicorp/terraform:0.9.8 plan
+
After executing the plan command, you should see output similar to:
You can go VPC via the AWS console, where you should now see a VPC in the list named blog. Now we have the VPC, we create a public and private subnet by adding the following to the main.tf.
Run now terraform plan again, it will show that the resources will be added, one for each subnet. Apply the change using terraform apply and inspect your changes again via the VPC section in the AWS console.
+
Next, we connect the public subnet via an internet gateway. Create a routing table and an internet gateway. And a aws_route_table_association to create an association between a subnet and routing table. Update your main.tf with the snippet below, and run a plan and apply.
We are now able to deploy an application to the public subnet and make it accessible via a security group. However, it is not yet possible to route traffic to the private subnet or let instances on the private subnet connect to internet. To do so, we add a NAT gateway and connect the gateway via a route table to the private subnet. The NAT gateway requires an elastic IP, which we will create first. Add the next terraform snippet to your main.tf and run a plan and apply to inspect and make the changes effective.
Setting up the VPC with subnets is quite verbose, and imagine then when you must support more availability zones, the code will almost double per zone. So we refactor it to a generic module to support the multiple subnets that are available in a zone.
+
Creating a VPC part 2
+
I have rewritten the code showed above to a generic module. With this model, it is easy to create a VPC with all availability zones per zone, and private subnets can be enabled on demand.
+
+
Modules in terraform are self-contained packages of gerraform configurations that are managed as a group. Modules are used to create reusable components, improve organization, and to treat pieces of infrastructure as a black box.
+
+
I will not describe all code in the module again but only will pay attention to the significant changes I made. In the code above the zone, availability zone where hard coded. This will not work in a generic module. A common way to solve this in terraform is by creating a map where a zone is mapped to a list of availability zondes. By passing the zone to the moudle, the module can find out which availability zones there are. A default map is available in the module but can be ovewritten as follow:
The cdir block in the code show above was hard coded, but terraform contains a function to calculate the cdir block. In the module I use the terraform functioncidrsubnet() to calculate the cdir block.
+
Now only one problem needs to be solved in order to create a generic module: how can we create a AWS resource for each element in a list? Do we create subnet or route table association for each availability zone? Or how can we avoid creating a private subnet at all? The solution is to use the count variable in a module to iterate over the list of availability zones, which is available on my github. We can now create a VPC similar to the one above with just a few line of code. Add the following lines to your terraform script to create a VPC.
\ No newline at end of file
diff --git a/2017/07/01/visualising-bezier-curves/index.html b/2017/07/01/visualising-bezier-curves/index.html
new file mode 100644
index 00000000..0d3753df
--- /dev/null
+++ b/2017/07/01/visualising-bezier-curves/index.html
@@ -0,0 +1,344 @@
+Visualising Bézier Curves | 040code
For reasons I might explain in another post, I was reading about Bézier curves on wikipedia the other day. Especially the paragraph on “Constructing Bézier curves” made me understand intuitively how these curves are constructed. I wondered if clojure could help me easily visualise some of these curves so I fired up a REPL (Read-Eval-Print-Loop) to investigate.
and reduce that result with the *' multiply function
+
+
;; so:
+(pow23)
+;; will lead to
+(reduce *' [222])
+;; which results in
+8
+
I’m using the *' multiply function instead of the normal * multiply function, because according to the docs *' supports arbitrary precision. You can see for yourself by typing (doc *') in your repl, or study the code by typing (source *') in your repl.
+
Bézier
+
Bézier functions work with control points. The minimum number of control points is 2 and Bézier curves with 2 control points are straight lines (although you can also create straight lines with more than 2 control points).
+
For the moment I only need Bézier curves with 3 control points: a start point, an end point and one point controlling the curve of the line between start and end.
+
Quoting the Wikipedia page: “A quadratic Bézier curve is the path traced by the function B(t), given points P0, P1, and P2”:
So given 3 points (P0, P1 and P2) I should be able to describe the curve with this math function. Conceptually a pen is drawing the curve by starting on P0 on moment t = 0 and following the curve described by this formula and the position of the 3 points until it hits the end point at t = 1. The only thing this resulting B(t) function needs is the moment t and it will calculate the X or Y coördinate at that particular moment.
As you can see, you’ll have to translate the infix notation to clojure’s prefix notation. The advantage is there are no precedence rules to remember anymore. These are all just lists where the first element of the list is interpreted as a function to call and the brackets are used to put them into context. (If you can’t live with that: you can also use infix notation in clojure. For instance incanter can also be fed with infix notation. However, you’ll lose the power of the 𝝺)
+
This function basically states:
+
+
define a function named bezier-3
+
let it have 3 arguments: P0, P1 and P2
+
make it return an anonymous function
+
+
This anonymous function:
+
+
takes t as an argument
+
has P0, P1 and P2 already ‘injected’
+
applies the Bézier math function
+
+
Excellent. Let’s try it:
+
(def test-b3 (bezier-3110))
+
+(test-b30)
+=> 1
+
Here I’m defining a variable test-b3 which holds the anonymous function returned by the bezier-3 function call. The 3 points are either all x or all y coordinates of the points P0, P1 and P2.
+
With the (test-b3 0) function call I’m calling the anonymous function with a t value of 0. This nicely returns an answer representing the x coordinates at moment t if you provided the (P0, P1 and P2) x coordinates when calling the bezier-3 function or the y coordinate at moment t if you provided the (P0, P1 and P2) y coordinates.
+
Since this test didn’t blow the stack or throw a NullPointerException or anything, let’s map this function over a range of t’s [0 0.25 0.5 0.75 1]:
The let form is clojure’s way of defining local variables. So b3x, xs, b3y, etc. can be seen as local variables with their values specified in the functions directly after their declaration.
+
This function basically states:
+
+
define a function named view-bezier-plot
+
let it have 3 arguments which are destructured into their 2D x & y coordinates
+
let the fourth argument be the title of the plot
+
let it have some local variables:
+
+
b3x takes all x coordinates of the 3 points
+
b3y takes the y coordinates
+
xs are the all x values resulting from applying the b3x anonymous function with all the range values
+
ys are the all y values resulting from applying the b3y anonymous function with all the range values
+
incanter can work with columns similarly to spreadsheets. dataset is an incanter dataset where 2 columns are ‘brought together’, or in lisp terms conj[oined]. By default, these columns are called col-0, col-1, … , col-n
+
xy-plot contains an incanter chart where dataset provides the data, the x-axis - and y-axis values are found in columns col-0 and col-1 respectively and viewing the points is set to true
+
+
+
and make it return an xy-plot
+
+
I love it when a plan comes together
+
+(view-bezier-plot[00][10][11]"increasing ascending")
+
+(view-bezier-plot[00][0 -1][11]"swoosh")
+
+(view-bezier-plot[-24][0 -4][24]"y = x^2 ?")
+
+(view-bezier-plot[-10][00][10]"my pulse after a useless meeting")
+
The generated charts will be opened in external java windows. Sometimes they stay hidden behind other windows on your screen, so you might have to bring your java windows to the foreground in order to see them.
+
Conclusion
+
In a relatively short repl session I was able to get a better understanding of (3 point) Bézier curves. Although the blogpost is long, not much code or time was needed to create the curves:
Actually, the hand-rolled pow function isn’t really needed. I could have used clojure’s java interop: (Math/pow 2 3). In that case I would have been calling the pow method directly on java’s Math class. I weighted the option of hand-rolling the pow function, showing some functional code and explaining how functions are defined in clojure versus explaining java interop.
+
Explaining in text what the code does took me 5 times as long as writing the code itself. That’s a great thing about lisps in general and clojure in particular: no fluff, just stuff!
+
Thanks Niek for posting my guest blog. Please share your comments, suggestions and thoughts about this blog with me on twitter.com/mmz_. Thanks for reading and Happy Coding!
REPL Driven Development - a blog which is a bit outdated on the test part, because nowadays, with clojure spec we easily generate our tests and test data by specifying our data structures, functions, etc. in clojure itself. With proper specs, you’ll get even more than generative testing; you’ll get validation, better error reporting, destructuring and type- and range checking in one go - without losing the power & pleasure of a dynamic programming language!
\ No newline at end of file
diff --git a/2017/07/12/visualizing-bezier-curves-part-2/index.html b/2017/07/12/visualizing-bezier-curves-part-2/index.html
new file mode 100644
index 00000000..7c39c74a
--- /dev/null
+++ b/2017/07/12/visualizing-bezier-curves-part-2/index.html
@@ -0,0 +1,312 @@
+Visualising Bézier Curves Part II | 040code
In my previous post I described how to visualise simple Bézier curves. I chose Clojure as implementation language. Unfortunately I had to ask you to jump through a couple of hoops in order to code along: installing leiningen, installing and configuring the lein-try plugin and maybe even installing the JVM.
+
This post will be easier for you - at least in terms of setup. You won’t have to leave your browser and can experiment with Clojure(Script) right here on this page!
+
NO Setup
+
In this post we’re going to use the very useful klipse plugin. Klipse is a client-side code evaluator pluggable on any web page. It can evaluate Clojure, ClojureScript, ruby, javascript, python, scheme, es2017, jsx, brainfuck, c++, reagent and probably much more languages in the near future. It’s like jsfiddle on steroids, right here in the 040code blog.
+
Let’s start with a very simple Klipse demonstration. I’m going to map the inc(rement) function over the numbers 1, 2 and 3. The result will not surprise you:
+
(map inc [1 2 3])
+
What might surprise you though, is that you can change the code in the klipse evaluator and see the results immediately in your browser. So for instance change (map inc [1 2 3]) into (map inc [41 999 2]) and be amazed. Or change it into something completely different like (filter even? (range 10)).
+
Defining functions and calling them? No problem:
+
+
+(defn factorial [n]
+ (if (= 1 n)
+ n
+ (* n (factorial (- n 1)))))
+
+(factorial 4)
+
+
+
Now call the factorial function with some other natural number (whole number > 0) and see the results.
+
For those of you who don’t know Clojure(Script), this function basically says:
+
+
define a function named factorial
+
let n be its argument
+
if n equals 1, return n
+
otherwise return the multiplication of n with the factorial of (n - 1)
+
+
Lisp in the browser. Embedded in this blog. No setup required. Excellent!
+
But hang on, better things will follow soon.
+
Clojure is designed to be a hosted language
+
Why reinvent your own platform, your own runtime, your own garbage collector, your own ecosystem, when all you need is a decent language? Clojure is designed to be a hosted language. It runs on:
Clojurescript - a compiler for Clojure that emits javascript - will be the tool we use in this blog to visualise Bézier curves. We are going to manipulate a html canvas right from a klipse plugin.
+
Bézier in ClojureScript
+
In my previous post I tried to explain how (simple) Bézier curves ‘work’. You might want to go there or scan the Wikipedia page on Bézier curves if you have no clue what I’m talking about. Otherwise, let’s dive right in and try to play with Bézier curves directly on this page.
+
+
+;; Get a grip on the html canvas element I inserted on this page
+(def canvas (.getElementById js/document "canvas-2d"))
+(def ctx (.getContext canvas "2d"))
+
+
+;; Draws a blue circle with radius `r` on the canvas on point [x y]
+(defn draw-point [x y]
+ (let [r 5] ;; radius
+ (set! (.-fillStyle ctx) "blue")
+ (.beginPath ctx)
+ (.arc ctx x y r 0 (* 2 Math/PI))
+ (.fill ctx)))
+
+
+;; Draws a red Bézier curve and its blue control points on the canvas
+(defn draw-bezier-curve [[x1 y1] [x2 y2] [x3 y3]]
+ ;; draw curve
+ (set! (.-strokeStyle ctx) "red")
+ (set! (.-lineWidth ctx) 2)
+ (.beginPath ctx)
+ (.moveTo ctx x1 y1)
+ (.quadraticCurveTo ctx x2 y2 x3 y3)
+ (.stroke ctx)
+ (.closePath ctx)
+ ;; draw control points
+ (draw-point x1 y1)
+ (draw-point x2 y2)
+ (draw-point x3 y3))
+
+
+;; Center the *drawing* canvas within the *html* canvas and draw the curves
+;;
+;; Reason for centering: keep the coordinates simple AND
+;; see the complete shapes (not cropped at the edge of the html canvas)
+;; make the ratio 1 to see what I mean with 'cropped at the edge'
+;;
+(let [wc (.-width canvas) ;; width of *html* canvas
+ hc (.-height canvas) ;; height of *html* canvas
+ ratio 0.9 ;; ratio of *html* canvas to use as *drawing* canvas
+ t (/ (- 1 ratio) 2) ;; translation constant
+ w (* ratio wc) ;; width of *drawing* canvas
+ h (* ratio hc) ;; height of *drawing* canvas
+ x (* 1/2 w) ;; x val in the middle of 0 and w
+ y (* 1/2 h)] ;; y val in the middle of 0 and h
+
+ ;; clear *html* canvas
+ (.clearRect ctx 0 0 wc hc)
+
+ ;; Center the *drawing* canvas in the *html* canvas
+ (.save ctx)
+ (.translate ctx (* t wc) (* t hc))
+
+ ;; Draw a grey border around the *drawing* canvas
+ (set! (.-lineWidth ctx) 1)
+ (set! (.-strokeStyle ctx) "grey")
+ (.strokeRect ctx 0 0 w h)
+
+ ;; Draw main 'anchor' points
+ ;; This is what I meant with 'simple coordinates':
+ ;; (draw-point 0 0)
+ ;; (draw-point x y)
+ ;; (draw-point w h)
+
+ ;; Increasing ascending curve
+ (draw-bezier-curve [0 h] [w h] [w 0])
+
+ ;; Swoosh
+ ;; (draw-bezier-curve [0 y] [0 h] [w 0])
+
+ ;; normal curve
+ ;; (draw-bezier-curve [0 h] [x 0] [w h])
+
+ ;; my pulse after a useless meeting
+ ;; (draw-bezier-curve [0 y] [x y] [w y])
+
+ (.restore ctx))
+
+
+
+
+
Before reading further you might want to experiment a bit by (un)commenting code and seeing the results. (Un)comment the different Bézier curves and different control points in the Klipse evaluator and see what happens.
+
Explanation
+
Scanning the code quickly without going into detail, this is what happens:
+
+
Get a grip on the html canvas element
+
Define functions to:
+
+
Draw (control) points
+
Draw Bézier curves with 3 control points
+
+
+
Draw the actual curves and points
+
+
Since Clojure is a hosted language, it must be able to access its host platform and libraries. Clojure and ClojureScript have good interopdocumentation so I won’t go into detail here, but we’re basically using these forms in this blog:
+
+
js/document => the global document object
+
(.beginPath ctx) => ctx.beginPath()
+
Math/PI => 3.141592653589793
+
(.-lineWidth ctx) => ctx.lineWidth
+
+
The last form is used for instance in (set! (.-lineWidth ctx) 1) and translates to ctx.lineWidth = 1. The more general syntax is (set! var-symbol expr).
+
The second form (.beginPath ctx) can also have arguments. The general syntax is (.instanceMember instance args*) in that case.
Thanks to the klipse plugin and a bit of preparation from my side, you can now play around with Bézier Curves in ClojureScript directly in this blog. I do realise this post is probably not a compelling case for using ClojureScript:
+
+
Javascript “in the small” is not really the place where ClojureScript shines, especially when the largest part of that small program is javascript interop
+
ClojureScript fits large browser applications better, where you need sane state management, immutable datastructures, lazy sequences, and a fast, stable and robust language
+
+
I’m not interested in ‘religious’ discussions about technology A versus technology B. Clojure and ClojureScript are THE sane way forward for me, in my context. I hope to share the fun I’m experiencing with it. Goethe said it best:
+
+
It is always better to say right out what you think without trying to prove anything much: for all our proofs are only variations of our opinions, and the contrary-minded listen neither to one nor the other.
+
+
But with all this talk about language, we’re almost forgetting what it’s all about: building useful stuff and having a great time doing it.
+
I hope you enjoyed experimenting with Bézier curves. The Klipse Blog has several great examples of using Klipse for ‘interactive programming’, for instance to write data driven documents.
+
Thanks Niek for sharing the 040code repo with me. Please share your comments, suggestions and thoughts about this blog post on twitter.com/mmz_. Thanks for reading and Happy Coding!
This week we (Jeroen and Niek) enjoyining Full Stack Fest 2017 Barcelona,
+a week-long full stack development conference with a focus on solving current problems with new & inspiring perspectives.
+
In this quite short blog we just summarizing briefly the talks and our opinion about the talks. We have rated all talks, the rating is subjective ass well.
+
The first two days are more backend focussed, the last two days have a frontend focus, read more here.
+
Day one
+
Keynotes - Science - Backup the internet
+
Rating: ⭐ ⭐ ⭐ ⭐
+
Karissa McKelvey (@okdistribute) discussed the topic to share fairly large data sets in the science community in a secure and distributed way. The dat project enables file sharing via peer to peer for example with the beaker browser.
+
Ehtereum, aeternity and the distributed wonders
+
Rating: ⭐ ⭐ ⭐ ⭐
+
Luca Marchesini (@kuzzleio) did a great job by explanining about blockchain and the differences between ethereum and bitcoin. Blockchain; the system where playing by the rules is more profitable than cheating.
+
Terraform in your delivery pipeline
+
Rating: ⭐ ⭐
+
We love the Terraform framework of HashiCorp but the talk not as good as we hoped, nothing new. The speaker Anton (@antonbabenko) is an expert on the topic and also maintains some great Terraform modules.
+
Migrating an existing service to serverless
+
Rating: ⭐ ⭐
+
For us a standard talk by Marcia (@mavi888uy) about migrating an existing monolitic applicaton to smaller components. We loved all the (angry) birds flying around.
+
Defending the human network
+
Rating: ⭐ ⭐ ⭐ ⭐
+
Good story about cyber security by Dr. Jessica Barker (@drjessicabarker). About how the mood of a user is one of the weakest link. How you can trick people into clicking a bad links. Socially engineering and use stress full situation let a user execute an (maybe) unwanted action.
+
Key take aways, learn to think as hacker and focus on the why not on how to enforce / restrict users to do something.
+
Rise and the fall of the full stack database
+
Rating: ⭐ ⭐ ⭐ ⭐
+
Comparison with the army knife. Don’t be afraid of new technologies.
+One tool for everything is not going to work.
+Example with analytics service, kafka stream, bus. Frank Lyaruu (@lyaruu) did a great job!
+
How booking.com serves deep learning model predictions
+
Rating: ⭐ ⭐ ⭐
+
Good explaination by Sahil (@sahildua2305) about how to train a model and how to deploy them.
+
What to do when complex systems fail
+
Rating:
+
Not relevant to our needs.
+
Day Two:
+
The internet of things: An army of devices ready for DDoS
+
Rating: ⭐ ⭐ ⭐ ⭐
+
Good talk by John Graham-Cummin (@jgrahamc) about how IoT device can be used to do a DDoS attack. About port 1900 showing all kinds of information which is sent to victims address. How to block DDoS attacks by putting the ip-address in a grid to search for patterns. Talked about Trump attacks and anonymous being stupid by setting the evil flag (We’re anonymous).
+
How to properly blame things for causing latency: An introduction to distributed tracing and Zipkin
+
Rating: ⭐ ⭐
+
Difference between logs, metric and trace. Good (known) story by Adrian Cole (@adrianfcole).
The story about testing in real live situations is not new, good te remember the importance to be prepared for failure. For not much new things in this talk since we here the chaos monkeys stories new for a couple of years. But we will plan our Chaos GameDay soon. Thanks to Aish (@aishrajdahal)
+
Flying Spaghetti monster: Verifying protocols with types and finite-state machines
+
Rating: ⭐ ⭐ ⭐ ⭐
+
Complexity of systems cause a lot of problems. Idris has a nice feature that Types are also values, so state changes can easialy be checked in compile time. talk.applaud() should only run after talk.give() and Idris has a way to check this easily during compile time.
+
Inspiration for doing more with new languages. Chris Ford (@ctford), thanks!
+
Smart grid: How iot fights climate change
+
Rating: ⭐
+
Kelsey Breseman (@ifoundtheme) explains us how complex the eletrical grids works which makes it hard as well to safe for example solar eneregy. It was great to hear the complecity of this relevant topic but unfortunately there was not much relation betwoon the world of of software.
+
Beyond patterns: Technological systems and the nature of order
+
Rating: ⭐ ⭐ ⭐ ⭐
+
Good to see the that our hipster shit is the same as the problems the manufacturing has already solved. Learn from that! Talk by Jerome Scheuring
+
Lighting Talks
+
Rating: ⭐ ⭐ ⭐
+
There where quite a lot fo lighting talks. In our opinion: hight quality but too many.
+
+
Docker talk about docker build was fun. Docker build takes long, and from now on I can’t ignore it anymore. THANKS!!! aka we should fix the layer thing.
+
Fullstack is a zoo. Complexity of our work environment.
+
http://exercism.io/: Nice code kata kind of thing. (not a talk, but worth to mention)
+
+
Gone in 60 milliseconds: Offensive security in the serverless age
+
Rating: ⭐ ⭐ ⭐ ⭐ ⭐
+
Awesome DDoS attack of slides by Rich Jones (@GUNdotIO). A lot of cool stuff. A very usefull and insight talk about how unsecure you serverless application can be. Serverless python with zappa and slides
\ No newline at end of file
diff --git a/2017/09/08/full-stack-fest-2017-frontend/index.html b/2017/09/08/full-stack-fest-2017-frontend/index.html
new file mode 100644
index 00000000..2425319c
--- /dev/null
+++ b/2017/09/08/full-stack-fest-2017-frontend/index.html
@@ -0,0 +1,215 @@
+Full Stack Fest 2017 | 040code
This week we (Jeroen and Niek) enjoyining Full Stack Fest 2017 Barcelona,
+a week-long full stack development conference with a focus on solving current problems with new & inspiring perspectives.
+
In this quite short blog we just summarizing briefly the talks and our opinion about the talks. We have rated all talks, the rating is subjective ass well.
+
The last days are more frontend focussed read here more about the backend days.
+
Day four
+
Keynote
+
Rating: ⭐ ⭐ ⭐
+
Jenn Schiffer (@jennschiffer) showed how to make coding actually friendly with glitch. And remember once you got bored you will not trying new cool stuff which makes you even unhappier and so on… And some cool pixel art
+
Once you go PRPL…
+
Rating: ⭐ ⭐ ⭐ ⭐
+
Service Workers by Houssein Djireden (@hdjirdeh) are stable and most serious browsers support them. SWs are a big help in improving the experience for the user. Workbox-cli is a great for configuring your app to cache usefull stuff only. Don’t cache too much. FoamTree and webpack-bundle-analytics show a good overview what to cache. Split up your js, with http/2, we do not need one big js file. React has good SW support.
+
Magic tricks with css houdini
+
Rating: ⭐ ⭐ ⭐ ⭐
+
Absolutely awesome to see the future already working in this talk by Sam Richard (@Snugug). A lot of visual stuff is currently blocking the main JS thread. With Houdini CSS a lot of css stuff can be handle on Worklets. ‘Much like Service Workers are a low-level JavaScript API for the browser’s cache Houdini introduces low-level JavaScript APIs for the browser’s render engine’. Currently only available in Chrome Canary and the definition of unstable, but ooooh so beautiful, slides in canary
+
Master the art of the ast (and take control of your js)
+
Rating: ⭐ ⭐ ⭐
+
Talk by Sears israel (@cowchimp) to understand most JS libraries you only need to know two things. AST and the visitor pattern. With this knowledge you can easily add ESLint, create Babel plugins or add jscodeshift refactor modules. A great tool to help with AST is https://astexplorer.net
+
SVG can do that?!
+
Rating: ⭐ ⭐ ⭐ ⭐ ⭐
+
Awesome what you can do with SVG by Sarah Drasner (@sarah_edo). This girl really knows her stuff. SVG is supported by all browsers including Opera Mini! Good integration with VueJS, react and other modern frameworks is possible. Lots of examples of really beautiful animations all working for all types of devices. Size does matter for images and animations. Normally it will result in ugly things, but with SVG this is absolutely not the case, slides.
+
Using react native + clojurescript to launch ios/android apps to 1m users
+
Rating: ⭐ ⭐
+
A long intro to react native, which is most likely not necessary on this conference. A nice twist with clojureScript. The code looks very clean. It is possible to create apps for both iOS and Android with a 99% simular codebase. The crashlogs of apps created with clojurescript and reactnative are unfortunately almost not readible. This set of tools is also great to use in combination with codepush. Now we can update the app on the fly without bothering appstores. Talk by Emin Hasanov (@hasanove)
+
Blasting react into space: building fluid, beautiful interfaces with react and webgl (and other exotic explorations)
+
Rating: ⭐ ⭐ ⭐ ⭐
+
A good talk by Chad Ostrowski (@chadoh), on how all the new technologies can help us in the future when live is possible on Mars. Using: HTTP/2, ServiceWorkers, LocalStorage, Bittorent + git, IPFS, blockchain, bitcoin, ethereum. “Tomorrow’s problems === Today’s problems but bigger.” Last year a simular talk was told on FSF, but it’s good to see that the technologies mentioned then are now in a much further state. Great talk and good choice from the organiser of this beautiful conference.
+
+
+
+
Day five
+
Bringing back the 1990s: The revenge of javascript stylesheets
+
Rating: ⭐ ⭐ ⭐
+
Steve Kinney (@stevekinney) did a lot of work in getting the Javascript Stylesheets, a technology only working in 1996 in Netscape 4) to work in 2017. In the beginning of the internet, people were debating about the split of concern regarding content and layout. One of the proposals was Javascript Stylesheets. Netscape 4 was the only browser capable of this feature. Steve showed us how we can do some really neat stuff in 1996 technology. A lot of this, we now take for granted, but we have to keep in mind that this was 1996. A lot of simularities with SASS, LESS, but also REACT like structures with the ECMA E4X for XML proposal. Conclusion: We still can learn from the past. Although some good initiatives take 20 years to become meanstream.
+
Rendering performance from the ground up
+
Rating: ⭐ ⭐ ⭐
+
Good presentation about how we can improve the performance a lot by using the GPU instead of CPU. A good explaination what the differences are between a CPU and GPU. Try to use translate and shaders because they don’t require repaintings. Thanks Martin Splitt (@g33konaut)!, slides
+
Web based virtual reality
+
Rating: ⭐ ⭐ ⭐
+
Shagufta Gurmukhdas (@shaguftamethwan) give a good talk ok the Web based virtual reality. Starting WebVR with WebGL is very hard. She showed us A-Frame. A-Frame was already mentioned three times in FSF, so we were already very curious about it. You can easily start your own VR world with this framework. It uses GPU under the hood and you can use one of your favourite webframeworks (vueJS, React, and so on) to change properties of the 3D models. Very impressive. With this tool, even the most suburn backenders, (we) can start with VR.
+
Exploring the world of decentralized networks with webRTC
+
Rating: ⭐ ⭐
+
Nikita Baksalyar (@nbaksalyar) started his talk with some technical issues, but he recovered quickly from that. Nikita tought us about how the internet can truly be implemented the way it was supposed to be implemented. Decentralised! Now-a-days we’re using a lot of things which are ment to be decentralised in a centralised way. Git and the internet are a nice examples of that. Serving the web, securely and encrypted, with webRTC through all client devices will make it imposible to censor the internet by governments. During the Q&A a question come up on what to do with unwanted content. The response of Nikita was amazing. We should educate people that they don’t want to see unwanted content instead of putting the responsibility of this on technology, resources
+
Developing for the next billion
+
Rating: ⭐ ⭐ ⭐
+
Look to your target group. Keep in mind that your apps might not be used by the people you’ve targetted on. Natalie Pispistunovich (@nataliepis) showed some examples from Kenya. What apps are used and why. The delivery of apps can be very different from ours. The way payments are done also.
+Nice reference to Fun with Flags
+
Progressive web apps and the internet of things
+
Rating: ⭐ ⭐
+
A story of with amongst other libraries, the johnny-five library. Today it’s easy to communicate with all kind of device with nodeJS. Well done, Luis Montes (@monteslu)
+
Lightning talks
+
+
PAKO a zlib for uploads in js. For uploading large files. :) NICE. @bombasarkadian
+
DevTools in Firefox: Alex Lakatos @lakatos88; BASIC
+
Internet for Everyone. Tom Enden en Liran Kurtz. Accessible. GOOD
+
Marek - ProRouter.js - Router.maniak.pro - No idea what this is all about.
+
Kirill Pimenov - Rust! Go write to Rust and go back to your own language and you will be beter. Awesome!
+
Raj - iOS Event Communication without internet, API and no loss of data. Way to fast. OMG… Peer to Peer connectivity with iPhone through Bonjour protocol
+
Paulo - Functional Programming: Elixir and Phoenix. I love Phoenix! See also git
+
FSF designer. Uses emoiji as classes. Awesome way of keeping the names short and compact. Really high quality frontend work. See Full Stack Fest web page and Inspect the code!
+
+
Making a lot of things
+
Rating: ⭐ ⭐ ⭐
+
Ben Foxall (@benjaminbenben) gave us two talks for the price of one!
+https://cojs.co is collaborate platform were people can co-create software. With blocks you can split the code. Adding a ! behind the url will fork the code. Other users can continue on the code in their own context, but with shared code. Awesome. The second part was about Microhacks. He explained how you should constrain your experiments to make them work. Plan your goal, not your work. Start it and finish it.
+
Awesome!!!
+
Many thanks to Codegram the company that organized once again the best conference!
\ No newline at end of file
diff --git a/2017/09/19/immutable-infrastructure/index.html b/2017/09/19/immutable-infrastructure/index.html
new file mode 100644
index 00000000..e936d8bd
--- /dev/null
+++ b/2017/09/19/immutable-infrastructure/index.html
@@ -0,0 +1,222 @@
+Immutable Infrastructure | 040code
The Docker slogan ‘Build, Ship, and Run’ advertises easy set up of immutable software builds, but it is not always that easy. Setting up immutable builds with Docker is pretty straight forward and shipping is just a matter of pushing the image to a repository. The next step is building the cloud infrastructure to run the containers. In the talk at the DevOps Days in Riga I have shown how to create an immutable infrastructure on AWS with Terraform. The example belows shows how you can run your micro services in docker containers on AWS.
+
In this talk I briefly touch upon building immutable software. But the main focus of the talk will be on creating an immutable infrastructure. In this talk I will show you how to create an immutable infrastructure on AWS with Terraform. I will use a real world example to explain and show live how easy you can get micro services live on AWS and continuously apply changes to the same cloud environment..
+
Slides
+
Below the slides that I used for the talk, the slides are available as well on GitLab. You can easy navigate through the slides with the spacebar.
+
+
+
+
To get started with Terraform the best starting points are:
During the talk, I demonstrated a hello world example, see the link above. This examples shows some basics of terraform by creating a ec2 instance and security group to server a simple web application. The example only works in the AWS region eu-west-1 since the AMI used is only available in this region.
+
ECS demo
+
The second example I have shown, is creating an immutable infrastructure to server docker containers. The picture below descibes briefly the enviroment that will be cretaed.
The example also creates log groups in cloudwatch to capture the logging of the ecs agent and the running containers (services).
+
Setup
+
Before you can start, you need an AWS account with sufficient rights (admin), and you should create a AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY to be able to access your account programmatically.
+
Clone the repository containing the ECS sample terraform code. To create the infrastructure step-by-step you can check out the tags vpc and ecs.
In the next steps we will create the network layers.
+
First, we initialize and plan our changes.
+
terraform init
+terraform plan
+
Terraform should print on the console that 19 needs to be added and 0 to change or destroyed. Next we apply the change be executing terraform apply.
+
Add ECS cluster
+
Now we have the network layer created, we will add the ECS cluster. By default the bastion host is disabled, the bastion can be enabled by updating the variable enable_bastion in the terraform.tfvars file. Time to plan and apply the new resources for ECS.
+
git checkout ecs
+terraform plan
+terraform apply
+
Once terraform is ready it will print the url of two applications on the console. One application, is a simple micro service that just prints the AWS availabilty zone where it is running, the other is a graphql micro service. It will take a few minutes before the services are available.
+
Adding your own service.
+
Time to have some more fun. Edit the main.tf and start making changes. Remove all the services, add your own service. By default for each service an application load balancer (ALB) is created. Add for each services that you want to deploy a configuration as below and update the variables as required.
+
module "your-service" {
+ source = "ecs-service"
+
+ service_name = "<service-name>"
+ image_url = "<docker-image>"
+ container_port = <port in the container where the service is listening>
+ desired_count = <number of instances>
+
+ aws_region = "${var.aws_region}"
+ environment = "${var.environment}"
+
+ vpc_id = "${module.vpc.vpc_id}"
+ subnets = "${module.vpc.public_subnets}"
+
+ cluster_id = "${module.ecs-cluster.cluster_id}"
+ ecs_service_role_name = "${module.ecs-cluster.ecs_service_role_name}"
+}
+
Now plan and apply your changes, Terraform will inform you that a few resources will be destroyed (the removed services) and a few will be added.
+
Clean up
+
Once done you can easily clean-up all created resources in AWS, just run terraform destroy.
\ No newline at end of file
diff --git a/2017/10/01/microhack-zappa-io/index.html b/2017/10/01/microhack-zappa-io/index.html
new file mode 100644
index 00000000..99d41cfe
--- /dev/null
+++ b/2017/10/01/microhack-zappa-io/index.html
@@ -0,0 +1,236 @@
+MicroHack Zappa.io | 040code
MicroHack Zappa.io
A quick test for using the Zappa.io serverless framework to run a python service in the cloud
I’ve visited one of the best conferences last september, Full Stack Fest in Barcelona. Since then, two talks are constantly in the back of my mind.
+
The first one is the talk by Ben Foxall (@benjaminbenben). He talked about Microhacks and how to become better in your experiments by constraining them. Write small reports / blogs of your Microhacks to so you have a nice overview of some of the experiments you’ve done. Guess what this blog is about..
+
The second talk I’m frequently think of was the talk by Rich Jones (@GUNdotIO) titled “Gone in 60 milliseconds: Offensive security in the serverless age”. A DDoS attack of slides, 452 in total. The talk main focus was about security and how you can hack the AWS infrastructure if things are not setup correctly. He knows his ways in AWS. That’s comes very handy if you want to build a cli which helps other people to run serverless python apps in the cloud. And this is exactly what he’s been doing with https://www.zappa.io.
+
What’s the experiment
+
I wanted to test the cli zappa.io with a simple python Flask app. The specs are simple: A service which reverses an input string running in the cloud with zappa.io. Serverless services are very interesting, because it’s based on lambdas and they typically run for a few milliseconds. So the costs are very low, utilization is very high.
+I’ve asked some colleagues to join me on a friday afternoon experiment and Jos Beijk wanted to help.
+
Let’s do it!
+
We need Python, a package manager (pip) and a tool to create isolated Python environments (Virtualenv).
+Store your AWS secrets in your .aws directory.
+
If you do not have an AWS account, don’t worry. Just signup and the first 1.000.000 requests per month are for free. https://aws.amazon.com/lambda/pricing
When you want to apply some changes, you can deploy it by running:
+
zappa update dev
+
If you add other libraries, make sure you add it in requirements.txt and install it locally in your virtual environment. All the packages in this folder will be shipped.
+
Debugging
+
You can do various things with the zappa cli.
+
You can view logging for example:
+
zappa tail dev
+
See help for more info:
+
zappa --help
+
Feel free to check the created services in AWS console. You will see API gateway, CloudWatch and Lambdas.
+
Cleanup
+
Remove the infrastructure by simply undeploy the environment:
+
zappa undeploy dev
+
Conclusion
+
Ofcourse this example has no real value, but I wanted to see how it works.
+I think zappa.io is very good in its default values, but is extremely flexible.
+You don’t like the JSON format and prefer YAML or TOML? no problem. Want to run it in a VPC, no problem. Don’t like the Keeping The Server Warm settings, you can change it.
+Look at https://github.com/Miserlou/Zappa for the complete list of options. There’s good community support with a slack channel, so I would encourage you to look at this project. Especially because you have a lot of good defaults in this setup and this guy knows a few things about AWS.
\ No newline at end of file
diff --git a/2017/10/19/graphql-geecon/index.html b/2017/10/19/graphql-geecon/index.html
new file mode 100644
index 00000000..27b8523f
--- /dev/null
+++ b/2017/10/19/graphql-geecon/index.html
@@ -0,0 +1,190 @@
+GraphQL | 040code
During the talk, I demonstrated GraphQL via graphcool, which is cloud service for creating a flexible GraphGL backend. I modelled a basic model of a Talk and Person with relation speakers and visitors on graphcool. Furthermore, I modelled two basic sample implementations of a GraphQL backend in Java and JavaScript that are available via GitHub. Those implementation are more limited than the online one, but both serve the following minimal queries.
+
Query for the name and the title of the talks given by all persons:
+
{
+ persons {
+ name
+ talks {
+ title
+ }
+ }
+}
+
And secondly, a query for all titles of talks with their speaker’s names.
+
{
+ talks {
+ title
+ speakers {
+ name
+ }
+ }
+}
Instead of yarn, you can use npm install && npm start. Once started, browse to http://localhost:8080/graphiql and play around with the GraphQL web interface. Implementing the JavaScript version was straightforward due to the many examples and good tutorials around.
+
Spring Boot Java sample
+
The Spring Boot Java sample is based on the Spring Boot Starter available for GraphQL and GraphQL java tools lib with schema first support.
Alternatively, you can build and run via gradle. Once started, browse to http://localhost:8080/ to play around with the GraphQL web interface.
+
Many languages already have support, so feel free to try out your favourite language. Currently the support on JavaScript looks like by far the best.
\ No newline at end of file
diff --git a/2017/12/09/runners-on-the-spot/index.html b/2017/12/09/runners-on-the-spot/index.html
new file mode 100644
index 00000000..51a4f7f1
--- /dev/null
+++ b/2017/12/09/runners-on-the-spot/index.html
@@ -0,0 +1,388 @@
+Runners on the Spot | 040code
GitLab CI is a first class citizen in GitLab to enable continuous integration and delivery to your project. Builds are orchestrated via GitLab Runners, which are agents registered to your GitLab account. An agent runs builds using a local shell or a container. Runnings these builds requires a well-defined infrastructure, both with respect to the type of server and the server capacity. Would it not be great if you can scale the infrastructure based on your needs? Indeed, GitLabs runner supports out of the box auto scaling using docker machine, which is discussed in the GitLab blog article: Autoscale GitLab CI runners and save 90% on EC2 costs.
+
The GitLab blog article nicely explains how to create your infrastructure in a manual way. However, we rather automate all things. Thus, we will automate the set up described in the blog article. In this post I will discuss how to create the infrastructure needed to run the build on AWS spot instances with Hashicorp Terraform. The GitLab Runner module discussed in the post is available in the Terraform Registery.
Before we start, we need to briefly discuss the GitLab runners. To execute the builds, GitLab uses an agent to orchestrate the build with a docker machine. A docker machine creates instances with docker engine to run docker containers. The first step for setting up a runner is to register a new runner. Because GitLab currently does not provide a fully automated way for this, we will do this manually.
+
Open you GitLab project and lookup the token to register a runner. Be aware that there are project tokens and a GitLab global token. Next, we use a docker container to register a runner. The command will ask a few details.
+
docker run -it --rm gitlab/gitlab-runner register
+
+
+
Provide the requested details and consult the GitLab manual for more details. Once done, you should see a new runner registered at your project or globally. Open the runner settings in edit mode and record the token. This token is needed later for connecting the agent.
+
Creating infrastructure for the runners
+
Now the runner is configured in GitLab, we can start creating the infrastructure on AWS. For setting up the network layers, we use Amazon networking scenario 2 to build a VPC with a public and private subnets. For more details see this post about creating a VPC in terraform. You can also simply use the official terraform module.
Next, we create a t2.micro instance using an autoscaling group in the private network. On this instance we install and configure the GitLab runner. Configuration of GitLab runners is done via the config.toml file. Below the parameterized version of this configuration file. In the Terraform module you will find that the configuration file is loaded via a data template.
All variables can be configured and most of them have default values. Only the name of the runner, the token and the GitLab URL need to be configured. The configuration contains a shared cache in S3 which will expire at the end of the next day (after creation) by default. Files will be deleted via S3 lifecycle management once expired, but the number of expiration days can be changed to your needs. You can find all the available variables in the variables.tf file of the module. Next we add the module to our Terraform file and define a minimal set of variables.
+
module "gitlab-runner" {
+ source = "npalm/gitlab-runner/aws"
+
+ aws_region = "<region-to-use>"
+ environment = "ci-runners"
+ ssh_public_key = "<contains-public-key"
+
+ vpc_id = "${module.vpc.vpc_id}"
+ subnet_id_gitlab_runner = "${element(module.vpc.private_subnets, 0)}"
+ subnet_id_runners = "${element(module.vpc.private_subnets, 0)}"
+
+ # Values below are created during the registration process of the runner.
+ runner_name = "<name-of-the-runner"
+ runner_gitlab_url = "<gitlab-url>"
+ runner_token = "<token-of-the-runner"
+}
+
The complete example is in GitHub. Now it is time to execute the scripts to create the actual infrastructure. Be aware that you need to configure AWS keys and have Terraform installed. The steps below should guide you through the setup.
The example directory contains the example as described above, so the properties for your newly registered runner in GitLab need to be configured. Please register a runner in GitLab (see docker command above) and update the terraform.tfvars file. That is all, now execute the Terraform code.
+
# genere SSH key pair
+./init.sh
+
+# initialize terraform
+terraform init
+
+# apply, or plan first
+terraform apply
+
+
+
After a few minutes the runner should be running and you should see it in your AWS console.
+
+
+
+
+
+
+
+
+
+
+
+The runner should be active as well in GitLab. Check the runner pages which should now indicate the latest moment of contact.
+
+
+
+
+
+
+
+
+
+
+
+
The modules also enable CloudWatch logging, all systemd logging is streamed. Go to CloudWatch to inspect the logging and you will see that the installation process was logged as well.
Finally we can verify that the runner is working properly by executing a build. The .gitlab-ci.yml below is an example to verify the runner is working properly. The build contains two stages. In the first stage an ascii art image is generated, stored in a file which is cached in S3. In the second stage the file is retrieved from the build cache.
Add the file above to the GitLab repository that has the created runner attached. Once you commit the file a build should triggered. The logging of the verification step should contain the ascii art image.
Be aware that prices of a spot instance can change over the time and instances can be terminated without a warning. Last black Friday prices of spot instances went up to over 2 dollar occasionally.
Last December at the AWS re:invent, AWS announced the new container service platform Fargate. Fargate is integrated to ECS. The key difference is that Fargate does not require you to have EC2 instances running to host your containers, which means we have serverless containers. A drawback is that Fargate is not globally available yet, today Fargate is only available in us-east-1, see also the list of supported regions. Later in December Fargate also become available in Terraform so time to see how it works.
+
In this post I will show how to deploy containers to Fargate using Terraform. As example docker image I will use the blog itself. The complete example in available on GitHub. First I show how you can deploy a container using Terraform to Fargate. In case you would like to experiment with AWS console instead of Terraform you should be able to execute the same steps direct in AWS. Only the VPC part could be a bit tricky, but even that part can be skipped and replaced with the default VPC. In that case you don’t have private subnets available.
+
Finally I will discuss how a deployment with Fargate in ECS compares to a deployment with EC2 in ECS. And show what steps have to be taken to move the deployment from Fargate to EC2 instances.
+
Prerequisites
+
Before you start you need to have programmatically access to an AWS account and Terraform (0.11+) installed. The tool tfenv let you manage multiple terraform version on your system.
+
Deploy serverless containers on Fargate
+
Before we can create our containers, we have to create a few infrastructural components. For this example we create an own VPC including public and private subnets. An ECS cluster for our containers, and a CloudWatch log group for centralized logging. The diagram below shows an abstract view of the deployment we are going to create, this view contains two in instead of the 3 availability zones we use.
We start with defining the VPC, we choose us-east-1 since Fargate is only available in this region. In the blog post Coding a VPC in Terraform you find more details about how this VPC module is structured.
+
provider "aws" {
+ region = "us-east-1"
+ version = "1.7.1"
+}
+
+provider "template" {
+ version = "1.0"
+}
+
+module "vpc" {
+ source = "npalm/vpc/aws" // https://registry.terraform.io/modules/npalm/vpc/aws
+ version = "1.1.0"
+
+ environment = "blog"
+ aws_region = "us-east-1"
+
+ create_private_hosted_zone = "false"
+
+ // us-east-1 is the only region that supports Fargate
+ availability_zones = {
+ us-east-1 = ["us-east-1a", "us-east-1b", "us-east-1c"]
+ }
+}
+
+
Next we define the ECS cluster, and CloudWatch log group. Since we will deploy to Fargate we don’t have to attache EC2 instances to our cluster.
+
resource "aws_ecs_cluster" "cluster" {
+ name = "blog-ecs-cluster"
+}
+
+resource "aws_cloudwatch_log_group" "log_group" {
+ name = "blog"
+}
+
Now the first part is defined we execute a terraform apply and inpect the results.
+
+
+
The next step is to deploy the docker image with the blog. In ECS you deploy a docker container with a task. And your task will be managed by a service. First we create the task definition which contains the container deployment definition as well. Fargate is using awsvpc as networking mode, in this mode each task definiton gets its own private ip address. This network mode requires a role for the task execution. In case you create your definition through the Amazon console a service linked role will be created for you. We will create this role also via code.
We have defined the execution rol for the task, next we define the task definion. This task definition consist of two parts. First we define a container definition via a template_file. In this container definition you see container port 80 is mapped to host port 80, the task will get its own private IP. And after that we define the task definition self. The following settings are required for a task that will run in Fargate: requires_compatibilities, network_mode, cpu and memory.
We have still nothing running but you can already see the different parts in de AWS console. We have now a VPC, CloudWatch log group, ECS cluster and task definition available. The next logical step in de AWS console would be to create the service, and find out at the latest step that you need to create a load balancer first. So in code we will define the load balancer first. The load balancer will route traffic via HTTP to the container.
We connect a target group to the load balancer. The same target group will be used later in the service, the service can register itself to the target group with the actual IP address once up and running. For the target group we have to specify the target type ip and not instance since containers running in Fargate will get their own IP. Actually this not Fargate but awsvpc behavior.
Again we verify our new code by executing a terraform apply.
+
+
+
We are almost there, next we define the service. A task running in in network mode awsvpc, requires a service that defines a network configuration. In the networking configuration we define a security group to control access to our containers, and we define the subnets in which the containers are hosted.
The last resource to create is the service. In the service we connect the task, load balancer and security group together. By setting the launch type to FARGATE we tell Amazon to deploy the container to Fargate.
Great we have now our container running in ECS with Fargate but what if we would like to move the container to a dedicated EC2 instances in ECS. Or what if we need that are not available in in Fargate, such as a volume mount? How difficult would it be to move our containers to an ECS cluster with dedicated EC2 instances? Time to do an experiment to see how difficult is is.
+
First we refactor the above code to some generic ecs service modules to be able to define our services with just a few lines of code. This module replaces all code of defining the load balancer, service and task. It still requires a VPC, ECS cluster, CloudWatch logging group, awsvpc security group and execution role. In the code you see a similar deployment of the blog but now with a generic ECS service module.
We still have our container not running in ECS on EC2 instances. For that we first have to create EC2 instances that will be attached to the same ECS cluster. For more details about create the EC2 instances see the ecs instance repo on GitHub.
Now we have our EC2 instances available we only have to copy the the blog module above and change the launch type to EC2, and our container will be deployed to a dedicated EC2 instance.
That is all, all sample code is available at GitHub see subdir fargate-ec2. Time to run a terraform apply to see if it works.
+
+
+
In the output you will find the two endpoint links to the blogs. After a few minutes both links will be active. You can also see the service running on the Amazon console, simply navigate to the ECS console and select the cluster. You should see now one service on Fargate and the second one on EC2.
\ No newline at end of file
diff --git a/2018/02/14/service-discovery-on-aws/index.html b/2018/02/14/service-discovery-on-aws/index.html
new file mode 100644
index 00000000..2a449f44
--- /dev/null
+++ b/2018/02/14/service-discovery-on-aws/index.html
@@ -0,0 +1,218 @@
+Service Discovery on AWS | 040code
Last year I wrote a post how to implement service discovery for Spring Boot applications running in Docker containers on AWS ECS. At that time the Amazon ECS agent does not have support to discover the docker exposed ports inside the container. In November 2017 Amazon released a feature in the ECS agent 1.15 to retrieve container meta data in the container. This feature makes the discovery agent obsolete.
+
The problem
+
Looking back to the last years problem. A Spring service running in a container needs its external ip address and port to register to Eureka for service discovery. A service running in a container does not know the external exposed port and docker does not support a way to obtain the port mapping. We could inject the ip address but for the port that would not work. Since the port is unknown at start time. Of course, this in only valid when using random ports. But fixing the port can eventually cause port conflicts. Besides that, you are not able to scale the container on the same host. Therefore, it makes sense to use automatic port assignment.
+
At that time I have solved this problem by adding an agent that can be called via a REST interface to lookup the exposed port based on the container id and internal port. The agent returns the external port, and the service can use this information to register itself to Eureka. This approach worked well but since the ECS agent is now shipped with a feature to retrieve the meta data we do not need longer an extra agent anymore.
The approach to register the container to the discovery service remains the same. During container startup, the start script obtains the external ip address, and docker exposed port. The discovered information will be passed to the Spring Boot application using environment variables. At application start the Spring service discovery client uses the environment variables to register the application to Eureka.
Before we are able retrieve information about the container (in the container) we need to enable the feature on the ECS instance. For more details about this feature see the Amazon documentation. To enable the meta data in the container you need to set ECS_ENABLE_CONTAINER_METADATA to true. This can be easily done by adding the line below the user data script.
Now each created container by ECS on the EC2 instances will have the meta data available in the file assigned to the environment variable ECS_CONTAINER_METADATA_FILE. An example of the meta data looks like:
We extract the port mapping from the metadata using jq. We assume the service will run in the container on port 8080. Otherwise you can make this configurable as well.
By using the Spring properties as environment variables, we can inject the information to the Spring application. Alternatively, you can use other names and assign the values in the application.yml file, see the example below.
With this approach the discovery agent becomes obsolete and we can retrieve the information for discovery through standard Amazon features. The example I have described in this post is just one way to implement service discovery in ECS, many alternatives are available.
\ No newline at end of file
diff --git a/2018/03/04/verlet-integration/index.html b/2018/03/04/verlet-integration/index.html
new file mode 100644
index 00000000..542d6b4e
--- /dev/null
+++ b/2018/03/04/verlet-integration/index.html
@@ -0,0 +1,475 @@
+Verlet Integration | 040code
This code uses the Quil library to render points
+and lines. In the main- function, the sketch macro from the Quil library is
+used to bind the physics domain to the UI.
Nothing special here. Just a map containing :points and :sticks. We’ll dive
+into the particulars of points and sticks soon.
+
Physics Simulation
+
This physics simulation deals with Points, and Sticks connecting Points. The
+Points are the things that seem to have direction and speed, are influenced by
+gravity and lose speed because of friction or because of bouncing against world
+borders. The Sticks try to keep their 2 Points apart according to the configured
+Stick length. The simulation loop boils down to:
This code can be read like this: ‘with state, first update-points, then
+apply-stick-constraints, and finally apply-world-constrains‘.
+
Points
+
Points are the main abstraction in this code. I decided to use a record to name
+them:
+
(defrecord Point [x y oldx oldy pinned])
+
A Point stores its current coordinates and the coordinates it had in the
+previous world state. On top of that it has a pinned property which indicates
+if a Point is pinned in space and - as a result - stays on the same coordinate.
+A pinned Point can be unpinned by clicking it with the mouse pointer.
+
The update-point function calculates the velocity of the Point and adds some
+gravity in the mix:
+
(defn update-point
+ [{:keys[x y oldx oldy pinned]:as point}]
+ (if pinned
+ point
+ (let[vx (velocity x oldx)
+ vy (velocity y oldy)]
+ (->Point(+ x vx)(+ y vy gravity) x y pinned))))
+
Reading superficially, this code says: if the Point is pinned then return the
+same Point. Otherwise, calculate it’s velocity (based on the current - and
+previous x & y coordinates), add some gravity in the y direction and return the
+newly calculated Point.
+
The function uses
+destructuring to name all
+the arguments of the incoming Point. With destructuring, you can bind the
+values in a data structure without explicitly querying the data structure. So
+instead of getting each and every value out of Point and binding it to a ‘new’
+name…
… you can very effectively tell the function that it will receive a map {}
+with keys x y oldx oldy pinned and that this function should have those
+variables available under the same name:
+
[{:keys[x y oldx oldy pinned]:as point}]
+
The code in my repository has some additional type hints, because I wanted the
+simulation to run smooth on my laptop and wanted to learn a bit more about type
+hinting and compiler optimizations.
+
And that’s it with regard to Points. In a couple of lines of code Points are
+already moving and reacting to gravity in the simulation.
+
World constraints
+
It’s time for Points to meet the harsh reality of life. Walls are harder than
+Points and therefore Points should bounce off of them:
Imagine a Point moved from A to B in the last Point update. This means a Point
+record is persisted with its x and y being the coordinates of B and oldx
+and oldy being the coordinates of A. If a wall line crossed the imaginary line
+A-B, the Point history should be rewritten. In essence, line A-B is mirrored in
+the wall it hits, giving rise to another imaginary line C-D where C mirrors A
+and D mirrors B. The miry (mirror-y) and mirx (mirror-x) variables in the
+code contain C coordinates mirroring A coordinates.
+
The simulation will take a little velocity loss into account when bouncing at
+walls. Therefore, D’ is calculated by using the x and y velocities (vx and vy)
+multiplied by a bounce factor (leading to the vxb and vyb variables in the
+code).
+
When the world state is drawn, the Point will fly off in exactly the right
+direction. Additionally, because of the history rewrite, subsequent Point
+updates will keep moving the Point in the right direction.
+
The picture shows the situation when hitting the ceiling. Hitting the floor and
+the walls works similar. And that’s all the math and code you need to simulate
+Points moving in a bounded space. Now let’s add Sticks to the simulation.
+
Stick constraints
+
Not only walls are restricting Points from free movement; Sticks also constrain
+them. A Stick connects 2 Points and has a configured length. The goal of the
+Stick constraint is to move the Points at the end of the Stick to the configured
+length of the Stick.
Instead of trying to calculate the solution that satisfies all Stick constraints
+at once, this code simply looks at one Stick at a time and ‘solves’ the
+constraint problem by repeatedly solving Stick constraints in isolation.
apply-stick-constraint takes a Stick and 2 Points (P0 and P1) as arguments.
+These arguments are heavily destructured. To understand what happens precisely,
+the next picture might be helpful:
Let P0 and P1 be two Points with a Stick S between them. P0 and P1 have already
+been updated by the update-point function and their new (x,y) coordinates are
+(40,50) and (70,90) respectively. This means:
+
+
dx = 30
+
dy = 40
+
distance = 50
+
+
Since the Stick between P0 and P1 was configured to have a length of 150, and
+we want P0 and P1 to be adjusted by the same amount, P0 should be located at
+Point S0 (10,10) and P1 should be located at Point S1 (100,130). This is
+exactly what apply-stick-constraint achieves:
+
+
The difference in the configured length of the Stick and the distance of P0
+and P1 is 100
+
The fraction of the distance between P0 and P1 that needs to be added to
+P0 AND to P1 is 1. So by extending the distance P0 and P1 with exactly 50
+(distance times fraction) in both directions, we’ll solve the Stick
+constraint.
+
This is done by using offsetX on P0x and P1x and using offsetY on P0y and
+P1y
+
+
Conceptually a scaled helper triangle is used to move P0 and an identical scaled
+helper triangle is used to move P1. The scale of this triangle is determined by
+the fraction which in this case is 1. These triangles are depicted in red.
+
Each Stick in the world is updated before a new world frame is drawn. It is very
+well possible that the neatly placed P0 and P1 are moved to different
+coordinates by another Stick constraint before the complete update is over. This
+sometimes results in wiggly behavior, although in most cases, the simulation
+is perceived to behave naturally.
+
Drawing
+
Finally, the Points and Sticks have to be drawn on screen:
Start by clearing the screen. If there is an :info-message entry in the
+state, draw the info-message. Otherwise draw all Points in state as a
+quil/ellipse and draw all Sticks as a quil/line. And that’s all folks!
+
Conclusion
+
As always, it was a pleasure working with Clojure. It turns out to be easy to
+work in a very mutable domain with mostly pure functions, also thanks to the
+excellent Quil library.
+
My current version of cloc tells me that
+core.clj is 230 lines long, but it counts the function documentation as code.
+Therefore, I ran a new cloc command without the doc strings. It then counts
+159 lines of clojure, approximately half of which are dedicated to UI
+interaction and the other half to physics simulation. Not bad at all!
+
I’m glad I decided to ‘bite the bullet’ and learn Emacs and CIDER. I now use
+Emacs for all my Clojure(script) development. In addition, I use Emacs for most
+of my text editing these days including writing and planning.
+
I’d like to thank Michiel Borkent a.k.a.
+@borkdude for reviewing an earlier version of
+the code and giving me very helpful feedback. Faults and not-so-idiomatic
+Clojure code remaining are my own.
+
Please share your comments, suggestions and thoughts about this blog post on
+twitter.com/mmz_. Thanks for reading and Happy
+Coding!
\ No newline at end of file
diff --git a/2018/05/26/immutable-infrastructure-taipei/index.html b/2018/05/26/immutable-infrastructure-taipei/index.html
new file mode 100644
index 00000000..7a58f941
--- /dev/null
+++ b/2018/05/26/immutable-infrastructure-taipei/index.html
@@ -0,0 +1,244 @@
+Immutable Infrastructure | 040code
The Docker slogan ‘Build, Ship, and Run’ advertises easy set up of immutable software builds, but it is not always that easy. Setting up immutable builds with Docker is pretty straight forward and shipping is just a matter of pushing the image to a repository. The next step is building the cloud infrastructure to run the containers. In the talk at
+the Cloud Native Meetup in Taipei I have shown how to create an immutable infrastructure on AWS with Terraform. The example belows shows how you can run your micro services in docker containers on AWS.
+
In this talk I briefly touch upon building immutable software. But the main focus of the talk will be on creating an immutable infrastructure. In this talk I will show you how to create an immutable infrastructure on AWS with Terraform. I will use a real world example to explain and show live how easy you can get micro services live on AWS and continuously apply changes to the same cloud environment..
+
Slides
+
Below the slides that I used for the talk, the slides are available as well on
+GitLab. You can easy navigate through the slides with the spacebar.
+
+
+
+
To get started with Terraform the best starting points are:
During the talk, I demonstrated a hello world example, see the link above. This examples shows some basics of terraform by creating a ec2 instance and security group to server a simple web application. The example only works in the AWS region eu-west-1 since the AMI used is only available in this region.
+
ECS demo
+
The second example I have shown, is creating an immutable infrastructure to server docker containers. The picture below descibes briefly the enviroment that will be cretaed.
The example also creates log groups in cloudwatch to capture the logging of the ecs agent and the running containers (services).
+
Setup
+
Before you can start, you need an AWS account with sufficient rights (admin), and you should create a AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY to be able to access your account programmatically.
+
Clone the repository containing the ECS sample terraform code. To create the infrastructure step-by-step you can check out the tags vpc and ecs.
You should also have Terraform installed (home brew install terraform) and
+tfenv to manage your terraform installation. Alternatively use a docker container to run the terraform commands, for example:
In this example you can swithc the reqion by changing the region variable
+aws_region to your own choice. Or set the following environment variable: export TF_VAR_aws_region=ap-northeast-1
+for Tokyo.
+
Create network layers (VPC)
+
In the next steps we will create the network layers.
+
First, we initialize and plan our changes.
+
terraform init
+terraform plan
+
Terraform should print on the console that 19 needs to be added and 0 to change or destroyed. Next we apply the change be executing terraform apply.
+
Add ECS cluster
+
Now we have the network layer created, we will add the ECS cluster. By default the bastion host is disabled, the bastion can be enabled by updating the variable enable_bastion in the terraform.tfvars file. Time to plan and apply the new resources for ECS.
+
git checkout taipei-ecs-1
+terraform plan
+terraform apply
+
Once terraform is ready it will print the url of two applications on the console. One application, is a simple micro service that just prints the AWS availabilty zone where it is running, the other is a graphql micro service. It will take a few minutes before the services are available.
+
Adding your own service.
+
Time to have some more fun. Edit the main.tf and start making changes. Remove all the services, add your own service. By default for each service an application load balancer (ALB) is created. Add for each services that you want to deploy a configuration as below and update the variables as required.
+
module "your-service" {
+ source = "ecs-service"
+
+ service_name = "<service-name>"
+ image_url = "<docker-image>"
+ container_port = <port in the container where the service is listening>
+ desired_count = <number of instances>
+
+ aws_region = "${var.aws_region}"
+ environment = "${var.environment}"
+
+ vpc_id = "${module.vpc.vpc_id}"
+ subnets = "${module.vpc.public_subnets}"
+
+ cluster_id = "${module.ecs-cluster.cluster_id}"
+ ecs_service_role_name = "${module.ecs-cluster.ecs_service_role_name}"
+}
+
Now plan and apply your changes, Terraform will inform you that a few resources will be destroyed (the removed services) and a few will be added.
+
Clean up
+
Once done you can easily clean-up all created resources in AWS, just run terraform destroy.
+
Serverless containers.
+
Since a few months Amazon support running containers without managing your
+ec2 instances, aka Serverless Containers. This services is called Fargate. I
+have written a blog
+post about
+how to run you containers serveless with terraform. Creating a mix of both
+ways of running of containers is explained as well. Be aware that Fargate is
+still only avaialable in us-east-1.
+
Have fun!
\ No newline at end of file
diff --git a/2018/08/08/microhack-brew-your-code/index.html b/2018/08/08/microhack-brew-your-code/index.html
new file mode 100644
index 00000000..296c96d0
--- /dev/null
+++ b/2018/08/08/microhack-brew-your-code/index.html
@@ -0,0 +1,226 @@
+Microhack Brew your code | 040code
Working on a Mac made my work a lot more fun. Especially when I was introduced to homebrew, the missing package manager for macOS.
+
MacPorts was fine, but it felt clumsy. Every tool seems to be switched to Homebrew and life was good.. Except when you encounter that one tool you need but is not available in brew…
+
Meet Gits
+
An example of this is the great shell script made by Roy Lines: gits.
+A shell script to share contribution between two git accounts when pair programming.
+
Unfortunately you have to install it by getting a remote bash script and add it in your /usr/local/bin directory with root rights.
When I see something like this, it’s another hurdle for using great scripts. It’s a good practice to check these scripts, before putting them into your bash with execution rights. Sometimes, I’m to lazy to do that..
+Also I always tend to forget these scripts, so they are never updated and never removed. Your system becomes a mess.
+
More people did not like that. An issue was raised with a question whether it could be installed with homebrew..
+
+
How can it install it on mac? I would prefer to use homebrew.
+
+
Well, let’s help Adam!
+
Create our own tap
+
Correct format
+
First step is to change the structure of your script so it can be used for brew.
+Scripts need to be in the bin/ directory so that is the first thing we need to do.
Once you have a release, we need the SHA-256 checksum of the release to ensure a proper version of the download.
+Browse to the release and copy the name of the tar-file (in our case: https://github.com/JeroenKnoops/gits/archive/0.1.0.tar.gz)
+
Download the tar and generate the SHA-256.
+
$ wget https://github.com/JeroenKnoops/gits/archive/0.1.0.tar.gz
+$ shasum -a 256 0.1.0.tar.gz
+13b778bf5a7f92285f54179cd88bea4b39f661a3de47a7ff0a84b9aa5d865962 0.1.0.tar.gz
+
We will need this SHA-256 for our formula.
+
Define the Formula
+
We have to define the formula. This can be done by creating a repo called homebrew-tap in your own github repo.
+It’s best practice to put some information in the README of this repo about the Formulas which are defined.
+
In our case:
+
## homebrew-tap
+Homebrew tap with formula for installing the following:
+
+- [gits](https://github.com/jeroenknoops/gits)
+
+
+#### Add this tap
+`brew tap jeroenknoops/tap`
+
+
+#### Install
+- gits: `brew install gits`
+
In the /Formula directory you must create a ruby file with the definition, in our case: gits.rb
+
class Gits < Formula
+ desc "A shell script to share contribution between two git accounts when pair programming."
+ homepage "https://github.com/jeroenknoops/gits"
+ url "https://github.com/JeroenKnoops/gits/archive/0.1.0.tar.gz"
+ version "0.1.0"
+ sha256 "13b778bf5a7f92285f54179cd88bea4b39f661a3de47a7ff0a84b9aa5d865962"
+
+ def install
+ bin.install "bin/gits"
+ end
+end
+
Push the repo to github and we’re ready to go.
+
Test it
+
Now people can use the script by installing it with homebrew.
+It’s still in my private homebrew-tap, so people first have to tap into our private tap.
+
brew tap jeroenknoops/tap
+
After that they can install gits with brew install gits
+
I do not want to tap into someone’s tap
+
To even raise the level of trust of the script / tool and install convenience, you can also put the Formula directly into the homebrew-core. Now people don’t need to tap into someones private tap.
+This will require some extra steps, like adding tests.
+
Brew has a way to test your formula by doing: brew audit --new-formula gits
+
This will show you what you still need to do, before you can submit it to homebrew. You can also find more information on that in the Formula-Cookbook
+
Improvements
+
There are more improvements I should make to release it to homebrew. F.e. add dependencies to git.
+For now I’m not going to dive into that. Homebrew has good documentation on that: https://docs.brew.sh/Formula-Cookbook
+
Conclusion
+
I really liked the way homebrew is set up. I like the way they name things, like brew, tap and bottle. By adding my own bottle I’ve learned how homebrew is working and now I even feel more confident in using it. I like the fact that only bottles with tests and with the proper dependencies check can be added in the core homebrew formulas.
+
Homebrew cask
+
Homebrew has another great feature, called homebrew cask. No more: “To install, drag this icon..“. Now you can install tools like atom or 1password by simply calling: brew cask install atom…
\ No newline at end of file
diff --git a/2018/08/22/maze-generation/index.html b/2018/08/22/maze-generation/index.html
new file mode 100644
index 00000000..1f2fce1a
--- /dev/null
+++ b/2018/08/22/maze-generation/index.html
@@ -0,0 +1,436 @@
+Maze Generation | 040code
During my summer holiday, in between long walks and nice food & drinks, I felt
+like writing some code for fun. Unfortunately, I didn’t bring my laptop.
+Luckily, Mike Fikes created a ClojureScript REPL
+for mobile devices to
+write some Clojure(script) without a laptop close at hand.
+
A REPL, or a Read Eval Print Loop, reads your input, evaluates it, prints the
+result and waits for new inputs to read. In the meantime, the code you wrote
+lives in REPL memory. So if I define a function in the REPL, it will be read and
+evaluated and in my next input I can call this function, because it lives in
+REPL space.
+
So, back to this REPL on my phone. Permanently saving work is not possible in
+this REPL, there are no debugging capabilities, no code completion, no test
+runners … just a REPL: 35 characters wide in portrait mode, 79 characters wide
+in landscape mode. To give you an idea of my ‘Integrated Development
+Environment’ and the resulting mazes, I recorded a 30 seconds screen cast:
+
+
At home, I created a gist containing all 70 lines of maze
+code (56
+lines according to the command-line tool cloc). In this blog I’m going to explain
+the concepts and code.
+
Grids, Cells and Neighbours
+
First of all, we’ll need a grid and a coordinate system to point to individual
+cells in the grid. The reason is that most maze generators expect to be able to
+traverse a grid and lookup information about cells and neighbours of cells.
+
A grid can be modeled as a vector of vectors, representing rows and columns:
+
(def grid
+ [[123]
+ [456]
+ [789]])
+
A cell can be pinpointed by its [row col] coordinate which matches perfectly
+with the get-in function in clojure. This get-in function is used to find
+items in nested data structures. For instance, (get-in grid [2 1]) will result
+in the value 8.
+
To find the neighbours of a cell, I created a couple of helper functions:
The function argument is destructured (taken apart and named) and the
+north-of, south-of, west-of or east-of neighbour coordinate is
+calculated by incrementing or decrementing the row or column number. For
+instance, (north-of [1 1]) results in the value [0 1].
+
Using the juxt function we can now calculate all neighbours in one go. juxt
+takes a variable number of functions as arguments and returns a new function.
+This new function returns a vector containing the result of applying each
+function to the arguments provided. So for instance ((juxt north-of south-of
+west-of east-of) [1 1]) results in the value [[0 1] [2 1] [1 0] [1 2]].
+
Unfortunately, these functions don’t care about the grid. ((juxt north-of
+south-of west-of east-of) [0 0]) will happily return [[-1 0] [1 0] [0 -1] [0
+1]] containing cell coordinates inside AND outside the grid. Therefore, in
+order to calculate all neighbours in the grid, we need to filter out the
+ones that are not part of the grid. This results in the following neighbours
+function:
Now working with the grid is easy, let’s move on to the maze. A maze is
+basically a grid, augmented with some information to determine if borders
+between cells are open or closed. That information can be easily stored in the
+cells themselves.
+
I liked the approach Mark
+Bastian
+took: in a cell, simply list the coordinates of neighbours the cell is connected
+to. We can even list the coordinates in a set #{}, since duplicating neighbour
+coordinates is useless. A 2x2 ‘maze’ with all borders closed looks like this:
+
[[#{}#{}]
+ [#{}#{}]]
+
A 2x2 ‘maze’ with an open border between the 2 cells on the left looks like this:
+
[[#{[10]}#{}]
+ [#{[00]}#{}]]
+
The two connected cells reference each other. It would be convenient to have a
+function that updates the grid when a border needs to be removed between 2
+cells:
Now it’s time for the maze generation. Luckily I had a copy of the ‘Mazes for
+Programmers
+book’
+on my phone. Because I like the texture of the mazes coming out of the recursive
+backtracker algorithm, I decided to implement that one.
+
The algorithm can start on any cell on the grid and I decided to start on [0
+0] every single time. Every cell that is visited by the algorithm is pushed on
+a stack which I have called the backtrackstack. The cell on top of the stack
+is considered the current cell.
+
From the current cell, a path is created (remove-border) to a randomly
+selected, previously unvisited neighbour. An unvisited neighbour can easily be
+detected, since it’s simply an empty set #{}. That previously unvisited cell
+(next in my code) is also pushed unto the stack. This process repeats
+continuously until a cell is visited that has no unvisited neighbours. At that
+point, that dead-end cell is popped of the stack, making the previous cell the
+current cell. Again, the algorithm will check if this cell has unvisited
+neighbours and will either create a path to that unvisited neighbour or pop
+another cell from the backtrackstack. This will continue until every cell has
+been visited and the backtrackstack is empty.
Nice! Now the only thing left is printing a more human friendly view of this
+maze.
+
Maze Printing
+
First, the top of the maze is printed:
+
+---+---+---+
+
Then, for every row the left border is printed | and 2 passes over the cells
+have to be made. The first pass determines if there should be a border between
+the current cell and its east-of neighbour, resulting in something like this:
+
+---+---+---+---+
+| | |
+
The second pass over the row adds a + and then determines if there should be a
+border between the current cell and its south-of neighbour, for instance
+resulting in this:
+
+---+---+---+---+
+| | |
++---+---+ + +
+
After repeating this for all rows, a maze like this is printed:
So there you have it: a maze generator in ~ 30 lines of code with a 30 lines of
+code ascii view on top of it, build with nothing but a REPL on a phone. I really
+enjoyed the puzzle and the small (and large) successes while working in the
+REPL. I hope you enjoyed this recap.
+
I’d like to thanks Mike Fikes for his excellent
+ClojureScript REPL, Mark Bastian for the
+inspiration for the maze data structure, Jamis Buck
+for his lovely ‘Mazes for programmers’ book and Gert
+Goet for reviewing an earlier version of this
+blog. Faults and not-so-idiomatic Clojure code remaining are my own.
+
Back home, I’m happy to be working in Emacs with the excellent Clojure CIDER
+plugin again. Which reminds me that during my holiday I visited a picturesque
+CIDER factory in France. Cheers!
This week we (Jeroen, Niek, Gertjan, Eric and Kike) are enjoying Full Stack Fest 2018 Barcelona,
+a week-long full stack development conference with a focus on solving current problems with new & inspiring perspectives.
+
In this quite short blog we just summarizing briefly the talks and our opinion about the talks. We have rated all talks, the rating is subjective as well.
+
This conference is split up in two parts. The first two days were focussed on Frontend development. You can read our post about it here
+
First of all we would like to thank Luca Marchesini for hosting the sessions.
+
Day one
+
Reinvent the social web
+
Rating: ⭐ ⭐ ⭐ ⭐
+
André Staltz (@andrestaltz) explained the problems we have with the current generation of social networks. Are the realy open, free, innovative? Can we trust and respect people in the network. He explained passionated how we can change the future with decentralized networks such as scuttlebutt.
+
Open Gateway - Decentralized IoT
+
Rating:
+
Kalyan Dikshit (@decode_dev) presents about the open gateway initiative from Mozilla. Not our topic.
+
Build and deploy a Blockchain web-app with Hyperledger Fabric and Hyperledger Composer
+
Rating: ⭐ ⭐ ⭐
+
Hyperledger is something we’re looking into right now, so this talk was very relevant. Horea Porutiu (@horeaporutiu) showed the basic things about hyperledger and that’s great for this audience. Great balance between technical information, examples and real applications of the chain. Their work with Brooklyn Roasting Company was a good example how to use the blockchain. It shows an audit trail on your coffee.
+
A Game of Theories: Why languages do what they do
+
Rating: ⭐ ⭐ ⭐ ⭐ ⭐
+
Rae Krantz (@rustbeltrae) quickly compared the ugly and beauty of Ruby, Python, Go, Clojure, JavaScript, Erlang, Rust and Elixir. Yeah we never will do Clojure and from tomorrow we only do Rust and Elixir. Awesome session. Best quote of the day: “JavaScript was written in the time of Netscape when Java was popular.” :)
+
The Future of ChatBots in the Travel Industry
+
Rating: ⭐ ⭐ ⭐
+
Prabh Simran Singh Baweja
+
Designing a chatbot is not that easy as it sounds. “Book me a hotel for this weekend.” What kind of hotel? Where? Weekend, is it friday and saturnday or is it saturday and sunday? How to combine data in a good way so people are not annoyed. Nice insights on how bookings.com is building these bots.
+Look at his slides
In modern worlds we use large datasets to tune our models. This is not without danger. Our datasets are often biased. Good tips on how to test if the models are digitally bias, by changing some of the fields like gender. Never use stock images. Nice example was shown if you google on CTO. This is not a good representation of the world. Enough food for thoughts!
+
Everything I always wanted to know about crypto, but never thought I’d understand
+
Rating: ⭐ ⭐ ⭐
+
Sasha Romijn (@mxsash) gave a fast introduction to cryptographic basic concepts that every developer should know. Quote: “Export in cypher builder should ring an alarm bell”
+A good overview of the magical world of Crypto. If you want to learn more about this, I would certainly recommend a free coursera course on this topic given by Dan Boneh.
+
Onion adventures: how to use onion services and the Tor network in your web endeavors
+
Rating: ⭐ ⭐ ⭐
+
Respect for Silvia Puglisi (@nopressure) for giving this talk when she has the flu. Very good to see all the activity around Tor. Important to keep on bringing the message it’s not that obvious to have freedom of speech and expression. /me installing TOR.
+
Everything in sketchnotes..
+
A big shoutout to Christin Westermann (@c16n) for creating these sketchnotes!
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Day 2
+
Observability for Emerging Infra: What Got You Here Won’t Get You There
+
Rating: ⭐ ⭐ ⭐ ⭐
+
Charity Majors (@mipsytipsy) gave a very good talk on how to look at current systems. In the current era, you cannot simply say: “Our application is up..” What does that mean? What’s up? Move away from the known-unknowns which can be monitored with dashboards, but look at the real problems users experience. Focus on the unknown-unknowns and make sure you can react and analyse your applications fast. Don’t aggregate data, because you will lose important details about problems. Come up with good sampling strategies. Build observability in from day one! Cool shirt, “I test in prod”. Quote: “What is unknown, we must leave uncoded.”
+
Do you need a Service Mesh?
+
Rating: ⭐ ⭐ ⭐ ⭐
+
A great presentation given by Matt Turner (@mt165pro) about Service Mesh. He demo-ed the principles of Service Mash by using istio.io. He used kubernetes to demo some of the configuration you can setup with only proxy-ing between different services. Very powerful concept. I’m wondering how this can be used in an infrastructure-as-code architecture.
+
HTTP and JSON for your gRPC Services
+
Rating: ⭐ ⭐ ⭐ ⭐
+
Michael Hamrah (@mhamrah) gRPCed all the things. First feeling was Oh no we go to do again CORBA, and there are similairities between gRPC and CORBA. In this talk Michael explained how you can build high performance application and left the all the plumbing code to the gRPC framweork. Just simply define an interface via IDL and only take care of the real business code. It was also nice to see how you can combine gRPC via rejoiner with GraphQL. Slides are available here
+
Software Engineering - A philosophical activity
+
Rating: ⭐ ⭐ ⭐ ⭐
+
Tom Enden (@tom_enden): Did you know that Plato invented OOP and the Prototype pattern is inspired by Nominalism which also dates back to Plato. Tom showed the parallels between philosophy and software engineering. According the pragmatic programmer we should learn at least one language every year. Since different languages solved things in different ways. And Tractatus says the limits of my language means the limit of the world. So looking on philosophy we can inspire our selves for new ideas. The slides
+
Take a (Client) Hint!
+
Rating: ⭐ ⭐ ⭐ ⭐
+
Jeremy Wagner (@malchata) explained nicely the power of client hints. Having a fast internet and reliable internet connection is still not common. For example a survey by Pew Research shows that one-third of American adults do not subscribe to any internet access faster than dial-up at their home. So building responsive clients on slow or unreliable networks still counts. Client hints comes in different types. By opting in (header ACCEPT-CH) a client can inform that it understands client hints, Device hints enables for example responsive images and netwrok hints can inform the server about the speed of the network of the client. The slides are here available.
+
Lost in transaction? Strategies to deal with (in-)consistency in distributed systems
+
Rating: ⭐ ⭐ ⭐ ⭐
+
Bernd Rücker (@berndruecker) is a (work-)flow automation enthusiast. He explained the strategies to deal with the temporary inconsistency and how you can convert it to consistency again using the starbucks as example. One can use different approaches to it. Chorography vs orchestration.
+These kind of problems can be solved by using a BPMN product like Camunda. System become complexer every day and this will become a bigger problem everyday. And what should we never forget: Zucchini and idempotency.
+
This talk triggered a lot of thoughts for me personally. This is one of the areas where blockchain might actually solve things. Use smartcontracts to atomically commit transactions all at the same time. Fortunately he already wrote a blog post about the topic. Great speaker!
+Bernd Ruecker’s blog
+
Lightning talks
+
Antiona Librada (@antoniolibrada) showed us how he improved a complex ci build pipeline which took > 45 minutes to run by using 2 lambdas and Amazon Code Build. The outcome was a more stable, cheaper and faster build pipeline. They increased the number of daily deployments drastically.
+
Peter van der Meulen (@HendrikPetertje) showed us his terminal. This shows again how important improving your daily workflow is. Some highights: Spotify in Vim, thefuck, circleci-cli, wee-slack. For more info: snippets and projects
+
Aaron Bassett (@aaronbassett) showed us some amazing (and probably illegal) things about SMS. Here you can find his slides. Really scary.
+
Two nice talks on our power as developers to shape the world. Let’s do this right this time.
+
Cats, The Musical! Algorithmic Song Meow-ification
+
Rating: ⭐ ⭐ ⭐ ⭐
+
Awesome to see how Beth Haubert (@haubertdashery) passionated hacked her way around to generated a cat meow song from songs using ruby. It was really fun to see, how the program evolved! Especially because at the end, the result was really terrible..
+
Everything in sketchnotes..
+
A big shoutout to Christin Westermann @c16n for creating these sketchnotes!
This week we (Jeroen, Niek, Gertjan, Eric and Kike) are enjoying Full Stack Fest 2018 Barcelona,
+a week-long full stack development conference with a focus on solving current problems with new & inspiring perspectives.
+
This conference is split up in two parts. The first two days were focussed on Backend development. You can read our post about it here
+
In this quite short blog we just summarizing briefly the talks for the second part, the front-end part. It’s our opinion about the talks. We have rated all talks, the rating is subjective as well.
+
First of all we would like to thank Reginald Braithwaite for hosting the sessions.
+
Day one
+
Building peer-to-peer apps with the Beaker Browser
+
rating: ⭐ ⭐ ⭐ ⭐
+
Tara Vancil, during the backend days we already are informed about the lack of freedom is our social networks, and how this could be solved with decentralized social networks. Tara showed that the web has the same problem. Do you remember Geocities? Yepz these sites are gone. And deploying or customizing web apps is still hard. Would it not be great we can fix it just in the browser the app we loved the most. Tara showed how easy you can run, customize and deploy a decentralized web app.
+
To push, or not to push?! - The future of HTTP/2 server push
+
rating: ⭐ ⭐ ⭐ ⭐
+
Patrick Hamann @patrickhamann, resource loading is still hard, performance in the browser is tightly coupled to latency, connection costs are high, and scipt execution is expensive. Patrick explained how to use resources wisely, make a difference between critical resources. resources in the users viewport, and non critical resources. By using preload, push and async push together with a strategy of critical resource a hugh step can be made for a high performing web app. See the slides here
+
The Future of Front-End Performance
+
rating: ⭐ ⭐ ⭐ ⭐
+
Sia Karamalegos, a great talk on performance of the front-end. How much impact does it has on your business. Staggering numbers were shown about AliExpress load time reducing and how that correlates to an increase conversion for new customers.
+Apps become bigger and bigger. Make sure you check your apps on size / speed by using tools like webpagetest.org. Optimize your images. Most of the web are images, but the most expensive bytes are javascript. Prioritize the code your shipping.
+Slides and more info are available.
Serg Hospodarets, great news for modern JavaScript development. Most browsers now support Native JavaScript modules ESM. This will change our way we are bundling our JavaScript using Webpack, Rollup and other bundlers. It’s also available in node and the extensions is called Micheal Jackson Solution aka .mjs
+Dynamically loading javascript modules in combination with HTTP/2 can save a lot of load time.
+Simular slides
+
Unity + HTML5 = Isomorphic Game Engine?
+
rating: ⭐ ⭐ ⭐ ⭐
+
Lerenzo Malcom @lerenzom, showed how to combine the world of Unity and the browser. Last year A-Frame was a big topic that address the same solution space but Lerenzo showed how to combine Unity in the browser instead of A-Frame. After discussing why unity and not A-Fram he spend attention how to setup bi-directional messaging, and finally gives some WebGL examples. Sorry we are not that AR/VR boys but with this talk Lerenzo did an awesome job, please see the slides (including animations) here
+
State of the Art Web User Interfaces with State Machines
+
rating: ⭐ ⭐ ⭐ ⭐ ⭐
+
David Khourshid (@davidkpiano), the front end is complex, it is difficult to understand, difficult to read, contain bugs, hard to enhance and new features make it even worse. What if we model our front end as an finite state machine (FSM). Nothing new in Computer Science, states machines are already introduces in “automata theory”, Specifying a the front end action (flow) in a state machine increase the predictability and even make it possible to automate testing. Just calculate you the shortest path via Dijkstra algorithm. David showed his open source framework xstate which is available on GitHub and npm. Slides are are here available
+
A Front-end dev’s hobbyist approach into Machine Learning
+
Susanna Wong, machine Learning is hard. It’s almost impossible to explain this in just one talk. Especially when you combine it with all kind of background information. We were very impressed by the architecture of the buildings, but unfortunately the talk was too much out of our reach.
+
Building a Modern Memex
+
rating: ⭐ ⭐ ⭐ ⭐ ⭐
+
Andrew Louis @hyfen, Imagine you can manage all your data with a single device… An idea that goes back to the 1945 when Vanneva Bush discribed the Memex. Andrew build his own Memex and demo the awesome combination fo all the data he collected over the years. Absolutely crazy to see that he is able to search back in time which podcast he was listing when driving from A to B. To be able to combine all your data from Google, Spotifiy, Maps, Whatsapp and so on, probably you need a complex system. But Andrew handles it all just with Rails, Ember, Electron, Node and Postgres. And how cool it is all open source. Slides are here.
+
Everything in sketchnotes..
+
A big shoutout to Christin Westermann (@c16n) for creating these sketchnotes!
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Day 2
+
Accessibility vs latest Web APIs. Can’t we just get along?
+
rating: ⭐ ⭐
+
Mauricio Palma (@palmaswell) showed pasionated how important it is to make the web accessible for everyone, also for the ones with a disability. For example, he show cased how to use the Google speech API. A quote to remember from Maricio: “If we use our own abilities as a baseline, we make things that are easy for some people to use, but difficult for everyone.”
+
The Web Authentication API - Imagine a World Without Passwords
Passwords were invented for a good reason. The world has changed and unfortunately, passwords are not sufficiently secure enough anymore. There are a lot of problems attached to the concept of passwords. The Web Authentication API allows us to authenticate our users using public key cryptography. This prevents malicious users to gain access to your data even if they have the public key. Modern computers have dedicated hardware to perform authentication, like touchID, faceID, camera’s, usb-tokens. This can now be used in our webapplications. This is very much in line with what we are want to achieve in our own project. Slides
+
Killing the checkout or: How I learned to stop worrying and love Payment Request
+
rating: ⭐ ⭐ ⭐
+
As member of the W3C Web Payments Working Group, Krystian Czesak (@krystostrerone), is heavily involved in setting a web standard on Payment Requests. The Response code 402 Payment Required is already there for a long time, but is never implemented. Lots of attempts are made, but none of them succeeded till now.. Currently the checkout procedure on most applications is not so user friendly. The Web Payments is initiated to improve this user experience. All modern browsers support it. All information about you is stored in your browser and not on each page. There’s an API available to use that information, so the user only has to add the information once and it can be used in every application. Chrome even supports Interledger. The presentation has a lot of information how to implement a client for this. See the slides for more info. Slides
+
A Real World PWA
+
Rating: ⭐ ⭐ ⭐ ⭐
+
Zack Argyle, @ZackArgyle stared with the story of Pinterest and what will happen with your uses once you decide to make a blocking page and only support a app. Smoothly continuing with the story how important it is to have a reactive app and what that means. A progressive web app is according to Zack: “A site that utilizes progressive enhancements, meaning that modern browsers will get a better experience than older browsers”. He explained how he used the principle of the 3R’s to build a progressive app. By Reduce, make it light. Use for example the webpack analyzer. Reuse, caching, server workers. Cache via a worker is faster then the browser cache. And Review, measure performance! Slides are available here
+
Purifying Typescript
+
rating: ⭐ ⭐ ⭐ ⭐
+
Timothy Clifford, @timothyclifford. Cool to see that we can do functional programming with JavaScript a language written in 10 days back in 1995 an nowadays the most populair language. We really liked the talk and clear examples how to program functional with TypeScript in JavaScript. Maybe the talk was little to much basic introduction to functional programming, but Timothy succeeds in his mission statement: “Understand how to apply functional programming techniques with TypeScript”. Slides and code are shared via GitHub.
+
Dynamic CSS: Layouts & Beyond
+
rating: ⭐ ⭐ ⭐ ⭐
+
This talk by -Tara Vancil-, I mean Miriam Suzanne (@mirisuzanne), was really fantastic. She integrated lots of photos from previous talks on the same day. But not only that was remarkable. The knowledge on CSS is outstanding. She showed us the history of how we used to do CSS, from tables to SASS, to flexbox and Houdini. But all of these methods needed some wicked tricks to work in some cases. Now there is something new and this is promising. CSS Grid. This is the future of layouts, so take a look at this. In the slides you will see several examples of how to use it. F.e. https://gridbyexample.com/ and the the Slides.
+
Lightning talks
+
1
+first one: @EleanorClarinet
+Nice corrolation between learning music and learning to code.
+Don’t be a “Mozart pianist”, be a musician
+Don’t be a “React developer”, be an engineer
2
+Isac. Rocket fuel for Ember.js and javascript teams.
+Splanky about jazz shows in Smalls NYC.
+
3
+2013 it was already possible to do a lot of reactive stuff with imba.io
+
4
+Robert Nickel about Mob Programming.
+We love Mob Programming, but it’s hard. See slides
+
5
+Andreaverlicchie.eu
+Better navigation in markup
+
6
+Berkay —> Sharing
+How to share components between multiple react projects
+
WebXR, if X = how?
+
rating: ⭐ ⭐
+
Diego González and Indira Knight introduced Extended reality, XR or should we say AR + VR in the browser. Physical and virtual worlds are merging, XR is coming to the browser, and the browser is coming to XR devices. The frameworks ((A-Frame && ThreeJS) || BabylonJS) && WebGl where demoed in the browser, both in a 2d and 3d view. For more details see the slides for more references.
+
Thanks to Codegram
+
We would like to conclude our post with a big applause for codegram, the Full Stack Fest was awesome again. Please never change the single track philosophy. And did we already mentioned the food is the best we ever had on a conference.
\ No newline at end of file
diff --git a/static/2019/03/13/git-bisect/git-bisect-1.json b/2019/03/13/git-bisect/git-bisect-1.json
similarity index 100%
rename from static/2019/03/13/git-bisect/git-bisect-1.json
rename to 2019/03/13/git-bisect/git-bisect-1.json
diff --git a/static/2019/03/13/git-bisect/git-bisect-2.json b/2019/03/13/git-bisect/git-bisect-2.json
similarity index 100%
rename from static/2019/03/13/git-bisect/git-bisect-2.json
rename to 2019/03/13/git-bisect/git-bisect-2.json
diff --git a/static/2019/03/13/git-bisect/git-bisect-3.json b/2019/03/13/git-bisect/git-bisect-3.json
similarity index 100%
rename from static/2019/03/13/git-bisect/git-bisect-3.json
rename to 2019/03/13/git-bisect/git-bisect-3.json
diff --git a/2019/03/13/git-bisect/index.html b/2019/03/13/git-bisect/index.html
new file mode 100644
index 00000000..3503d117
--- /dev/null
+++ b/2019/03/13/git-bisect/index.html
@@ -0,0 +1,343 @@
+Git Bisect | 040code
I want to tell you a little story about what happened to me a few days ago.
+I went to the office and a colleague of mine was not very happy. This is strange because I work for a very nice company.. What happened? I asked him what was wrong. It turned out that he updated a lot of dependencies and some code for a certain service. And now the SonarQube reporting was not working anymore with a very strange message: HTTP Error 502 Bad gateway..
+
I decided to help him, because we don’t work alone and especially when you’re stuck (and frustrated) at something it’s best to include other human beings.
+
Since there we’re already several commits done I’ve decided to use git bisect to find the bug-introducing commit.
+
I’ve noticed that it was new to my colleague so let’s write a blog post about it.
+
Prerequisites
+
In order to follow the instructions, you need the following things:
The rust-example is very simple. It’s a rust library with four functions namely: add, substract, multiply and division. By no means this should ever be used in real systems, this repo only exists as example for this blog post. It has 9 commits.
+
$ git log --pretty=format:"%h - %an, %ad : %s"
+4d7f719 - Jeroen Knoops, Sun Aug 12 17:10:07 2018 +0200 : Adds link to blog post on git bisect
+380d12e - Jeroen Knoops, Sun Aug 12 17:09:04 2018 +0200 : Refactors tests in seperate directory
+1ae37e1 - Jeroen Knoops, Sun Aug 12 17:07:32 2018 +0200 : Adds edge cases for division
+0580aec - Jeroen Knoops, Sun Aug 12 17:06:41 2018 +0200 : Adds information on how to run tests and create the documentation
+d619b4d - Jeroen Knoops, Sun Aug 12 17:04:50 2018 +0200 : Adds function 'division'
+abc4c40 - Jeroen Knoops, Sun Aug 12 17:03:29 2018 +0200 : Adds function 'multiply' with a runnable example in the documentation
+2a5a057 - Jeroen Knoops, Sun Aug 12 17:01:27 2018 +0200 : Adds readme.md
+3d75d3d - Jeroen Knoops, Sun Aug 12 16:59:58 2018 +0200 : Adds travis-ci pipeline
+030b725 - Jeroen Knoops, Sun Aug 12 16:58:49 2018 +0200 : Adds functions 'add' and 'substract'
+
It starts with adding two functions, adding travis-ci, add some documentation, more functions, some refactoring… everything you expect from a ‘real’ project.
+
Of course it has some tests and I’ve tested everything manually before I commit… But for some reason at the end, the substract function is not working anymore..
+Whoops.. I’ve tested it manually, but for some reason I’ve forgot to add a test for it..
+
Let’s create a test
+
We’re creating a test in /tests/bug.rs to show the bug.
+We will use this test to check in which commit the bug was introduced.
When implementing multiply, I’ve accidentally broken the substract function.
+Git bisect will also add refs so you can which commits have been tested.
+
The correct implementation would be:
+
pubfnsubstract(a:i32, b:i32)->i32{
+ a - b
+}
+
The test we’ve written to find the bug can be renamed and also added to prevent this from happening again.
+
Conclusion
+
This was a very simple test / bug to find. By doing a git blame on the line, we would found the same
+error, but this is not always the case.
+Imagine a website which overtime became very slow. Try to create a test which measures the performance of a certain
+page. This test should fail when the performance is below a certain threshold. This can be a complete end-to-end test.
+Now you will find the exact place where the introduction of a small stupid javascript library caused a big performance
+problem.
\ No newline at end of file
diff --git a/2019/03/15/mob-programming/index.html b/2019/03/15/mob-programming/index.html
new file mode 100644
index 00000000..005d8071
--- /dev/null
+++ b/2019/03/15/mob-programming/index.html
@@ -0,0 +1,175 @@
+Mob Programming | 040code
Last week I was lucky to be able to attend a workshop about “mob programming”, given by Woody Zuill. As somebody who considers himself “agile” (in the original sense of the word) and who is a proponent of extreme programming, I liked it a lot. In this article I’m sharing some of my findings and learnings.
+
+
Mob Programming
+
Simply put, it means that the entire team is working on a single computer, all together at the same time. As an outsider, you might see a few big screens next to each other, and team members sitting next to each other facing those screens. One person is behind the keyboard, and the others are looking at the screens. One of them seems to have a special role, because he is telling the person behind the keyboard what to do.
+
Looking a bit closer, you notice that one person is not really looking at the screens very much, but paying close attention to what the rest are doing. He notices you and quietly comes over to you. He introduces himself as the team’s coach.
+
You: This looks interesting, can you explain what is going on here?
+
Coach: Sure! This team is learning mob programming. They have been at it for a while now and are approaching the point where they don’t need me anymore. They understand how to do it and why they are doing it. They are learning how to improve on this by themselves.
+
You: So why are they doing it?
+
Coach: There are two important reasons. The first is that the team wanted to learn to work together more effectively. They want to have a common understanding of things like architecture, coding standards, and code quality. Before they started working this way, they were in trouble. They were working on a project that was already a year late, and they didn’t know how to make the drastic improvements that were needed. When I was hired, I noticed that cycle times for bug fixes where sky-high: it took forever for a bug that was reported to be fixed. When I looked into the code base and I was not very enthusiastic about the quality. When asking about this to the individual team members, they reacted very differently. Some didn’t feel there was a problem. Others did, but they found they couldn’t address it because the rest of the team “just doesn’t get it”. There clearly was an issue with the team’s ability to operate as a team instead of just a bunch of individuals.
+
You: I have seen situations like that myself occasionally. Has it improved?
+
Coach: Oh yes, it can be difficult still sometimes, but they have learned to discuss such issues with each other. They have learned to raise such issues, and they have learned to do that in a respectful manner.
+
You: How did that happen?
+
Coach: I let them play.
+
You: Come again?
+
Coach: Let me ask you a question: what is the most important prerequisite for learning?
+
You: That’s easy. We have known forever; that’s why we don’t allow bullying at schools, for example. It is security, the knowing that it’s ok to be wrong, to make mistakes.
+
Coach: Indeed! So that’s what we did. Every Friday afternoon, we’d have a coding dojo together, where we practiced coding kata’s. The word “dojo” comes from martial arts: a place where you practice. A “kata” is a series of made-up exercises designed to help you learn something in a safe situation. And we found that it was so much easier to discuss code quality with each other when we didn’t have the pressure of “work” attached to it! Nowadays, instead of using Friday afternoons for this, we start every day with a coding dojo of an hour. Through this, we have learned to treat each other with kindness, consideration, and respect.
+
You: That sounds like it might not only be effective, but also fun! So you mentioned that there are two reasons for doing mob programming. What’s the other one?
+
Coach: The second reason is even more important than the first one. What harms project progress more than anything else?
+
You: Ehh…
+
Coach:(gives encouraging look)
+
You: Well…
+
Coach: Exactly, delays! Waiting time, queues, inventory! This is all very well known from lean manufacturing. It is common sense that keeping people busy while waiting for important answers is counter-productive. It’s not about being busy, it is about getting the most important thing done as fast as possible. A waiting time of an hour probably costs at least two hours because of context switching. It’s all about flow. Mob programming is a multi-discipline approach. Everybody who is needed should be present, including designers, testers, product owner. And if key people cannot be present, they have to make sure that they are otherwise available and get back to questions within minutes. Unless the project is not important of course, but then you should probably find something else to do.
+
You: Right. It’s surprising how much lean is common sense in one discipline, and how novel it appears to be in software engineering.
+
Coach: And it gets better. Because “flow” has another meaning, from the field of psychology. When you’re “in flow”, you are highly concentrated, you forget distractions, every ounce of your mental powers is used to solve the problem at hand. One of the original goals of pair programming was to improve flow: when one of the pair needs to step away for a moment, the other continues. When the first returns, she can get right back into flow again. Mob programming reinforces this even more.
+
The Workshop
+
So that was what the workshop was about: why do it, and how to do it. There was a hands-on coding dojo of a few hours that was designed to get a flavour of what you need to learn as a team to be more effective. I won’t spoil it for you, but to me it struck a good balance between fun and “suffering” (being confronted with things that you didn’t know you were doing ineffectively).
+
Focussing on flow feels like a good idea. When you do that as a team, you can come up with strategies to deal with problems like managers trying to interrupt too often, for example. But also things that need studying can interrupt your flow; you might create a list of study topics to park these things on, and have a dedicated hour of learning every day for those topics (coding dojo!).
+
There is so much more to talk about. For example, the roles of mob programming and the underlying principle: all ideas must go from somebody’s head into the computer through somebody else’s hands.
+
Want to learn more? Hire Woody Zuill and/or buy his book “Mob Programming – A Whole Team Approach”. Like what you read? Contact me and let’s see how we can leverage this!
\ No newline at end of file
diff --git a/2019/03/21/spring-boot-graphql/index.html b/2019/03/21/spring-boot-graphql/index.html
new file mode 100644
index 00000000..ed1b4947
--- /dev/null
+++ b/2019/03/21/spring-boot-graphql/index.html
@@ -0,0 +1,435 @@
+GraphQL + Spring Boot | 040code
Now GraphQL is not anymore that new kid on a block it would be nice if we can implement a service with a GraphQL interface. Spring Boot is one of the well known frameworks to build your micro services in Java. Time to assess how easy, or hard it will be to implement a GraphQL service. In this post we will not explain how GraphQL works. We focus on a simple Java Spring Boot micro services with a JPA layer. In the GraphQL language we cover the tree major concepts, a query, a mutation and a subscription.
+
GraphQL is API is based on three main concepts. Query the backend for data using a graph based query. The query is in a JSON like style and the response is valid JSON. A major difference with REST is that wit a query you can specify excatly what you expect or want from the backend. Mutation are the way to change data in the system based on flat input object. In this example we use mutation to add new information. Finally we have subscription to subscribe for changes and get notified once something is changed via a web socket.
+
Java, Spring and GraphQL.
+
For implementing a GraphQL service you have two choices. Either schema first, or code first. So similar concepts as we used with SOA XML services. When starting with code you have to write more plumbing code, but you have more freedom. And on the other side starting of the schema reduce all the plumbing code and gives a faster start.
+
In this blog we first create a simple JPA enabled Spring Boot service. Then we add step-by-step GraphQL features to the service. We start from the schema and show which small additions you have to do to our application. Starting from a schema will not result in generated coded, you still have to write the matching code yourself. For the Java implementation we will use the libraries from graphql-java-kickstart.
+
Sample Model and API
+
For simplicity we use in this blog a simple domain model. Consisting of two classes. A person and notes, a person can have multiple notes. And a note always has one author, a person.
First we create a simple Spring JPA services, to store the Notes and the Authors of type Person. The implementation will be straightforward. We create an entity class for Person and Note. Fore readability the imports and other plumbing is omitted. See GitHub for a complete implementation.
So that is all to create a simple service with basic JPA (persistence) capacity. Time to focus on adding GraphQL to our service.
+
Implement a query
+
As base library we use graphql-java-tools this library requires a GraphQL schema to implement GrahpQL. The first step is to define a schema for our queries. The schema defines our GraphQL root, for now only the query. Next the queries we will implement and the types we use. We define two queries. The first one to look op a note and the second one to find all notes based on a filter. Next, we define the types. As you can see a Note has a relation to a Person object. By querying for a note we can get immediately the authors name for example.
We have now defined a first version of a GraphQL schema. The second step is to connect the schema to the Note Service. GraphQL java tools expects for each query, a resolver function. Therefore, we need to implement GraphQLQueryResolver from graphql-tools.
This is all, we have to do to be able to handle the queries as specified in the schema. We can now start the spring boot application and open http://locahost:8080/graphiql and query form some. Graphiql is a simple IDE to execute GraphQL queries. A much richer tool to play with your queries is GraphQL Playground. Actually no fun yet, we can only execute queries without getting any result. We simply have an empty system. We need a way to add notes to the system. In GraphQL you mutate data via a mutation. In the next step we add a mutation to the schema and update the code for executing the mutation.
+
In the schema we add the mutation to the root and add a mutation to add a note to the Mutation type. Quite similar to defining the query above. For the input we define an input object for both Note and Person.
+
+schema {
+ ...
+ mutation: Mutation
+}
+
+type Mutation {
+ addNote(note: InputNote!, author: InputPerson!): Note
+}
+
+# Input type for the author of the note
+input InputPerson {
+ name: String!
+ email: String
+}
+
+# Input type for a new Note
+input InputNote {
+ note: String!
+}
+
Similar to implementing the query we have to implement an interface and declare methods for the mutation. Mutations have to be implemented in a class implementing the interface GraphQLMutationResolver. In this implementation we defined for the input objects converters to convert the GraphQL input object to domain objects that can be consumed by the service.
Add some more notes and experiment with the query. All standard features will work such as aliases, fragement and variables.
+
Implement Subscriptions.
+
Another cool feature is the subscriptions. With a subscription you can subscribe to updates via a websocket. The same philosophy is followed for the subscription. You will gat your data in same structure as specified in the subscription call.
+
To add subscription to our Java implements, we update the schema and add code to handle updates. To add a subscription for new notes we update the GraphQL schema first. Again update schema root and add the subscription, define the available subscriptions and add the required types.
+
schema {
+ ...
+ subscription : Subscription
+}
+
+type Subscription {
+ # Subscribe to all new comments
+ notes: NoteUpdate!
+}
+
+type NoteUpdate {
+ # The note text
+ note : String
+ # The name of the author
+ authorName : String
+ # Time when the note was received
+ createdOn : String
+}
+
And once again we have to implement a resolver, this time we implement the GraphQLSubscriptionResolver.
The notePublisher is responsible to update messages via a WebScoket so the subscribers get the update. For the implementation check out the GitHub repo. Time to test the subscription. An easy way to test is topen the GraphiQL editor on two screens. Invoke in the first one the subscription and in the second one add a note. You will see you get an update of the new note on the first screen.
Another nice feature provided by graphql-spring-boot is the test utils. Testing your GraphQL queries becomes really simple. The library provide a GraphQLTestTemplate which makes it easy to invoke the GraphQL endpoint by with a query. The result is the JSON object. Let’s have a quick look on some code. First define the query in queries/find-notes.graphql.
That is all, of course you should add some assert to check the content as well. For more details check out the GitHub repo of this blog.
+
Some thoughts
+
The example above shows how you can enable your Spring Boot service with a GraphQL API. Adding a GraphQL layer to your existing (simple) service is fairly easy with the tools. But GraphQL will become really powerful when you combine multiple (REST) API in one GraphQL endpoint to create a fluent and understandable API.
+
If a java based approach is in that case the best way to go, I don’t know yet. In that case it could be interesting to dive into Rejoiner.
\ No newline at end of file
diff --git a/2019/04/02/react-native-navigation-and-clojurescript/index.html b/2019/04/02/react-native-navigation-and-clojurescript/index.html
new file mode 100644
index 00000000..c9533d14
--- /dev/null
+++ b/2019/04/02/react-native-navigation-and-clojurescript/index.html
@@ -0,0 +1,579 @@
+React Native Navigation and ClojureScript | 040code
React Native Navigation and ClojureScript
A beginner's guide to realizing proper navigation in a React Native application using ClojureScript and shadow-cljs
That’s a big bunch of words. Let me briefly introduce them to you, so that I can
+then dive deeper into some of them. “Beginner” in this case means that, a week
+before I started doing this, I had practically zero experience with
+Clojure(Script) and React Native. I do have quite a bit of experience using
+React, and also writing native iOS applications (Swift/Objective-C).
+
“Guide” means that I’m going to explain to you how I accomplished my goal.
+You’ll have to interpret it and follow along if you want to do the same. Future
+versions of any of the tools may break this guide.
+
React Native Navigation is a framework for React Native. Which is a
+framework for building cross-platform apps for iOS and Android using JavaScript,
+but you probably already knew that part. ClojureScript is a functional
+programming language. It’s a variant of Clojure that compiles to JavaScript.
+“Proper navigation” is, admittedly, a bit subjective; I’ll explain this part in
+more detail below. Finally, shadow-cljs is the toolchain I used
+as an easy way to integrate a bunch of ClojureScript code into an originally
+Javascript-based project.
+
A summary of this guide is:
+
+
Setup a new React Native project (using react-native cli).
+
Add shadow-cljs to be able to use ClojureScript in that project.
+
Add and setup react-native-navigation dependency.
+
Create a bit of infrastructure on the ClojureScript side to be able to use
+the React Native Navigation APIs while using shadow-cljs’ hot reloading.
+
+
So let’s get started. I executed all these steps exactly as I’m describing here,
+the result of which you can find in my GitHub repo. Commit messages
+include all details about the commands I used. If you want to read my
+rant about wanting to use native navigation, keep reading till the very end of
+this post :)
+
+
Please note that this post does not include all the code that you need
+to make it work, that would be too much for a web page. Please refer to
+the final solution in my GitHub repo if you want to reproduce all of it.
+
+
1. Create React Native Project
+
This step is very easy. First, if you haven’t got it yet, install
+react-native-cli:
+
npminstall-g react-native-cli
+
Then create a new project:
+
react-native init CLJSReactNativeNavigation
+
So far so good, nothing new here, let’s move on.
+
2. Add shadow-cljs
+
We’re going to use the standalone version of shadow-cljs,
+as recommended. So we add the npm dependency for shadow-cljs:
+
yarnadd--dev shadow-cljs
+
Now we need to configure shadow-cljs a bit. We create a file called
+shadow-cljs.edn and give it the following content:
+
{:source-paths
+ ["src/main";; production code
+ "src/test"];; yes we're going to add tests as well!
+
+ :dependencies
+ [[reagent "0.8.1"]]
+
+ :builds
+ {:myapp;; the target definition
+ {:target:react-native;; the target type
+ :init-fn myapp/init ;; react native's entry point
+ :output-dir"build"}}};; where to put the built JS
+
This is telling shadow-cljs that there is a target called “myapp”, that
+it’s a React Native target, that the entry point of our app is the function
+init in the myapp namespace, and that the output should be written to the
+folder build.
+
Now let’s create that myapp namespace. In src/main create a file called
+myapp.cljs with the following contents:
When using shadow-cljs, you also get its variant of hot reloading. You don’t
+need to use the developer menu to enable it, but you do need to add a bit of
+code to make it work: (a) you need to had a function that performs the reload,
+and (b) you need to enable hot reloading in the config.
+
Add reload function
+
Let me just give you the code of myapp.cljs and then explain what’s going on.
The text content (“Hi Shadow!”) is extracted into a separate component
+(content), because the call forceUpdate that we’ll use
+updates everything below the application root component, not the root
+component itself.
+
The app-root component is annotated with a component-did-mount handler
+(updatable-app-root). This handler stores the actual JavaScript object
+that represents the root component into the atom component-to-update.
+
A function reload is added, which takes the value of that atom and calls
+the method forceUpdate on it.
+
+
One final step remains: enabling hot reloading in shadow-cljs.edn. It’s new
+content is:
+
{:source-paths
+ ["src/main";; production code
+ "src/test"];; yes we're going to add tests as well!
+
+ :dependencies
+ [[reagent "0.8.1"]]
+
+ :builds
+ {:myapp;; the target definition
+ {:target:react-native;; the target type
+ :init-fn myapp/init ;; react native's entry point
+ :output-dir"build";; where to put the built JS
+ :devtools{:autoloadtrue}}}};; enables hot-reloading
+
If you reload the app, changing the text in the content component should
+cause the app to automatically update!
+
3. Add React Native Navigation
+
This bit is a somewhat tedious I’m afraid. You’ll have to go through the
+instructions to setup React Native Navigation. Add the npm
+dependency, update your Xcode project, update iOS source code, update Android
+build files, and update Android source code. Tip for Android part: Please don’t
+blindly copy-paste. Some of the instructions refer are not up to date, some
+parts are not really needed, etc. If you want you can have a look at
+how I did it.
+
Obviously you’ll skip the last step of the instructions, namely the part where
+the JavaScript code is updated to use React Native Navigation. We’ll address
+that in ClojureScript next.
+
4. Integrate and Wrap
+
Ok, roll up your sleeves, because here comes the interesting part.
+
Wrapper functionality
+
In JavaScript we would need to do something like this:
Obviously in ClojureScript we need to do something similar. There’s a problem
+though: we need a handle to the actual JavaScript component in order to call
+forceUpdate on it (for hot reloading). React Native Navigation has made the
+design choice that it creates new root components for screens that you push
+on the navigation stack. So also for those components we need a handle and
+call forceUpdate. We accomplish this by not registering the component itself
+with Navigation, but a wrapper of that component.
+
This causes another problem though. React Native Navigation gives components
+that you register a componentId. It uses this for its internal registration
+so that it can make navigation work. For example, when you push a new screen
+onto the navigation stack, it uses the componentId to find the screen from
+which you are pushing. The problem is that we registered the wrapper, but
+we’re navigating from the wrapped component. Which does not have a
+componentId, because we never registered it with Navigation. Solution:
+make the wrapper in such a way that it passes its componentId on to the
+wrapped component.
+
But there is more! React Native Navigation defines some additional life cycle
+methods, such as navigationButtonPressed. And for that to work, you need
+to call Navigation.bindComponent. So our wrapper also calls
+bindComponent and forwards navigationButtonPressed. Forwarding other
+life cycle methods is left as an exercise for the reader.
+
Here’s the main code for the wrapper (full version):
Initially, in myapp/init we called React Native’s registerComponent. Now
+we call our env/register instead. We could just call (env/register "App"),
+but we want to pass some options for the navigation bar.
+
Furthermore we need to call the Navigation.events().registerAppLaunchedListener
+JavaScript function to set the navigation root for our app.
That’s it! Done! Just four simple, easy, almost trivial steps! Well maybe not
+so trivial. I guess a ClojureScript beginner couldn’t have come up with this.
+Well, in fact I am a ClojureScript beginner and I did not come up with this
+solution myself. All the credits go to Thomas Heller, the author of
+shadow-cljs. He has been amazing in his support by answering
+all of my beginner-level questions, and then he ended up conjuring this
+solution and committing it to my repository. He actually spent hours on this
+I believe, and that level of support from a community is truly awesome
+(and rare). He doesn’t seem to be advertising it very much, but you can become
+his patreon.
+
Why React Native Navigation
+
What’s special about React Native Navigation is that it is implemented using the
+real platform native components, specifically
+UINavigationController on
+iOS. I seem to be at odds with most of the rest of the world on this, but I
+happen to think that it is very important to present the user with an
+experience that is (as much as possible) identical to that of native apps. You
+shouldn’t be able to tell from the user experience whether the app was written
+using native technology or cross platform technology. Not even when you update
+your OS to a new major version. So if my app uses a navigation stack, it has to
+be the native one. Maybe I’m more sensitive to this then others, but I get
+really upset by apps that don’t support the normal gesture for going back up the
+navigation stack. I also get annoyed when the animation that is used while going
+back is slightly non-standard. Using the native components is the only way to
+accomplish that. Other components can come close, but not close enough for me.
+
Versions
+
Like I said in the beginning, future versions of any of the tools may break
+this guide. So it is only fair to mention which versions I was using for this:
+
+
+
+
Tool
+
Version
+
+
+
+
+
node
+
11.10.1
+
+
+
npm
+
6.5.0-next.0
+
+
+
react
+
16.8.3
+
+
+
react-native
+
0.59.3
+
+
+
react-native-cli
+
2.0.1
+
+
+
react-native-navigation
+
2.16.0
+
+
+
shadow-cljs
+
2.8.26
+
+
+
Xcode
+
10.2
+
+
+
iOS SDK
+
12.2
+
+
+
Android SDK
+
API levels 26, 27, 28
+
+
+
+
Build tools 27.0.3, 28.0.3
+
+
+
+
System images: android-28
+
+
+
\ No newline at end of file
diff --git a/static/2019/04/18/cloudrun/build.json b/2019/04/18/cloudrun/build.json
similarity index 100%
rename from static/2019/04/18/cloudrun/build.json
rename to 2019/04/18/cloudrun/build.json
diff --git a/static/2019/04/18/cloudrun/deploy.json b/2019/04/18/cloudrun/deploy.json
similarity index 100%
rename from static/2019/04/18/cloudrun/deploy.json
rename to 2019/04/18/cloudrun/deploy.json
diff --git a/2019/04/18/cloudrun/index.html b/2019/04/18/cloudrun/index.html
new file mode 100644
index 00000000..cd8fb1be
--- /dev/null
+++ b/2019/04/18/cloudrun/index.html
@@ -0,0 +1,249 @@
+Cloud Run | 040code
Last week at the Google Next in San Francisco, Google announced Cloud Run. A new platform to enable developers without any infrastructure to run Docker containers on Google Cloud.
+
+
Cloud Run is a managed compute platform that enables you to run stateless containers that are invocable via HTTP requests. Cloud Run is serverless: it abstracts away all infrastructure management, so you can focus on what matters most — building great applications. It is built from Knative, letting you choose to run your containers either fully managed with Cloud Run, or in your Google Kubernetes Engine cluster with Cloud Run on GKE.
Time the see how this new feature on the Google Platform really works. There are two ways to use Cloud Run. You can choose to run the container serverless, which means you do not have to create any infrastructure. The other option is deploy the container via the Cloud Run to a GKE cluster. From developers perspective there is no difference, the developer can still use the same Cloud Run API.
+
Lets quickly compare Cloud Run to some other option to run an application sereverless in the cloud. For example you can use Now from Zeit to quickly run a serverless application build in many languages, some major are Node, Rust, Static, Python, Go and NextJS. The free tier get you started quickly. In the paid tier you pay for traffic and invocations. Fargate the the AWS approach to enable serverless Docker containers. But in Fargate you container is always running and you pay per second for CPU and memory. Read here more about deploying to Fargate.
+
Deploy the 040 blog to Cloud Run
+
Why not see if we can publish this blog to Cloud Run. Starting is quit simple, just follow the Google documentation. Go to the Google Cloud console and navigate to Cloud Run. You will see here that deploying a container requires two steps. First publish the Docker image to the Docker registry in your Google project. Next deploy the docker image to Cloud Run.
+
Prepare your image
+
On the Cloud Run create service page is clearly mentioned that the containers needs support injection of a port that listen for HTTP traffic via the environment variable PORT. This is for example easy to support via the Spring Boot framework. But for the blog we allready have an NGINX container which not support injection of the port.
+
The first step will be to enable port injection to our blog runtime container. Which can be done by starting the container via a wrapper script which update the NGNIX configuration. We define a simple server configuration in which we can replace the NGNIX_PORT environment variable via envsubst.
Finally we update the Dockerfile to add the new assets and ensure the start.sh script is invoked at container start.
+
FROM jekyll/jekyll:3.8.3 AS build
+
+... OMITTED ...
+
+FROM nginx:1.15.3
+COPY--from=build /build/_site /usr/share/nginx/html
+
+### copy wrappers and configuration template
+COPY nginx/default.conf /etc/nginx/conf.d/mysite.template
+COPY nginx/start.sh /usr/bin
+
+CMD ["start.sh"]
+
This was the hard part, our image is ready. You can test your image locally. For example on port 8081
+
Docker build -t blog .
+Docker run -e PORT=8080 -p 8081:8080 blog
+
Publish the image to Google Cloud
+
You can publish the Docker image to the gcr.io registry by building locally and push the Docker image. Or by using Google Cloud Build to build the container. We use the power of the cloud and let Goolge build the image.
In the web interface it is straightforward to deploy to Cloud Run. In the cli it is not hard bu you need te be aware that cloud run is beta which requires to install the beta components.
+
gcloud components install beta
+
Next you specify the same properties as you should do via the web interface. Currently Cloud Run is only available in the region us-central1. Since we run a public site we allow unauthenticated traffic.
+
gcloud beta run deploy blog \
+ --image gcr.io/<project-id>/blog:latest \
+ --region us-central1 --allow-unauthenticated
+
+
+
Some final thoughts
+
The first impression is that Cloud Run is quit easy to set up. Good to mention is that is based on the Open Source project Knative. At the moment the feature is in beta and there is no support in other regions.
+
In this post we show just the basics of CLoud Run. It is really super easy to deploy you docker containers to the cloud for low costs. Could be handy for example to review branches. But there is a lot more, containers can be protected and only accept authenticated traffic. Furthermore with integration to Cloud Build it is easy to setup a CI/CD pipeline for GitHub, Bitbucket and Cloud Source Repository. With GitLab CI setting up a pipeline for Cloud Run is almost a no brainer. And for deployment GitLab already supports Knative to deploy to a Kubernetes cluster.
\ No newline at end of file
diff --git a/2019/04/20/discovery-agent/index.html b/2019/04/20/discovery-agent/index.html
new file mode 100644
index 00000000..96f30f05
--- /dev/null
+++ b/2019/04/20/discovery-agent/index.html
@@ -0,0 +1,228 @@
+Discover the exposed port | 040code
Update 2018-02-14, since Amazon has introduced a feature to extract meta data at runtime in a container the discovery agent described in this post becomes obsolete for the ECS case, see more this post for more details.
+
This post describes a solution for implementing the discovery of the unknown exposed port by a Docker container. A specific situation where the problem occurs is when you deploy Spring Cloud micro services as Amazon ECS services to the cloud.
+
The problem
+
Setting up a micro services architecture requires that there is a way services can find each other. Spring Cloud provides a services discovery service out of the box: Eureka. Services running in the micro services landscape are responsible to register themselves to the discovery service. Then, any service running in the micro services landscape can locate them based on their name. Deploying those services to a docker based cloud, such as AWS, does however create some difficulties.
+
A service running in a container does not know the external exposed port. The port map can be fixed, for example by mapping the external port on the same port as the internal one but this will eventually cause port conflicts. Besides that, you are not able to scale the container on the same host. Therefore, it make sense to use automatic port assignment. But in that case, services must be able to discover the port, assigned by docker, for registration.
+
Running micro services in docker containers on AWS ECS looks as follows.
Each micro service is defined as task which will run as service (a docker container). The docker container runs on a Amazon EC2 instance that also runs docker agent to manage the cluster. Unfortunately, the agent has no API to lookup the exposed port.
+
Solution
+
To lookup the exposed port an extra agent is added to the EC2 instance. The discovery agent uses an API to discover the port for a given container id, internal port and protocol. The only drawback is that the discovery agent needs access to the docker socket which is a potential security risk.
Now the agent is running, it can be tested. For testing we start another container, remember the problem is that we need to find the exposed port from within a container. The local ip address will be added as extra host to the container. The test container is just a linux (CentOS) container containing the tool curl and for which port 80 is exposed to a random port.
We are now in the shell of the CentOS container, the host file contains a DNS entry that points to the host. First, we have to search for the id of the current container.
+
CONTAINER_ID=$(cat /proc/self/cgroup | grep "cpu:/" | sed 's/\([0-9]\):cpu:\/docker\///g')
+
Then, we ask the discovery agent for our external mapped port by sending the container id and the internal port.
The result is a JSON string container with the hostIp and hostPort, for example:
+
{
+ "HostIp": "0.0.0.0",
+ "HostPort": "32779"
+}
+
Alternative to passing the host ip address, it is possible to register a static ip address and use this to find the dicovery agent.
+
sudo ifconfig lo0 alias 172.16.123.1
+
Now we can discover the exposed port, let’s give some direction on how this can be combined with Spring Cloud on ECS. First deploy Eureka (service discover) as container. Register an ALB (internal) to the service. Now we have an endpoint that we can use to inject the service that needs to register itself. Next create a spring service that is aware of discovery. Before the services can be started, the exposed port needs to be found. Add the snippet below to the start script to start the services in the container.
The last step is to update the application.yml and add a few lines to force the ip address and port during service registration.
+
eureka:
+ client:
+ serviceUrl:
+ defaultZone: ${EUREKA_URL} // Needs to be injected as environment variable
+ instance:
+ preferIpAddress: true
+ ip-address: ${DOCKER_HOST}
+ non-secure-port: ${EXPOSED_PORT}
+
Alternatives
+
Discovery of the unknown exposed port is a highly discussed topic and there are many alternative solutions, Please see docker issue, which is a good starting point for alternatives. The following alternatives came to my mind:
+
+
Same approach, but run the discovery service directly on the host.
+
Implement the discovery capability in each servie as a library, but the drawback is that all the services need the docker socket.
+
Choose another discovery mechanism for example based on ALB.
\ No newline at end of file
diff --git a/2019/07/01/cross-cluster-service-mesh/index.html b/2019/07/01/cross-cluster-service-mesh/index.html
new file mode 100644
index 00000000..22ae19ce
--- /dev/null
+++ b/2019/07/01/cross-cluster-service-mesh/index.html
@@ -0,0 +1,462 @@
+Service Mesh | 040code
The last years we have seen a huge adoption of micro services architectures. Typically micro services brings a lot ff benefits such as flexibility, modularity, autonomy. But deploying and managing micro services architectures brings other difficulties. How dow you know what is running, how do you know your services are compliant? Another pattern that we see is that micro services ar typically heavy loaded with common dependencies for logging, authentication, authorization, tracing and many more cross cutting concerns.
+
A service mesh brings transparency to the chaos of micro services. A mesh can help with implementing, enforcing and managing requirements such as authentication, authorization, traceability and, data integrity. It also provides features as orchestration and collection of telemetry. There are currently several service meshes out, for example App Mesh from Amazon, Consol from HashiCorp, Linkerd from the CNNF and Istio launched by IBM, Lyft and Google in 2016. Istio is a fully open source solution based on the high performance proxy Envoy.
+
Another trend in the industry is multi cloud or hybrid cloud. Confusing terms, no clear definitions. But it look likes common sense when we speaking about multi cloud we point to combining public clouds. And hybrid cloud is when you mix and match public with private cloud. When you start running cross cloud clusters it becomes even harder to manage all your micro services. Be confident that policies are implemented well or compliant. In this multi / hybrid topology, abstraction from cross cutting concerns to a mesh becomes even more important. In this blog we go to build a multi cloud Kubernetes cluster with an Istio service mesh.
+
A bit more about a Service Mesh
+
Before we dive into building a multi cloud service mesh, a few words about how a mesh works. An Istio mesh consists of two parts. A data plane, intelligent proxies (Envoy) deployed as sidecars. Those proxies mediate and control the network traffic. The second component is the control plane which manages and configures the proxies, and enforce policies.
+
+
To create a cross cluster mesh with Istio there are two topologies. In the first scenario there is a single control plain that controls all the cluster. In the second scenario a single control plain is deployed to every cluster and you have to ensure the same configuration is pushed to each cluster. In this post we will create an example based on the second option, a control plane in every cluster.
+
+
Building a cross cluster mesh
+
Lets get started with building a multi cloud service mesh. A quite similar similar example as below is available on my GitHub. The example on GitHub is scripted with a set of simple shell scripts and created 3 clusters in 2 clouds. For this post we limited our selves to just 2 clusters in 2 clouds.
+
We will create 2 clusters and install in both clusters the Istio service mesh, one cluster on AWS (EKS) and the second on Google (GKE). For creating clusters you need to setup your environment with the right tools and credentials. For AWS the AWS CLI is required, for Google you need the Google Cloud SDK. Furthermore you need to install kubectl to interact with Kubernetes. And helm as packagem manager for Kubernetes.
+
Setup and configure EKS cluster
+
First we create EKS clusters with eksctl. But cluster has a quite minimal configuration. See the page of weaveworks ekstcl or AWS fom more details. Creating a cluster roughly takes 20 minutes. With the command below we create a default EKS cluster, with only one node it’s own VPC.
You should now have a Kubernetes cluster running on AWS. Next we download, install and configure Istio for our service mesh. In the steps below we use for simplicity the sample certificates provided by the Istio distribution. Should I really mention that you should replace those certificates for a real life setup…
The last step could take a minute or so, check with
+kubectl get crds | grep 'istio.io' | wc -l if the Istio custom resources are created. Once finished there should be 53 custom resources created. The last step for installing the service mesh is to create the Istio control plan.
For this example we simply enable sidecar injection for every pod in the default namespace. The sidecar will function as proxy and intercepts all network traffic to the containers in the pad.
Services in a local Kubernetes share a common DNS suffix (e.g. svc.cluster.local). To ba able to route our remote service we have to stub the Kubernetes DNS for the domain name .glabal. Service in a remote cluster can then be addressed with the naming convention <name>.<namespace>.<global>. Therefore we need to stub the Kubernetes DNS. EKS is using CoreDNS, so we add the config map below.
The service mesh on EKS is now ready to start serving applications.
+
The Postcard sample application
+
In this blog post we use a simple polyglot postcards application. It is always fun to write an application in yet another language. The postcard application consists of twe services. The first service is the greeter, the greeter is written in NodeJS and generates a webpage with a postcard. The message on the postcard should be provided by the second application, the messenger. THe messenger is a Rust application that will run on the second cluster. I returns a string message based on configuration.
+
+
+
+
+
+
+
+
+
The greeter app will print an error message in case the messenger is not available. We deploy now the greeter to the Kubernetes cluster on AWS. We create a standard deployment for the pod, a service, a Istio Gateway, and a Istio Virtual service. The Istio resources are created to make the postcard app public available via an ingress. You can check out the configuration files here. During deployment we update the configuration with name of the cluster.
+
# Deploy greeter pod and service
+curl -L \
+ https://raw.githubusercontent.com/npalm/cross-cluster-mesh-postcard/master/mesh/demo/greeter/greeter.yaml \
+ | sed "s/CLUSTER_NAME/${CLUSTER_NAME}/" | kubectl apply -f -
+
+# create gateway for greeter service
+kubectl apply -f \
+ https://raw.githubusercontent.com/npalm/cross-cluster-mesh-postcard/master/mesh/demo/greeter/gateway.yaml
+
+
Once deployed we should already be able to see our postcard via a browser. You can construct the URL with the command below. It can taka few minutes before Load Balancer is ready to accept traffic.
+
export ISTIO_INGRESS=$(kubectl -n istio-system \
+ get service istio-ingressgateway \
+ -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')
+open http://${ISTIO_INGRESS}/greeter
+
The postcard shows no message from the second cluster. But our greeter app is up and running. The next steps is creating the second cluster.
+
+
+
+
+
+
+
+
+
Setup and configure GKE cluster
+
For the second cluster we create a GKE cluster on Google Cloud. You can replace by a second cluster in Amazon or one in another cloud. The first step is similar to creating a cluster in AWS but now one in Google. Open a new terminal to avoid conflicting environment variables.
The cluster will created in roughly 5 minutes. Next you have to install Istio. The steps are exactly the same as above. Only the last step where we configure the DNS is different since GKE use kube-dns instead of core-dns. Execute the command below to stub the DNS.
On the second cluster we deploy the messenger app that simply sends a message back. The message can be configures via an environment variable. Install the messenger the app with the kubectl command below.
+
# Deploy greeter pod and service
+curl -L \
+ https://raw.githubusercontent.com/npalm/cross-cluster-mesh-postcard/master/mesh/demo/messenger/messenger.yaml \
+ | sed "s/MESSAGE_TEXT/All good from Google Cloude/" \
+ | kubectl apply -f -
+
Only one step left now. We need to configure the first cluster with a service entry so the mesh knows how to route calls for messanger.default.global. We switch back to the terminal where we have created the first cluster on Amazon. And lookup using the kubeconfig from the second cluster the ip address of the ingress loadbalancer. Next we create a service entry.
+
# Lookup the ingress ip address
+export CLUSTER_GW_ADDR=$(kubectl \
+ --kubeconfig=kubeconfig-istio-2.config \
+ get svc --selector=app=istio-ingressgateway -n istio-system \
+ -o jsonpath='{.items[0].status.loadBalancer.ingress[0].ip}')
+echo $CLUSTER_GW_ADDR
+
+# Create a service entry
+kubectl apply -f - <<EOF
+apiVersion: networking.istio.io/v1alpha3
+kind: ServiceEntry
+metadata:
+ name: messenger
+spec:
+ hosts:
+ - messenger.default.global
+ location: MESH_INTERNAL
+ ports:
+ - name: http1
+ number: 3000
+ protocol: http
+ resolution: DNS
+ addresses:
+ - 127.127.42.69
+ endpoints:
+ - address: $CLUSTER_GW_ADDR
+ ports:
+ http1: 15443 # Do not change this port value
+EOF
+
Now our postcard application should get the message from the second cluster, go back to the browser and refresh the page. You should now see card as below.
+
+
+
+
+
+
+
+
+
That is all you need to do for creating a cross cluster service mesh.
+
Cleanup
+
The cleanup steps below assume you created fresh cluster and don’t use them for hosting other applications. The Amazon cluster was created via eksctl. This tool actually create cloud formation stacks in Amazon. Before you delete the stack, you need to delete the load balancer created by Istio.
Now you can delete the cluster, the deletion is a asynchronous process. Ensure you verify the deletion went well. In case it fails, happens many times to me. Delete the VPC and retry the delete via cloud formation.
The example used in the blog is inspired by a great talk from Matt Turner at the KubeCon - CloudNativeCon 2019 in Barcelona. His talk and example is available in the blog post Cross-cluster Calls Made Easy with Istio 1.1.
Last year Amazon released Amplify Console, a feature to easily deploy any branch for your web application to the cloud. Connected Git systems are, GitHub, GitLab, BitBucket and CodeCommit. Which limitted the usage if you code resides in a self hosted Git. Later Amazon added a manual deploy option to deploy based on a zip file. The zip archive can be uploaded via the web interface, an HTTP link or via an S3 bucket. Would be handy if we can automate this process to utilize this AWS feature in a self hosted Git server.
+
+
+
+
+
+
+
+
+
Let’s explore how we can automate the manual option for a self hosted Git server.
+
Step 1 - Create your app
+
The first step is to create you app in Amplify, you can do this in the web console but, we don’t like manual steps. Creating your app is as simple as the the one liner below.
+
exportAPP_ID=$(aws amplify create-app --name blog | jq -r'.app.appId'
+
The output we capture in a variable APP_ID for later use.
+
Step 2 - Create branch
+
Before you can deploy a branch, we need to create a branch in Amplify. To avoid we run the command if a the branch already exists we do a quick check. You can also add a password to protect our web app via basic authentication.
Finally we can deploy our app. Finding a way to deploy automated in the case Amazon was call manually, was a bit hard due to the lack of documentation and posts. For doing a deployment from a non connected Git (aka manual deployment) you have the option in we web console to provide a public link, upload file, or ink to file in S3. The AWS CLI docs does not mention the S3 option. The post Deploy files stored on Amazon S3, Dropbox, or your Desktop to the AWS Amplify Console suggest to upload the file ot a S3 and create a lambda that will a deployment for a new file. We choose a slightly different approach. First we send our app as zip to S3.
Now we need to find a way to trigger the deployment. The option --source-url for starting a deployment does not mention that a S3 url is an option as well. So we tried it and it works perfect.
I’ve been visiting Full Stack Fest from the very first edition back in 2012. I think it’s a remarkable single track conference with topics which are really relevant for modern developers.
+
Previous years we also wrote blogs about it, these can be found here:
This year the conference is three days and there’s no clear distinguish between frontend and backend days.
+The location has changed. We’re now in Sitges, a beach place near Barcelona.
+
The first day - 2019-09-04
+
Host
+
We were welcomed by Sara Vieira. She works at codesandbox, which we really like. I also like her humor. She is very direct and I like it very much.
+
Lee Byron - We’re gunna program like it’s 1999
+
I’ve seen Lee Byron at ReactEurope in Paris and this talk had some of the same elements. I really admire Lee for all his work and in showing everybody how we need to continue to change the abstractions, syntax and mental models. He shows the past in the future. Also liked the Svelte code example.
I’m not sure anymore how the talk was called, but she had an amazing talk on making developers tools.
+In her work for Rust / WebAssembly she sees that it’s very important to understand how developers use your tool. You don’t make a product, your enriching a developer.
+There is something like Language Strangeness Budget. When creating tools or clis, this is very important to keep in mind.
+
Our products are like the Fire Flowers. They turn Mario (a developer) into a Fire Mario! Senior developers should become Fire Mario Enablers.
Nice comparison with Baseball. We’re using a JAMstack for this blog (gatsby). Brian showed what the benefits are of JAMstacks. The static pages can be served everywhere. Github has some good workflows to automatically publish sites.
+Showing Github Actions, now, netlify and so on.
+He showed a project Open-sauced. It’s a frontend with github API as backend.
Interesting talk on how Yan migrated a monolith to a s8s environment. How it saves a lot of money. How it impacted the lead times for features and how he approached it. He showed a little monolith diagram of the application and a big diagram with the current lambda implementation. The latter looked a lot bigger, but as a matter of fact, was a lot less complex. The first diagram had simple boxes with a lot of functionality in it. The last one had a lot of boxes with isolated functions.
+He suggested to put all features in separate github repositories and give the task of deployment to each repository. Monorepos were not easy to use.
A programming language for a continuous delivery world.
+
by Ellen Chisa
+Ellen complained about how long it takes to build a simple app. She wanted a simple app to register her mood, but months later, she was finally ready with setting up Rails, Heroku, Twillio, and so on. The end result is workable for herself, but can not even be shared, so there is something wrong. Too much accidental complexity.
+
She’s no co-founder of dark-lang. Dark is a holistic programming language, structured editor, and infrastructure, for building backend web services. It’s for frontend, backend, and mobile engineers.
+
This is all in Alpha testing, but she did some cool live coding to show the basic of dark-lang. We were all very impressed and kinda speechless.
+
Applied Accessibility: Practical Tips for Building More Accessible Front-Ends
Very good presentation on the topic of a13y. She showed why it’s so important and how you can improve your apps.
+Although it was already late, people wanted to hear her complete story and that’s a compliment!
+
It has a nice contrast with the prior presentation… this feels like a lot of accidental complexity.
Una did a great job in showing the possibilities of Houdini and how this can change the way websites will look. Houdini is not available in all brwosers, but it’s getting there. In Chrome you can already use it when you enable a flag; chrome://flags/#enable-experimental-web-platform-features Once enabled, you can start programming in CSS. See extra-css for more examples.
Angie showed how you can miss a lot of things, because people are focussed on only a particular part of the app. She showed a clip Whodunnit? to prove it.
+By running automated tests, you can improve a lot. You’re testing exactly what text should be on the screen and what buttons to click. This seems all fine, but can still cause problems. Are the texts visible? Are they overlapping? Are the labels located correct? These kind of bugs are not rare. All big websites sometimes suffer from these errors. Angie showed a framework which takes screenshots and uses ML to analyse the differences with a baseline. Not pixel by pixel, because that is very brittle.
Paul explained why P2P on the web deserves another chance. He works on the beaker browser. He showed the history of P2P. Napster, Kazaa, LimeWire, and so on. For some reason P2P was very normal, but for some reason P2P fell off. That’s a pity. Beaker re-introduces P2P nodes with the dat:// protocol. P2P gives us Decentralization, Data Privacy, Data Ownership, No dev-ops and the code is truely open source. No hidden boxes on a server, just all code on each peer. Personally I’m passionate about P2P. We’re building an Identity Box using IPFS.
This was a very advanced talk, with probably the most impact on the world of software. WASI (WebAssambly System Interface) is an initiative to run WebAssembly outside the browser. WebAssembly Interface Types interoperate with all the things; languages, wasm, but also browsers and operating systems. This can have impact on everything we currently know. The slides by @codecartoons were very beautiful. I would definitely recommend watching the presentation in the link below.
Philip Wadler has a lot of energy, especially when he’s excited about something. In this case functional programming and proofing that your program is doing what you claim it does. In a language called Agda, he can write proofs. He uses github to openly write a book, and emphasises how wonderful that is. Functional Programming is becoming more and more accepted in for example banking. Philip uses Haskell and Agda in Blockchain to prove SmartContracts.
Sarah showed us a lot of examples how animations will improve the user experience. She showed a lot of (Vue / Nuxt) code examples how to achieve this.I recommend watching the presentation.
A great talk on how you can make teams work. I especially liked the part where she talks about the mismatch between team goals and individual incentives. When they are not aligned, it’s very hard to build a team. I recognized this a lot.
Daniel is part of the TC39. He explained how the stages work in TC39. He’s currently responsible for 3 proposals in Stage 3; BigInt, Class Public Instance Fields & Private Instance Fields and Private instance methods and accessors, and 1 proposal in Stage 2; Decorators. The last one you might already have seen in ember, Angular or MobX. Aligning the implementations is a very hard job. They all seem to have a slight different idea on decorators. It’s up to Daniel (and TC39 which have members of the ember, angular in it) to come up with a solution that doesn’t break the web.
Daniel is the creator of curl, so he knows a few things about HTTP. HTTP/2 did improve a lot, but Daniel explained how the boxes in between the server and the client are very helpful. They see traffic passing by and they start helping us. This might look useful, but it is not. Especially not when you want to improve the network layer. That’s why HTTP/3 will use UDP instead of TCP. Of course this gives a lot of other interesting problems. Company networks often see UDP as a DDoS attack and blocks it, but in order to improve the speed of the internet, we need to change the protocol. The Head-on-line problem shifted from HTTP level to TCP level when switching to HTTP/2. This causes all streams to be blocked if a package is missing. HTTP/3 with QUIC can resolve the HOL problem on TCP level. This means only one stream is blocked when a package is missing. But it’s a long way… There are a lot of boxes in between..
I love Rust, so I’m not going to talk a lot about this presentation. Steve explains a lot about it and how Rust and the Rust Toolchain can be used for WASM. I recommend everybody to see the presentation. WASM (together with WASI) can be the future of s8s. Currently Docker is the container, but a browser can also be the container with WASM / WASI.
On October 2019, AWS released a nice feature on AWS ECR (Elastic Container Registry). They introduced the ability to scan docker images hosted within ECR in order to detect vulnerabilities.
+
ECR scanning is free of charge, but you can only scan the same image every 24 hours. You get throttled if you make more than 1 request within 1 day. Naturally, scans are only triggered when an image is pushed. This blog explains how to trigger the scans at your own pace (for example, on a daily basis).
+
Since we love automation, we have decided to automate scanning task using a full serverless application that sends via slack a notification when there is a change in the scanning results.
+
Why?
+
Docker images are pulled left and right. You find yourself constantly looking for that new image which solves the majority of your problems. You find it. You pull it. You execute it. You like it. You decide to push it to your own ECR Registry because you’re afraid the original publisher eventually deletes it from DockerHub, or you simply need a private docker registry. But did you really check what was inside it? Are you aware of all the libraries and packages this image is using? What if there are critical security issues with that image that may compromise the software you’re running? What if your company loses tens of thousands (or maybe millions) of dollars just because a vulnerability that was never known?
+
It may sound scary, and it actually is! Vulnerabilities are everywhere and there’s very little you can do to mitigate the problem on your own. Usually, responsible publishers fix these vulnerabilities as soon as they are detected. However, not using a compromised image already is a step in the right direction. Automating this process is even nicer because you can react on it and decide what path to take: whether tailoring your own Docker images or just temporarily suspending the use for a compromised one.
+
How?
+
There are mainly two ways which you can scan your Docker images for vulnerabilities:
+
+
Enabling Scan on push for a given ECR registry, which will scan every image and output the results within the ECR console.
+
Using the AWS SDK to invoke the APIs in an automated way.
+
+
We like #2 because the more automated a process is the better. For example using Terraform:
You don’t want to be notified every single time a image was scanned, do you? You only want to be notified when there were vulnerabilities detected for a new image or if the vulnerabilities for a previously scanned image have changed. This means that this state must be stored somewhere. DynamoDB is a great choice here because we can easily store such state there and we can also react to any Table changes using DynamoDB Streams. Do you smell an Event Driven Architecture already?
+
+
+
+
+
+
+
+
+
Getting our hands dirty
+
Scanning images
+
For our use case, we decided to put all the ECR repositories to be scanned in a .json file within an S3 bucket. Everytime the Lambda function runs, it downloads that file, parses it and issues listImages for every repository defined and startImageScan for every image found.
+
Using the Node.js SDK and AWS SDK, we can easily put it together, see github for the full code.
This task is as easy as that. It invokes the SDK to start an image scan. This process will now be run asynchronously in AWS.
+
Analysing the results for a Scanned image
+
This could be achieved by reacting to CloudWatch events in AWS, but we decided to go the easy way and periodically poll ECR for the Scan Findings. If something was found, we store the findings in DynamoDB.
The code snippet above always inserts data into DynamoDB, because we don’t want to check during analysis time whether something has changed or not. This would require a new query to DynamoDB and extra logic to this function. We want to keep the functions small and concise, so we will create another function very soon to check whether there have been changes in a given image or not.
+
Notifying should there be vulnerability changes
+
Should there be any vulnerability changes to a previously scanned image or any vulnerabilities at all for a new image, we want to be aware of that.
+
Since we are already using DynamoDB, we can use DynamoDB Streams to react on write events. If you remember correctly, our previous function always pushes data to DynamoDB, meaning a Stream will always be triggered. Let’s use that in our favor and have a Lambda function react upon it. This new function will be smart enough to check whether something changed or if a new image was scanned. In any case, it will trigger a notification to a Slack channel. If there are no changes at all, the Lambda function then does nothing but happily terminate.
When changes are detected to either a new image or to an existing image, a Slack message notifying there are vulnerabilities for a given image will be sent.
+
+
+
+
+
+
+
+
+
You can now decide on your own if the vulnerabilities detected are worth investigating. If you go to AWS’s console and inspect the image you were notified of, you can see a list of all the vulnerabilities and their corresponding CVEs
+
+
+
+
+
+
+
+
+
Putting all together.
+
In this post we have mostly concentrated on the lambda code required to create the components. Apart from the Lambda code, the GitHub repo also contains the Infrastructure Code (Iac) required to create the resources in AWS in order to glue them together. Just follow th instruction smentioned in this repo and you should have your docker images scanner up in minutes.
+
+
+
For the purposes of this post, we used Slack. But the way the Lambda function is built, it could really be any HTTP endpoint, meaning you could even notify your internal systems of potential vulnerability changes in a Docker image.
\ No newline at end of file
diff --git a/2020/05/25/getting-help-from-your-compiler/index.html b/2020/05/25/getting-help-from-your-compiler/index.html
new file mode 100644
index 00000000..ddd814a3
--- /dev/null
+++ b/2020/05/25/getting-help-from-your-compiler/index.html
@@ -0,0 +1,577 @@
+Implementing Form Validation Without Conditional Logic | 040code
Implementing Form Validation Without Conditional Logic
Using clever data types to reduce the possibilities for mistakes
I’m learning about functional programming. I’m having loads of fun doing that,
+and I also think my code is improving because of it. In this post I want to
+share something that I learned about using Either to do form validation in
+TypeScript.
[Update June 4, 2020] When continuing to work on this code, I improved one
+important aspect. At the end of this post I added the section Making the
+validation functions reusable to
+explain. Also the code in the github repo was updated to reflect
+this.
+
Less Bugs
+
Today I want to show you a neat technique for form validation. Almost every web
+app will eventually have a form somewhere, and the user input of that form
+usually needs to be validated. You can do that by writing lots of conditions all
+over the place, but in this post I will show you that you can do it without any
+single condition except for the validation logic itself. But the rest of the
+app, including the UI, doesn’t have conditional logic. That’s great, because it
+significantly reduces the number of mistakes (bugs) you can make. To understand
+this post, you don’t need any specific knowledge or experience.
+
Clever Data Types
+
Let’s explore a concrete example. Suppose you’re writing a GUI that contains a
+form, and you need to validate the elements of that form. Making it even more
+specific: a registration web page where a user can sign up for a new account by
+providing information like e-mail, phone number, and a password (twice, to
+prevent typing mistakes).
+
This kind of code can easily become a big mess of conditional logic: if the
+e-mail field is not empty and it is not a valid e-mail address, then
+show an error. And if the phone number field is not empty and it is not
+a valid phone number, then show another error. And if the password is
+not empty and does not meet the minimum requirements, then show yet
+another error. And if any of the above ifs were true, then disable
+the submit button. And so on. Yes, you can write that code, but it’s going to be
+ever so easy to forget cases and combinations of cases.
+
So why not use a cleverly designed data type? I’m not making this up myself,
+smarter people have already done so, and they call the type Either. Typically
+it represents either an error or a valid value. The error case is (by
+convention) called the left of the Either, the valid value is the right.
+So a variable of the Either type could either be left('that e-mail address is
+invalid') or it could be right('me@example.com').
+
This data type comes with an API. This API is designed in such a way that it
+becomes hard to “do the wrong thing”. For example, the function that allows you
+to get the valid value out of the Either if it’s a right, requires you to
+also specify what should happen when it’s a left. You can’t forget, because it
+won’t compile1.
+
So how do you use that? Back to our registration page, let’s define the
+following validation function (in TypeScript):
On line (1), you see the declaration of the validation function. It’s argument
+is that string that we want to validate, and the return type is an Either of
+which both the left hand side and the right hand side are a string. That’s
+because the error (left) is a string, and the valid value, namely the email
+address, is also a string. On line (2) we call a library function to parse the
+provided string and check whether the return value is null. When it’s not, the
+email address was valid and we return the right on line (3), or when it is
+null then we return the left on line (4). Nice and simple.
+
Let’s define a few more.
+
const phoneValid =(phone:string): Either<string,string>=>
+ /[0-9]{10}/.test(phone)?right(phone):left('invalid phone number')
+
+const equalPasswords =(
+ p1:string,
+ p2:string
+): Either<string,string>=>(p1 === p2 ?right(p1):left('passwords differ'))
+
+const minLength =(s:string): Either<string,string>=>
+ s.length >=8?right(s):left('password must be at least 8 characters')
+
+const oneCapital =(s:string): Either<string,string>=>
+ /[A-Z]/g.test(s)?right(s):left('password must have at least one capital')
+
+const oneNumber =(s:string): Either<string,string>=>
+ /[0-9]/g.test(s)?right(s):left('password must have at least one number')
+
Zooming in on the password, we note that there’s a bunch of different ways in
+which a password can be wrong, and we want to capture all of them. So we want to
+call all of the related password-checking functions, each of which will give an
+Either as a result, and then combine those Eithers. Combining the right
+is easy. After all, if it’s a right then the right contains the password, so
+the combined either should still have the password as a value. How do we combine
+the left? By converting the left from a string to an array of strings, and then
+concatenating those arrays. This conversion is called lifting: we lift the
+single error string into an array containing that single error string. And now
+that the lefts are lifted into arrays, we can combine them by concatenating
+those arrays. If you’re interested in the source code of lift, you can find it
+in the appendix below, but for now it’s good enough if you conceptually
+understand what it does: it converts left('passwords differ') into
+left(['passwords differ']), for example.
+
The Scary Bit 😱
+
The library that I’ve been using for this style of (functional) programming in
+TypeScript, is fp-ts. It has some concepts that may seem scary if
+you’ve never seen them before. But fear not, it’s actually not that difficult to
+understand and use.
+
First up: semigroup. A semigroup is just something that supports
+combining things. Or concating if you like. You can see why we need this,
+right? We need to combine the error strings in the lefts of all the validation
+results.
+
Then there’s the function getValidation(), which returns a special kind of
+Either that knows that the lefts have to be combined somehow. How, you ask?
+That’s defined by the semigroup that you give it.
+
🤯
+
Ok ok, relax, you’ll get there. The following piece of code just means: give me
+a special Validation-kind of an Either that knows how to combine lefts by
+concatenating the string arrays that they contain:
If you want more details / explanation, I recommend reading their
+article about the topic.
+
The last scary piece of the puzzle consists of sequenceS and sequenceT. They
+are basically the same, so let’s start with sequenceT (T is for tuple, aka
+array). It needs the applicativeValidation we just defined, and then a list of
+our validation functions. It spits out an array of the return values of the
+validation functions. Well, sort of. It puts it in an Either first. That is,
+unless one or more of those Eithers was a left, in that case sequenceT spits
+the left which is the array of errors that the applicative validation has
+created.
This uses the simple validation functions that we defined above, as well as
+lift which we already mentioned, and lift2 which is the same thing but then
+to lift a validation function that takes two parameters instead of just
+one.2 It returns an Either. When everything is ok, it returns:
+
right([p1,p1,p1,p1])
+
One p1 because the return value of line (2) is
+right(p1), one because line (3) returns right(p1), one for line (4) and
+one for line (5) (because equalPasswords also returns p1 when all was ok).
+However, if one or more of those return values was a left, the whole thing is
+a left. For example when the passwords don’t have a number or an upper case
+character, the return value is:
+
left(['password must have at least one capital',
+ 'password must have at least one number'])
+
In order to use this as a building block for the entire form validation, we want
+a passwordValid function that returns only right(p1), not the array of four
+p1s, so we map the Either and get the following password validation
+function.
This map conceptually is the same as the one you already know for arrays: it
+takes whatever value is captured (the things in the array, or the right-hand
+side of the either), and applies a function to it. The pipe function pipes the
+value of an expression into a pipeline of functions. See
+example. The function constant is a function that always
+returns the given value. This is needed because map requires a function to
+be applied to the element (the right in this case). Instead of constant(p1)
+we could also write () => p1, but this way the intention is more explicit.
+
Now it’s a small step to define the entire form validation function. In this
+case we use sequenceS instead of sequenceT. Where sequenceT makes an array
+of elements in the right, sequenceS makes an object:
Look at that! We have a function that takes all the individual form input
+values, and it spits out an object that happens to be my internal user-profile
+representation, or a list of errors!
+
😎
+
Using This In The UI
+
For the UI part I’ll be using React. Because of it’s declarative nature it
+matches very well with the above function style of doing validation. The
+component that we are creating is called RegistrationForm. It uses a bunch of
+state hooks for the individual form input values:
Easy enough. Now for the interesting part. After declaring the variables for the
+various state hooks, we add the call to our validation function. You don’t need
+to put the type there, TypeScript will infer it for you, but I did so anyway to
+remind you of what our validation function is returning.
In Semantic UI, the Form element has to know whether there’s an error in the
+form. That’s easy, we use the isLeft API function of Either. So we replace
+the <Form> element with:
What should happen when we click the ‘Register’ button? In my case, since I’m
+also using Redux, it should dispatch an action that takes the RegistrationData
+as a parameter. We can do that using map again. On an Either, map performs a
+function on value if it’s a right, and leaves it alone if it’s a left.
The final piece of the puzzle is the list of error messages. To show that, we
+want to swap the Either, meaning that it replaces left and right. That’s
+because we want to do something with the errors which are left, but “doing
+something with an Either” mostly means doing it on a right. Also, we use
+getOrElse, which is one of those safe API methods that I mentioned. Not only
+does it return the right of an Either, but also do you need to specify what
+has to happen when the Either was a left. Here we simply generate an empty
+array in case of a left, and we have to make the TypeScript compiler happy by
+saying what type that empty array has.
+
So we get the array of error message out of the Either and pass it into a
+Semantic UI Message element like this:
+
<Message
+ error
+ header={t(
+ 'registration.formErrors',
+ 'Het formulier is niet goed ingevuld'
+ )}
+ list={getOrElse(constant([]asstring[]))(swap(validationResult))}
+ />
+
🥳
+
Conclusion
+
Do you realize what we just did? We created an entry form with input validation.
+It shows a full list of errors all the time, and updates it as you type. And
+we did so without using a single conditional outside of the individual
+simple validation functions like oneCapital, which is where they belong. And
+as a bonus all those simple validation functions are super-easy to unit-test and
+highly reusable. The other conditionals that you would normally need are now
+abstracted away in the Either and Validation, so that you can’t do it wrong
+anymore. I don’t know about you, but I’m happy. 😀
+
The full source code and a completely working example can be found on
+github. As a bonus, it also contains a couple of Jest matchers for
+checking Eithers in unit tests.
+
A New Vocabulary
+
There is one last observation that I’d like to make. All your “C-style”
+programming languages are basically the same. A loop (for, while), a condition,
+some stuff about objects/classes, a switch. When you can write Java, you can
+also write Swift or C#. Sure, you need to learn some new APIs, but the basic
+building blocks are all the same. It’s a common vocabulary shared by all these
+languages, that allow you to talk and reason about code.
+
As I challenge myself to use the functional programming concepts more and more,
+like I did in this article, I find that it gives me a whole new vocabulary in a
+similar way, but on a higher abstraction level. Learn how to use fp-ts in
+TypeScript, and you use the same constructs in Haskell, PureScript, Scala, etc.
+So any investment that you make in learning this stuff is not limited to the
+specific programming language that you learn it for. And it’s going to allow you
+to write your code using higher-level abstractions, thereby reducing hopefully
+the number of bugs.
+
Appendix: lift and lift2
+
As promised, below are the definitions for lift and lift2. The actual code
+is quite simple. When you pass a validation function into it (called check
+here), this returns a new function that, when called, first calls the original
+validation function and then uses mapLeft to put the left value in an array.
+It ignores right values. All the type stuff around it may make it look a bit
+daunting, but if you give it a hard stare you should probably be able to figure
+it out. 🧐
+
import{ pipe }from'fp-ts/lib/pipeable'
+import{ Either, mapLeft }from'fp-ts/lib/Either'
+import{ NonEmptyArray }from'fp-ts/lib/NonEmptyArray'
+
+/**
+ * Lifts the error of a validation function into a (non-empty) array.
+ * This way multiple validation functions can be composed while appending
+ * their errors to the array. This variant is for a validation function
+ * with one parameter.
+ *
+ * lift :: ( a -> Either e,b ) -> ( a -> Either [e], b )
+ *
+ * @param check single-arg (validation) function of which the error is lifted
+ * @see lift2
+ */
+exportfunctionlift<E,A,B>(
+ check:(a:A)=> Either<E,B>
+):(a:A)=> Either<NonEmptyArray<E>,B>{
+ return a =>
+ pipe(
+ check(a),
+ mapLeft(e =>[e])
+ )
+}
+
+/**
+ * Lifts the error of a validation function into a (non-empty) array.
+ * This way multiple validation functions can be composed while appending
+ * their errors to the array. This variant is for a validation function
+ * with two parameters.
+ *
+ * lift :: ( a -> b -> Either e,c ) -> ( a -> b -> Either [e], c )
+ *
+ * @param check double-arg (validation) function of which the error is lifted
+ * @see lift
+ */
+exportfunctionlift2<E,A,B,C>(
+ check:(a:A, b:B)=> Either<E,C>
+):(a:A, b:B)=> Either<NonEmptyArray<E>,C>{
+ return(a, b)=>
+ pipe(
+ check(a, b),
+ mapLeft(e =>[e])
+ )
+}
+
Making the validation functions reusable
+
(This section was added on June 4, 2020)
+
There is one thing about the code above that kept bothering me. For example, take
+the validation for checking that passwords contain a number:
The problem with this approach is that the function combines two
+responsibilities: (1) the check whether or not the string satisfies the
+conditions, and (2) attaching the proper error message when the check fails. Do
+you see the problem with this? It’s not reusable. Because in another form we
+maybe want to use the same check, but with a different error message. We can do
+better!
+
First, we can make a completely generic and reusable check function, which even
+lives in a separate more generic file:
This function returns an Option instead of an Either. An Option is either
+some(value), or none (which corresponds to the value being null or
+undefined). In other words: the function atLeastOneNumber(str) gives us
+none when the check fails, or some(str) when it succeeds.
+
The second step is specific to the registration form validation, namely
+attaching the proper error message. We do that by taking the output of
+atLeastOneNumber and putting it in an Either using fromOption. This API
+function fromOption converts the some to a right, and the none to a
+left with the given value. So we get this (flow is left-to-right function
+composition3):
So now we have an even more generic and completely reusable digits checker,
+that you can pass the desired number of digits, and a specialized
+phoneValidator that uses it for the registration validation.
In TypeScript you can probably define lift in such a way that it works for both single-parameter and two-parameter functions, but that’s where I draw the line for now.↩
+
So flow creates a new function that is the sequential combination of the functions you put into it. The call flow(f,g)(x) is equivalent to g(f(x)). You can read it as “first do f, then do g” on whatever you pass into it.↩
+
+
\ No newline at end of file
diff --git a/2020/05/25/scaling-selfhosted-action-runners/index.html b/2020/05/25/scaling-selfhosted-action-runners/index.html
new file mode 100644
index 00000000..c46bb437
--- /dev/null
+++ b/2020/05/25/scaling-selfhosted-action-runners/index.html
@@ -0,0 +1,644 @@
+Scaling GitHub Action Runners | 040code
Scaling GitHub Action Runners
Serverless scalable self-hosted runners on AWS spot instances
Last year GitHub released self-hosted action runners which already empower you to run GitHub actions on your own machines. At the time of, the feature was quite limited; it lacked an API and only supported repository level runners. We could already automate the deployment of a single self-hosted runner for one repo with Terraform on AWS. But the clear drawback here is that you will pay AWS for hosting your instances, even if there is no workload to execute. Over the last months, GitHub released an API for self-hosted action runners, and support for org level runners was added. Therefore, now is a good moment to have a second look on how to utilize self-hosted runners, and scale the self-hosted runners up and down based on workload using the GitHub API. Similar to scaling runners for GitHub we wrote an article on how to scale runners for GitLab which Terraform community module you find here.
+
Why?
+
Why we are using self-hosted runners? First of all, by running on our own hardware, we created our own trusted environment for our builds that we can connect to networks not available via public runners. Note that because of this reason, we choose to limit the use of self-hosted runners to our private and internal repositories. While using self-hosted runners on public repositories is possible, be aware that any pull request will run on your hardware which could have serious security risks. Therefore you should carefully consider the usage of self-hosted runners in this context. A second advantage of hosting our own runners, is that we can fine-tune the required hardware. Finally, local runners allow you to control, and maybe even save, costs, in contrast to public runners that are not free for non-public repo’s. The setup we will show will have almost no costs if there are no builds, and by executing the workflows on AWS spot instances we save again some pennies.
+
The idea
+
The GitHub self-hosted action runner is an agent that you need to install on any machine that needs to pickup and execute workflow. To connect the agent to GitHub you can either choose to register the agent on repo level or org level. For registration you request a registration token (API) and this token is used during the setup of agent. This part is easy to automate, so let’s have a look how we can orchestrate the process of creating runners.
+
In GitHub everytime a workflow is triggered, a check run event is created. Can we use this event to decide whether to scale a runner? Yes, if we find a way to receive and listen for the event. Actually, there are two options: using the webhook at the repo or org level, or using the GitHub app. Here, we will use the GitHub app option. The GitHub app can sent a message to our webhook for every check run event and we can easily implement this webhook with a AWS Lambda function. This lambda can verify the event, run some other checks and decide to create a EC2 spot instance to facilitate the execution of the workflow. The EC2 spot instances will be linux based with docker so we can run any containerized workload.
+
+
+
+
+
+
+
+
+
Being able to create runners once a workflow is triggered brings us the possibility to scale up. Question remains now how to scale down? Again we could listen for event and next try to scale down. Since a runner instance is not dedicated to only executing one workload. At the moment another workflow requires a build the, the runner will process this build as well. This makes scaling down based on events less trivial. So we took a slightly different and much simpler approach: every x minutes all runners that are not busy are removed.
+
You might think why not using Kubernetes? The reasons for us are quite simple. First, we would like to have a solution that could scale really from zero without making any costs. Second, with Kubernetes we enter a other model of running the GitHub agent, which is not supported (yet?). So, this would introduce some extra obstacles as managing docker in docker. Hence, we favoured this simple and straightforward approach.
+
Architecture
+
Before we build scalable runners on AWS, we discuss the architecture in a bit more detail. On AWS we need several infrastructure components and a component to orchestrate the life cycle of the action runners. All infra structure will be created using HashiCorp Terraform. The orchestration layer is build with AWS lambda. All the lambda functions are written in TypeScript and compiled to NodeJS single file with zeit ncc.
+
For the orchestration we build 4 different lambda’s: First a lambda to handle the GitHub events, two lambda’s for scaling up and down. And a last one to synchronize the runner binary. The self-hosted runners will run on AWS spot instances.
+
+
Webhook
+
This lambda receives events via the API gateway sent via the GitHub App for each check run event. The lambda will check if the event is signed with correct secret. Next it filters only for the check run event that requires an action runner, other events are simply dropped. In other words the we only keep the events that are created once a new workflow is triggered. The events we accept are published on a SQS queue. Messages on the queue are delayed for 30 seconds by default. This allows possible idle action runners to start the workflow execution.
+
Scale up
+
This lambda will receive messages from the SQS queue. For each message the lambda first verifies if the workflow has not yet been started. When the workflow is still queued, the lambda will create an AWS spot instance via a launch template. Before creating the instance, a new runner will be registered to GitHub. The token will be temporary stored in the AWS parameter store (SSM). During the boot the AWS instance installs all requirements via a user_data script. The script will install docker, git and the action runner, which is fetched from a S3 bucket. Finally the runner is configured with parameters fetched from the parameter store, and is then ready to consume workloads.
+
Scale down
+
This lambda is triggered by a cloud watch event. Every time it runs it will try to remove all the registered runners from GitHub. Only inactive runners can be removed. Trying to remove an active one will simply fail, resulting in a 500 response. For all the removed runners from GitHub the lambda will terminate the EC2 spot instances so the bill stops. The scale up and scale down lambda share the same code base, and are released as one zip.
+
Update Binary
+
This lambda will download the action runner binary to avoid unnecessary latency during creation of the runners. We found out in our first week of running our setup that downloading the action runner binary could take up to 15 minutes, therefore we build a simple lambda to cache the artifact in a bucket.
+
Build it
+
Time to build the idea. In the next steps we create a GitHub app, setup Terraform, configure an example repo to test our runners.
+
GitHub app
+
As mentioned above there are two scenario’s two options for running the self-hosted runner, repo level or org level. Our module supports both but for this blog we use the repo level runner as example. The example below is based on the default example in the Terraform module.
+
The process starts with the event that we need to receive. So first we create a GitHub App. To create an app in your organization, go to you organization settings and select GitHub apps. On the app creation page you only need to fill in the name, homepage, and uncheck the webhook. We will activate the webhook once we know our endpoint.
+
+
+
+
+
+
+
+
+
Next hit create app, you will be redirected to the app overview page. Make a note of the following generated id’s, and generate the ssh private key on the bottom of this page.
Before we can actually deploy the infrastructure we need to build or download the lambda functions. You can clone the module repo and use the script in .ci/build.sh to build the 3 lambda zip files. But the easier option is just to download the release artifacts from the repo. We even provide a small utility module to download the artifacts.
+
Create a new workspace (directory) and add the following terraform code.
Next run terraform init && terraform apply the lambda artifacts will be download to your local machine.
+
Deploy the action runners
+
Now we will create all required AWS resources and deploy the lambda functions. Besides this module, we also need a vpc as well. We have a VPC with a public terraform AWS VPC module, you can use your own VPC as well. Create a new workspace and the code below to the main.tf.
You can already run and apply the terraform code, once we have finished with the setup. We can now add the GitHub action runner module. Events sent to the webhook will be signed with a secret, therefore we generate a password with Terraform random provider. Ensure you set the variables for the lambda zip to the download lambda artifacts, and also add the id’s, keys and base64 encoded secret for the GitHub app to the configuration.
Save your configuration and run terraform init and terraform apply. Once Terraform is finished we will trigger the lambda function for the syncing the distribution one time manually to ensure our cache is filled.
The module we released for creating the infrastructure supports IAM permissions boundaries for those who have to run on shared AWS environments. For more details checkout the example.
+
+
Configure the GitHub App part 2
+
Go back to your GitHub app and activate the webhook, and provide the endpoint and secret. Don’t forget to save the changes. Now you can subscribe for events. Go to the section permissions and events, grant permission on repo as follow, check your terraform output for the values. (terraform output -json webhook):
+
+
Administration (read, write) - required to register runner.
+
Repository permissions Checks (read) - required to get check run events
+
+
Scroll all the way to the bottom and check the box to subscribe to the check event.
+
+
+
+
+
+
+
+
+
+
Note: for org level runners you need to grant the administration permission on org level instead of repo level.
+
+
Putting all together
+
We are almost done; next we configure the self-hosted runner for a test repo. First we create a test repo in GitHub. In case you are using a runner and GitHub cannot find any runner that could run the workflow the build will directly fail. Since the scalable runners only create a runner once an event is received and remove the runners after an idle time, there are no candidate runners registered by default. For the time being we have a work around: register a runner for the repo (or org) which we will never use, and start that runner. The registered runner will prevent the workflow from failing. Be aware that this runner will be removed by GitHub after 30 days. To add this runner, go to your repo settings (or org settings), select actions and choose “Add runner”. Run the provided instruction for your OS and give it a descriptive name such as “dummy”. In case you provided the terraform module extra labels, provide them here as well.
+
+
+
+
+
+
+
+
+
THe final step is to install the GitHub app we have created. Go to your app settings and select install, now you can choose how to install the app. To only receive relevant events we choose to be repo specific and install the app to our test repo.
+
+
+
+
+
+
+
+
+
That’s it, time to test. First create a simple action workflow in an empty directory.
After 30 seconds you should see your new spot instance starting in AWS.
+
+
+
+
+
+
+
+
+
A bit later, the runner should be registered to your GitHub repo.
+
+
+
+
+
+
+
+
+
And once registered it should only be a matter of seconds before the build starts.
+
+
+
+
+
+
+
+
+
Every 5 minutes the scale down lambda is executed, and if you not running any job you should see the runner is terminated.
+
+
+
+
+
+
+
+
+
In case you build runners are not starting or registering. The best you can do is follow the trace. In the GitHub app on there is an advanced settings page where you can see the events sent to the webhook. When the event is not accepted double check the endpoint and secret. Next you can check the logs for the webhook and scale up lambda in cloud watch. Finally you can inspect the EC2 user data logging. Access via SSM (MP: should this be ssh?) is by default enabled. Just select connect to the instance and inspect the log /var/log/user_data.log.
+
Final thoughts
+
The abilities of GitHub self-hosted runners are already quite neat. But creating the setup as discussed requires some hacks and shortcuts. First of all it would be much easier once GitHub delivers the orchestration component, which will reduce the whole setup to only managing a bit of cloud resources. For example having runners on a Kubernetes cluster including scaling can make it a lot easier to use.
+
By delivery an API for actions GitHub gives you the option to build your own setup, the fact that we always need to have a runner registered is quite an hack. Also, the way for scaling down will not win a beauty contest but is necessary because the API does not provide an option to verify whether a runner is still active. Furthermore, the cache of the distribution is bit of a hack to work around some crazy slow downloads that we had experienced.
+
But given all those hacks we are still quite happy with this setup. Until GitHub releases the components need to orchestrate the life cycle of an action runner, we will happily use this setup.
+
Acknowledgements
+
The module discussed in this article was developed together with Gertjan Maas.
\ No newline at end of file
diff --git a/2021/12/15/run-debug-lambda-local/index.html b/2021/12/15/run-debug-lambda-local/index.html
new file mode 100644
index 00000000..db95cd8f
--- /dev/null
+++ b/2021/12/15/run-debug-lambda-local/index.html
@@ -0,0 +1,618 @@
+Run and debug an AWS Lambda locally | 040code
Imagine you are building an AWS serverless function. As a good developer you have designed and written your code with paradigms like clean code and TDD. Your code is clean, maintainable and well (unit) tested. But the moment you deploy your function to the cloud, it does not work as you have expected. Would it not be great if you can locally run, debug and iterate fast. And when deploying to the cloud be 100% certain that your lambda will work smooth?
+
So what is the problem? The real world is not so nicely isolated as your test environment for unit testing. There are no mocks or stubs. The environment could be different. The function is invoked differently. And IAM permissions are applied once running in the cloud. Of course, this is the difference mostly between a unit test and integration test environment.
+
Time to find out how close can we get to behavior in the real world, aka cloud while running as much locally.
Let’s first build a simple Lambda. We use TypeScript as language for the Lambda which requires a Node runtime environment for execution. This add an extra step between writing the code and running, we need to compile the TypeScript code to JavaScript.
+
We write a simple program that sends a message to a SQS queue, Amazon’s Simple Queue Service. Actually it does not matter which AWS services we use. We only need integration points to the cloud, so we are also depending on the cloud services for local excution. For simplicity we limit ourselves to only one.
+
Our tiny program will read the SQS queue url and a message from the environment. If no message is set the famous string hello world! will be send to the queue.
While, mocking is a great way to check the logic, it does not guarantee that an actual deployment in the cloud will work. For testing the lambda working with a realistic scenario, connected to dependent services, integration testing is a common practice. But integration testing is not always speeding up the development flow. Sometimes we need an easy way, to check quickly the code changes. For this reason we explore running a test locally, so we can easily adapt changes and connect to a debugger.
+
Required cloud resources
+
The Lambda function above requires in terms of AWS the following resources.
+
+
A Lambda function, this we will not create since we focus on running locally.
+
A SQS queue, the queue to which we publish message. Since the goal is to run the Lambda as realistic as possible, without any mocking, we need an actual SQS queue.
+
An execution role, this is the role that defines which resources the Lambda can access. The Lambda needs to send messages to SQS. We also going to use this role locally.
+
+
Today we have dozen of ways to create cloud resources, manually via the web console, AWS CloudFormation, Terraform, AWS CDK, Pulumi, Serverless Framework, e.g. Here we will use Terraform which is an IaC eco-system with no extra tooling to streamline Serverless development. When developing only a serverless application the SAM Framework, CDK or the Serverless Framework may seem a more logical choice. But using Terraform shows us how we can mix and match if we cannot chose every component.
+
Time to create the cloud resources with Terraform. This setup assumes you have an admin role. Clone the repo and cd in the Terraform directory to create the resources.
+Expand for Terraform resources
+
First we create the role, we add an extra principal to allow you to switch to the role used by the Lambda.
+
+locals{
+ namespace="blog"
+}
+
+resource "aws_sqs_queue""test"{
+ name="${local.namespace}-test"
+}
+
+resource "aws_iam_role""Lambda"{
+ name="${local.namespace}-test"
+ assume_role_policy= data.aws_iam_policy_document.assume.json
+}
+
+data "aws_caller_identity""current"{}
+
+data "aws_iam_policy_document""assume"{
+ statement{
+ actions=["sts:AssumeRole"]
+
+ principals{
+ type="Service"
+ identifiers=["Lambda.amazonaws.com"]
+ }
+
+ # Assume you have admin rights, this allows you to switch locally to this role.
+ principals{
+ type="AWS"
+ identifiers=["arn:aws:iam::${data.aws_caller_identity.current.account_id}:root"]
+ }
+ }
+}
+
Next, we create a policy that grant the Lambda the required access to AWS resources, in this example the SQS queue.
The last step of the cloud setup is to configure your AWS profile in <user_home>/.aws/config. This profile make it easy to assume (switch) the role we have created to execute the Lambda.
By now you should have created a SQS queue and a role for Lambda in your AWS account. There is no Lambda deployed at all. You can test your setup with the AWS CLI.
+
# send
+aws sqs send-message --queue-url <QUEUE_URL> --message-body "Test" --blog-test
+
+# receive with wrong profile, so causes an error
+aws sqs receive-message --queue-url <QUEUE_URL>--profile blog-test
+
+# receive with your admin role
+aws sqs receive-message --queue-url <QUEUE_URL>--profile admin-role
+
+# purge messages on the queue, should be empty but already
+aws sqs purge-queue --queue-url <QUEUE_URL>--profile admin-role
+
Enough said about creating infra and testing, remember we just showed a way to create the cloud resources with IaC. Feel free to use your own preferred (or required) framework.
+
Testing locally
+
For running the Lambda locally, we explore several options. Let’s first list them briefly. The list will not be complete but should include the most logical options.
+
+
+
Using NodeJS: Running the Lambda locally in NodeJs, for example using a watcher like nodemon. Invoking the Lambda needs to be faked by calling the handler function.
+
+
+
Using SAM: Run the Lambda function using AWS Serverless Application Model (SAM). The function will run via SAM and we can just pass an event to the function, similar to running in the cloud.
+
+
+
Using Serverless Framework: Like AWS SAM, the Serverless framework simplifies the process of building Serverless applications, including Lambda. The framework also provides an option to run the function locally.
+
+
+
NodeJs
+
We try to keep as close to the NodeJS ecosystem. To run the Lambda locally we will start a small wrapper program via a nodemon watcher to invoke the Lambda on every single code change. Or just start the program after each change without a watcher.
+
Generate the sample event with SAM. Note that the sample event does not match the TypeScript definition of the event, the version property was missing.
+
sam local generate-event cloudwatch scheduled-event
+
Next we write a small wrapper, check the source code for the full details.
This wrapper we can use to directly start in VSCode via a launcher, check the config here. This will make debugging your code trivial.
+
+
+
+
+
+
+
+
+
Another option would be using nodemon in a watch mode and re-run after every change, run yarn watch:local. Ensure you have set the correct AWS profile in the terminal before you run the yarn command.
+
Serverless framework
+
The Serverless Framework let you develop and deploy serverless stacks to various clouds. Here we only quickly explore if we can use the framework to invoke the function locally.
+
The framework requires you define your function in a serverless.yml. Below we define the test function in a serverless manifest.
Before we invoke the function ensure you have not set the environment variable AWS_PROFILE. And run yarn run build to generate the JavaScript dist bundle.
We leave it by here, and not cover integration in VSCode, and or attaching debuggers.
+
AWS SAM
+
The last option we explore is AWS SAM, the framework provided by AWS for building serverless applications. Amazon provides a VScode plugin, AWS toolkit as plugin to easy integrate SAM into VSCode.
+
For SAM applications needs to be defined in a manifest file named template.yml. Below a minimal version of the SAM manifest required to run our Lambda function.
We can simply invoke the Lambda via a CLI command. Before invoking SAM, ensure you have build the sources, yarn run build. And set the profile for AWS, export AWS_PROFILE=blog-test.
+
sam local invoke
+
The log output should look very familiar to the one you know from CLoudWatch.
+
+
+
+
+
+
+
+
+
Another option is to start the Lambda and use the AWS CLI to invoke SAM.
+
sam local start-Lambda
+
Open a second terminal and use the AWS CLI and invoke the Lambda, since our Lambda does not require any input we do not provide an event.
This all works, but it requires first building sources. And attaching a debugger would require most likely several Google searches to find the right way to configure your IDE. But there is an easier option, for which you do not have to take care of compiling code. And you can easily attach a debugger in VSCode. The AWS Toolkit helps you create a launch configuration, see the AWS detailed instruction here. We added the following launch configuration. The sample below was generated with the AWS Toolkit, ensure you update the ref to the LambdaHandler, set the credentials and environment variables.
Now we can start a debugging session via VSCode. Almost similar to the option for Node but without a wrapper. Drawback is the boot time is a bit slower. SAM is building the sources, and starting a Docker container to before the function is invoked.
+
+
+
+
+
+
+
+
+
Conclusion
+
We have explored several options for running and debugging an AWS Lambda in a local developer environment. I can recommend AWS Toolkit for VSCode with SAM. It lets you easily test and debug the Lambda locally. If you really want to bring speed to your development, you could consider a tiny wrapper and avoid the heavy lifting of the AWS Toolkit. In case you are all in with AWS SAM or the Serverless Framework I would stick to that framework.
\ No newline at end of file
diff --git a/2022/12/02/oidc-part-1/index.html b/2022/12/02/oidc-part-1/index.html
new file mode 100644
index 00000000..a578aa57
--- /dev/null
+++ b/2022/12/02/oidc-part-1/index.html
@@ -0,0 +1,383 @@
+Are you still using keys? | 040code
Are you still using keys?
Practical guide to deploy with GitHub Actions and Terraform to AWS.
This blog post is the first in a series of two in to explain how you can deploy keyless with GitHub Actions to AWS. This first part covers the basics of setting up OIDC integration between GitHub Actions and AWS. In the second more advanced practices are explained, such as deploying using GitHub Action shareable workflows with custom OIDC claims. This blog is practical and shows how to use OIDC with GitHub Actions to deploy to AWS. The same patterns apply to deploy to other OIDC enabled clouds like Google or Azure.
Are you still using secrets to deploy from GitHub Actions to your cloud provider? Do you still define secrets in your workflow jobs and rotate them regularly? Do you know there is no need anymore to define secrets to connect to common cloud providers? You can avoid the need of using secrets when deploying with GitHub Actions to a provider supporting OpenID Connect (OIDC). New to OIDC, watch the talk “OAauth 2.0 and OpenID Connect in plain English”.
+
With OIDC you define a trust relation between GitHub Actions end your cloud, in our case AWS. You define a trusted relationship between your AWS account and the OIDC provider of GitHub Actions. And then define conditions under what circumstances an AWS IAM role can be assumed. New to assuming a role in AWS, watch this two minute tutorial.
+
Once the integration is set up, your workflows can obtain a short-lived token. This token is allowed to assume a role to access resources in AWS, based on the policies defined for the role. This post provides you with a practical guide on how to set up the required infrastructure resources with IaC, and shows a simple example to validate the setup.
+
Setup OIDC for GitHub Actions and AWS
+
GitHub provides detailed documentation for authenticating Actions to AWS that nicely describes all the required steps. It is highly recommended to read the article on GitHub to get a deep understanding. Since we love to automate everything we are not setting up the required resources manually. Instead we use Terraform to automate the creation of the resources.
+
The example in this blog is using the repository 040code/blog-oidc-github-actions-aws, you find a full example here. You can run the example by cloning or forking the repository and run the Terraform code by setting the variable repo to your repository, e.g owner/repo.
+
OIDC Provider
+
The first step is to define the OIDC provider. To define the OIDC provider we need to look-up the thumbprint for the GitHub Actions SSL certificate. The AWS web console can do this for you. The Terraform TLS provider has a data source that can do the look-up for us. Once we know the thumbprint, defining the OIDC provider is straightforward.
Now run terrafrom apply as a result the OIDC provider for GitHub is created in your AWS account, this is a global resource you only have to create it once per AWS account.
+
Next, define a role and trust relationship, this trust defines from which repository and under what condition the role can be assumed.
It is important that you take care defining the trust relation. One mistake, and you could open your AWS account to any repository on GitHub.
+
Assuming the role is allowed when all conditions are met. In general conditions are combined in a logical and and values in the test with a logical or. Ensure you are familiar with IAM conditions.
+
+
+
+
+
+
+
+
+
In the example above the iss, aud and sub attributes are checked in the policy. The first two are straightforward. Checking the sub requires care. When you accept any value in sub or even don’t check the field. You will allow anyone access to your AWS role via GitHub Actions. And likely you don’t want to open our AWS account to the whole universe!
+
For example, the condition below allows any Action running in the repository 040code/blog-oidc-github-actions-aws from any ref (branch / tag) to perform the assume role operation and access the resources granted via this role.
By adding a condition like the one below, you can deny any pull request to assume the role. Similar checks can be done or allow or deny the ref, or context.
It should be clear, that you need to take care of defining the conditions for which you define the trust. In part 2 you can read more about advanced subjects checks by customizing the sub field of the JWT token.
+
The final step is to allow access to some resources via the created role. This blog post only shows the pattern by example. Remember that it is good practice to write policies with the least privileged access needed. For simplicity we have created a quite coarse-grained role for the blog. Time to define the required Terraform resources.
+
+
An S3 bucket (no policies, no encryption in this example)
+
A policy that grants the role created before access to the bucket.
For convenience add the following outputs to your terraform script. Those are handy when you create the workflow later. To apply this configuration, just run terraform apply
+
+output "role" {
+ value = aws_iam_role.github_actions.arn
+}
+
+output "bucket" {
+ value = aws_s3_bucket.blog.name
+}
+
✨ Test locally ✨
+
The role created can only be assumed via GitHub Actions, sometimes it is convenient to test from your own environment. This can be done by adding the following statement to the trust condition. After applying you can assume the role locally. Replace your the user_id by your user id ARN.
Time to show this setup works, up to here no GitHub repository was needed. To test the setup, create a GitHub repository. Which can be a private or public one in any org or user space; it really does not matter. The easiest way is just to fork the example repository. Once you have created the repository, create a secret to hide you AWS Account ID. In this post we use a secret named AWS_ACCOUNT_ID. The ID should not be a secret but at least you make it one small step harder for an attacker.
+
Earlier you have defined the OIDC provider in your AWS account, this provider is used to provider AWS keys based on the JWT token provided by the GitHub OIDC provider. AWS provides an action: aws-actions/configure-aws-credentials to obtain an AWS STS token via OIDC. This action makes the whole process of obtaining short-lived secrets painless; you only need to add the action to your workflow and configure the role to assume. All secret handling is done by the action.
+
The workflow below brings it all together. The workflow shows how you can access an S3 bucket from GitHub Actions without configuring a secret.
Once you have pushed the workflow to your repository, a GitHub Action job starts running. The result shows some ascii art.
+
+
+
+
+
+
+
+
+
That was all, no keys needed anymore, no key rotation required. Time to stop using keys and start using OIDC. In part 2 we discuss a more advanced use-case in which you can use OIDC to deploy with a shared workflow.
\ No newline at end of file
diff --git a/2022/12/02/oidc-part-2/index.html b/2022/12/02/oidc-part-2/index.html
new file mode 100644
index 00000000..280e2692
--- /dev/null
+++ b/2022/12/02/oidc-part-2/index.html
@@ -0,0 +1,357 @@
+Are you still using keys? | 040code
Are you still using keys?
Deploy with GitHub Action using shared workflows to AWS
This second post explains more advanced practices of deploying keyless with GitHub Actions to AWS. In the first post explains the basics of deploying to AWS with GitHub Actions using OIDC. In this second post you learn how to debug the OIDC claim, and how to allow shared workflows to deploy resources to AWS.
In the first post we have seen that deploying with GitHub Action and OIDC to Amazon is straightforward. Defining a trust condition for the repository allows you to safely deploy without using keys. What if we want to do some more generic deployments. For example we want to deploy files from several repositories to a single S3 bucket, where each repository has its own path to deploy files. A use-case here is deploying Backstage techdocs to S3. Another use-case is publishing docker images from several repositories to Amazon ECR.
+
You can replicate the setup for each repository by duplicating the workflow and updating the IAM policy. This pattern is hard to maintain, and in most cases creates a central bottleneck. Each team having a repository is getting blocked by a central team that needs to update IAM permission. This last part could be automated, but let’s explore what our other alternatives are.
+
With GitHub Action you have the option to create a shared workflow in your organisation, this shared workflow can take care of those common deployments. Using shared workflows removes duplication of your workflows, but not the complexity of maintaining IAM policies to allow access via OIDC to AWS. Let’s see how to simplify this.
+
The solutions
+
The problem that we need to solve is: how can we allow the caller of the workflow to assume a role in AWS. And can we check what the caller is allowed to with a dynamic policy based on the repository name? Before exploring the options, we have a closer look at the information in the claim. You can check the GitHub OIDC configuration here. Another option to understand the claim; is debugging the OIDC claim running in an Action. Add the following workflow to debug the OIDC claim to your repository.
+
name: Debug OIDC
+on:
+ workflow_dispatch:
+
+jobs:
+ oidc_debug:
+ permissions:
+ id-token: write
+ runs-on: ubuntu-latest
+ name: A test of the oidc debugger
+ steps:
+ -name: Debug OIDC Claims
+ uses: github/actions-oidc-debugger@main
+ with:
+ audience:'https://github.com/github'
+
This workflow uses the GitHub Action to debug OIDC. Run the workflow and check the log for the details in the claim.
This looks promising, the claim contains an attribute repository. It would be nice if we can use this in AWS to dynamically check if the workflow caller has access to a location in the bucket that matches the repository name. Or publish an ECR container matching the repository name to the registry name. But this is not possible because AWS does not support those custom fields, but instead only allows a limited set of keys. Circumventing this by writing dynamic policies is also not allowed. So unfortunately we cannot use the OIDC field repository to check if a role is allowed to invoke the operation.
+
Amazon does support setting principal details via session tags. A claim would look like this:
Now we can check the principal in IAM dynamic with the dynamic field ${aws:PrincipalTag/Repository} and match it against a path in the bucket or registry name in ECR. But this requires that GitHub sets this information in the JWT token, which is currently not the case. This means that also this second option is a no go. Hopefully support will be added soon, follow this issue to track the status.
+
Only one option is left at the moment. GitHub allows you to change the default subject (sub) and set any of the available custom fields, or combinations. We can set for example the workflow reference in the subject field. By adding the workflow reference to the subject, we can define a trust relation for the shared workflow. And allow the shared workflow to run the cloud deploy. Checking or enforcing the caller in accessing the correct path in the bucket has to be done in the workflow. Doing the check in the IAM policy is of course preferably. The other problem is that a subject claim needs to be altered for any repository calling the shared workflow, which increases the complexity. The subject claim can be set on repo or org level, doing this on org level can cause unwanted and unexpected side-effects in case the repository uses OIDC for other connections.
+
Altering the subject claim
+
Time to work out an example. We create a shared workflow that syncs files to a location in a S3 bucket based on the repository name. The first step is to update the subject field in the OIDC claim with the workflow reference. This is required to define a trust based on our shared workflow. The only way to change the fields is via the REST API. The GitHub cli seems not to support this PUT operation, we have to fall back to invoke the REST API directly. Below an example of the REST calls, set the GITHUB_REPOSITORY to org/repo and the GITHUB_REPOSITORY to a GitHub access token.
The setup above can also be provided via a shareable workflow, see here an example. The workflow can be called from a caller repository to customise the OIDC claim.
+
For setting up the OIDC provider on the cloud, we use almost the same Terraform configuration as in part 1. In case you have used the setup in part 1, ensure you remove the setup by running terraform destroy. Next you run the setup for part 2.
+
We add an extra condition to the AWS IAM policy for assuming the role. This condition is allowing the shared workflow to assume the deployment role. Any repository that can call the shared workflow, can deploy to AWS using this workflow. But since we lock down the workflow the caller cannot assume the role directly. Ensure the inputs you inject are safe, and cannot cause unwanted access to your cloud account.
+
For the repository we only check it matches one of the repositories in our Org. Update the conditions to fit your needs.
When replaying the blog example, you can simply run the Terraform code provided here and set the variables for the repository and the workflow.
+
To deploy files per repository to a central bucket we create a shared workflow. The shared workflow is running in a separate job, so we cannot use any build output directly. Therefore we store our generated file in a temporary artifact. Remember for this blog the workflow is not hardened, no extra checks are done!
The final step is creating a workflow that calls the shared workflow to deploy. The workflow below is generating some data, stores it as an artifact, and calls the shared workflow. The shared workflow can then deploy the generated artifact to S3. The shared workflow is allowed to assume the role because you have updated the claim on the calling repository before.
That was all, next run you workflow by pushing or manually triggering.
+
+
+
+
+
+
+
+
+
Final thoughts
+
We have explored how we can run deployments with GitHub Actions to AWS using shared workflows. There is currently not a good way to check the caller dynamic to match permissions in AWS. With the customised OIDC claim we can deploy using shared workflows, but it still requires the caller to alter the subject claim. And we need to run checks in the shared workflow instead of using IAM policies when we want to avoid a maintenance burden on our IAM policies. It would be great when AWS and GitHub could work out a way to support session tags to check the principal.
\ No newline at end of file
diff --git a/2023/01/06/github-event-aws-eventbridge/index.html b/2023/01/06/github-event-aws-eventbridge/index.html
new file mode 100644
index 00000000..f213f632
--- /dev/null
+++ b/2023/01/06/github-event-aws-eventbridge/index.html
@@ -0,0 +1,873 @@
+Micro Hack | 040code
Micro Hack
Hands-on guide to deliver GitHub events to AWS EventBridge
With the growing usage of GitHub as well as scaling more self-hosted runners we found a strong need to act more on events sent by GitHub. For the Self-Hosted runners we already receiving GitHub events via a webhook. The events are processed by the control plane for scaling the runners. As you can see in the image below events are directly passed from a Lambda to the SQS queue, which makes it hard to use the same event for another independent task.
+
+
+
+
+
+
+
+
+
With this runner’s solution, we already had the need to process the workflow job evens for monitoring purposes. For example, to calculate the time a job is started after the first event is created, e.g. the job was queued. We introduced a secondary queue to deliver events for analytics. A simple hack but it does immediately not feel right.
+
+
+
+
+
+
+
+
+
The webhook provided by the runner solution is tailored to only accept workflow_job events, but we are interested in more events. There are numerous events in which we are interested. Examples are triggering a compliance process when a repository is created, triggering an alarm if even an admin makes a repository public, gather analytics for the lead time of a pull request. Even when we don’t combine a solution, the current solution is not flexible enough to process all kinds of different events with different needs.
+
What’s the experiment
+
Since GitHub and AWS Cloud are a given, we looking for a better solution to handle the events. And avoiding creating a new webhook, or app for every event we are interested in. We are looking for an event-driven solution to process GitHub events. We distinguish two main use cases. A hot stream of events to act on directly, and a cold stream for analytics. AWS typically provides you the building blocks, allowing you to compose the solution in several ways. Let’s quickly explore our options:
+
+
Simple Notification Service (SNS): A lightweight serverless option. A producer (webhook) publishes messages on a topic and subscribers can receive notification via an HTTP/HTTPS endpoint, email, Kinesis, SQS, Lambda, and SMS. With a filter, a subset of messages can be received.
+
EventBridge: A lightweight serverless option. A producer can (webhook) can publishes messages on the bus. With rules messages can be delivered to targets. Example targets are Lambda, API endpoints, SQS, SNS, Redshift, and many more. The EventBridge also provides an option to redeliver events based on a message Archive.
+
Kinesis: A serverless option for handling continuous streams of data in real-time. Kinesis is based or inspired by Kafka, AWS native, and opinionated.
+
Managed Kafka (MSK): A server-based option to handle continuous data streams. Kafka is highly configurable and will support most of the use-cases but also is more expensive and requires more knowledge to manage.
+
+
We are looking for a Serverless approach since we have in general not a constant or equally distributed load. And we have a strong preference for a SaaS-managed service instead to have to manage our own services. With these requirements, the AWS EventBridge seems a logical candidate to investigate.
+
+
+
+
+
+
+
+
+
The AWS EventBridge allows you to publish from several sources messages. Based on rules messages can be transformed and routed to several targets such as SQS, SNS, Lambda, RedShift, Firehose Data Steam, API endpoint, and many more.
+
In this POC we quickly explore how hard it is to set up an integration between GitHub and the AWS EventBridge to capture events and deliver the to different targets. For the experiment, we limit ourselves to the following targets.
+
+
CloudWatch LogGroup: can we get the integration working?
+
Lambda: Can we trigger custom business logic based on events?
+
Firehose Data Stream: Can we build a Data lake in S3?
+
+
+
+
+
+
+
+
+
+
As the final step, we will re-play the events sent earlier.
+
+
⚠️ Code and Terraform examples are not optimized or hardened ⚠️
+
+
Let’s do it!
+
Deliver GitHub events to CloudWatch LogGroup
+
As mentioned we use Terraform as IaC framework. The first resource we need is the event bus. To be able to replay messages later, we create an event archive as well.
The next question is, how can we deliver events from GitHub to the EventBridge? In GitHub, you can define an App with a webhook and subscribe to events. Events will be delivered to the webhook signed with a secret. Be aware that only events for repositories are sent when the app is installed in that repository. An easier alternative is to create a webhook on the enterprise, organization, or repository level.
+
Now we know how we can send events, we need a way to receive them in AWS and put them on the EventBridge. AWS recently announced a quick start to deliver GitHub events to the EventBridge.
+
+
+
+
+
+
+
+
+
The AWS quick start creates a Lambda function with an endpoint that you can configure in GitHub as a webhook. The Lambda checks the signature before delivering the messages to the Event Bus. Amazon provides a CloudFormation template that deploys a webhook to receive GitHub events and publish them on the bus. This setup through the web console is straightforward. It is not clear how the Lambda function code is maintained. Also, there seems not a way to lock the version of the code, the only way to do this is to maintain the code and CloudFormation template yourself. The provided Lambda does not provide any debug logging or configuration option in case you would change the source. GitHub also does not guarantee a max message size, and the AWS EventBridge only allows messages smaller than 256KB. When you would handle large messages differently, Lambda provides no option. Amazon suggests in those cases to download the Lambda from the console and maintaining it yourself. You can also fetch the Lambda zip from S3.
Since we already build a webhook to capture the GitHub events, stripping this function down and delivering the messages to the event bus is fairly simple. For now, we build our own function to keep more control. Giving us the option to take action on messages we cannot accept or add more logging. Later we can still decide to move to the AWS route. A webhook to handle the event in TypeScript looks as below, check the full sources on GitHub.
Next, we deploy the function with a function endpoint, keep in mind this endpoint is open to the world. The signature check validates the message. Creating a Lambda with a function url requires Terraform resources as described below. See here the full example.
After creating the Terraform resources we can test the webhook. A simple way to test events is by creating a test issue and ensuring your App or webhook is subscribed to events on issues. For the App, you can check the status of events in the advanced section of the App settings. You can redeliver the event here as well, quite handy for testing!
+
+
+
+
+
+
+
+
+
After creating an Issue we should see in AWS that our Lambda is triggered. On the Eventbridge, we can’t see the event. The next step is to deliver the event to a target. The first target we define for the test is a CooudWatch Log Group. Just for testing purposes.
+
Before events can be delivered to a target, an event rule needs to be created. For now, we forward all events from GitHub to the rule.
To an event rule, a target can be connected. Check the AWS developer documentation for the options. We limit ourselves to a few targets. First, we create the target log group.
The module we declared creates an event rule target to a log group and ensures the right IAM permissions are set. A similar pattern we apply later for Lambda and Firehose as targets of an event rule.
Time to trigger another event. As a result a log message in CloudWatch should appear.
+
+
+
+
+
+
+
+
+
Trigger a Lambda
+
Let’s check if we can do some more interesting things. Like sending a message to a Lambda. For that, we need a simple function. We keep it very simple and only echo the message. But you can imagine that you can replace this with any business logic required.
Details for the echo Lambda function are implemented in a module, similar to the webhook discussed earlier. This means we only have to add the modules to our main.tf. The module echo is implementing the Lambda function and declares the required Terraform resources. The second module, event_rule_target_lambda is similar to the one we created fom the log group before. But now tailored to a Lambda target.
When you trigger a new event or update the issue created before. The Lambda will be invoked and in the logging of the lambda, the event is printed. Not very useful, but remember we only check the pattern here.
+
+
+
+
+
+
+
+
+
Build a Data Lake in S3
+
The last target we checking now is Firehose delivery stream. With the Firehose stream, we deliver the message to S3 to mimic a Data Lake. With the following Terraform resources, you create a Firehose delivery stream to S3, configuration is standard. This means that data is cached till 5MB is received or 300 seconds are passed.
And to test our Data Lake we have to trigger some events again. After 5 five minutes you should see the first objects appearing in the S3 bucket.
+
+
+
+
+
+
+
+
+
Replay
+
The final question we would like to answer is, can we replay messages. We already have created a message archive. A replay can be initiated via the web console or CLI. In the web console navigate to the EventBus and select Replay. Next set the time windows and source for which you replay messages. After starting the job you should see messages appearing again on the targets.
+
+
+
+
+
+
+
+
+
Conclusion
+
Delivering messages from GitHub to AWS is straightforward with the AWS Eventbridge. As mentioned have some doubts to use the Amazon integration with GitHub today. Since it is not clear how to code is managed. And when not using Terraform you have to extract the webhook code from the CloudFormation template or Lambda. But on the other hand, this is nothing keeping us back. You should make your own judgment here. At this moment we will most likely build an maintain our own Lambda to ensure we can deal properly with cases when the messages exceed the maximum size of 256KB supported by the EventBridge.
+
Messages delivered to the EventBridge are forwarded to targets based on rules. In a rule, you specify matching criteria. This can be coarse-grained like we did. We matched the event only based on the source. When there is no EventRule that matches a message, the message ends up in /dev/null and you will get no notification that you missed something. This is not a problem, but should be considered and is more or less the opposite of SNS where you get all the messages unless you filter.
+
Looking at our current hacky approach, the way we delegate events for monitoring purposes to a second SQS queue, a move to AWS EventBridge seems much more flexible. And avoid tailoring the code. With a move to the AWS EventBridge, our solution could transform as follow.
+
+
+
+
+
+
+
+
+
The EventBridge is priced roughly with 1$ per 1 million messages published calculated in blocks of 64KB. Today we handle roughly 1 million messages a month for only the workflow_job event. When we start listening for many events this will likely double a few times. Besides that, you have to think about the costs of running the Lambda to handle the events. And finally, you will make costs to process the events.
+
Rate limits and quotas should also be considered. The EventBridge quotas for PutEvents per region vary from 10.000 per second to 400 per second. The messages posted on the EventBirdge should not exceed 256KB, similar to SQS and SNS. It seems many events in GitHub are relatively small, but for example, the push event can be big in case many branches and tags are pushed at once. When you start writing Lamba’s to handle events that require GitHub API calls, you should also think about the GitHub API rate limits. Using a personal access token (bad idea) you have a limit of 5000 API calls per hour. For a GitHub App this can be up to 15.000 per hour.
+
The AWS EventBridge seems to be a good option to start building a solution to handle GitHub events for building a Data lake as well as acting directly on events. In case you keep your architecture nicely lost coupled you can always move to the Ferrari, Kafka, for Eventing.
\ No newline at end of file
diff --git a/2023/01/12/just-in-time/index.html b/2023/01/12/just-in-time/index.html
new file mode 100644
index 00000000..45752972
--- /dev/null
+++ b/2023/01/12/just-in-time/index.html
@@ -0,0 +1,796 @@
+Just In Time | 040code
We trust you have received the usual lecture from the local System
+Administrator. It usually boils down to these three things:
+ #1) Respect the privacy of others.
+ #2) Think before you type.
+ #3) With great power comes great responsibility.
+
Maybe you remember this warning, the warning was shown to a user when using the sudo command to run tasks as administrator on Unix/Linux. As a GitHub Org admin, you do not explicitly switch with sudo to an admin view, you are admin all the time and to each action, so you should keep the sudo warning in mind. In this blog, we explore how we can become just-in-time admin, like using sudo on Linux.
+
The problem
+
Are you an admin for a GitHub Org? Then you likely use the same user that you use for day-to-day work, like developing code, creating pull requests, reviewing pull requests, and more. When you perform a task for changing privileged settings you have to authenticate yourself like when you do a sudo operation.
+
+
+
+
+
+
+
+
+
But how do you then differentiate between the access you have as a normal user because you are a member of a team or collaborator on a repository, and the full access you are granted as administrator? The answer is; there is no easy way to see the difference.
+
Checkout detailed examples
+For example, when non-admin team members work on the code, normal rules are in place based on the branch protection and access level to the repository. But what are the rules for you as an admin? And should the rules not be the same as for the rest of the team when you are just working on the code?
+
Or when you contribute to the repository from another team, which is shared as InnerSource, the process and possibilities for a non-admin Org member are clear, and mistakes cannot be made as a member. Depending on how InnerSource is enabled, a user has to create a fork or branch, and create a pull request for the change. However, as an Org admin you can do anything on that repository because there are no safeguards for you. To act properly, you should check the settings and guides for what the expected way of working is. Are you checking this every single time?
+
+
Because your user is always admininistrator, you can always do anything in your Org. You can see any repository, make any change, operate on any pull request, push commits to any repository, and much more. For most of those operations no explict authentication like sudo is required. Only when you change settings with high impact an extra authenticaton is triggered. And normal members in the Org can only act on repositories to which they are explicit granted acces.
+
Just in Time Admin with IssueOps
+
A solution for the problem is using two users. One user you use as admin, and the second as a normal member. Besides the costs of an extra license, a mistake can be made still easily. And with the standard SSO including MFA having two users is in many organizations not trivial. Simply because you are logged in with the wrong user. Another way of solving the problem is becoming just-in-time (JIT) admin, similar to sudo.
+
At the moment of writing GitHub does not provide an option to become just-in-time admin. The GitHub Service team open-sourced an action and JavaScript CLI which automate the promoting and demoting of normal Org members to admin based on the JIT principle.
+
+
+
What is IssueOps? IssueOps is automating the process around GitHub Issues with GitHub Action. With optional approvals before automation kicks in.
+
+
+
We are going to use this action to automate “just-in-time” promotion to Org admins with IssueOps. The action will execute highly privileged operations on our GitHub Org and thus it requires adminstration permissions for your Org. We will be very careful to who we provide the token because malicious actions could misuse the token. When trusting third-party actions it is strongly recommended that you lock the action on the Git SHA. Locking the action to the exact version is just the first step in hardening. Think also about checking the source code, libraries used, and what you run. For many JavaScript actions, you simply don’t know if the generated runtime bundle is matching the sources. The topic of securing your software supply chain is a topic on itself and we not cover in detail in this post.
+
Fork
+
To ensure we keep our admin token safe we take a copy of the repository, check the code and ensure we generate the execution bundle ourselves. We fork the admin action repository and make a few modifications. It is up to you if you fork or copy the repository. But make sure you always check the license. In this case, the license is MIT which means no limitations in usages or rules to contribute back.
+
The readme of the admin-support-issueops-actions repository is a good guide for setting up the automation, including directions on how to extend it. Creating your own extensions is great, but we believe those should be contributed back as open source. Because it is much better to use each other’s power than build the same in our closed environments over and over again.
+
Read how we setup the fork to secure the action.
+Besides the advice given in the repository, we think it is good practice to leave our upstream as untouched as possible, this ensures we can keep it aligned with upstream changes. As already addressed, we want to guarantee that our runtime matches our sources. There a many ways to safeguard this. We choose a simple two-step approach. First, extend the standard CI workflow for pull request builds and add a [step](https://github.com/040code/admin-support-issueops-actions/blob/d37694f3d0293442ae89034ad3b5e320e0722b02/.github/workflows/ci.yml#L35-L49) to check the distribution is up-to-date, and if not update the branch of the PR.
+
Second, add a release workflow based on please-release. This release workflow creates a branch and PR for the next release. By providing a token on behalf of an App instead of the default GitHub Action token we ensure builds on the created release PR are triggered. The App has read/write access to the contents and pull request of the repository. Later in the post, we cover App creation as well.
We have now safeguarded that for our releases the runtime matches the sources. For each upcoming release, the release action is maintaining a PR. On this PR the standard CI is triggered, with our extension which checks for updates to the runtime, and if needed applies those updates. When the PR has been merged, a release is created with an up-to-date distribution.
+
+
+
+
+
+
+
+
+
This are all the changes to our fork. As said, we try to keep the sources as close as possible to the upstream. Now we can implement the promotion and demotion in a second repository using this action.
+
+
+
+
+
+
+
+
+
+
Promotion and demotion workflow
+
The admin support action implements a simple promotion/demotion process. Key in the process is that only members have access to the repository for admin request issues. This means the repository has to be ⚠️ private ⚠️. The diagram illustrates the process of demotion and demotion.
+
+
+
+
+
+
+
+
+
+
A member of the Org that has access to the admin’s repository creates an issue to request admin add access. Next, an action starts and handles the issue to promote the user. The issue is kept open as long the user is busy with his or her admin task. Once the admin task is finished the user closes the issue. The closure of the issue will start the demotion workflow to demote the user back to a normal member. As a fallback, scheduled workflow ensures that after a time-out the user is always demoted back to a standard member. The user can set the time-out in the issue which should be between 1 and 8 hours.
+
Action setup
+
Now we have discussed the working of the action it is time to set up the actual workflows. First, we need to create a repository to handle the issues for requesting admin promotion. Remember this repository must be private and you should only grant access to members in your Org who are allowed to become an admin. Create a new repository, for example via the GitHub CLI:
+
gh repo create my-org/admins --private
+
We have already discussed the process for promotion and demotion. This process requires three different workflows:
+
+
Promotion workflow that acts on issue creation
+
Demotion workflow that acts on issue closure
+
Timeout check to ensure a user is demoted after a maximum duration that acts on a schedule (cron).
+
+
+
PAT vs App token
+
The admin support action suggests using a PAT (personal access token). We prefer to use an App token instead of the PAT. It avoids user accounts that are required to run automation. With the old PAT, the scope of the token was far too wide, with the new PAT, a token can be limited in the same way as an App token. A second issue of the PAT is the rate limit, although that may not be a problem for this use case but could be in general a challenge. Finally, it saves you a license seat. The action can run with an App token but the template workflows require a few modifications.
+
+
Create the Admin support App
+
As discussed we will use an App token for authorization against the GitHub API. Go to your Org developer’s settings and create a GitHub App. Set the following settings for the new App:
+
+
Disable the webhook
+
Repository permissions: contents read/write
+
Organization permissions: Administration read/write, Members read/write
+
+
Save your App, download the SSH key of the App, and make a note of the App id. The last step is to install the App to the created repository to manage admins. This grants the App access to the repository.
+
+
Repository configuration
+
The admin support action requires a configuration file config.yml in the root of the repository. We set up the admin promotion/demotion for a single Org. If needed you can set up the action for multiple Orgs. Add a similar file like below to your admin’s repository.
The automation we are going to run requires labels on the repository. The example workflows are using the actions-ecosystem/action-add-labels action. This action is not well maintained, therefore we replace the label created by simple gh CLI commands. Drawback: we have to create the labels first. You can create the labels with this simple script.
Finally, the workflow needs to get a token for the created GitHub App. This requires you to define the following two secrets. First add the id of the App as APP_ID. Next, add the base64 encoded string of the private ssh key as APP_PRIVATE_KEY_BASE64. You can convert the ssh key to a base encoded string by running cat key-file.pem | base64 | pbcopy..
+
+
Issue template
+
We are going to promote and demote users with IssueOps which requires that we can automatically process the issue with Actions. To help the user to create an issue in a predictable format we use an issue template. Note that issue templates are not supported in private repositories for Orgs on the free plan. Create the directory .github/ISSUE_TEMPLATE and add create an issue template yaml file.
+
---
+name: Request administrator permission in the organization
+about: Allows the support team to request temporary admin permission in an organization
+title: Request administrator permission
+labels:''
+assignees:''
+---
+
+Organization: my-org
+Description:
+Duration:2
+Ticket:0
+
As you can see several fields are required. You can use the setup for multiple organizations, for now, we use it for one Org. The Duration can be set to max eight hours but we will set the default to two hours. The tickets are meant to refer to an external ticket system and because we use GitHub issues we can ignore this field.
+
+
Promotion workflow
+
The promotion workflow is triggered once an issue is created. After parsing the issue the workflow promotes the user to admin. This and the following workflows are using third-party action based on version. Be aware that a version is mutable; it is strongly recommended to lock your actions on SHA instead of a tag.
The workflow should be easy to read, but lets discuss the most interesting details. Since we are using an App for executing API calls to promote users, we first need to obtain a token for the App. We use an action that can get a token for an app [#1]. For all API calls that don’t need the admin token, we use the repo-level secret injected by GitHub Actions, aka GITHUB_TOKEN. After parsing the issue and setting labels the user is granted access, here we use the token from the App [#3]. Once the user is promoted, a comment is made on the issue to inform the user.
+
We can already test our promotion. Create an issue based on the template. You should shortly see that the issue is updated like below.
+`
+
+
+
+
+
+
+
+
+
+
Demotion
+
Now we can promote the user, the next step is to dethrone the user and revoke the admin privileges. The second workflow acts on closing the issue.
+
name: Demote a user
+on:
+ issues:
+ types:[closed]
+
+jobs:
+ demote-workflow:
+ name: Demoting a user for closing an issue
+ runs-on: ubuntu-latest
+ permissions:
+ issues: write
+ contents: write
+ env:
+ GH_TOKEN: ${{ github.token }}
+ DEMOTION_ERROR_NOTIFY:"@npalm"
+
+ steps:
+ -name: Checkout repository
+ uses: actions/checkout@v3
+
+ -uses: philips-software/app-token-action@v1.1.2
+ id: token
+ with:
+ app_id: ${{ secrets.APP_ID }}
+ app_base64_private_key: ${{ secrets.APP_PRIVATE_KEY_BASE64 }}
+ auth_type: installation
+
+ -name: Add label automation-running
+ if: always()
+ run: gh issue edit --add-label automation-running ${{ github.event.issue.number }}
+
+ -name: Parse the issue submitted
+ id: issue_parser
+ uses: 040code/admin-support-issueops-actions/admin-support-cli@v1.0.0
+ with:
+ action:"parse_issue"
+ issue_number: ${{ github.event.issue.number }}
+ ticket: ${{ github.event.issue.number }}
+
+ -name: Parse issue_parser json output
+ id: parse_issue_output
+ run:|
+ target_org=$(echo ${{toJSON(steps.issue_parser.outputs.output)}} | jq -r .target_org)
+ description=$(echo ${{toJSON(steps.issue_parser.outputs.output)}} | jq -r .description)
+ duration=$(echo ${{toJSON(steps.issue_parser.outputs.output)}} | jq -r .duration)
+ echo "target_org=$target_org" >> $GITHUB_OUTPUT
+ echo "description=$description" >> $GITHUB_OUTPUT
+ echo "duration=$duration" >> $GITHUB_OUTPUT
+
+ -name: Demote user
+ id: demote_admin
+ uses: 040code/admin-support-issueops-actions/admin-support-cli@v1.0.0
+ continue-on-error:true
+ with:
+ action:"promote_demote"
+ username: ${{ github.event.issue.user.login }}
+ target_org: ${{ steps.parse_issue_output.outputs.target_org }}
+ role:"member"
+ admin_token: ${{ steps.token.outputs.token }}
+
+ -name: Add a comment on the issue to confirm the demotion
+ uses: actions/github-script@v6
+ if: success()
+ with:
+ github-token: ${{secrets.GITHUB_TOKEN}}
+ script:|
+ await github.rest.issues.createComment({
+ issue_number: context.issue.number,
+ owner: context.repo.owner,
+ repo: context.repo.repo,
+ body: `✅ We have executed the request and now the user **@${{github.event.issue.user.login}}** has been demoted from ${{steps.parse_issue_output.outputs.target_org}}. \n\n This issue will be locked to avoid new interactions
+
+ <sub>
+ Find details of the automation <a href="https://github.com/${context.repo.owner}/${context.repo.repo}/actions/runs/${{github.run_id}}">here</a>.
+ </sub>
+ `
+ })
+ await github.rest.issues.lock({
+ issue_number: context.issue.number,
+ owner: context.repo.owner,
+ repo: context.repo.repo
+ })
+
+ -name: Add a comment to notify the team that this automation failed
+ uses: actions/github-script@v6
+ if: failure()
+ with:
+ github-token: ${{secrets.GITHUB_TOKEN}}
+ script:|
+ await github.rest.issues.createComment({
+ issue_number: context.issue.number,
+ owner: context.repo.owner,
+ repo: context.repo.repo,
+ body: `Demoting the user has failed. ${{env.DEMOTION_ERROR_NOTIFY}} have a look to make sure the user is left in a correct state.
+
+ <sub>
+ Find details of the automation <a href="https://github.com/${context.repo.owner}/${context.repo.repo}/actions/runs/${{github.run_id}}">here</a>.
+ </sub>
+ `
+ })
+
+ -name: Add labels user-demoted, manual-demotion
+ if: ${{ success() && github.event.sender.login == github.event.issue.user.login }}
+ run:|
+ gh issue edit --add-label user-demoted ${{ github.event.issue.number }}
+ gh issue edit --add-label manual-demotion ${{ github.event.issue.number }}
+
+ -name: Add labels user-demoted, manual-demotion
+ if: ${{ success() && github.event.sender.login != github.event.issue.user.login }}
+ run:|
+ gh issue edit --add-label user-demoted ${{ github.event.issue.number }}
+ gh issue edit --add-label automatic-demotion ${{ github.event.issue.number }}
+
+ -name: Remove label user-promoted
+ if: success()
+ run: gh issue edit --remove-label user-promoted ${{ github.event.issue.number }}
+
+ -name: Remove label automation-running
+ if: always()
+ run: gh issue edit --remove-label automation-running ${{ github.event.issue.number }}
+
The demotion workflow is similar to the promotion workflow, but in reverse. After getting a token and parsing the issue, the user is reverted back to normal Org member. Again a comment to the issue is made to inform the user. In case of any failure, a comment is made to trigger a notification to a predefined user. The admin support action also supports retrieving the relevant parts from the audit log, and writing them as an audit record back in the repository.
+
Now we can demote our user back to normal user by just closing this issue. After closing the issue will be updated like below.
+
+
+
+
+
+
+
+
+
+
Timeout
+
The timeout is a safeguard. We run a scheduled workflow every hour to check the open issues. In case the user has reached the maximum duration of being admin, one to eight hours. This workflow will close the issue which triggers the default demotion workflow that runs on the closure of an issue.
+
name: Provisioning check to see if a user needs to be demoted
+on:
+ schedule:
+ -cron:"0 * * * *"
+ workflow_dispatch:
+
+jobs:
+ provisioning-check:
+ name: Close issues with expired duration
+ runs-on: ubuntu-latest
+ steps:
+ -uses: philips-software/app-token-action@v1.1.2
+ id: token
+ with:
+ app_id: ${{ secrets.APP_ID }}
+ app_base64_private_key: ${{ secrets.APP_PRIVATE_KEY_BASE64 }}
+ auth_type: installation
+
+ -name: Checkout repository
+ uses: actions/checkout@v3
+
+ -name: Run through all the issues and close them if they are expired
+ id: issue_parser
+ uses: 040code/admin-support-issueops-actions/admin-support-cli@v1.0.0
+ with:
+ action:"check_auto_demotion"
+ ticket: ${{ github.event.issue.number }}
+ # Require a non default action token, otherwise it won't trigger a job on issue close
+ admin_token: ${{ steps.token.outputs.token }}
+
This workflow does not need much explanation. As mention, it checks on hourly based the open issues using the admin-support-issueops-action. When running it once in an hour it could result that the issue being closed almost an hour after the time-out, but nothing holds you back to run the workflow more frequently.
+
Fireside chat
+
In this article, we have explained how you can promote members to admin and back to a normal member with IssueOps. We run IssueOps via action, which makes you dependent on GitHub Actions. When GitHub Acton is down your promotion/demotion workflows are broken. An alternative could be hooking into the GitHub events with a webhook. Reduce your dependencies, but you are still dependent now on the GitHub eventing system. When moving to Just In Time access via IssueOps it sounds smart to keep a backup user available, one that normally not is used, but that is available for disaster recovery.
+
The solution provided by GitHub via the admin support action makes it possible to have just-in-time admin. But it would be much better to have this feature by default in the platform. The setup with a private repo gives you also slightly more control over who can add an admin. By distinguishing user permission levels on this repository.
+
Now our Org admins are no longer admin anymore by default and are only promoted to admin for doing specific tasks. Should we not ask ourselves, why do we have all the time repository admins? Is this required? Should we not apply a similar mechanism to them as well?
\ No newline at end of file
diff --git a/404/index.html b/404/index.html
new file mode 100644
index 00000000..03bf1877
--- /dev/null
+++ b/404/index.html
@@ -0,0 +1,112 @@
+Page Not Found | 040code
404 Page Not Found
👻
Looks like you've followed a broken link or entered a URL that doesn't exist on this site.
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+