Build Java Container Images in seconds using Quarkus CLI

Here's a quick 7 minute video I threw together that shows you how to use the Quarkus CLI to build images using Podman / Docker and Jib and runs the container using Podman Desktop.

TL;DR If you just want to try it out, just run this:
quarkus image build


As a Java developer, getting started with building containers can get complicated real quick. At the face of it you just copy your artifact into a base image and do a docker or podman build and that's it right?! On a serious note, creating a container from your application requires quite a few more considerations than that. To begin with, you have to start with a base image: do you start from scratch, or use a base image that already has a jvm implementation included? Or maybe you've opted to do a native build of your application and you don't need a jvm running in your container. Either way, you're going to need to find a trusted source that provides tested, secured, possibly signed, and stable images for the particular Java version you're targetting. And hopefully this source will be able to support you as well as being able to maintain and backport patches going forward. I'm of course a little bit biased, but I always start from Red Hat's Universal Base Images (UBI). These UBIs are free to use without restrictions which is pretty cool, and they are at the core a subset of the Red Hat Linux (RHEL) distribution so you know they're going to be stable and trusted. There are several JVM based UBI images, as well as micro UBIs that can be used if you're going to deploy a native binary.

Once you have decided on a base image, you will need a way to build your container. Docker build is the most known way of building containers, but certainly not the only one. You can use Podman, a pure open source alternative with more security capabilities built in, or you can use BuildPacks or Jib, the latter being a container build tool specifically created for building container images with Java. The downside is that for each of these tools, you need to know how to build a container image; and moreover, you'll need to have knowledge of the above mentioned base images and how to write a Dockerfile/Containerfile to copy your artifacts and dependencies into them.

Quarkus & Containers

Quarkus wants to bring Developer Joy and it is not different when it comes to building containers. When you create a Quarkus project, whether you use Quarkus CLI quarkus create app, code.quarkus.io or one of the quarkus plugins in VS Code or IntelliJ, Quarkus will create a folder in your src/main called 'docker'. And in this src/main/docker, you'll find a few prebaked Dockerfiles that are ready to go using those UBI base images I mentioned before. What's even better is that the Quarkus plugin has knowledge of these Dockerfiles so you can build images using your favorite project build tool (Maven or Gradle). Basically, you can do an image build with eg. mvn quarkus:image-build (make sure to mvn compile your code first) and quarkus will automatically build a container image using, by default, the Dockerfile.jvm that came with your Quarkus project. Pretty sweet right? But it gets better.

The Quarkus CLI

The Quarkus CLI is basically 'just' a convenience wrapper around Maven or Gradle (depending on your preference) that lets you create projects, manage extensions and do essential build and dev commands. This is where things get interesting, because you can do really simple commands that are easy to remember such as quarkus image build. Behind the scenes, Quarkus will invoke your build tool (by default Maven, but can be Gradle as well) and run both `mvn compile` and `mvn quarkus:image-build`. So you can literally compile your code and build the container image with the opinionated base images provided by Quarkus with one simple command. That's pretty cool if you ask me. Actually, it could get even better because if you want to push your built container to a registry as well, it could be done in one go with the artifact and container build like so quarkus image push --also-build!

What if I don’t have Docker or Podman on my machine?

Quarkus has extensions for other container build tools such as Jib or Buildpacks. You just have to add the extension, eg quarkus ext add jib and subsequently specify your build strategy when you invoke the quarkus build command: quarkus image build jib.

Podman Desktop

Another alternative is to use Podman Desktop. This tool is FOSS (Free Open Source Software) and makes managing containers really easy. When you install it, it'll help you install the underlying Podman engine. Once installed, you can spin up your container using the Podman desktop UI. Just go to the Images tab and hit the play button next to your container image. Podman will automatically configure the port the app will run on (usually 8080 by default in Quarkus) but it’ll also automatically figure out if that port is already being used and reassign the container to use a different port instead.

podman desktop view images
Podman Desktop's "Images" view, where you can see the created container image and the play button to start it

podman desktop start container from image view
Podman Desktop's "Create Container" view, where you can define settings and start the container

podman desktop view containers
Podman Desktop's "Containers" view listing the containers on your machine

podman desktop view container logs
Log output from the running container

I hope you liked this post. If you have any questions or comments, feel free to share them in the comments section. And don't forget, sharing is caring :)

Using Kafka with Camel Quarkus

I recently spoke at Devoxx Belgium about Camel Quarkus (you can find the recording here: https://youtu.be/FBWgbhp8FG8) and for this talk I wanted to demonstrate just how easy it is to integrate different components using Camel. I also wanted some interaction with the audience and make the session a little more fun by having them participate in the demo.

My initial idea was a voting app where the audience could vote for their favorite IDE by using Twitter. It seemed like a neat idea, I could use the Twitter component of Apache Camel which I had already demo'd during a talk I did for the Openshift Coffee Hour (https://youtu.be/aaEV19c6GhQ), but ultimately I decided against it because I really wanted more than just a handful of votes (if that). Since my talk was also about using Camel Quarkus in a Serverless way I needed a LOT of votes from the audience :P.

What I came up with was to build a small web app where the audience could vote through the UI which would send a REST Get request to an API that I would expose using Camel; and then update a database that I could poll to show the results live on the big screen. I didn't want the API requests to immediately update the database, but instead stream them to a Kafka topic first and then handle the requests and update the database under a controlled load. That would allow me to also demonstrate how easy it is to integrate Camel with Kafka. It's basically one line of code:

to("kafka:{{kafka.topic.name}}")

Easy right?

I still wanted to show the Twitter component as well as an additional feature, and finally, just in case there weren't going to be too many votes, I prepared another rest endpoint that I could call with jmeter to 'cheat' the result so I could show the audience how the system would scale pods out rapidly under load. Using Quarkus and Knative that's super easy to do since Quarkus starts up super fast, and Knative scales out rapidly based on requests.

Architecture

The final architecture looks a little bit like this (forgive my crappy architecture design skills)

Camel Quarkus Voter Architecture

The UI application features a, well, UI, that shows the results of a poll and a form to vote for your favorite Java stack. When you vote, a REST POST event gets sent to the 'ingester' app, which will translate the result and add it to a Kafka topic.
The 3rd application (processor) consumes the Kafka messages at its own pace and updates the database accordingly. (eg. if someone voted 'Vim', the counter for vim would be incremented by 1) The 'retriever' application (currently embedded in the processor app) has a REST GET endpoint to get the results from the DB.

Anatomy

So ultimately the application is composed of the following components:

UI
The ui application displays a list of java stacks/frameworks that you can vote for by clicking the respective button next to it. This action calls the ingester app. The page also displays a bar chart of the results so far. The app is built with Quarkus Qute templating and some crappy javascript/jquery code :P

Ingester
The ingester Camel Quarkus application receives requests from the user (via HTTP) and forwards the requests to the Kafka broker. The main component of the application:

  • RestToCamelRoute : Camel route that receives a rest call and forwards it on to a Kafka broker

Processor
The processor Camel Quarkus application receives the vote requests from Kafka, processes them, and writes results into the votesdb Postgres DB table. As of right now it also returns the results through a /getresults endpoint. The application has the following Camel Routes:

  • processor/VotesRoute consumes messages from a kafka topic (in json format), extracts the value of 'shortname' and increments the counter of the java stack that matches with this shortname.
  • RestRoute returns data from the votes table in json format

Twitter
The twitter Camel Quarkus application polls twitter for matches to my handle (@kevindubois) and scans for matches to the various IDEs I'm using.

  • TwitterRoute polls twitter, and forwards any matches to the same Kafka topic messages

And this is basically the UI of the voting app:

You can find the source code in my github: https://github.com/kdubois/CamelQuarkusVoter

Using Camel Quarkus to send alerts on Telegram when web content changes

Ahh… Covid-19. From thinking it was just another little outbreak in Asia to months long lockdowns and seeing friends and family succumb to this nasty little virus, it's been a surreal year-and-a-half. Earlier this year we finally got some hope with the rollout of vaccines, and I personally was very eager to get vaccinated and hopefully leave my anxiety to leave the house behind me. I currently reside in Brussels, Belgium, and unlike in the US, the vaccine rollout was, at least initially, painfully slow. The Brussels government did however set up a nice little web application where they would keep everyone up to date with the current age group that was eligible to get vaccinated. The only caveat was that there was no way to keep informed of any changes to the age group other than refreshing the website. Which I did several times a day for a day or two. But, I wouldn't work in tech if it didn't seem a bit silly to perform a manual repetitive task right? :D

I saw a while ago a nice little demo of one of my colleagues at Red Hat who used Red Hat Fuse (based on Apache Camel) to send a message to Telegram, and this seemed like a pretty good way to go since I use Telegram on a daily basis anyway. The advantage of using an Enterprise Integration Pattern based system is that it makes the solution quite flexible, so if at some point I want to send the message to another endpoint then all I need to change is the "to" parameter of my Camel route and I'm good to go. Likewise, if I want to adapt this code to scrape bits of data from another website, then that's a very easy change as well.

Anyway, take a look at the video and let me know what you think!

The source code can be found in my Github: https://github.com/kdubois/datascraper. Feel free to scrutinize and send pull requests with improvements!

Gitlab CI/CD on Openshift with Quay integration

Openshift has pretty cool built in CI/CD solutions (based on Jenkins or Tekton), but it can easily be used with external CI/CD tools too. I recently had to create a small POC for a customer who wanted to see how Openshift integrates with Gitlab, and they were also curious how they could incorporate container scanning and the Quay registry. I figured I might as well share this setup with the rest of the world, so below you can find the steps to set it up. Let me know what you think in the comments!

You can find the source code here: https://gitlab.com/kdubois/quay-openshift-gitlab-ci

Gitlab CI/CD on Openshift (with Quay container scanning integration)

This project demonstrates how a Python application could be built and deployed to Openshift using Gitlab CI/CD. It also demonstrates how a Quay container registry could be leveraged to store and scan the container image, and how the deployment can be interrupted if any vulnerabilities are discovered in the image.
Gitlab provides a certified operator to deploy a Gitlab runner. This is the only secure way to integrate Gitlab CI/CD with Kubernetes.

Steps to reproduce:

Clone the project

Clone this repository in your Gitlab repository. It contains a very basic Flask app; a .gitlab-ci.yml file that defines the pipeline, a kubefiles folder that contains the kubernetes yaml files referenced below in the instructions; a Dockerfile which can be used to build a Python base image without vulnerabilities; and Dockerfile in the gitlab folder that's used for the runner's container image. It includes some utilities such as Skopeo and openshift client tools (oc, kubectl, kn)

Configure Gitlab

First, we're going to create a project for the Gitlab CI/CD tools. Log in to Openshift and create a gitlab ci/cd namespace / project: oc new-project gitlab
In this project, we'll install an operator that's going to manage the lifecycle of a Gitlab runner instance. To do this, go to the OperatorHub in your Openshift cluster UI (Administrator > Operators > OperatorHub), search for 'Gitlab' and install the Operator. Alternatively you can also install the operator by applying the following yaml: oc apply -f kubefiles/gitlab-operator.yaml
Next, we're going to deploy the actual Gitlab runner instance, but before we can do that, we need a Gitlab token so the runner can be mapped back to Gitlab.
To create/retrieve a Gitlab token, log in to your Gitlab repository, go to settings > CI / CD > Runners>(expand). Then under 'Specific Runners' -> 'Set up a specific Runner manually' you will find the token.
The runner is configured to use the token through a 'secret' that's referenced in the runner config. To create the secret: oc create secret generic runner-token-secret –from-literal runner_registration_token=__YOUR_TOKEN__ -n gitlab
Now we can deploy the runner with oc apply -f kubefiles/gitlab-runner.yaml (you can do it through the 'Installed Operators > Gitlab > 'Create instance' in the UI as well).
Go back to Gitlab again, and verify that the runner has been registered (settings>CI/CD->Runners>Runners activated for this project). If everything went well you should see the runner in the list. Click on the edit icon and check the 'Run untagged jobs'.

Configure the Build

At this point we can create a project for the application we're going to deploy. eg. oc new-project python-project
Since the runner is deployed in a different namespace (gitlab) and Openshift by default isolates namespaces, we will need to explicitely allow the gitlab runner's service account to have access to the python-project namespace. This can done with the following command: oc policy add-role-to-user edit system:serviceaccount:gitlab:default -n python-project (feel free to create a more specific service account, for the sake of this demo we're just going to use the default service account of the gitlab project)
One of the nice things of Openshift is that it allows you to build images directly on the cluster, so you don't need to have a local container build/run environment and pull secrets etc configured. For this demo, you can just apply the img-build.yaml that's included in the project, but if you'd like to learn more about Openshift builds, check out the documentation: https://docs.openshift.com/container-platform/4.5/builds/understanding-image-builds.html.
The img-build.yaml tells Openshift to create a container image using the location of this git repository, and a given Python base image. Apply it with oc apply -f kubefiles/img-build.yaml -n python-project

Quay

The pipeline is configured to push the built image to a Quay registry. While the built-in Openshift registry does a great job for basic registry functionality, many organizations opt to use an external registry, especially when they have multiple clusters. The external registry can live inside of Openshift, but for this use case it's using a Quay.io repository. One of the nice things about Quay is that it has a built in container scanner. Our demo pipeline will check for the result of this scan (using a custom scanresult.py script included in this repository) before promoting the image and deploying it, unless there are high/critical vulnerabilities – in this case the pipeline will be marked as failed.
Connect Quay to Gitlab: In Quay.io, log in (or create a free account if you don't have a login yet), and create a new repository 'python-app'. If you don't have a robot account yet, go to user settings -> Robot Accounts, and create a robot account. Then click on the new account, and click on the 'Robot Token'. With these credentials, you will need to create two environment variables in Gitlab. In the Gitlab repository, go to Settings -> CI/CD -> Variables and add a new variable 'REG_CREDS', with value : . (the robot username and the token, separated by a colon). Create the second variable 'REGISTRY_NAMESPACE' to set the Quay namespace, which should correspond with your Quay username (not the Robot username).

Set up Knative/Openshift Serverless

The pipeline is configured to deploy the python app in a 'serverless' way. This is a 'bonus' capability of Openshift, deployable as an operator. To deploy it, go to the operatorhub in your Openshift instance and install the 'Openshift Serverless' operator. Once it's installed, create a new 'knative-serving' project and in it, go to the Installed Operators > Openshift Serverless Operator > Knative Serving > Create Knative Serving. The default options should be fine for this use case.

Kick off the build

Everything should be in place to run the CI/CD pipeline now. In your Gitlab repository, go to Pipelines and click the 'Run Pipeline' button. You'll notice that the pipeline starts running (if it's in stuck status, make sure you checked the 'run untagged jobs' for the runner as described in the 'Configure Gitlab' section above). If there are no vulnerabilities in the base image or code, then the build should pass and the application gets deployed using Openshift's Serverless Serving capability. This means the application is going to start up, wait for requests, and if there are no requests coming in to the application, it will scale down to 0 and wait for requests to scale back up.
That's it!

Extras

ArgoCD Automated Deploy:

If you'd like to manage the application configuration with ArgoCD:
To install ArgoCD on your cluster:
Install the ArgoCD (community) operator through the UI; or with oc create -f argocd-operator.yaml -n argocd
Deploy ArgoCD instance: oc create -f kubefiles/argocd.yaml -n argocd
Once it's deployed, get the admin password: oc get secret argocd-cluster -n argocd -o jsonpath="{.data.admin\.password}" -n argocd and log in to the UI with the route that was exposed: oc get route argocd-server -n argocd -o jsonpath="{.spec.host}"
Once logged in to ArgoCD, create a new app, in fill out the form with the required information. Note that the 'Project' is an ArgoCD terminology so you can set this to 'default'. For Destination, if you deployed ArgoCD on the same OCP cluster you can use "https://kubernetes.default.svc", otherwise use your cluster's info. Set namespace to "python-app".
At this point you can go ahead and sync the project and it should automatically deploy the application. Feel free to remove the 'deployment' step from the .gitlab-ci.yml since it too will try to deploy.

Vulnerability scan

To see the pipeline fail due to vulnerabilities, you can configure the kubefiles/img-build.yaml to use a different base image, or you could try adding to the threshold_list in the scanresult.py on line 34 (adding 'Medium' did the trick at the time of writing)

Burst Requests to the application

Since the application is leveraging Knative / Openshift Serverless, it scales very rapidly based on the incoming requests. To see this in action, there is a 'knburst.sh' script included in this repository. To run it, open a terminal, make sure you're logged in to your Openshift cluster and are using the python-project project, and then launch the script and watch what happens to the number of pods of the application. Play around with the memory and cpu limits to see what effect it has on the number of pods that get deployed.

Custom git repo

If you would like to use a custom git repo not hosted at gitlab.com, you will need to change the URL in kubefiles/gitlab-runner.yaml

Synchronize between Subversion and Git using Continuous Integration Pipelines and Containers

I recently worked on a project for a client that (still) uses Subversion for their code repositories. Unfortunately they had no real short-term interested in migrating their code to Git, which left the team without such goodies as merge/pull requests, easy branching and all the other advantages that Git provides. Git actually has a built-in tool called 'git-svn'. Its main purpose is to migrate from Subversion to Git, or at the most do one-way syncing. In my case, two-way synchronization was the only option, since the Subversion repository was also used by another independent team.

After some experimentation I created a script that was able to do 2-way synchronization without too many problems, though this was just on my local machine, and we needed a solution for the entire team.

Jenkins, Containers & Gitlab to the rescue!
What I came up with was to build a Jenkins pipeline script that does the heavy lifting of synchronizing between git and subversion, while also integrating a Continuous Integration pipeline so that the main codebase is kept stable and thus *should* never break.

Fortunately the client was nice enough to provide us with a bare-bones RHEL 7 VM, and from there, I created a containerized CI/CD suite with containers for Jenkins, Gitlab, Sonar and a local Artifactory for snapshot build management. I created 2 Jenkins pipeline scripts for each project, one to do the git-svn synchronizations, and one to run a ci/cd pipeline that sets up an ad-hoc Docker container for the app (and one for the DB), and runs build & tests against them.

This CI/CD pipeline is triggered both by opened merge requests into the develop branch (no one is allowed to commit directly to develop/master branches), and by the git-svn sync pipeline script. The latter synchronizes between the master branch of git and the subversion repository and triggers the CI/CD pipeline in reverse order, that is, it rebases SVN code into master, and after the CI/CD pipeline successfully passes, merges the code into develop. This way, the develop branch is protected from broken build commits both from SVN and from feature branches.

I created a demo project with examples of the pipeline script and a full-fledged containerized solution you can use as a starting point:
https://github.com/kdubois/git-svn-pipeline