Using Kafka with Camel Quarkus

I recently spoke at Devoxx Belgium about Camel Quarkus (you can find the recording here: https://youtu.be/FBWgbhp8FG8) and for this talk I wanted to demonstrate just how easy it is to integrate different components using Camel. I also wanted some interaction with the audience and make the session a little more fun by having them participate in the demo.

My initial idea was a voting app where the audience could vote for their favorite IDE by using Twitter. It seemed like a neat idea, I could use the Twitter component of Apache Camel which I had already demo'd during a talk I did for the Openshift Coffee Hour (https://youtu.be/aaEV19c6GhQ), but ultimately I decided against it because I really wanted more than just a handful of votes (if that). Since my talk was also about using Camel Quarkus in a Serverless way I needed a LOT of votes from the audience :P.

What I came up with was to build a small web app where the audience could vote through the UI which would send a REST Get request to an API that I would expose using Camel; and then update a database that I could poll to show the results live on the big screen. I didn't want the API requests to immediately update the database, but instead stream them to a Kafka topic first and then handle the requests and update the database under a controlled load. That would allow me to also demonstrate how easy it is to integrate Camel with Kafka. It's basically one line of code:

to("kafka:{{kafka.topic.name}}")

Easy right?

I still wanted to show the Twitter component as well as an additional feature, and finally, just in case there weren't going to be too many votes, I prepared another rest endpoint that I could call with jmeter to 'cheat' the result so I could show the audience how the system would scale pods out rapidly under load. Using Quarkus and Knative that's super easy to do since Quarkus starts up super fast, and Knative scales out rapidly based on requests.

Architecture

The final architecture looks a little bit like this (forgive my crappy architecture design skills)

Camel Quarkus Voter Architecture

The UI application features a, well, UI, that shows the results of a poll and a form to vote for your favorite Java stack. When you vote, a REST POST event gets sent to the 'ingester' app, which will translate the result and add it to a Kafka topic.
The 3rd application (processor) consumes the Kafka messages at its own pace and updates the database accordingly. (eg. if someone voted 'Vim', the counter for vim would be incremented by 1) The 'retriever' application (currently embedded in the processor app) has a REST GET endpoint to get the results from the DB.

Anatomy

So ultimately the application is composed of the following components:

UI
The ui application displays a list of java stacks/frameworks that you can vote for by clicking the respective button next to it. This action calls the ingester app. The page also displays a bar chart of the results so far. The app is built with Quarkus Qute templating and some crappy javascript/jquery code :P

Ingester
The ingester Camel Quarkus application receives requests from the user (via HTTP) and forwards the requests to the Kafka broker. The main component of the application:

  • RestToCamelRoute : Camel route that receives a rest call and forwards it on to a Kafka broker

Processor
The processor Camel Quarkus application receives the vote requests from Kafka, processes them, and writes results into the votesdb Postgres DB table. As of right now it also returns the results through a /getresults endpoint. The application has the following Camel Routes:

  • processor/VotesRoute consumes messages from a kafka topic (in json format), extracts the value of 'shortname' and increments the counter of the java stack that matches with this shortname.
  • RestRoute returns data from the votes table in json format

Twitter
The twitter Camel Quarkus application polls twitter for matches to my handle (@kevindubois) and scans for matches to the various IDEs I'm using.

  • TwitterRoute polls twitter, and forwards any matches to the same Kafka topic messages

And this is basically the UI of the voting app:

You can find the source code in my github: https://github.com/kdubois/CamelQuarkusVoter

Using Camel Quarkus to send alerts on Telegram when web content changes

Ahh… Covid-19. From thinking it was just another little outbreak in Asia to months long lockdowns and seeing friends and family succumb to this nasty little virus, it's been a surreal year-and-a-half. Earlier this year we finally got some hope with the rollout of vaccines, and I personally was very eager to get vaccinated and hopefully leave my anxiety to leave the house behind me. I currently reside in Brussels, Belgium, and unlike in the US, the vaccine rollout was, at least initially, painfully slow. The Brussels government did however set up a nice little web application where they would keep everyone up to date with the current age group that was eligible to get vaccinated. The only caveat was that there was no way to keep informed of any changes to the age group other than refreshing the website. Which I did several times a day for a day or two. But, I wouldn't work in tech if it didn't seem a bit silly to perform a manual repetitive task right? :D

I saw a while ago a nice little demo of one of my colleagues at Red Hat who used Red Hat Fuse (based on Apache Camel) to send a message to Telegram, and this seemed like a pretty good way to go since I use Telegram on a daily basis anyway. The advantage of using an Enterprise Integration Pattern based system is that it makes the solution quite flexible, so if at some point I want to send the message to another endpoint then all I need to change is the "to" parameter of my Camel route and I'm good to go. Likewise, if I want to adapt this code to scrape bits of data from another website, then that's a very easy change as well.

Anyway, take a look at the video and let me know what you think!

The source code can be found in my Github: https://github.com/kdubois/datascraper. Feel free to scrutinize and send pull requests with improvements!

Synchronize between Subversion and Git using Continuous Integration Pipelines and Containers

I recently worked on a project for a client that (still) uses Subversion for their code repositories. Unfortunately they had no real short-term interested in migrating their code to Git, which left the team without such goodies as merge/pull requests, easy branching and all the other advantages that Git provides. Git actually has a built-in tool called 'git-svn'. Its main purpose is to migrate from Subversion to Git, or at the most do one-way syncing. In my case, two-way synchronization was the only option, since the Subversion repository was also used by another independent team.

After some experimentation I created a script that was able to do 2-way synchronization without too many problems, though this was just on my local machine, and we needed a solution for the entire team.

Jenkins, Containers & Gitlab to the rescue!
What I came up with was to build a Jenkins pipeline script that does the heavy lifting of synchronizing between git and subversion, while also integrating a Continuous Integration pipeline so that the main codebase is kept stable and thus *should* never break.

Fortunately the client was nice enough to provide us with a bare-bones RHEL 7 VM, and from there, I created a containerized CI/CD suite with containers for Jenkins, Gitlab, Sonar and a local Artifactory for snapshot build management. I created 2 Jenkins pipeline scripts for each project, one to do the git-svn synchronizations, and one to run a ci/cd pipeline that sets up an ad-hoc Docker container for the app (and one for the DB), and runs build & tests against them.

This CI/CD pipeline is triggered both by opened merge requests into the develop branch (no one is allowed to commit directly to develop/master branches), and by the git-svn sync pipeline script. The latter synchronizes between the master branch of git and the subversion repository and triggers the CI/CD pipeline in reverse order, that is, it rebases SVN code into master, and after the CI/CD pipeline successfully passes, merges the code into develop. This way, the develop branch is protected from broken build commits both from SVN and from feature branches.

I created a demo project with examples of the pipeline script and a full-fledged containerized solution you can use as a starting point:
https://github.com/kdubois/git-svn-pipeline

'Add More' form fields with javascript / jquery

A few years ago I wrote a post on how to add more fields without reloading the page using javascript, but that method was kind of clunky and involved a hack to get it to work with the DOM.  The following is a much easier way, and (as far as I've tested) compatible with all major browsers.

First off, download the latest version of jquery and include it on your page, or just include the following line if you trust google to host your jquery file:

<script type="text/javascript" src="https://ajax.googleapis.com/ajax/libs/jquery/1.6.4/jquery.min.js"></script>

Here's a sample html form – adjust according to your needs:

<form action='mypage' method='post' accept-charset='utf-8'>    
 <ul id='myfields'>
  <li>
   <input type='text' name='mytextfield[]' /> 
  </li>
 </ul>
 <div id='addmorelink'>
  <a href='javascript:addField()'>add more</a>   
 </div>
</form>

What's important is the ul with the 'myfields' id, and the link that calls the 'addField()' js function.

The javascript code is very short and straightforward – basically copy your list item with the form field code into the content of the 'newContent' variable, make sure you escape quotes if you enclose the string with the same quotes as in your form field, and then use the jquery 'append' function to add the next field to your form, et voila, that's it!

<script type="text/javascript"> 
 function addField(){
  var newContent = "<li><input type='text' name='mytextfield[]'/></li>";
  $("#myfields").append(newContent); 
 }
</script>

I haven't mentioned one last piece that's going to come in handy on the server side where you're going to process the fields.  You probably noticed the [] brackets behind the field name in the form – that turns the field into an array.  On the server side, depending on what programming language you use, you just need to handle your post variable as an array – so to retrieve each value you'll need to loop through the array.  Assuming you're programming in php and you're not using a framework that handles post variables differently:

<?php
 if(isset($_POST['mytextfield'] &amp;&amp;  is_array($_POST['mytextfield']){
  foreach($_POST['mytextfield'] as $value){ 
   // do something with each $value 
  }
 } 
?>

Creating an advertisement widget for external websites using JavaScript

If you work with Google Analytics or Google Adsense, you've undoubtedly worked with the little code bits they ask you to include in your website.  This tutorial will teach you how to create your own widget that people can include on their website by pasting a similar piece of javascript.

First off, you'll need to create a javascript file, accessible from 3rd party systems.  Typically you won't have to do anything for this – just make sure you can access the file from any browser and host with the http protocol.

1) The simple and easy (but limited) solution

So let's go ahead and create mywidget1.js, and tell it to output the string 'Hello World'.

document.write("hello world!");

Now, we'll write the code bit that will be included on the 3rd party sites:

<script type="text/javascript" 
src="https://www.kevindubois.com/demos/js/mywidget1.js"></script>

Simple, right?  If that satisfieds your needs, all the better, but most people will want a little more functionality.

Either way, you can see a demo for this one at https://www.kevindubois.com/demos/widget1.html (the js code is at https://www.kevindubois.com/demos/js/mywidget1.js)

There are ways to pass variables and there are ways to handle secure servers, so if that's what you're interested in, please continue to read!

2) The more advanced (but still pretty easy) solution:

The files stay the same, only the code within will get a little more chunky.  My sample will be based on the assumption that you're sending an advertisement widget to the 3rd party sites – but it really could be anything you want, just change the parameters in whatever you want.

Let's start with the 3rd party code this time.  We'll write the javascript tags again,  but this time we're not going to include the location to the js file in the tag but rather, we're going to tell the browser to write another javascript codebit from within the first javascript tag.  This is necessary to pass some parameters to the js file; and to write some code around secure sites (or not) issues:

<script type="text/javascript">
 referral_code = "MQI74";
 ad_code = "aa";
 ad_width = "200"; // optional
 var myHost = (("https:" == document.location.protocol) 
? "https://www." : "http://www."); document.write(unescape("%3Cscript src='" + myHost +  
"kevindubois.com/demos/js/mywidget2.js' type='text/javascript'%3E%3C/script%3E")); </script>

referral_code, ad_code and ad_width are parameters that are being sent to the js file.  myHost detects whether we're on a secure site or not, and will change the path to reflect that.  Note that this will only work if you have a secure certificate on your end too.  The reason why I included this, is because some browsers will return a warning on the 3rd party secure site if your js file is not secure, saying that some items on the site are not secure.  The html tags are being escaped ( < = %3C and > = %3E ) and then unescaped, because otherwise the browser will interpret these tags incorrectly.

Let's go back to the js file and add some more code to handle the parameters and the security:

Start the js file by figuring out the security again:

var myHost = (("https:" == document.location.protocol) ? 
"https://www." : "http://www.");

The parameters you passed are directly available, so you can start using them immediately. In my case, I added a switch statement to show different ads (sort of like google adsense, where you can show different ad sizes and types by changing the ad code).  Make sure you escape opening and closing tags because you will get problems with the browser interpreting things incorrectly.

var ad_code = ad_code ? ad_code : '1a';
var referral_code = referral_code ? referral_code : '000';
var ad_width = ad_width ? ad_width : 200;

// switch code to see what we want to serve up
switch(ad_code){

   case '1a':
     document.write(unescape("%3Cdiv  
style='width:"+ad_width+"px; font-family: verdana, arial, helvetica,  
sans-serif; cursor:pointer; font-size:11px;'%3E"));      document.write(unescape("%3Ca href='" + myHost + "kevindubois.com/index.php? 
referral="+referral_code+"' style='text-decoration:none;color:white'%3E"));      document.write(unescape("%3Cdiv style='border-top:5px solid #0F0442;  
width:"+ad_width+"px; background-color:#FF6204; color:#FFF; padding-bottom:10px '%3E"));      document.write(unescape("%3Cul%3E"));          document.write(unescape("%3Cli style='margin-left:-15px'%3EThis is a  
demo for %3Ca href='https://www.kevindubois.com/2009/11/16/creating-advertisement-widgets-for-3rd-party-websites'%3E 
https://www.kevindubois.com/2009/11/16/creating-advertisement-widgets-for-3rd-party-websites%3C/a%3E%3C/li%3E"));     document.write(unescape("%3Cli style='margin-left:-15px'%3bla bla bla%3C/li%3E"));     document.write(unescape("%3Cli style='margin-left:-15px'%3Ebla bla bla%3C/li%3E"));     document.write(unescape("%3Cli style='margin-left:-15px'%3Ebla bla bla%3C/li%3E"));     document.write(unescape("%3C/ul%3E  "));     document.write(unescape("%3Cdiv style='text-align:center;font-weight:bold; 
font-size:12px; '%3E LEARN MORE %3C/div%3E"));     document.write(unescape("%3C/div%3E"));     document.write(unescape("%3C/a%3E"));     document.write(unescape("%3C/div%3E")); break; case '2a': document.write(unescape("%3Cdiv style='width:"+ad_width+"px; font-family: 
verdana, arial, helvetica, sans-serif; cursor:pointer; font-size:11px;'%3E")); document.write(unescape("%3Ca href='" + myHost +  
"kevindubois.com/index.php?referral="+referral_code+"' style='text-decoration:none;color:white'%3E")); document.write(unescape("%3Cdiv style='border-top:5px solid #0F0442;  
width:"+ad_width+"px; background-color:#FF6204; color:#FFF; padding-bottom:10px '%3E")); document.write(unescape("%3Cul%3E"));     document.write(unescape("%3Cli style='margin-left:-15px'%3EThis is a demo for  
%3Ca href='https://www.kevindubois.com/2009/11/16/creating-advertisement-widgets-for-3rd-party-websites'%3Ehttps://www.kevindubois.com/2009/11/16/creating-advertisement-widgets-for-3rd-party-websites%3C/a%3E%3C/li%3E"));     document.write(unescape("%3Cli style='margin-left:-15px'%3bla bla bla%3C/li%3E"));     document.write(unescape("%3Cli style='margin-left:-15px'%3Ebla bla bla%3C/li%3E"));     document.write(unescape("%3Cli style='margin-left:-15px'%3Ebla bla bla%3C/li%3E"));     document.write(unescape("%3C/ul%3E  "));     document.write(unescape("%3Cdiv style='text-align:center;font-weight:bold;  
font-size:12px; '%3E LEARN MORE %3C/div%3E"));     document.write(unescape("%3C/div%3E"));     document.write(unescape("%3C/a%3E"));     document.write(unescape("%3C/div%3E")); break; // and so on ... }

The only thing I haven't mentioned is the links and the referral code: you can track widgets and clicks from each 3rd party by supplying a referral code. The link within the widget has this referral code and in the linked page you can then write some code to track/count/whatever you want.
From here on, I think you should have all the tools to start customizing your own widget. You can make everything even more dynamic by using some ajax; you can add more parameters or really any content you want.

You can find a demo for the 2nd widget at https://www.kevindubois.com/demos/widget2.html (the js code is at https://www.kevindubois.com/demos/js/mywidget2.js)