Posted on Aug 30 • Originally published at softwaresennin.dev In today's tech industry, setting up a CI/CD pipeline is quite easy. Creating a CI/CD pipeline even for a simple side project is a great way to learn many things. For today we will be working on one of my side projects using Portainer, Gitlab and Docker for the setup.As the founder of Apoti Development Association (A.D.A.) an NGO, I like organizing technical events in the Buea area (SW Region of Cameroon, Africa). I was frequently asked if there is a way to know all the upcoming events (the meetups, the Jugs, the ones organized by the local associations, etc.) After taking some time to look into it, I realized there was no single place which listed them all. So I came up with https://apotidev.org/events, a simple web page which tries to keep an update of all the events. This project is available in Gitlab.Disclaimer: Even though this is a simple project, the complexity of this project is not important here. The different components of our CI/CD pipeline we will detail can also be used in almost the same way for more complicated projects. However they are a nice fit for micro-services.To make things as simple as possible, we have an events.json file in which all new events are added. Let's look at a snippet of itOur mustache template is applied to this file. It will help us to generate the final web assets. Once our web assets have been generated, they are copied into an nginx image — the image that is deployed on our target machine.Thanks to Gitlab's multi-stage build, our build is in two parts:Let's look at the Dockerfile used for the buildBefore we proceed, we need to test the generation of our site. Just clone the repository and run the test script. This script will create an image and run a container out of it.Now let us run our test scriptThis is what our output looks lkeWe can now access our webpage using the URL provided at the endAs you have probably noticed, this web site is not critical (only a few dozen visits a day), and as such it only runs on a single virtual machine. This one was created with Docker Machine on AWS, the best cloud provider.Given the scale of our project, you must have noticed that it is not that critical (barely a few visits a day), so we will only one one virtual machine for it. For that we created ours with Docker on Exoscale, a nice European cloud provider.We configured our VM (virtual machine) above so it runs the Docker daemon in Swarm mode so that it would allow us to use the stack, service, secret primitives, config and the great (very easy to use) orchestration abilities of Docker swarmThe file below -> ada.yaml defines the service which runs our Nginx web server that contains the web assets.Let's break this down Our stack is executed with the command:Portainer is a really great web UI which will help us to manage all our Docker hosts and Docker Swarm clusters very easily. Let's take a look at its interface where it lists all our stacks available in the swarm.As you can see above, our current setup has 3 stacks:If we go ahead and list the details of the www service, in the ada_events stack, we can easily see that the Service webhook is activated. This is a new feature, available since Portainer version 1.19.2. This update allows us to define a HTTP Post endpoint which we can call to trigger an update of our service. As you will notice later on, our Gitlab Runner is in charge of calling this webhook.Note: As you see in the screenshot above, I use localhost:8888 to access Portainer. Since I don't want to expose our Portainer instance to the external world, access is done through an ssh tunne; which we open with the command belowOnce we have done this, all requests targeted at our local machine on port 8888 -> localhost:8888 are sent to port 9000 on our VM through ssh. Port 9000 is the port where Portainer is running on our VM but this port is not opened to the outside world. We used a security group in our AWS config to block it.NB: Note that in the command above, the ssh key that was used to connect to the VM was the one generated by our Docker Machine during the creation of the VM.A gitlab runner is a continuous integration tool that helps automate the process of testing and deploying any and all applications. It works in with GitLab CI to run any job defined in the project's .gitlab-ci.yml file.So in our project, our GitLab runner is in charge of executing all the actions we defined in the .gitlab-ci.yml file. On Gitlab, you have a choice of using your runners or using the shared runners available. In this project, we used a VM on AWS as our runner. First we register our runner providing a couple of commands:As you see above, we have $PROJECT_TOKEN as one of the needed options. We will get that from the project page on Gitlab, used to register new runnersOnce we have registered our gitlab runner, we can now start itOnce our VM has been setup as a Gitlab runner, it will be listed in the CI/CD page, under settings of our projectNow that we have a runner, it can now receive work to do every time that we have a new commit and push in our git repo. It will sequentially run the different stages in our .gitlab-ci.yaml file. Let's look at our .gitlab-ci.yaml file -> the file that configures our Gitlab CI/CD pipelineLet's break down the stagesOur Gitlab runner is running inside a container in our Docker swarm. Like mentioned before, we could have instead used a shared runner (publicly available runners which share their time between the jobs needed by different projects hosted on GitLab) — but, in our case, since the runner must have access to our Portainer endpoint (to send the webhook), and also because I don't want our Portainer app to be publicly accessible, I preferred setting it this way, having it run inside the cluster. It is also more secure this way.In addition to that, because our runner is in a docker container, it is able to send the webhook to the IP address of the Docker0 bridge network, to connect with Portainer through port 9000 which it exposes on the host. Thus, our webhook has following format: http://172.17.0.1:9000/api[…]a7-4af2-a95b-b748d92f1b3bUpdating a new version of our app follows the workflow belowIt starts with a developer pushing some changes to our GitLab repo. The changes in the code mainly involve adding or updating one or more events in our events.json file while also adding some sponsors' logo.After this, the GitLab runner performs all the actions that we defined in the .gitlab-ci.yml file.Then the GitLab runner calls our webhook that is defined in Portainer.Finally, upon the webhook reception, Portainer deploys the newest version of the www service. It does this, calling the Docker Swarm API. Portainer can access to the API because the /var/run/docker.sock socket is bind mounted once it is started.Now our users can access the newest version of our events website.Let's test our pipeline by doing a couple changes in the code and then committing / pushing those changes.As you can see in this screenshot below, our changes triggered our pipeline On Portainer side, the webhook was received and the service update was performed. Also, although we cannot see it clearly here, one replica has been updated. Like we mentioned before, that still left the website accessible through the other replica. The other replica also was updated a couple of seconds later.Although this was a tiny project, setting up a CI/CD pipeline for it was a good exercise. First it helped me get more familiar with GitLab (which has been on my To-Learn list for quite some time). Having done this project, I can say that it is an excellent, professional product. Also, this project was a great opportunity for me to play with the long awaited webhook feature available in updated versions of Portainer. Lastly, choosing to use Docker Swarm for this project was a real no-brainer - so cool and easy to use!Hope you found this project as interesting as I did. No matter how small your project is, it would be a great idea to build it using CI/CD. What projects are you working on and how has this article inspired you? Please comment below.Templates let you quickly answer FAQs or store snippets for re-use. Are you sure you want to hide this comment? It will become hidden in your post, but will still be visible via the comment's permalink. Hide child comments as well Confirm For further actions, you may consider blocking this person and/or reporting abuse Ahmed Zidan - Aug 20 Zeno Rocha - Aug 10 Victor Isaac Oshimua - Aug 1 UponTheSky - Aug 1 Would you like to become an AWS Community Builder? Learn more about the program and apply to join when applications are open next. Once suspended, aws-builders will not be able to comment or publish posts until their suspension is removed. Once unsuspended, aws-builders will be able to comment and publish posts again. Once unpublished, all posts by aws-builders will become hidden and only accessible to themselves. If aws-builders is not suspended, they can still re-publish their posts from their dashboard. Note: Once unpublished, this post will become invisible to the public and only accessible to Lionel Tchami ♾️☁️. They can still re-publish the post if they are not suspended. Thanks for keeping DEV Community safe. Here is what you can do to flag aws-builders: aws-builders consistently posts content that violates DEV Community's code of conduct because it is harassing, offensive or spammy. Unflagging aws-builders will restore default visibility to their posts. DEV Community — A constructive and inclusive social network for software developers. With you every step of your journey. Built on Forem — the open source software that powers DEV and other inclusive communities.Made with love and Ruby on Rails. DEV Community © 2016 - 2023. We're a place where coders share, stay up-to-date and grow their careers.