<img height="1" width="1" style="display:none;" alt="" src="https://dc.ads.linkedin.com/collect/?pid=58103&amp;fmt=gif">
Skip to content
All posts

Alternative CI Pipeline Tools Comparison

  Shadow-Soft Team  ·  8 minute read

Introduction

I can still recall with clarity from over a decade ago when one of my job responsibilities was ensuring the source control management, building, packaging, quality assurance, delivery, and release of an enormous monolithic software tool suite comprised of C, C++, FLEX, YACC, BISON, FORTRAN77/90, Ada, Perl, Python, and all sorts of other languages I cannot even begin to recall at the moment. Although I acquired multifarious skills and proficiency in many languages when I debugged and rectified the code, the amount of time spent bisecting commits for root cause was ridiculous. Then one day I had the idea to schedule automated nightly builds and regression tests, and everything became significantly better almost immediately.

The point I am attempting to demonstrate here is that continuous integration pipeline tools were not created for fun, or because someone arrived at the idea randomly. They exist to dramatically improve the software development lifecycle. After cobbling together my own homegrown solution at the time, months later I discovered Hudson (later Jenkins). I architected and implemented a pipeline to continuously integrate and deploy software using shell scripts, RPM in the mock environment, Spacewalk, and Puppet (time capsule much?). Today we have many tools that can be leveraged both for and within pipelines.

Although Jenkins has the lion’s share of the market, and is the most robust and capable tool in the industry, it also requires a significant amount of upkeep and maintenance. Not every company has the personnel to satisfy these requirements. Not every company may even see the value in satisfying these requirements.

In this article we will explore a comparison between the major CI pipeline tools other than Jenkins.

Build Agents

  • Circle CI: Utilizes both container and virtual machine images. The container images are subject to registry availability, and the machine images are a decent variety of pre-built Linux, Windows, and MacOS.
  • Gitlab CI: Utilizes any container or virtual machine image that you have available in your environment supporting the Gitlab Runner.
  • Github Actions: Utilizes a machine image with a decent variety of pre-built images for selection.
  • Travis CI: Utilizes a machine image with a decent variety of pre-built images for selection, but fewer than Github.
  • Concourse: Utilizes any container or virtual machine image that you have available in your environment, subject to some minor constraints, that supports the Concourse worker configuration.

Ranking 1. Gitlab 2. Concourse 3. Circle 4. Github 5. Travis

Libraries and Code Support

  • Circle CI: Supports Orbs, which are collections of shell commands. This leads to the normal issues with robustness, logging, and functionality. There are generally few advantages to using these with Circle other than DRY.
  • Gitlab CI: Supports libraries with the Enterprise Edition, which are collections of shell commands. Similar to the above, There are generally few advantages to using these other than DRY. The sole advantage Circle possesses over Gitlab in this category is the ease of packaging, retrieving, and utilizing the shared code.
  • Github Actions: Supports robust and rich enhancements with Javascript in the Node.js framework, and the Actions Toolkit SDK. These can greatly assist with functionality and enhancements in a pipeline.
  • Travis CI: Supports importing shared configs, which are collections of Travis configs. In practice these are somewhat unclear to efficiently implement.
  • Concourse: Supports libraries in basically almost any language. These need to be packaged into the build agent (most conveniently container image) in expected paths, and utilize the bindings to the Concourse API.

Ranking 1. Concourse 2. Github 3. Circle 4. Gitlab 5. Travis

Speed and Efficiency

  • Circle CI: Absolutely crushes in this category. Circle often finishes an entire job before other tools finish the first step. It also supports robust intra-job configurations for stage branching and asynchronous concurrency.
  • Gitlab CI: Nothing special here. Gitlab typically executes sequentially in an expected amount of time.
  • Github Actions: Basically the same as Gitlab. Nothing really speedy or slow happening here. Supports parallel job matrices. Has a slightly slower startup time relative to Gitlab.
  • Travis CI: Often requires a significant amount of startup time, and then may execute slowly afterwards. Supports parallel job matrices.
  • Concourse: Requires a decent amount of startup time, but nowhere near as much as Travis. Supports intra-job parallelization similar to Circle, but not stage branching dependencies. Also supports parallel job matrices.

Ranking 1. Circle 2. Concourse 3. Gitlab 4. Github 5. Travis

Ease of Use

  • Circle CI: This requires a decent amount of hunting through documentation and ramp-up before implementing a config. This is partially due to the extra features in Circle versus other YAML-based config tools. However, the existence of Orbs in an available repository can simplify usage somewhat.
  • Gitlab CI: This is basically the same as Circle, except for the lack of a repository with libraries. Therefore, Gitlab requires more user knowledge and effort for implementation relative to Circle. Otherwise, the ease of use factor is equivalent.
  • Github Actions: These are super simple and easy to setup and use. There are plenty of Github Action packages that streamline entire blocks of setup in the config. A typical fully functional config for integration and deployment can be as few as twenty lines of YAML.
  • Travis CI: These are also simple and easy to setup, but do require more lines of configuration due to lack of config libraries in a repository. Configs are straightforward to implement, and the Travis gem allows for config validation.
  • Concourse: These are definitely the most difficult to implement. Concourse has some unique features and functionality, and these require learning the specific terminology and functionality before implementing. The config also has a very real DRY problem relative to the other pipeline tools, and this causes the normal issues with config organization and maintenance. Configuration also requires use of the fly CLI and its non-intuitive usage, and a web portal that can be cumbersome to navigate.

Ranking 1. Github 2. Travis 3. Circle 4. Gitlab 5. Concourse

Debugging

  • Circle CI: This is rather amazing. Circle enables the capability to re-execute a specific job in a build with an inserted breakpoint, and then quickly and easily SSH into the build agent with the same keypair used for committing the code. This is all from within one page in the dashboard. I cannot overstate how awesome this is, and also how I want every CI pipeline tool to implement this feature.
  • Gitlab CI: On the other hand, Gitlab falls at the other end of the spectrum for debugging support. It involves old school displaying commands to the stdout descriptor, and runner configuration for indirect access.
  • Github Actions: Github is slightly better than Gitlab in and that it additionally supports extra verbose debug logging options. This can assist with additional information such as build agent and job step information.
  • Travis CI: Travis also allows agent access through SSH, and additionally supports extra debugging features through the CLI provided by the Travis gem. In the vast majority of situations the extra debugging features are not necessary, but they are still nice to have. Travis only falls slightly short of Circle because its debugging requires extra effort.
  • Concourse: Concourse supports a variety of debugging options through the functionality provided by its CLI. In addition, the agents can be accessed indirectly, and the logging verbosity can be increased. The features and ease of use fall just slightly short of Travis.

Ranking 1. Circle 2. Travis 3. Concourse 4. Github 5. Gitlab

Functionality and Customization

  • Circle CI: Circle seems to be the closest to Jenkins in feature parity. Many of the features implemented in Jenkins are also available in some manner within Circle. The dynamic and customizable caching in Circle is also super helpful.
  • Gitlab CI: Gitlab features a large variety of customization available for usage. The available specifications for customization of intrinsic Gitlab CI steps will normally exceed what is actually required.
  • Github Actions: This currently does not support a large array of features and customization. However, that is also not really the objective of Github Actions in the first place, and also it is the newest of all of these tools anyway. Github Actions focuses more on ease of use in its config and integrations.
  • Travis CI: Despite existing for longer than many of the competitors, Travis does not offer as many features as you would probably expect. For example: it supports basic caching, but not to the degree you may actually require. At some point you may find yourself frustrated with the inability to fulfill requirements.
  • Concourse: Concourse falls just slightly short of Circle and Gitlab for features and customization. It offers a number of interesting and unique features, but these are typically rarely useful individually (e.g. “check”). While the quantity of features in Concourse is on par with Circle and Gitlab, one will often find their usefulness somewhat questionable.

Ranking 1. Circle 2. Gitlab 3. Concourse 4. Gitlab 5. Github

Integrations

  • Circle CI: Circle’s integrations are primarily dependent upon partners implementing Orbs. The shortcomings of Orbs have already been discussed previously. While the integrations are quite numerous, their quality is overall rather suspect due to Orb limitations. The integrations are easily customizable for setup and specific steps.
  • Gitlab CI: Gitlab’s integrations are primarily centered around authentication. For example, an integration with Vault’s JWT/OIDC authentication engine is very possible. However, integrations with platforms and tools are virtually nonexistent. It also (obviously) primarily integrates with Gitlab for SCM.
  • Github Actions: Similar to Gitlab, Github Actions (of course) primarily integrates with Github for SCM. Beyond that, its integrations are explained similarly to Circle. They are reliant on partners and third parties developing Actions. These are more robust than Orbs for reasons described previously. Therefore Github ACtions wins slightly for general integration functionality, but loses in SCM integrations specifically (which may be a deal breaker if you are not on Github). However, the integrations also typically only involve setup actions and not subsequent steps, and therefore it drops slightly behind Circle after this consideration.
  • Travis CI: Travis’s integrations are primarily comprised of various different SCM software tools. You can use Travis quite easily with Github, Bitbucket, etc., but any integrations beyond SCM would likely need to be custom built yourself.
  • Concourse: Concourse’s integrations are very similarly explained compared to Circle. Both (like Github) are reliant on partners and third parties developing resource types. The robustness and flexibility of the resource types wins over Actions, and crushes Orbs. However, the quantity of integrations is fewer than Circle or Github Actions, and you may need to build something yourself if the resource type does not yet exist, or is no longer supported. Interfacing with the integrations allows parameter inputs, but it is very awkward to enable custom steps in the jobs.

It is honestly very difficult to rank among Circle, Github Actions, and Concourse in this category, but I believe the following to be a decent estimate here. However, a different perspective could easily reorder the three. It is also worth noting that this is a category where Jenkins very much dominates completely, and as such it is difficult to compare among these offerings when they are all in the shadow of Jenkins.

Ranking 1. Concourse 2. Circle 3. Github 4. Travis 5. Gitlab

Billing

  • Circle CI: Circle provides a surprising number of features and capabilities in its free tier, and it is entirely possible that a small business would not even require purchasing a subscription. However, the offerings per subscription tier are quite generous. It is still a SaaS though, so it would still be cheaper (in money and not time) to custom design your own on-demand build agent infrastructure if your business has the personnel to support it.
  • Gitlab CI: If you use the SaaS Gitlab, then the subscription includes a specified number of minutes, and more can be purchased. If you are self-hosting, then you save money by designing your own build agent infrastructure.
  • Github Actions: It is free in the SaaS, and self-hosted if your implementation is also self-hosted. There is not much else to say other than this is obviously the least expensive.
  • Travis CI: Even a small business would likely need to purchase the lowest tier of the subscription for the necessary features and capabilities. Additionally, Travis bills time consumed waiting in its internal queue for resources and startup prior to actually initiating the builds (which can be up to 80% of total time), and that can be a significant shock to the wallet.
  • Concourse: This relies upon self-hosted build agent infrastructure, and therefore you can customize to optimize for reduced costs.

Ranking 1. Github 2. Gitlab 3. Concourse 4. Circle 5. Travis

Other Tools

Appveyor

Appveyor vies with Travis for total time of existence, and was basically known as “that alternative to Travis if you needed a Windows OS build agent”. It is primarily noted here for historical reasons because it initially was the second most popular SaaS compared to Travis, but has fallen in popularity with the abundance of new and better options in recent years.

Bamboo

Bamboo has also existed for a long time, and primarily exists if you really need those first class Atlassian integrations because you are already using their tools extensively. You can leverage it with both self-hosted and SaaS. It is subject to the normal stipulations around Atlassian licensing costs. If those Atlassian integrations are not super important, then you may not be considering Bamboo as an option.

Cloud (CodePipelines, AzureDevOps, CodeBuild/CodeDeploy)

These are offered as a component of the AWS, Azure, and GCP vendor suites. If you are considering these offerings, then it is likely because you are exclusively using the cloud vendor platform offerings. These platform tools prioritize integrations with the rest of the cloud tools over everything else. If you are exclusively leveraging a cloud’s tooling, then these are interesting options to consider. If you prefer features, robustness, and flexibility over cloud integration convenience, then these are probably not an interesting option.

Conclusion

This article explored a categorical comparison of the CI pipeline tools other than Jenkins. We discussed the strengths and weaknesses of each option intrinsic to themselves, and relative to each other. It should hopefully now be clearer what is the best option for you and your organization based on your environment’s priorities. You should be able to improve your workflows and processes with the confidence that you made an informed decision with respect to tooling.

If your organization is interested in exploring or migrating to a new CI pipeline tool, and/or architecting and implementing integration and deployment pipelines for your applications and software, then contact Shadow-Soft.