5 Criteria for evaluating Continuous Integration Servers
Many Continuous Integration Servers exist, some are free, some are very expensive and there are lots of others in between. Each tool is different: different architecture, different features, different user interface. Before you invest in a Continuous Integration solution, it is important to evaluate the different options, and it helps to have criteria for evaluating these options.
In designing FinalBuilder Server, VSoft researched FinalBuilder users' needs to understand what our customers wanted from a Continuous Integration tool. We uncovered a variety of needs, but found a number of criteria which were common to most of our customers. Five of these criteria are discussed below.
1. Choose software you can trust and which is well supported.
First and foremost, you must be able to trust the tool. The Continuous Integration Server is going to be a foundation for building the software you ship to your customers. If the software has been around for some time and other companies have invested their trust in using the software, this is a good indication the Continuous Integration Server is robust.
It is also essential to choose a tool that is backed with outstanding support. When something does go wrong, either a bug or a usability issue, how and where do you get support? Does the tool vendor offer free support for the life of the product? Is there a community forum for discussing ideas, offering suggestions and reporting bugs? How quickly does the vendor respond to bug reports?
2. The Continuous Integration Server should support the tools you currently use and may use in the future.
Do you really care what command line parameters you need to use with your Version Control System? It's much easier to select your source control and have the Continuous Integration Server do the monotonous work for you.
Good Continuous Integration Servers support, out-of-box, the tools you already use; for example your compilers, testing tools and version control systems. However, the best Continuous Integration Servers will abstract the complexity of interfacing with these tools to provide you with a quick an easy solution for achieving tight integration with these tools.
It's also important to consider what tools you'll use in the future. If you're thinking of changing to a different testing tool, is this also supported? If you'll need to update to the latest version of your version control system, is this supported? Examine both the breadth of tools already supported by the Continuous Integration Server and the time it takes for new tools and new versions of existing tools to be supported.
3. Account for the angle bracket tax.
We often hear positive feedback from users who have moved on from complicated free products which require writing XML for configuration. Yes, these options are open source products and free, but they typically take a lot longer to configure and are far more difficult to manage than other options. Remember, your time isn't free.
Paying the XML angle bracket tax can quickly add significant expenses to an investment that started out free. Look for a server which you find easy to configure and which makes it easy for you to schedule, trigger, monitor and run your automated build process. Also consider that someone in the future may take over the build management; how easy will it be for a new person to learn and understand how to use the Continuous Integration Server?
4. The user interface is easy to use.
The user interface is very important. Look for a UI which is fast, rich and easy-to-use.
Do you find the UI easy-to-use? Does the UI make it easy to customise every aspect of your project? Is it quick and easy to find information in the log (eg. find all steps in error)? And is there an easy way to see the progress of the build process, including logs, while the build is running?
5. The essential feature set.
VSoft have been in the business of builds for quite some time. Listening to the continuous feedback from our customers, we have developed a good idea of what features are essential for any Continuous Integration Server:
- A flexible and advanced, yet easy-to-use user interface
- Ease of configuration
- Continuous Integration support for your version control system
- Support for testing tools, compilers and the other tools you use in your automated build process
- Comprehensive logging and presentation of logs that is intuitive and searchable
- Build Metrics for gaining quick insight into your build process
- Running builds in parallel, essential for reducing the time taken to run your builds
- Build notifications, via email, RSS, IM clients or other communication mechanisms
- Role based permissions for team management
- Build responsibility flags to have broken builds fixed immediately
- Build Queue management
As it was designed with our customers' criteria in mind, FinalBuilder Server fulfils all of these criteria and a lot more. Download the 30-day trial or try the live demo now to see for yourself!