Understanding Distributed Builds

Introduction

The Distributed Builds is an extension to the base Continuum functionalities that gives us the ability to process multiple independent builds beyond the capacity of a single server's processing power. It also enables us to execute builds on multiple different platforms while retaining a unified view of all project builds.

Architecture

Continuum follows a Client-Server model using XML-RPC as the protocol. However, since it uses a bi-directional XML-RPC implementation, we instead distinguish the components by calling them Master and Build Agent.

The Master is a Continuum instance that has the ability to delegate the builds to registered Build Agents.

The Build Agent is a standalone Jetty-bundled webapp that listens for any build requests from the Master it is assigned to.

There is a one-to-many relationship between the Master and the Build Agents. A Master may have many Build Agents, but each Build Agent can only have one Master.

Behavior

Distributed Builds happen at the project group level of Continuum. When the entire project group is built in the Master, independent projects (single project or multi-module project) are distributed to a single registered Build Agent.

In a project group containing a mix of projects, the distribution of work goes through the following steps:

  1. In the Master, a build in the project group is triggered.
  2. Every independent project within the project group is identified, whether as a single project or a multi-module project. Projects with inter-dependencies cannot be distributed separately, so multi-module projects are delegated to a Build Agent as one build.
  3. For each independent project, the Master iterates over the list of registered Build Agents and queries each if available. The query is an XML-RPC ping() followed by a getBuildSizeOfAgent() invocation.
  4. If there is a Build Agent available, the Master collects the information necessary for the build (SCM URL, project id, etc.) and passes it when invoking buildProjects() to the Build Agent with the smallest number of tasks in its queue. The Master also passes the name of the local repository of the project's project group. Make sure that a local repository with the same name is configured in the Build Agent's configuration file since that is what will be used when the project is built in the build agent. Otherwise, it will use the local repository set in the build agent's settings.xml file.
  5. In the Build Agent, the build request is processed: the build is queued and executed. Upon execution, the Build Agent first performs an SCM checkout or an SCM update followed by an SCM changelog to get the latest update date, then the actual build follows.
  6. At this point, when the build is running, the Master can invoke cancelBuild() which returns a transient build result, and getBuildResult() that updates the build output viewed in the Master.
  7. After the build, the Build Agent returns the complete build result to the Master by invoking the callback method returnBuildResult(), which the Master aggregates to provide a unified view of projects.

    A ping() is always called before each XML-RPC invocation to check if the agent is available. If it's not, the agent will be disabled.

Setup

  • Enable the Distributed Builds option and set the Shared Secret Password in the General Configuration.
  • Install and Configure one or more Build Agents.
  • Add your Build Agents to the Continuum Master.
  • Add your Build Agents to a Build Agent Group.
  • Add your Build Agent Group to a Build Environment.
  • Configure a project to use that Build Environment.
  • Make sure that the local repository of the project group is properly configured. In this case, it should point to the path of the repository where the Build Agent is installed/running.

Warning: You need to have a central remote repository to store the artifacts created from the Build Agent so that other agents will be able to use the new artifacts.

Limitations

  • Only system administrator can enable/disable distributed builds
  • Credentials (i.e. svn credentials) are passed along if specified, but if server cache is used it will need to be done individually on the Build Agents
  • There is no tracking of SCM changes
  • The Build Agent needs a configuration web interface
  • All projects in a project group will be distributed to the same Build Agent

Future Enhancements

  • Remote builders
    • Builders can be installed on remote machines, a Continuum manager will send actions to run to builders. An action can be something to run on all builders, on some of them or eventually only to an available builder if we don't want to run more than one build. Actions can be sent with JMS and builders can apply some filters if they don't want to receive all actions. With that, we can do some parallel builds but the dependency tree must be respected for the build order. To work correctly with dependencies, each builder must use a central local repository. Maybe we can use an internal Archiva.
    • With Continuum builders configured to receive all commands, users can run multi-platform build for each build definition execution.
    • With Continuum builders configured to receive only some project types, users can use a different builder by project group. In this case, the build of all projects will be done quickly because commands are balanced on several servers.
    • With Continuum builders configured to build something when it is available, users can install builders on several machine to balance the charge. In this case, it will be possible to run some parallel builds.
    • When the builders work is done, a message will be sent to the manager to notify the end of the process.
    • With JMS used for the communication, we can add some listeners to create reports/statistics, log some information.
  • Policy-based distribution
    • Next available
    • Load balanced
    • Targeted environment matching