Upgrading dependency versions for Cloud Foundry

Page last updated:

This topic tells you how to upgrade a dependency version in a custom buildpack. These procedures enable Cloud Foundry (CF) operators to maintain custom buildpacks that contain dependencies outside of the dependencies in the CF system buildpacks.

Cloud Foundry buildpacks team process

The procedures in this topic refer to the tools used by the CF buildpacks team, but they do not require the following specific tools. You can use any continuous integration (CI) system and workflow management tool to update dependencies in custom buildpacks.

The CF buildpacks team uses the following tools to update dependencies:

When the New Releases job in the notifications pipeline detects a new version of a tracked dependency in a buildpack, it creates a Tracker story about building and including the new version of the dependency in the buildpack manifests. It also posts a message as the dependency-notifier to the #buildpacks channel in the Cloud Foundry Slack channel.

Building binaries

For all dependencies, you must build the binary from source or acquire the binary as a tarball from a trusted source. For most dependencies, the CF buildpacks team builds the binaries from source.

The following steps assume you are using a Concourse deployment of the buildpacks-ci pipelines and Pivotal Tracker.

To build the binary for a dependency:

  1. Go to the public-buildpacks-ci-robots directory and verify no uncommitted changes exist.

  $ cd ~/workspace/public-buildpacks-ci-robots
  $ git status

  1. Run the git pull command in the directory to ensure it contains the most recent version of the contents.

  $ git pull -r

  1. Go to the binary-builds directory.

  $ cd binary-builds

  1. Locate the YAML file for the buildpack that you want to build a binary. The directory contains YAML files for all the packages and dependencies tracked by the CF buildpacks team. Each YAML file correlates to the build queue for one dependency or package, and uses the naming format DEPENDENCY-NAME.yml. For example, the YAML file tracking the build queue for Ruby is named ruby-builds.yml and contains the following contents:

    ---
ruby: []

  2. Different buildpacks use different signatures for verification. Determine which signature your buildpack requires by consulting the list in the buildpacks section of this topic and follow the instructions to locate the SHA256, MD5, or GPG signature for the binary:

  • For the SHA256 of a file, run shasum -a 256 FILE-NAME.
  • For the MD5 of a file, run md5 FILE-NAME.
  • For the GPG signature (for Nginx), see the Nginx Downloads page.

  1. Add the version and verification for the new binary to the YAML file as attributes of an element under the dependency name. For example, to build the Ruby 2.3.0 binary verified with SHA256, add the following to the YAML file:

    ---
ruby:
- version: 2.3.0
  sha256: ba5ba60e5f1aa21b4ef8e9bf35b9ddb57286cb546aac4b5a28c71f459467e507

    Do not preface the version number with the name of the binary or language. For example, specify 2.3.0 for version instead of ruby-2.3.0.

    You can enqueue builds for multiple versions at the same time. For example, to build both the Ruby 2.3.0 binary and the Ruby 2.3.1 binary, add the following to the YAML file:

    ---
ruby:
- version: 2.3.0
  sha256: ba5ba60e5f1aa21b4ef8e9bf35b9ddb57286cb546aac4b5a28c71f459467e507
- version: 2.3.1
  sha256: b87c738cb2032bf4920fef8e3864dc5cf8eae9d89d8d523ce0236945c5797dcd

  2. Use the git add command to stage your changes:

    $ git add .

  3. Use the git commit -m "YOUR-COMMIT-MESSAGE [#STORY-NUMBER]" command to commit your changes using the Tracker story number. Replace YOUR-COMMIT-MESSAGE with your commit message, and STORY-NUMBER with the number of your Tracker story.

    $ git commit -m "make that change [#1234567890]"

  4. Run git push to push your changes to the remote origin.

    $ git push

  5. Pushing your changes initiates the binary building process, which you can monitor at the binary-builder pipeline of your own buildpacks-ci Concourse deployment. When the build completes, it adds a link to the Concourse build run to the Tracker story specified in the commit message for the new release.

    Binary builds are run by the Cloud Foundry Binary Builder and the dependency-builds pipeline.

Updating buildpack manifests

After you build the binary for a dependency that you want to access and download from a URL, follow the instructions to add the dependency version to the buildpack manifest

The following steps assume you are using a Concourse deployment of the buildpacks-ci pipelines and Pivotal Tracker.

To add the dependency version to the buildpack manifest, use these steps:

  1. Go to the directory of the buildpack for which you want to update dependencies and run git checkout develop to check out the develop branch. 

    $ cd ~/workspace/ruby-buildpack
$ git checkout develop

  2. Edit the manifest.yml file for the buildpack to add or remove dependencies.

    dependencies:
  - name: ruby
    version: 2.3.0
    md5: 535342030a11abeb11497824bf642bf2
    uri: https://pivotal-buildpacks.s3.amazonaws.com/concourse-binaries/ruby/ruby-2.3.0-linux-x64.tgz
    cf_stacks:
      - cflinuxfs4

* Follow the current structure of the manifest. For example, if the manifest includes the two most recent patch versions for each minor version of the language, you can also include the two most recent patch versions for each minor version of the language, such as both `ruby-2.1.9` and `ruby-2.1.8`.
* Copy the `uri` and the `md5` from the `build-BINARY-NAME` job that ran in the Concourse dependency-builds pipeline and add them to the manifest.

In the PHP buildpack, you might see a <code>modules</code> line for each PHP dependency in
the manifest. Do not include this <code>modules</code> line in your new PHP dependency
entry. The <code>modules</code> line is added to the manifest by the <code>ensure-manifest-has-modules</code> Concourse job in the <code>php-buildpack</code> when you commit and push your changes. You can see this in the output logs of the <code>build-out</code> task.

  1. Replace any other mentions of the old version number in the buildpack repository with the new version number. The CF buildpack team uses Ag for text searching.

    $ ag OLD-VERSION

  2. Run the following command to package and upload the buildpack, set up the org and space for tests in the specified CF deployment, and run the CF buildpack tests:

    $ BUNDLE_GEMFILE=cf.Gemfile buildpack-build

    If the command fails, you might need to fix or change the tests, fixtures, or other parts of the buildpack.

  3. Once the test suite completely passes, use git commands to stage, commit, and push your changes:

    $ git add .
$ git commit -m "YOUR-MESSAGE[#TRACKER-STORY-ID]"
$ git push

  4. Monitor the LANGUAGE-buildpack pipeline in Concourse. Once the test suite builds, the specs-lts-develop job and specs-edge-develop job, pass for the buildpack, you can deliver the Tracker story for the new Dependency release. Copy and paste links for the successful test suite builds into the Tracker story.

Buildpacks

The following list contains information about the buildpacks maintained by the CF buildpacks team:

Go:

Godep:

Node.js buildpack

Node:

  • Verified with: The SHA256 of the node-vVERSION.tar.gz file listed on https://nodejs.org/dist/vVERSION/SHASUMS256.txt For example, for Node version 4.4.6, the CF buildpacks team verifies with the SHA256 for node-v4.4.6.tar.gz on its SHASUMS256 page.
  • Example: Enqueuing binary builds for Node 4.4.5 and 6.2.0.

Python buildpack

Python:

  • Verified with: The MD5 of the Gzipped source tarball, listed on https://www.python.org/downloads/release/python-VERSION/, where VERSION has no periods. For example, for Python version 2.7.12, use the MD5 for the Gzipped source tarball on its downloads page.
  • Example: Enqueuing binary build for Python 2.7.12.

Java buildpack

OpenJDK:

  • Built from: The tarred OpenJDK files managed by the CF Java Buildpack team.
  • Verified with: The MD5 of the tarred OpenJDK files.

Ruby buildpack

JRuby:

Ruby:

Bundler:

PHP buildpack

PHP:

  • Verified with: The SHA256 of the .tar.gz file from the PHP Downloads page.
  • To enqueue builds for PHP, you need to edit a file in the public-buildpacks-ci-robots repository. For PHP5 versions, the CF buildpacks team enqueues builds in the binary-builds/php-builds.yml file. For PHP7 versions, the CF buildpacks team enqueues builds in the binary-builds/php7-builds.yml file.
  • Example: Enqueuing binary builds for PHP 7.2.5 and 7.0.30.

Nginx:

HTTPD:

Composer:

  • Verified with: The SHA256 of the composer.phar file from the Composer Downloads page.
  • For Composer, there is no build process as the composer.phar file is the binary. In the manual process, connect to the appropriate S3 bucket using the correct AWS credentials. Create a new directory with the name of the composer version, for example 1.0.2, and put the appropriate composer.phar file into that directory.

For Composer v1.0.2, connect and create the php/binaries/trusty/composer/1.0.2 directory. Then place the composer.phar file into that directory so the binary is available at php/binaries/trusty/composer/1.0.2/composer.phar.

The buildpacks-ci dependency-builds pipeline automates the process of detecting, uploading, and updating Composer in the manifest.

Staticfile buildpack

Nginx:

Binary buildpack

The Binary buildpack has no dependencies.

Create a pull request or raise an issue on the source for this page in GitHub