Upgrading Dependency Versions

Page last updated:

This topic describes how to upgrade a dependency version in a custom buildpack. These procedures enable Cloud Foundry (CF) operators to maintain custom buildpacks that contain dependencies outside of the dependencies in the CF system buildpacks.

Cloud Foundry Buildpacks Team Process

The CF buildpacks team uses the following tools to update dependencies:

Note: The procedures in this topic refer to the tools used by the CF buildpacks team. However, the procedures do not require the specific tools mentioned above. You can use any CI and workflow management tool to update dependencies in custom buildpacks.

When the New Releases job in the notifications pipeline detects a new version of a tracked dependency in a buildpack, it creates a Tracker story about building and including the new version of the dependency in the buildpack manifests. It also posts a message as the dependency-notifier to the #buildpacks channel in the Cloud Foundry Slack.

Build the Binaries

For all dependencies, you must build the binary from source or acquire the binary as a tarball from a trusted source. For most dependencies, the CF buildpacks team builds the binaries from source.

Note: The steps below assume you are using a Concourse deployment of the buildpacks-ci pipelines and Pivotal Tracker.

To build the binary for a dependency, perform the following steps:

  1. Change into the buildpacks-ci directory and check that there are no uncommitted changes.
    $ cd ~/workspace/buildpacks-ci
    $ git status
    
  2. Check out the binary-builds branch. This is an orphan branch of buildpacks-ci that the CF buildpacks team uses as a separate resource on Concourse to trigger the binary building process.
    $ git checkout binary-builds
    
  3. Pull the branch to make sure it is up to date.
    $ git pull -r
    
  4. Locate the YAML file for the buildpack you want to build a binary for. The directory contains YAML files for all the packages and dependencies tracked by the CF buildpacks team. Each YAML file correlates to the build queue for one dependency or package and the naming format is DEPENDENCY-NAME.yml. For example, the YAML file tracking the build queue for Ruby is called ruby-builds.yml and contains the following contents:

    ---
    ruby: []
    
  5. Different buildpacks use different signatures for verification. Determine which signature your buildpack requires by consulting the list below and follow the instructions to locate the SHA256, MD5, or GPG signature for the binary:

    • For the SHA256 of a file, run shasum -a 256 FILE.
    • For the MD5 of a file, run md5 FILE.
    • For the GPG signature (for Nginx), see the Nginx Downloads page.
  6. Add the version and verification for the new binary to the YAML file as attributes of an element under the dependency name. For example, to build the Ruby 2.3.0 binary verified with SHA256, add the following:

    ---
    ruby:
    - version: 2.3.0
      sha256: ba5ba60e5f1aa21b4ef8e9bf35b9ddb57286cb546aac4b5a28c71f459467e507
    

    Note: Do not preface the version number with the name of the binary or language. For example, specify 2.3.0 for version instead of ruby-2.3.0.

    You can enqueue builds for multiple versions at once. For example, to build both the Ruby 2.3.0 binary and the Ruby 2.3.1 binary, add the following:

    ---
    ruby:
    - version: 2.3.0
      sha256: ba5ba60e5f1aa21b4ef8e9bf35b9ddb57286cb546aac4b5a28c71f459467e507
    - version: 2.3.1
      sha256: b87c738cb2032bf4920fef8e3864dc5cf8eae9d89d8d523ce0236945c5797dcd
    
  7. Stage your changes for commit:

    $ git add .
  8. Commit your changes using the Tracker story number.

    $ git commit -m "YOUR-COMMIT-MESSAGE[#STORY_NUMBER]"
  9. Push your changes to the remote origin.

    $ git push
  10. Pushing your changes triggers the binary building process, which you can monitor at the binary-builder pipeline of your own buildpacks-ci Concourse deployment. When the build completes, it adds a link to the Concourse build run to the Tracker story for the new release.

    Note: Binary builds are executed by the CF Binary Builder and the binary-builder pipeline.

Update Buildpack Manifests

After you build the binary for a dependency that you can access and download from a URL, follow these instructions to add the dependency version to the buildpack manifest.

Note: The steps below assume you are using a Concourse deployment of the buildpacks-ci pipelines and Pivotal Tracker.

  1. Change into the directory of the buildpack for which you want to update dependencies and check out the develop branch.

    $ cd ~/workspace/ruby-buildpack
    $ git checkout develop
    

  2. Open the manifest.yml for the buildpack and remove or add dependencies.

    dependencies:
      - name: ruby
        version: 2.3.0
        md5: 535342030a11abeb11497824bf642bf2
        uri: https://pivotal-buildpacks.s3.amazonaws.com/concourse-binaries/ruby/ruby-2.3.0-linux-x64.tgz
        cf_stacks:
          - cflinuxfs2
    
    • Follow the current structure of the manifest. For example, if the manifest includes the two most recent patch versions for each minor version of the language, do the same, such as both ruby-2.1.9 and ruby-2.1.8.
    • Paste in the uri and the md5 from the build-BINARY-NAME job that ran in the Concourse binary-builder pipeline.

      Note: In the PHP buildpack, you may see a modules line for each PHP dependency in the manifest. Do not include this in your new PHP dependency entry. This will be added to the manifest by the ensure-manifest-has-modules Concourse job in the php-buildpack when you commit and push your changes. You can find this in the output logs of the build-out task.

  3. Replace any other mentions of the old version number in the buildpack repository with the new version number. The CF buildpack team uses Ag for text searching.

    $ ag OLD-VERSION
  4. Run the following command to package and upload the buildpack, setup the org and space for tests in the specified CF deployment, and run the CF buildpack tests.

    $ BUNDLE_GEMFILE=cf.Gemfile buildpack-build

    If the command fails, you may need to fix or change the tests, fixtures, or other parts of the buildpack.

  5. Once the test suite completely passes, push your changes:

    $ git add .
    $ git commit -m "YOUR-MESSAGE[#TRACKER-STORY-ID]"
    $ git push
    
  6. Watch the LANGUAGE-buildpack pipeline in Concourse. Once the test suite builds pass for the buildpack (the specs-lts-develop job and specs-edge-develop job), you can mark the Tracker story for the new Dependency release as delivered. Paste links for those successful test suite builds in the Tracker story.

Buildpacks

The following list contains information about the buildpacks maintained by the CF buildpacks team.

Go Buildpack

Go:

Godep:

Node.js Buildpack

Node:

  • Verified with: the SHA256 of the node-vVERSION.tar.gz file listed on https://nodejs.org/dist/vVERSION/SHASUMS256.txt For example, for Node version 4.4.6, the CF buildpacks team verifies with the SHA256 for node-v4.4.6.tar.gz on its SHASUMS256 page.
  • Example: Enqueuing binary builds for Node 4.4.5 and 6.2.0

Python Buildpack

Python:

  • Verified with: the MD5 of the Gzipped source tarball, listed on: https://www.python.org/downloads/release/python-VERSION/ where VERSION has no periods. For example, for Python version 2.7.12, use the MD5 for the Gzipped source tarball on its Downloads page.
  • Example: Enqueuing binary build for Python 2.7.12

Java Buildpack

OpenJDK:

  • Built from: the tarred OpenJDK files managed by the CF Java Buildpack team.
  • Verified with: the MD5 of the tarred OpenJDK files.

Ruby Buildpack

JRuby:

Ruby:

Bundler:

PHP Buildpack

PHP:

  • Verified with: the SHA256 of the .tar.gz file from the PHP Downloads page.
  • For PHP5 versions, the CF buildpacks team enqueues builds in the php-builds.yml file in the binary-builds branch. For PHP7 versions, the CF buildpacks team enqueues builds in the php7-builds.yml file in the binary-builds branch.
  • Example: Enqueuing binary builds for PHP 5.5.37, 5.6.23, and 7.0.8

Nginx:

  • Verified with: the gpg-rsa-key-id and gpg-signature of the version. The gpg-rsa-key-id is the same for each version/build, but the gpg-signature will be different. This information is located on the Nginx Downloads page.
  • Example: Enqueuing binary build for Nginx 1.11.0

HTTPD:

Composer:

  • Verified with: the SHA256 of the composer.phar file from the Composer Downloads page.
  • For Composer, there is no build process as the composer.phar file is the binary. In the manual process, connect to the pivotal-buildpacks S3 bucket using the correct AWS credentials. Create a new directory with the name of the composer version (ex. 1.0.2) and put the appropriate composer.phar file into that directory. For Composer v1.0.2, connect and create the php/binaries/trusty/composer/1.0.2 directory. Then place the composer.phar file into that directory so the binary is available at php/binaries/trusty/composer/1.0.2/composer.phar.

    Note: The buildpacks-ci binary-builder pipeline automates the process of detecting, uploading, and updating Composer in the manifest.

  • Example: Automated enqueuing of binary build for Composer 1.1.2

Staticfile Buildpack

Nginx:

  • Verified with: the gpg-rsa-key-id and gpg-signature of the version. The gpg-rsa-key-id is the same for each version/build, but the gpg-signature will be different. This information is located on the Nginx Downloads page.
  • Example: Enqueuing binary build for Nginx 1.11.0

Binary Buildpack

The Binary buildpack does not have any dependencies.

View the source for this page in GitHub