edit

Release Notes

Note

Cloud Foundry Container Runtime (CFCR) was formerly known as Kubo. Some CFCR assets still use the Kubo name.

v0.44.0

Release Date: July 16, 2020

Downloads

Features and Updates

  • Updated to Kubernetes v1.17.9

v0.43.1

Release Date: July 13, 2020

Downloads

Fixes

  • Fixed Windows workers, which were missing a configuration entry causing Docker engine to not start

v0.43.0

Release Date: July 11, 2020

Downloads

Features and Updates

  • Updated to Kubernetes v1.16.8
  • Updated to etcd 3.3.22
  • Updated to Stemcell 621.76
  • Fixed an issue with etcd 3rd node addition by reverting etcdctl to 3.3.12

NOTE: Do NOT use this with Windows workers, please use 0.43.1

This one got majorly delayed due to COVID and other constraints. 1.17 and 1.18 shouldn't be as long of a wait, with some luck.

v0.42.0

Release Date: March 17, 2020

Downloads

Features and Updates

  • Updated to Kubernetes v1.15.7
  • Updated to etcd 3.3.18
  • Updated to Stemcell 621.59

CFCR is back, and will be publishing bumps to Kubernetes 1.16 in the coming days, followed shortly thereafter by extra features, other dependency bumps, etc.

v0.41.0

Release Date: October 29, 2019

Downloads

Features and Updates

  • Updated to Kubernetes v1.15.5 and fixes window change cidrs

v0.40.0

Release Date: October 23, 2019

Downloads

Features and Updates

  • Updated to Kubernetes v1.15.5

v0.36.0

Release Date: October 2, 2019

Downloads

Features and Updates

  • Updated to Kubernetes v1.15.4

v0.35.0

Release Date: October 1, 2019

Downloads

Features and Updates

  • Updated to Stemcell v456.27
  • Updated to go1.12.9.linux-amd64
  • Updated to kubernetes v1.14.5
  • Updated to metrics-server v0.3.5
  • Updated to coredns 1.4.0
  • Added jobs that were previously part of kubo-release-windows

Bug Fixes

  • Fixed a bug which prevented creating persistent volumes on vSphere when there were no resource pools set in the cloud config.

v0.34.0

Release Date: June 21, 2019

Downloads

Features and Updates

  • Add flannel etcd certs to Windows
  • Configure Kubelet force drain settings
  • Allow customization of vxlan VNI and port
  • Updated to go1.12.6.linux-amd64
  • Updated to Stemcell v315.41

v0.33.0 (skipped due to internal issue)

Release Date: N/A

v0.32.0

Release Date: May 3, 2019

Downloads

Features and Updates

  • Updated to Kubernetes 1.14.1
  • Updated to go1.12.4.linux-amd64
  • Updated to metrics-server version v0.3.2
  • Updated to Stemcell v315.11
  • Updated to bpm version 1.0.4
  • Enabled user authorization on etcd

v0.31.0

Release Date: March 25, 2019

Downloads

Features and Updates

  • Updated to Kubernetes 1.13.5
  • Updated to Stemcell v250.23
  • Updated golang to v1.12.1
  • Added the ability to deploy Windows Workers through the kubo-release-windows. See ops-files/windows/add-worker.yml for details. Currently, kubo-release-windows is only compatible on vSphere in an internet-connected environment.
  • Added an ops-file to set fs.inotify.max_user_watches.

Bug Fixes

v0.30.0

Download Release

Download Deployment Manifests

Release Date Mar 4, 2019

  • Bumped Stemcell to v250.9
  • Bumped Docker to 18.06.3-ce

Fix Quote and escape strings in cloud-provider's ini formatting to accept special characters in cloud provider configuration.

Fix Deployment CFCR manifests reference a release of docker with the runc CVE-2019-5736 addressed.

v0.29.0

Download the release artifact.

Release Date: Feb 14, 2019

  • Important: Replaced the etcd certificates and authorities with a unique CA (etcd_ca) to address CVE-2019-3779. See the Action Required section for details.
  • Bumped Docker to v18.06.2-ce.
  • Upgraded flannel to v0.11.0.
  • Upgraded bpm to v1.0.3.
  • Removed kube-dns as an option from apply-addons.

Action Required

Users upgrading from earlier versions of CFCR must perform a series of upgrade steps to facilitate a smooth rotation of the etcd certificates.

With BOSH, certificate rotations require a three-phase process that generates a CA, generates and utilizes a set of end-user certificates, and retires the original CA.

The three phases are as follows:

  1. Generate the new etcd_ca and add it to the list of trust anchors for etcd.
  2. Generate the new client and server certificates from etcd_ca and begin using them.
  3. Remove the old CA (kubo_ca) from etcd's trust anchors.

Perform the steps below.

Steps

  1. Redeploy the existing CFCR deployment, applying an additional ops-file for phase one: 

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
    - type: replace
  path: /variables/-
  value:
    name: etcd_ca
    options:
      common_name: etcd.ca
      is_ca: true
    type: certificate

- type: replace
  path: /instance_groups/name=master/jobs/name=etcd/properties/tls/etcd/ca
  value: ((tls-etcd-v0-17-0.ca))((etcd_ca.certificate))

- type: replace
  path: /instance_groups/name=master/jobs/name=etcd/properties/tls/etcdctl/ca
  value: ((tls-etcdctl.ca))((etcd_ca.certificate))

- type: replace
  path: /instance_groups/name=master/jobs/name=etcd/properties/tls/peer/ca
  value: ((tls-etcd-v0-17-0.ca))((etcd_ca.certificate))

  2. Redeploy the existing CFCR deployment, removing the ops-file for phase one and applying the ops-file for phase two: 

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
    - type: replace
  path: /variables/-
  value:
    name: etcd_ca
    options:
      common_name: etcd.ca
      is_ca: true
    type: certificate

- type: replace
  path: /variables/-
  value:
    name: tls-etcd-v0-29-0
    options:
      ca: etcd_ca
      common_name: '*.etcd.cfcr.internal'
      extended_key_usage:
        - client_auth
        - server_auth
    type: certificate

- type: replace
  path: /variables/-
  value:
    name: tls-etcdctl-v0-29-0
    options:
      ca: etcd_ca
      common_name: 'etcdClient'
      extended_key_usage:
        - client_auth
    type: certificate

- type: replace
  path: /instance_groups/name=master/jobs/name=etcd/properties/tls/etcd
  value:
    ca: ((tls-etcd-v0-17-0.ca))((etcd_ca.certificate))
    certificate: ((tls-etcd-v0-29-0.certificate))
    private_key: ((tls-etcd-v0-29-0.private_key))

- type: replace
  path: /instance_groups/name=master/jobs/name=etcd/properties/tls/etcdctl
  value:
    ca: ((tls-etcdctl.ca))((etcd_ca.certificate))
    certificate: ((tls-etcdctl-v0-29-0.certificate))
    private_key: ((tls-etcdctl-v0-29-0.private_key))

- type: replace
  path: /instance_groups/name=master/jobs/name=etcd/properties/tls/peer
  value:
    ca: ((tls-etcd-v0-17-0.ca))((etcd_ca.certificate))
    certificate: ((tls-etcd-v0-29-0.certificate))
    private_key: ((tls-etcd-v0-29-0.private_key))

  3. Upgrade to v0.29.0, completing phase three.

v0.28.0

Download the release artifact.

Release Date Feb 7, 2019

We upgraded to Kubernetes 1.13.3

Default stemcell bumped to Xenial v250.4

Added support for AWS tagging using kubernetes.io/cluster/<cluster_tag> format

Removed bundling heapster and influxdb

Bumped version of CoreDNS to 1.3.1

Updated the system specs to be inline with the samples provided for k8s 1.13.3

DOC FIX: Changed documentation on encryption configuration

DOC FIX: Recommended values for NO_PROXY when using proxy configuration

v0.27.0

Download the release artifact.

Release Date: Jan 18, 2019

CFCR now defaults to using CoreDNS instead of kube-dns - When upgrading you may want to remove kube-dns

CFCR now provides an ops-file for deploying with kube-dns

CFCR now ships with etcd 3.3.11

CFCR now ships with docker-bosh-release 33.0.1

CFCR now provides an ops-file for deploying workers with persistent disks

CFCR now provides documentation for disabling Linux swap.

Default stemcell bumped to Xenial v170.21

BUG FIX: Configure audit logging to preempt log rotation to avoid corruption of audit logs due to kubernetes/kubernetes#52865

BUG FIX: Unmount overlay2 mounts during docker shutdown

v0.26.0

Download the release artifact.

Release Date: Dec 19, 2018

We upgraded to Kubernetes 1.12.4

We upgraded the Dashboard to v1.10.1, which includes the patch for (CVE-2018-18264)

CFCR now ships with go 1.11.4

CFCR now ships with JQ package 1.6

CFCR now allows audit policy to be configurable via the manifest

CFCR now allows configurable timeout for kubectl drain

CFCR now ensures that BOSH DNS will be chosen first by Kube DNS during resolution

CFCR now ships as a pre-compiled release

CFCR now allows the audit log flag values to be configured by ops file

vSphere Wait for all disks to be detached before shutting down worker nodes

GCP Change GCP to use hostname override due to change of behaviour in stemcell 170.13

BUG FIX: The default flag values for audit log now allow the log file be handled correctly by BOSH logrotate

v0.25.0

Download the release artifact.

Release Date: Dec 5, 2018

We upgraded to Kubernetes 1.12.3, which includes the patch for (CVE 2018-1002105)

BREAKING: Bosh-dns was removed as an addon. CFCR now assumes it's already in the director's runtime config. The ops-file for using runtime config is now empty and will be removed in future versions of CFCR.

--allow-privileged and --keep-terminated-pod-volumes flags in kubelet configuration within the default manifest have both been removed since they have both been deprecated.

Added an ops-file to enable data encryption at rest.

CFCR now uses pre-compiled releases for kubo-release to make deploys faster.

Added an ops-file the configure the number of workers.

Added an ops-file to change certificate duration.

CFCR now ships with Docker 18.06.

CFCR versions are now tied to a specific version of a stemcell in manifest to ensure stemcell compatibility.

Azure: Added support for CIFS volumes for Azure Files or FlexVolumes.

Azure: Azure cloud provider can now use useManagedIdentityExtension property

BUG FIX: Azure load balancers now work with default configuration.

BUG FIX: Now possible to use services with Local externalTrafficPolicy in vSphere & AWS.

BUG FIX: Fixed OIDC ops-file to work with flexible flag.

BUG FIX: Fixed missing kubelet logs.

v0.24.0

Download the release artifact.

Release Date: Nov 9, 2018

BREAKING CHANGE In order to expose all of the kubernetes configuration the manifest format has changed. All the properties for Kubernetes jobs, for example kube-apiserver, kubelet, kube-proxy, cloud-provider etc. have been placed under k8s-args section. Now, every flag that Kubernetes job has can be passed to the Bosh release. All properties in kubo-deployment have been changed. If you have custom operation files to modify the manifest, you will need to change them before you upgrade.

In addition to kubernetes job properties being exposed, cloud provider flags are now exposed to allows more flexible configuration. Any configuration in the source cloud provider will be exposed in the release, for all IaaS.

For more information on modifying kubernetes arguments refer to this page in our docs.

NOTICE While all the properties have been exposed in the release, CFCR team does not support properties that are not part of the kubo-deployment manifest or ops-files.

Kube-apiserver Admission Controllers

In exposing the kubernetes configuration, we now expose the admission-controllers. We support the kubernetes defaults, and adding the following additional admission controllers: DenyEscalatingExec, SecurityContextDeny and PodSecurity Policy.

  • SECURITY The DenyEscalatingExec and SecurityContextDeny admission control plugins are no longer enabled by default. This change was made to better align CFCR with the Kubernetes admission controller defaults and to make it easier to for ops-files to incrementally enable or disable the additional plugins. We recommend that you harden your cluster by applying these ops-files to your manifest.:
  • kubo-deployment/manifests/ops-files/enable-denyescalatingexec.yml

  • kubo-deployment/manifests/ops-files/enable-securitycontextdeny.yml

  • SECURITY we have modified our system add-ons PodSecurityPolicy controller works with system addons out-of-the-box. To enable the controller and use policies apply the following ops-file kubo-deployment/manifests/ops-files/enable-podsecuritypolicy.yml. CFCR does not create default Policy for applications. Please, follow the documentation to enable them https://kubernetes.io/docs/concepts/policy/pod-security-policy/. You need to create PodSecurityPolicy in advance if you upgrading the cluster in order to prevent application downtime. This can be done manually or using new post-start-custom-specs property on kubernetes-roles job.

Other changes

  • It is now possible to backup and restore the cluster control plane when there is 3 masters. This was done by updating the BBR scripts in our etcd release. Thank you Platform Recovery Team for their PR

  • Azure deployment now in Beta stage, with a number of changes in v0.24. Use stemcell from xenial 170 line to deploy it. We expect a number of changes to come in future releases to complete our support coverage.

  • We have exposed the live-restore flag in the docker release, and setting this to true in our manifest. This is to improve workload stability in failure scenarios where the docker daemon is restarted.

  • In order for us to support NFS Storage Class in Kubernetes, we now install the the nfs-common library. This was removed in the Xenial stemcell line. – bug

Component Versions

The following table lists the component versions for CFCR v0.24.0:

Component Version
Kubernetes 1.11.3
Flannel 0.10.0
ETCD 3.3.9
Docker 17.12.1-ce
CNI 0.7.1

v0.23.0

Download the release artifact.

Release Date: Oct 16, 2018

  • We have extended our multi-cloud support to Azure. PR #223 We consider this experimental as we learn, iterate and extended our test infrastructure.

  • As previously announced, the route-sync functionality is now removed. #158559777

  • In order to help automate the maintenance load balancers for master nodes during deploy and upgrade on GCP, we have added the backend-service in our default vm_extensions - #160417211

  • Fix: Fixed an issue seen during drain: #251

Component Versions

The following table lists the component versions for CFCR v0.23.0:

Component Version
Kubernetes 1.11.3
Flannel 0.10.0
ETCD 3.3.9
Docker 17.12.1-ce
CNI 0.7.1

v0.22.0

Download the release artifact.

Release Date: Sept 28, 2018

  • Upgraded to Kubernetes v1.11.3

  • In order to allow workloads that have a security context configuration, the SecurityContextDeny admission controller can now be disabled through a ops-file, it will continued to be enabled by default. The allow-privilege property and corresponding ops-file will no longer disable the SecurityContextDeny admission controller. This is a change in behaviour, if you were using allow-privileged to disable SecurityContextDeny, you should now use the disable-security-context-deny.yml ops-file

  • The kubo-release version set in manifest is configured to the most recent released version. It will no longer be latest or bundled within the kubo-deployment release artifact. The kubo-deployment folder name specifies the version.

  • For GCP deployment we have allowed for the setting of the sub-network name. Doing this will allow for services using internal loadbalancers to be deployed – GH issue

  • CFCR no longer supplies scripts to deploy BOSH. We assume a BOSH director has been provisioned, with some standard cloud config dependencies. If you need to deploy BOSH we recommend using BOSH Boot Loader. See the guide below for migration away from deploy_bosh.

  • Added an ops-file to use vm_extensions to on vSphere, namely disk.enableUUID

  • Documentation We have provided instructions on how to configure Kubernetes with a Cloud Provider for each Iaas in our docs. The cloud provider interfaces with the IAAS to provision TCP Load Balancers, Nodes, Networking Routes, and Persistent Volumes.

deploy_bosh Migration Guide

Going forward, we're encouraging users to discontinue using the deprecated deploy_bosh and deploy_k8s scripts. To help with the transition from the scripts to standard practices for managing bosh deployments, we've outlined a few changes to be aware of:

Cloud Config

Previously the deploy_bosh scripts would generate a compatible cloud config for CFCR clusters. The generated cloud-configs are still valid and can be used. See our documentation for cloud-config for requirements. If you are configuring CFCR with a cloud provider, we've moved to using vm_extensions defined with BOSH generic configs. See our cloud-provider documentation to see how to set this up.

Deployment Manifest

Instead of generating a manifest via the deploy_k8s script, we encourage users to follow our deployment docs and create a bosh deploy command with ops-files appropriate for their CFCR clusters. One of the big reasons we deprecated these scripts was to decouple the bosh director provisioning steps when creating CFCR clusters. Previously, deploy_bosh would create an artifact called director.yml and this file was used in deploy_k8s to provision CFCR clusters. The director.yml file contains credentials that can still be used as vars to bosh deploy command. However the file itself is not required.

Component Versions

The following table lists the component versions for CFCR v0.22.0:

Component Version
Kubernetes 1.11.3
Flannel 0.10.0
ETCD 3.3.9
Docker 17.12.1-ce
CNI 0.7.1

v0.21.0

Download the release artifact.

Release Date: August 29, 2018

Note

The Kubernetes 1.11 release kicked off the deprecation timeline for the Heapster component, see https://github.com/kubernetes/heapster/blob/master/docs/deprecation.md for more info. As a result, we're in the process of replacing Heapster with Metrics Server in an upcoming releases of kubo-release. Once that process is complete, Heapster will no longer be a default addon applied in 'apply-specs' errand.

  • We are deploying metrics server by default as part of the apply-specs errand. Heapster is still deployed but will be removed in a future release – story

  • With metrics server deployed securely, CFCR supports Horizontal Pod Autoscaler. This includes optional HPA configuration flags as part of CFCR release spec. – story

  • Fix: We found and responded to an issue in kubernetes, which was causing issues with updating StatefulSet workloads on AWS. We have improved our drain behvaviour, and there is a manual workaround – bug If you find the rare case, where a pod that uses volumes is stuck in ContainerCreating state after upgrade, also you can see FailedMount message when you run kubectl describe pod command. To manually fix it, recreate a worker using bosh recreate worker/

  • We have included CoreDNS as an optional addon. This can be set by applying the use-coredns.yml ops-file before running the apply-specs errand – story CoreDNS should be deployed in place of kube-dns. kube-dns deployment should be deleted from existing clusters, after deploying CoreDNS

  • Fix: There is an issue deploying v0.20 in environments not connected an internet registry. We fixed the name in a packaged docker image – bug

  • We have enabled the HostPort feature, to allow pods to open external ports on the worker node. This includes an upgrade to the CNI component to v0.7.1 – story

  • We have incorporated Kubernetes release v1.11.2

Component Versions

The following table lists the component versions for CFCR v0.21.0:

Component Version
Kubernetes 1.11.2
Flannel 0.10.0
ETCD 3.3.9
Docker 17.12.1-ce
CNI 0.7.1

v0.20.0

Download the release artifact.

Release Date: August 8, 2018

Known Issue There is an issue with deploying cfcr v0.20 in airgapped environments, related to an image dependency. This will be fixed in the next release.

Features

  • Upgraded Kubernetes version to v1.11. For more information about the upgrade, see story #159328985 in the CFCR Tracker. For more information about Kubernetes v1.11, see the Kubernetes release notes.

  • Upgraded the default stemcell to Ubuntu 16.04 Xenial. For more information about the upgrade, see story #158004796 in the CFCR Tracker.

    Tip

    If you used CFCR’s ./deploy_bosh script to deploy your BOSH director, we recommend updating the BOSH runtime config, as it has a reference to Ubuntu Trusty. For more information, see the BOSH documentation.

  • This release includes BOSH Backup and Restore (BBR) scripts to restore the state of a cluster from a backup. For more information about the feature, see stories #158945642 and #158946553 in the CFCR Tracker. For more information about BBR, see the BBR documentation.

    Note

    The BBR functionality is verified against a single-master cluster running stateless workloads. We will support further verified scenarios in future versions.

  • Upgraded the ETCD release to v3.3.9.

  • Deprecation notice: Heapster is officially deprecated in Kubernetes v1.11. We will remove it by default in favor of metrics-server in a future release.

Component Versions

The following table lists the component versions for CFCR v0.20.0:

Component Version
Kubernetes 1.11.1
Flannel 0.10.0
ETCD 3.3.9
Docker 17.12.1-ce
CNI 0.5.2

v0.19.0

Download the release artifact.

Release Date: July 18, 2018

  • Deprecation notice Future release will remove support of the CF routing feature – #157695924

  • Allow CIDRs to be configured for pods and services – pr #220,#157480131

  • Update the admission-controllers based on kubernetes recommendations – #156525910 Added DefaultTolerationSeconds and ValidatingAdmissionWebhook. We removed NamespaceExists, as it is redunant.
  • Kubernetes v1.10.5 version – #158527191
  • Changed the docker storage driver from overlay to overlay2 – #158495554 When upgrading, the old images will remain on each worker in the /var/vcap/data/docker/docker/overlay directory. Recommended mitigation is to manually delete the directory after upgrading
  • Allow NTLM formatted usernames for vSphere – pr #229
  • Fix: improve drain script for upgrades on large clusters – #158782574

Component Versions

The following table lists the component versions for CFCR v0.19.0:

Component Version
Kubernetes 1.10.5
Flannel 0.10.0
ETCD 3.3.1
Docker 17.12.1-ce
CNI 0.5.2
Stemcell 3586.25

v0.18.0

Download the release artifact.

Release Date: July 5, 2018

  • Deprecation notice Removal of ABAC as an authorization mode option #157695924

  • Improved the security of internal communication between kubernetes components. Stories: #158357421 #158356113 #158356114

  • Allow storage classes to be configured with vSphere storage policies. #157477844

  • Ensure nodes will recover correctly with vSphere HA #158142038 #158180949

Other minor changes

  • Improved our tooling to deploy CFCR on BOSH-lite. readme
  • Changed the set_kubeconfig input parameters. Note: credhub login required before running set_kubeconfig. story

Component Versions

The following table lists the component versions for CFCR v0.18.0:

Component Version
Kubernetes 1.10.4
Flannel 0.10.0
ETCD 3.3.1
Docker 17.12.1-ce
CNI 0.5.2
Stemcell 3586.24

v0.17.0

Download the release artifact.

Release Date: May 30, 2018

  • Hardening and validation of running multiple Kubernetes Master Nodes across Availability Zones. The BOSH native manifest will deploy with three masters by default, our script based deployment will continue to deploy one. -- epic

  • Upgraded to Kubernetes v1.10 -- story

  • Added smoke-tests to verify a deployed cluster -- story, readme

  • Allow operators to control feature gates on Kubernetes components. -- PR

Other changes

  • Enable kube-controller-manager to issue certificates PR:1,2,3
  • Support AWS LoadBalancers when externalTrafficPolicy is set to Local -- story
  • Add BOSH instance IDs to the node labels -- PR
  • Re-enable rpcbind to allow support for NFS -- story
  • Remove unused terraform variable from kubo-lbs.tf -- PR
  • Operator rename.yml to set deployment name and etcd certificate alter -- PR
  • Drain script improvements -- story
  • Enabled the automated vSphere cloud provider (ref), and remove references to the cloud provider in vSphere deployed workers. -- stories:1,2

Announcement

In version v0.18 we are going to remove support for ABAC as a authorization mode.

Component Versions

The following table lists the component versions for CFCR v0.17.0:

Component Version
Kubernetes 1.10.3
Flannel 0.10.0
ETCD 3.3.1
Docker 17.12.1-ce
CNI 0.5.2
Stemcell 3586.16

v0.16.0

Download the release artifact.

Release Date: April 06, 2018

  • A new etcd bosh release is being used (cfcr-etcd-release), this is to enable deployments with multiple etcd nodes -- story.
  • Prevent unnecessary route creation in default kube-controller-manager config -- story.
  • System specs can be applied even when a cloud provider is not configured -- story.
  • Enable inter-container communication in flannel, while retaining the original source ip. -- story.

  • CFCR can be deployed to BOSH-lite -- story.

  • Fix: Made the node drain (even) more robust -- story.

Other minor changes

Component Versions

The following table lists the component versions for CFCR v0.16.0:

Component Version
Kubernetes 1.9.6
Flannel 0.10.0
ETCD 3.3.1
Docker 1.13.1
CNI 0.5.2
Stemcell 3541.10

Conformance Tests Results

v0.15.0

Download the release artifact.

Release Date: March 20, 2018

  • Removed edge case where timeouts during upgrades could lead to Etcd data loss -- story.
  • Added the ability to configure the HTTP(s) proxy to be used by the Kubernetes control plane -- story.
  • Made Kube-DNS use its own configuration so that the Kubelet configuration is not exposed in the Kube-DNS container -- story.
  • Fix: Allowed \ and | to be used in vCenter passwords -- cloudfoundry-incubator/kubo-release#180.
  • Fix: Made the node drain more robust -- story #155549518, story #156008895 and cloudfoundry-incubator/kubo-release#181.
  • Fix: Included a kube-proxy dependency (conntrack) to fix error logs -- story.
  • Fix: Can use kubectl top node against a CFCR cluster. Caveat: the --heapster-scheme='https' flag needs to be included -- story.
  • GCP: Stopped the Kube-Controller-Manager from creating unnecessary routes -- story.
    • This was causing high Google Cloud API usage.
  • AWS: Added the ability to provide AWS credentials in the BOSH manifest -- story.
    • Previously, AWS access could only be granted by setting the IAM profile in the cloud config. In cases of BOSH Directors that are used by multiple deployments, it is necessary to provide the credentials in the BOSH manifest.

Other minor changes

Component Versions

The following table lists the component versions for CFCR v0.15.0:

Component Version
Kubernetes 1.9.5
Flannel 0.10.0
ETCD 3.3.2
Docker 1.13.1
CNI 0.5.2
Stemcell 3541.9

Conformance Tests Results

v0.14.0

Release Date: February 20, 2018

  • Kubernetes v1.9.3 -- cloudfoundry-incubator/kubo-release#176.
  • Flannel v0.10.0 -- cloudfoundry-incubator/kubo-release#169.
  • BOSH DNS v0.2.0 -- cloudfoundry-incubator/kubo-deployment#261.
  • GOVC v0.16.0.
  • Golang v1.9.4.
  • BOSH Stemcell v3541.4.
  • CFCR can now be deployed on an environment paved by BBL -- story.
  • Exposed OpenID authentication properties -- cloudfoundry-incubator/kubo-release#101.
  • logging-level BOSH property can be used to control the logging level of kube-proxy -- cloudfoundry-incubator/kubo-release#163.
  • HTTP(s) Proxy BOSH properties will be used for Kubernetes interactions with the IaaS -- cloudfoundry-incubator/kubo-release#130.
  • Nodes can now be deployed across multiple AZs on GCP -- story.
  • Nodes get tagged appropriately by Kubernetes to ensure that workloads are properly spread across AZs.
  • System workloads are now applied as part of the apply-addons BOSH errand -- story.
  • System workloads have been a cause of many deployment issues.
  • Enabled the API server audit logs -- story.
  • Audit logs can be disabled if the kube-apiserver.enable_audit_logs BOSH property is set to false.
  • Disabled the read-only port in the Kubelet -- story.
  • Disabled cAdvisor in Kubelet -- story.
  • Disabled the security context manipulation when privileged containers are off -- story.
  • The API server will not try to fix malformed requests anymore for security reasons -- story.
  • The API Server will clean up terminated pods more often to avoid running out of disk space -- story.
  • The API server will unmount volumes of terminated pods for security reasons -- story.
  • Most BOSH jobs switched to use BPM -- story.
  • From the BPM readme: "[BPM] crucially provides a security barrier such that if one of the jobs on your machine is compromised then the incident is limited to just that job rather than all jobs on the same machine".
  • OpenStack: Exposed cloud-provider.openstack.ignore-volume-az BOSH property for the OpenStack Cloud Provider -- cloudfoundry-incubator/kubo-release#166.
  • OpenStack: Exposed region BOSH variable for the OpenStack Cloud Provider -- cloudfoundry-incubator/kubo-deployment#262.
  • Fix: UAA credentials and vCenter passwords are now redacted in BOSH logs -- story.
  • Fix: to ensure that workers will pick the correct node name during rolling upgrades -- cloudfoundry-incubator/kubo-release#170.
  • Fix: to ensure that nodes get properly drained before they stop, in order to minimize workload downtime during a rolling upgrade -- story.
  • vSphere Fix: vCenter password with special characters (&, #, etc) can now be used with CFCR without breaking the deployment -- story.
  • Experimental: An ops-file can now be used in conjunction to the kubo-deployment in order to experiment with the multi-master setup -- story.

Component Versions

The following table lists the component versions for CFCR v0.14.0:

Component Version
Kubernetes 1.9.3
Flannel 0.10.0
ETCD 3.2.14
Docker 1.13.1
CNI 0.5.2
Stemcell 3541.4

Conformance Tests Results

v0.13.0

Release Date: January 25, 2018

Download the release artifact.

  • Kubernetes 1.9.2 -- story and story.
  • Flannel 0.9.1 -- story.
  • RBAC as the default authorization mode.
  • Support for VM power-offs and restarts -- story.
    • Reliance on certain functionality provided by BOSH was causing restarting VMs to fail.
  • Secure communications between system specs (Dashboard, Heapster and InfluxDB) -- story and story.
  • Ability to configure the timeout for system specs -- story.
    • The BOSH property is kubernetes-system-specs.timeout-sec and is set to 20 minutes by default.
  • Ability to update addon specs without experiencing API downtime -- story.
  • Ability to get diagnostic information if a system pod fails to be applied -- story.
  • Ability to have the default storage class be used in PVCs that do not specify a storage class -- story.
  • Ability to rotate the Kubernetes API certificate -- story.
  • Ability to use the syslog addon in a CFCR deployment -- story.
  • Fix: to not print secrets in user-facing scripts -- story.
  • Fix to not have more than one nodes go down during an upgrade -- story.
  • vSphere Fix: to avoid a synchronization issue that was causing master to fail to start -- story.
  • OpenStack Fix to have CFCR properly configure Kubernetes in order to communicate securely (TLS) to OpenStack -- cloudfoundry-incubator/kubo-release#156.

Component Versions

The following table lists the component versions for CFCR v0.13.0:

Component Version
Kubernetes 1.9.2
Flannel 0.9.1
ETCD 3.2.14
Docker 1.13.1
CNI 0.5.2
Stemcell 3468.20

Conformance Tests Results

v0.12.0

Release Date: January 10, 2018

Download the release artifact.

  • Use Kubernetes 1.8.6 -- story.
  • Enable secure access to the Dashboard via a NodePort when using RBAC -- story.
  • Privileged container support is turned off by default. There is a new property named allow_privileged_containers in director.yml which can be used to enable the feature -- story.
  • also cloudfoundry-incubator/kubo-deployment#252 and cloudfoundry-incubator/kubo-release#153
  • Don't update the master node when scaling up workers -- story.
  • Switch to use cfcr.internal as a TLD instead of .kubo -- story.
  • Disable all profiling / tracing endpoints by default -- story.
  • Always validate ServiceAccount tokens exist in etcd as part of authentication -- story.
  • Stop the Kubernetes API Server from serving unsecured and unauthenticated access in localhost -- story.
  • Remove unnecessary flag from kube-proxy -- story.
  • Make applying addon specs not fail if the specs are empty -- story / cloudfoundry-incubator/kubo-release#150.
  • Bump system specs timeout to work with slower environments -- story.
  • Implement logic to never lose more than one worker nodes during update -- story.
  • Restrict the data directory permissions for etcd -- story.
  • Use SSL for etcd peer connections -- story.
  • We are currently running single-node etcd clusters so no peer connections are established. Nevertheless, etcd would listen for peer connections over plain HTTP.
  • Openstack: openstack_tenant is not required in the director.yml as it is obsolete for OpenStack Keystone v3. The property still exists as it is needed by OpenStack Keystone v2 -- story.
  • Openstack: VMs deleted from the IaaS in OpenStack do not appear as ghost nodes -- story.
  • The fix for this issue introduces new director.yml properties as the OpenStack K8s Cloud Provider needs to be configured:
    • auth_url, openstack_username, openstack_password, openstack_project_id, openstack_domain
  • Caveat: the BOSH director needs to have human_readable_vm_names set to false in order for the Kubelet to register with the API Server successfully. See K8s issue: kubernetes/kubernetes#57765.
  • Fix: deploy_k8s to fail when the addon specs are not successfully applied -- story.
  • Fix: regression in abac authorization mode -- story.
  • vSphere PR Support CFCR deployments on vSphere environments with multiple datacenters -- cloudfoundry-incubator/kubo-release#127 / cloudfoundry-incubator/kubo-release#148.

BOSH Release

Component Versions

The following table lists the component versions for CFCR v0.12.0:

Component Version
Kubernetes 1.8.6
Flannel 0.5.5
ETCD 3.2.10
Docker 1.13.1
CNI 0.5.2
Stemcell 3468.13

Conformance Tests Results

Download the conformance test results.

v0.11.0

Release Date: December 20, 2017. Download the release artifact.

  • Rename the CF Routing properties in director.yml to follow a consistent naming style. story.
    • [ACTION REQUIRED]: If you use CF Routing in CFCR, you must update your director.yml with the new CF Routing property names.
  • Implement ability to deploy CFCR on an existing BOSH Director and make the experience for BOSH-native users comparable to the one in cf-deployment, bosh-deployment, and concourse-deployment. PR.
    • [ACTION REQUIRED]: If you deploy CFCR by means of bosh deploy, you should re-examine the names of the manifest and the ops-files as various changes have been made.
  • vSphere: Support persistent workloads on vSphere environments that do not use Resource Pools. Resource Pools are optional in vSphere and CFCF already supports vSphere environments that make use of Resource Pools. story.
  • Looked into an issue with the kubernetes-system-spec post-start script. story / Github Issue.

Theme: Security

  • AWS: Reduce the permissions of the AWS IAM policies for the master and worker nodes so that if credentials leak from an AWS CFCR node, the permissions associated with it are minimal. story.
    • [ACTION REQUIRED]: If you deploy CFCR on AWS, you must re-run Terraform to update the IAM Policies.
  • Use stemcell version 3468.13. story.
  • Disallow anonymous requests to the API server. story.
  • Alter the permissions of files with sensitive information so that they cannot be read by non-root users. story.
  • Make ETCD only listen to TLS connections so that ETCD-bound traffic in the cluster cannot be sniffed. story.
  • Disallow exec and attach commands to privileged pods. story.
  • vSphere: Escape back-slashes for vSphere users in cloud config. PR.

Component Versions

The following table lists the component versions for CFCR v0.11.0:

Component Version
Kubernetes 1.8.4
Flannel 0.5.5
ETCD 3.2.10
Docker 1.13.1
CNI 0.5.2
Stemcell 3468.13

Conformance Tests Results

Download the conformance test results.

v0.10.0

Release Date: December 8, 2017. Download the release artifact.

  • New property addons_spec_path in director.yml. Operators can use this property to provide a K8s spec file that is applied to the cluster when it comes up. story.
  • New property worker_count in director.yml. Operators can use this property to configure the number of K8s workers. story.
  • Enabled the K8s aggregation layer to support API server extensions. story.
  • CFCR was tested to run with 20 workers and "chatty" workloads. story.
  • Exposed the K8s API connection properties via a BOSH link. story.
  • [ACTION REQUIRED]: The HAProxy (proxy) routing mode is not longer supported. story.
  • [ACTION REQUIRED] GCP: The service_account property in director.yml is no longer supported. It has been replaced by service_account_master and service_account_worker which can be used to reference GCP service accounts that are provided to master and worker VMs separately. story.
  • GCP: New properties service_key_master and service_key_worker in director.yml. Operators can use these properties to enable the K8s cloud provider to use a GCP service account without having to change the BOSH cloud config. story.
  • GCP: The GCP K8s Service Catalog was tested on CFCR. story.
  • GCP: Predefined a standard storage class to be applied when CFCR is deployed on GCP. It uses the gce-pd PV provisioner. story.

Community contributions

Component Versions

The following table lists the component versions for CFCR v0.10.0:

Component Version
Kubernetes 1.8.4
Flannel 0.5.5
ETCD 3.2.10
Docker 1.13.1
CNI 0.5.2
Stemcell 3468.5

Conformance Tests Results

Download the conformance test results.

v0.9.0

Release Date: November 22, 2017. Download the release artifact.

Features

  • CFCR has been added to the Certified Kubernetes Conformance Program. See the Conformance Test Results below.
  • The Docker BOSH release has been updated to v30.1.4.
  • The ETCD and master nodes are colocated on the same VM. Deployments of v0.9.0+ have 3 worker nodes and 1 master/ETCD node.
  • BOSH has been updated to v264.1.
  • The Kubernetes Dashboard is accessible with RBAC mode as cluster admin. The Dashboard needs to be exposed via kubectl proxy.

Bug Fixes

Component Versions

The following table lists the component versions for CFCR v0.9.0:

Component Version
Kubernetes 1.8.2
Flannel 0.5.5
ETCD 3.1.8
Docker 1.13.1
CNI 0.5.2
Stemcell 3445.11

Conformance Tests Results

Download the conformance test results.

v0.8.1

Release Date: November 10, 2017. Download the release artifact.

Features

  • Upgraded Kubernetes version to v1.8.2

Bug Fixes

  • Bug in authorization switch mechanism: all clusters were deployed in RBAC by default. New property in director.yml to set desired mode (ABAC|RBAC).

Component Versions

The following table lists the component versions for CFCR v0.8.1:

Component Version
Kubernetes 1.8.2
Flannel 0.5.5
ETCD 3.1.8
Docker 1.11.0
CNI 0.5.2
Stemcell 3445.11

v0.8.0

Release Date: November 3, 2017. Download the release artifact.

Features

  • Upgraded Kubernetes version to v1.8.1
  • Bosh DNS replaces Power DNS
  • Memory limit is configurable
  • Kubelet resource reservation flags exposed: kube-reserved, system-reserved, eviction-hard. See Kubernetes docs for more information.
  • Internal routing from workers to masters through BOSH DNS -- no need for HAProxy or LB to route cluster internal traffic
  • User can load balance traffic from external load balancers

Improvements

  • Removed worker_node_tag property to set worker tags automatically for GCP load balancers

Bug Fixes

Component Versions

The following table lists the component versions for CFCR v0.8.0:

Component Version
Kubernetes 1.8.1
Flannel 0.5.5
ETCD 3.1.8
Docker 1.11.0
CNI 0.5.2
Stemcell 3445.11

Upgrading from v0.7.0

Perform the following steps to upgrade an existing CFCR v0.7.0 cluster to v0.8.0:

  1. Clone the new version of kubo-deployment.
  2. Log in to the CredHub server on your BOSH Director with the CredHub CLI.
  3. Delete the current Kubernetes certificate from CredHub:

    $ credhub delete -n "${director_name}/${deployment_name}/tls-kubernetes"

  4. Verify that the appropriate stemcell is installed in BOSH. To view the uploaded stemcells, run the following command.

    $ bosh stemcells

    To upload a new stemcell, run bosh upload stemcell STEMCELL_URL.
  5. See the BOSH Bootloader repository for information on how to update the BOSH Director.

  6. See CFCR repository for information about upgrading your CFCR cluster.

  7. Deploy the CFCR BOSH release. See the CFCR repository for instructions.

v0.7.0

Release Date: September 7, 2017. Download the release artifact.

Features

  • kubo-release tarball bundled with kubo-deployment
  • The deploy-k8s script deploys local release by default
  • Cluster self-healing capabilities enables the recoveru of worker VMs
  • Support for persistent volumes in GCP, AWS, and vSphere
  • Improved documentation to install on GCP and AWS

Bug Fixes

  • Removed AWS-related tags for other platforms
  • Password issues in vSphere: GitHub issue #102