README.md

# Renovate

[Renovate](https://docs.renovatebot.com/) is a free and open source tool that automatically updates dependencies. Renovate supports multiple languages and platforms such as docker, golang, node, etc. See [here](https://docs.renovatebot.com/modules/datasource/) for a complete list.

The pluggable constructs in Renovate include:

* **manager**: extracts dependencies and assigns a datasource.
* **datasource**: fetches releases for a dependency

Renovate handles opening Gitlab Merge Requests for out-of-date dependencies and can be customized for individual projects.

## How we use Renovate in Ironbank

Renovate is an opensource project which we have implemented in Repo1's Gitlab.  Our renovate-bot runs daily so any vendor that wants to create their own configuration of Renovate is free to do so.  The follow subsequent sections outline how we utilize Renovate on opensource projects that are mangaged by the Container Hardening Team to watch for updates to upstream resouces in those projects.  Ultimately, the vendor is responsible for the updating of their container in Ironbank, the use of Renovate is not required but is an option we provide for their use.

Renonvate has a capability set of updating dependancies from a wide range of sources, however, due to the nature of our container hardening process we utilize a very narrow subset of Renovate's capabilites.  Gitlab projects in Repo1 do not contain any of the application source code, only the process of how that application is hardened into a container image.  Therefore, managers and datasource that are targeted at updating source code depenancies are not used.

The Ironbank pipeline requires all upstream resouces to be declared in the hardening_manifest.  Due to this we use Renovate, more specifically a custom manager call `Ironbank` and the `Regex Manager`, to parse the hardening_manifest for declared versions and check for updates using the supported datasources in our `Ironbank Manager` or what is outlined in the renovate.json file for the `Regex Manager`.

---

**Note**

There is a limitation for this usage.  Your upstream resource must be stored in a location for which renovate has a datasoure (a full list of supported datasources can be found [here](https://docs.renovatebot.com/modules/datasource/) ).  Unfortunatly, this is a limiting factor for some projects.  For these cases we use an alternative approach, while not as elegant as Renovate it works.

> **_Important:_** It is important to remove any references to a specific application version in the filename of the hardening_manifest.yaml file when downloading artifacts that are not images. This way, if the artifact version is bumped, the COPY statement included in the Dockerfile will be able to handle this updated version, without failing because it attempts to copy an older version which is no longer being downloaded.

> **example:** For example, use the filename kubernetes.tar.gz instead of the filename kubernetes-1.18.3 because the Dockerfile COPY kubernetes-1.18.3.tar.gz statement will throw an error that a resource cannot be found if the Kubernetes artifact version is bumped to 1.18.4. COPY kubernetes.tar.gz will handle any version bumps that occur and result in successful copying of the hardening_manifest.yaml artifact for the build.
---

## Hardening_manifest

Specifically, there are three pieces of data in the `hardening_manifest` that Renovate needs to watch and update.

    - tags
    - labels
    - resources

*The following are example snippits from the ArgoCD project hardening_manifest that identify these:*

```yaml
tags:
- "v1.8.7"

labels:
    org.opencontainers.image.version: "v1.8.7"

resources:
- url: "docker://docker.io/argoproj/argocd@sha256:ce34acd7bac34d5a4fdbf96faf11fa5e01a7f96a27041d4472ca498886000cbf"
  tag: "argoproj/argocd:v1.8.7"

```

We use two Renovate managers to fully update the `hardening_manifest`: `Ironbank` and `Regex`

## IronbankManager

The `Ironbank Manager` is a custom **manager** that was developed for use in Repo1 to handle extracting dependencies from the `hardening_manifest.yaml` file resource section.  If you have a hardening_manifest.yaml file and a renovate.json file the `Ironbank Manager` will automatically parse the resource section and look for updates for each resource that comes from a supported datasource.

`docker`, `github-releases`, `github-tags`, `ruby-gems`, and `pypi` **datasources** are currently the only supported datasource, but other native datasources can be added as well upon request. The manager parses the `url` key to determine the type of dependency.

```yaml
- url: docker://{registry}/{repo}@{digest}
- url: https://github.com/${repo}/.../${version}/${archive}
- url: https://rubygems.org/downloads/{import}-{version}.gem
- url: https://files.pythonhosted.org/../.../{import}-{version}...whl
```

Any other datasource in your hardening_manifest.yaml file will not be automatically renovated.  When selecting where you are pulling your resources from please take this into consideration, all things being equal we prefer you pull from one of the supported datasources.

---

**Note**
In order for you to take advantage of the custom Ironbank Manager you only need to have an empty renovate.json file in the base of your Gitlab project. The existance of the renovate.json file is what we use to identify projects to run the Renovate-bot against. The Ironbank Manager will parse the hardening_manifest (if it exists) and check for updates for supported sources by default without any additional configuration.

## RegexManager

Renovate includes a **regex manager** that can extract dependencies with a regular expression. This is useful for file formats that do not have an associated manager. Within Ironbank, the regex manager is used to handle updating the `hardening_manifest` **tags** and **labels** section.

The example `renovate.json` below will update the **tags:** and **labels:** strings if there is an update to the argocd docker image found on docker.io.

```json
{
  "baseBranches": ["development"],
  "regexManagers": [
    "fileMatch": [
        "^hardening_manifest.yaml$"
    ],
    "matchStrings": [
        "org\\.opencontainers\\.image\\.version:\\s+\"(?<currentValue>.+?)\"",
        "tags:\\s+-\\s+\"(?<currentValue>.+?)\""
    ],
    "depNameTemplate": "argoproj/argocd",
    "datasourceTemplate": "docker"
    }
  ]
}
```

> **_Important:_** You only need to add the `Regex Manager` to your renovate.json file if your main resource (that is the resource that dictates the tag and label) comes from a supported datasource.  

There are cases where the main resource does not come from a supported datasource but other supporting resource in the hardening_manifest do have supporting datasources.  In those cases you will still need a renovate.json file but you won't use a `Regex Manager`.  See [Workflow](##RenovateWorkflow) on minimum renovate.json requirements 

When using the `Regex Manager` you will need to provide the what datasource it is and the dependancy name. Notice the `depNameTemplate` and `datasourceTemplate` that defines this information in the above example.

---
**Note**
The following examples can help you determine what you will put for the `depNameTemplate` and `datasourceTemplate`

  Docker:

    if from docker hub the depNameTemplate will be the docker hub path 

      - "depNameTemplate": "argoproj/argocd",
        "datasourceTemplate": "docker"

    any other docker registry the depNameTemplate will be the full path 
      
      - "depNameTemplate": "gcr.io/tekton-releases/github.com/tektoncd/triggers/cmd/controller",
        "datasourceTemplate": "docker"

  Github:

    if the source is github and the url contains 'archive' or 'tags' you will use github-tags 
    as the datasourceTemplate and github path as the depNameTemplate
      
      - "depNameTemplate": "postgres/postgres",
        "datasourceTemplate": "github-tags"

    else it will be github-releases

      - "depNameTemplate": "gohugoio/hugo",
        "datasourceTemplate": "github-releases"

  Pypi and Rubygems:

    These will just the name of the package depNameTemplate 
    with the corrisponding datasourceTemplate
    
      - "depNameTemplate": "fluentd",
        "datasourceTemplate": "rubygems"

### Major Versions

Repositories that a pinned to a major version (i.e. `postgresql10`) should disable major revision bumps.

```json
{
    "major": {
        "enabled": false
    }
}
```

### Reviewers

Add reviewers for each repository so `MAINTAINERS` will be notified of a new pull request.

```json
{
    "reviewers": [
        "jeason"
    ]
}
```

### Automerge

Renovate can also be configured to automerge pull requests (currently under testing).

```json
{
    "automerge": true,
    "gitLabAutomerge": true,
}
```

---

**Note**
Renovate is very configurable and provides many configration options that can be put in the renovate.json file. For more indepth documentation and configuration options you can visit the official documentation [here](https://docs.renovatebot.com/configuration-options/).

## RenovateWorkflow

In order to begin using Renovate in your project you must have a renovate.json file in the base of your project. Any project with a renovate.json file the renovate-bot will detect and run against it.

---

**Note** This is an example of the most basic renovate.json you can have:

```json
{
    "baseBranches": ["development"]
}
```

This will ensure the `Ironbank Manager` will run against your resources list in your `hardening_manifest` without updating either the tag or labels.

---

You only need to add a renovate.json file if there are resources in your `hardening_manifest` file that have supporting [datasources](##IronbankManager). You must also add your [regex managers](##RegexManager) to update the tag and label to the correct version if your main resource has a supported datasource.  If your main resouce has a supported datasource and you have proper regex managers in the renovate.json file any updates to that resource will produce a completely updated hardening_manifest in the generated merge request in Gitlab.

If an update is detected a merge request will be created with those changes.  An Issue will be created and associated with that merge request.  It is then required that the normal container hardening process if followed with that merge request.  The pipeline must pass, merged into development, and any new CVEs must be justified before requesting it to be merged to master.  

## Limitations

### Renovate Datasources

The base image in this repository does not support many of the native Renovate datasources. The image provides support for the minimum feature set required by Ironbank (eg, docker, github-releases, github-tags, rubygems, pypi).

Also metioned before your upstream resource must be stored in a location for which renovate has a datasoure.  Renovate ships with several datasources such as docker, github-releases, rubygems, and pypi. A vendor's upstream resource may not publish to one of these supported platforms and as such could be a limiting factor for the use of Renovate. A custom datasource would have to be implemented to support fetching release information for these resources.

### YUM

Renovate does not currently support package managers. An open feature request can be found [here](https://github.com/renovatebot/renovate/issues/6188).

One challenge with package manager support is dealing with **transitive** dependencies. In the Dockerfile below, installing `java-11-openjdk-headless` with `dnf` will include any of its dependencies as well.

    ```shell
    RUN dnf install -y java-11-openjdk-headless && \
        dnf clean all && \
        rm -rf /var/cache/dnf
    ```

In this example, Renovate needs to gather dependency metadata to handle upgrades. But where should it get this data?

There are a few approaches that can be applied to solve this issue:

1. Do no use Renovate! Trigger a new build periodically with `dnf update -y` regardless of whether or not a dependency has been updated. This mechanism could be extended to first pull the existing image and check for updates before triggering a build:

```shell
    docker run --rm --entrypoint=/bin/bash -it [repository]/[image]:[tag] dnf list updates
```

2. Define a manifest for the image and include it within the repository. Renovate could trigger updates for each dependency in the manifest. For example:

dsop/opensource/openjdk/openjdk11/packages/manifest.txt

```
...
copy-jdk-configs-3.7-1.el8.noarch
javapackages-filesystem-5.3.0-1.module_el8.0.0+11+5b8c10bd.noarch
lcms2-2.9-2.el8.x86_64
alsa-lib-1.1.9-4.el8.x86_64
lua-5.3.4-11.el8.x86_64
libjpeg-turbo-1.5.3-10.el8.x86_64
avahi-libs-0.7-19.el8.x86_64
tzdata-java-2019c-1.el8.noarch
cups-libs-2.2.6-28.el8.x86_64
freetype-2.9.1-4.el8.x86_64
libpng-1.6.34-5.el8.x86_64
lksctp-tools-1.0.18-3.el8.x86_64   
java-11-openjdk-headless-11.0.7.10-1.el8_1.x86_64
...
```

3. Write a custom manager that inspects the existing docker image for updates similar to (1). However it must bump something in the repository to create a valid pull request. This could be a CHANGELOG file for example.

## Challenges

Automated dependency updates present some challenges.

Software upgrades are not always as simple as bumping a version. A `Dockerfile` may require additional changes that accompany a version bump.

    - How are we made aware of these changes?
    - How can we keep these changes in sync?

Renovate is one tool that can be used to solve some of these issues but additional work needs to be done to move those updats through the hardening