Moved over resources from /big-bang/terraform-modules/air-gap-deployment

docs/airgap/README.md

+# Airgap
+
+Currently this is in proof of concept mode, so play around with this to get an idea of it.
+
+This work was quickly developed to entertain certain paths for image packaging and deployment.
+
+## Image Packaging / Deployment
+
+`package_images.sh` - Proof of concept script for image packaging
+
+* Dependencies
+  * `docker` - The docker CLI tool
+  * `images.txt` - A list of all requires airgap images
+  * `jq` - The jq CLI tool
+* Deliverables
+  * `registry:package.tar.gz` - Modified `registry:2` container loaded with airgap images
+    * NOTE - `registry:2` vs `harbor` vs anything else is trivial, we can use whatever we want
+    * Packaged images are loaded and retrievable immediately upon container start
+    * `/var/lib/registry-package` is created and populated with images
+    * `/etc/docker/registry/config.yml` is templated to use new registry folder
+    * This is due to the fact that `/var/lib/registry` is a docker volume
+
+`deploy_images.sh` - Proof of concept script for image deployment
+* Dependencies
+  * `docker` - The docker CLI tool
+  * `registry:package.tar.gz` - Modified `registry:2` container loaded with airgap images
+* Deliverables
+  * Running `registry` container with airgap images deployed and retrievable
+
+Hack commands:
+
+* `curl -sX GET http://localhost:5000/v2/_catalog | jq -r .`
+  * Verify the catalog of a local running registry container
+
+# Repository Packaging / Deployment
+
+Airgap Deployment is a form of deployment which does not have any direct connection to the Internet or external network during cluster setup or runtime. During installation, bigbang requires certain images and git repos for installation. Since we will be installing in internet-disconnected environment, we need to perform extra steps to make sure these resources are available.
+
+## Requirements and Prerequisites
+
+### General Prereqs
+- A kubernetes cluster with container mirroring support. There is a section below that covers mirroring in more detail with examples for supported clusters.
+- BigBang(BB) [release artifacts](https://repo1.dso.mil/platform-one/big-bang/bigbang/-/releases).
+- Utility Server.
+
+### Package Specific Prereqs
+
+#### Elastic (Logging)
+
+Elastic requires a larger number of memory map areas than some OSes support by default. This can be change at startup with a cloud config or later using sysctl.
+
+```
+MIME-Version: 1.0
+    Content-Type: multipart/mixed; boundary="==MYBOUNDARY=="
+    
+    --==MYBOUNDARY==
+    Content-Type: text/x-shellscript; charset="us-ascii"
+
+    #!/bin/bash
+    # Set the vm.max_map_count to 262144. 
+    # Required for Elastic to run correctly without OOM errors.
+    sysctl -w vm.max_map_count=262144
+```
+
+
+
+## Utility Server
+
+Utility Server is an internet-disconected server that will host the private registry and git server that are required to deploy bigbang. It should include these commandline tools below;
+
+- `docker`: for running docker registry.
+  - `registry:2` image
+  - `openssl` for self-signed certificate.
+- `curl`: For troubleshooting registry.
+- `git`: for setup git server.
+
+
+
+## Git Server
+
+As part of  BB release, we provide `repositories.tar.gz` which contains all the git repositories that BB depend on for deployment. 
+
+### Process
+
+You can follow the process below to setup git with `repositories.tar.gz` on the Utility Server.
+
+- Create Git user and SSH key
+
+```bash
+$ sudo useradd --create-home --shell /bin/bash git
+$ ssh-keygen -f ~/.ssh/identity
+```
+
+- Create .SSH folder for `git` user
+
+  ```bash
+  $ sudo su - git
+  $ mkdir -p .ssh && chmod 700 .ssh/
+  $ touch ~/.ssh/authorized_keys && chmod 600 ~/.ssh/authorized_keys
+  $ exit
+  ```
+  
+- Add client ssh key to `git` user `authorized_keys`
+
+  ```bash
+  $ sudo su
+  $ cat /[client-public-key-path]/identity.pub >> /home/git/.ssh/authorized_keys
+  $ exit
+  ```
+
+- Extract `repositories.tar.gz` to git user home directory
+
+  ```bash
+  $ sudo tar -xvf repositories.tar.gz --directory /home/git/
+  ```
+
+- To test the client key;
+
+  ```bash
+  GIT_SSH_COMMAND='ssh -i /[client-private-key-path] -o IdentitiesOnly=yes' git clone git@[hostname/IP]:/home/git/repos/[sample-repo]
+  
+  #For example;
+  PRIVATEIP=$( curl http://169.254.169.254/latest/meta-data/local-ipv4 )
+  GIT_SSH_COMMAND='ssh -i ~/.ssh/identity -o IdentitiesOnly=yes' git clone git@$PRIVATEIP:/home/git/repos/istio-controlplane
+  ```
+
+  
+
+## Private Registry 
+
+Images needed to run BB in your cluster is packaged as part of the release in `images.tar.gz`. You can see the list of required images in `images.txt`. In our airgap environment, we need to setup a registry that our cluster can pull required images from or an existing cluster where we can copy images from `images.tar.gz` into.
+
+### Set Up
+
+To setup the registry, we will be using `registry:2` to run a  private registry with  self-signed certificate.
+
+First, untar `images.tar.gz`;
+
+```bash
+tar -xf images.tar.gz -C .
+```
+
+Use the script [registry.sh](./scripts/registry.sh) to create registry;
+
+```bash
+$ sudo ./registry.sh
+
+Required information:
+Enter bit size for certs (Ex. 4096): 4096
+Enter number of days to sign the certs with (Ex. 3650): 3650
+Enter the 'Country' for the cert (Ex. US): US
+Enter the 'State' for the cert (Ex. CO): CO
+Enter the 'Location' for the cert (Ex. ColoradoSprings): ColoradoSprings
+Enter the 'Organization' for the cert (Ex. PlatformOne): PlatformOne
+Enter the 'Organizational Unit' for the cert (Ex. Bigbang): BigBang
+Enter the 'Common Name' for the cert (Must be a FQDN (at least one period character) E.g. myregistry.com): myregistry.com
+Enter the 'Subject Alternative Name' for the cert(E.g. 1.2.3.4): 10.0.52.144
+
+Generating certs ...
+mkdir: cannot create directory ‘certs’: File exists
+Generating RSA private key, 4096 bit long modulus
+.............................................................................................................++
+.....................................++
+e is 65537 (0x10001)
+Generating RSA private key, 4096 bit long modulus
+......................................................................................................................++
+.......................++
+e is 65537 (0x10001)
+Signature ok
+subject=/C=US/ST=CO/L=ColoradoSprings/O=PlatformOne/CN=myregistry.com
+Getting CA Private Key
+
+Launching our private registry ...
+def21e7025c7d4ea7bbb30603955e0b7da14d077592851b327e59d78a849cb7d
+
+Installation finished ...
+
+Notes
+=====
+
+To see images in the registry;
+
+=========================
+curl https://myregistry.com:5443/v2/_catalog -k
+=========================
+
+```
+
+A folder is created with TLS certs that we are going to supply to our k8s cluster when pulling from the registry.
+
+You can ensure the images are now loaded in the registry;
+
+```bash
+ curl -k https://10.0.52.144:5443/v2/_catalog 
+{"repositories":["ironbank/anchore/engine/engine","ironbank/anchore/enterprise/enterprise","ironbank/anchore/enterpriseui/enterpriseui","ironbank/big-bang/argocd","ironbank/bitnami/analytics/redis-exporter","ironbank/elastic/eck-operator/eck-operator","ironbank/elastic/elasticsearch/elasticsearch","ironbank/elastic/kibana/kibana","ironbank/fluxcd/helm-controller","ironbank/fluxcd/kustomize-controller","ironbank/fluxcd/notification-controller","ironbank/fluxcd/source-controller","ironbank/gitlab/gitlab/alpine-certificates","ironbank/gitlab/gitlab/cfssl-self-sign","ironbank/gitlab/gitlab/gitaly",...]
+```
+
+
+
+### Mirroring
+
+ 			The images specified as part of the helm charts in BB are expected to be sourced from `registry1.dso.mil` hence this registry needs to be mirrored to the one setup above. To reduce the amount of work needed on the developer part, we will be taking advantage of container mirroring which is supported by `containerd` as well as `cri-o`. Check if your container runtime supports this as it is required for smooth developer experience when deploying BB.  You should also check documentation on how your cluster supports passing these configuration to the runtime. For example, TKG and RKE2 support such configuration for `containerd` below to enable `registry.dso.mil` and `registry1.dso.mil` . 
+
+​			You need to also configure your cluster with appropriate registry TLS. Please consult your cluster documentation on how to configure this.
+
+If you need to handle mirroring manually, there is an example Ansible script provided that will update the containerd mirroring and restart the container runtimes for each node in your inventory. (copy-containerd-config.yaml)
+
+#### Konvoy Cluster
+
+Modify the `cluster.yaml` file and apply. More details can be found on the [D2iQ Konvoy documentation](https://docs.d2iq.com/dkp/konvoy/1.6/install/install-airgapped/).
+
+```yaml
+kind: ClusterConfiguration
+apiVersion: konvoy.mesosphere.io/v1beta2
+spec:
+  imageRegistries:
+    - server: https://registry1.dso.mil:443
+      username: "myuser"
+      password: "mypassword"
+      default: true
+    - server: https://registry.dso.mil:443
+      username: "myuser"
+      password: "mypassword"
+      default: true
+```
+
+#### TKG Cluster
+
+```yaml
+...
+      - path: /etc/containerd/config.toml
+        content: |
+          version = 2
+          [plugins]
+            [plugins."io.containerd.grpc.v1.cri"]
+              sandbox_image = "registry.tkg.vmware.run/pause:3.2"
+              [plugins."io.containerd.grpc.v1.cri".registry]
+                [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
+                  [plugins."io.containerd.grpc.v1.cri".registry.mirrors."registry1.dso.mil"]
+                    endpoint = ["https://myregistry.com:5443"]
+                  [plugins."io.containerd.grpc.v1.cri".registry.mirrors."registry.dso.mil"]
+                    endpoint = ["https://myregistry.com:5443"]
+ ...
+```
+
+#### RKE2 cluster
+
+```yaml
+mirrors:
+  registry.dso.mil:
+    endpoint:
+      - https://myregistry.com:5443
+  registry1.dso.mil:
+    endpoint:
+      - https://myregistry.com:5443
+configs:
+  myregistry.com:5443:
+    tls:
+      ca_file: "/etc/ssl/certs/registry1.pem"
+```
+
+
+
+## Installing Big Bang
+
+Untar bigbag
+
+```bash
+$ tar -xf bigbang-[version].tar.gz -C .
+$ cd bigbang
+```
+
+Install flux
+
+Install Flux 2 into the cluster using the provided artifacts. These are located in the scripts section of the Big Bang repository.
+
+    kubectl apply -f ./scripts/deploy/flux.yaml
+
+
+After Flux is up and running you are ready to deploy Big Bang. We will do this using Helm. To first check to see if Flux is ready you can do.
+
+    kubectl get all -n flux-system
+
+We need a namespace for our preparations and eventually for Big Bang to deploy into.
+
+    kubectl create ns bigbang
+
+Installing Big Bang in an air gap environment currently uses the Helm charts from the **[Big Bang Repo](https://repo1.dso.mil/platform-one/big-bang/bigbang)**.
+
+All changes are modified in the custom [values.yaml](./examples/values.yaml) file. Modify as needed and replace IP.
+
+Change the hostname for the installation. It is currently set to the development domain:
+
+    # -- Domain used for BigBang created exposed services, can be overridden by individual packages.
+    hostname: bigbang.dev
+
+Add your registry URL. This will be the IP address or URL of the utility server or the registry in which you have loaded all of the Big Bang images (note: it is possible that your registry doesn't have a username or password, there will be ignored for insecure registries.):
+
+    # -- Single set of registry credentials used to pull all images deployed by BigBang.
+    registryCredentials:
+      registry: 10.0.52.144
+      username: "asdfasdfasdf"
+      password: "asdfasdfasdfasdfasdf"
+      email: ""
+
+For your Git repository you have two options for setting up the credentials.
+
+Option 1: Use an existing secret.
+
+    cd ~/.ssh
+    ssh-keygen -q -N "" -f ./identity
+    ssh-keyscan <YOUR GIT URL HERE> ./known_hosts
+    
+    kubectl create secret generic -n bigbang ssh-credentials \
+        --from-file=./identity \
+        --from-file=./identity.pub \
+        --from-file=./known_hosts
+
+In the above example we created a new set of keys to use, you could also use an existing set of keys. These are just SSH keys, so any SSH key pair should work. The second command is going to create a known hosts file. There is no way to answer yes to the unknown hosts prompt, this alleviates that neeed. 
+
+Once we have our private key, public key and the known hosts file, we place all of those into the secret using kubectl. This creates a BASE64 encoded secret of these values. !!! It is VERY important that the names of the files match above. So if you are using your own keypair change the names. Kubernetes uses the names of the files to create the keys inside of the secret.
+
+If you want to create your secret and store in the Kubernetes format you can add the -o yaml --dry-run to the above command to get that output.
+
+    kubectl create secret generic ssh-credentials \
+        --from-file=./identity \
+        --from-file=./identity.pub \
+        --from-file=./known_hosts \
+        -o yaml --dry-run
+
+Once your secret is created you can add that value to the values.yaml that we were modifing above.
+
+    git:
+      # -- Existing secret to use for git credentials, must be in the appropriate format: https://toolkit.fluxcd.io/components/source/gitrepositories/#https-authentication
+      existingSecret: "ssh-credentials"
+
+** Note that we substituted the name of the secret from the example to the secret created above. This value is arbitrary, so if you created your secret with a different name use that name instead.
+
+Option 2: Put the values of your ssh keys directly in the values.yaml file.
+
+You can also elect to just put the key values and the known hosts directly into the chart's values.yaml file.
+
+
+    ssh-keygen -q -N "" -f ./identity
+    ssh-keyscan <YOUR GIT URL HERE> ./known_hosts
+    
+    cat identity
+    cat identity.pub
+    cat known_hosts
+
+Take the values from each of these files and place in the correct fields in the values.yaml.
+
+    git:
+        # -- SSH git credentials, privateKey, publicKey, and knownHosts must be provided
+        privateKey: |
+          -----BEGIN RSA PRIVATE KEY-----
+          MIIEowIBAAKCAQEAwcG6YKsqDC6728XZ7/8oiqnQaw3OkQnvMBrzvZjxd//PsEog
+          xVc+F9YqW4FIeTH57wN6JXIC4iMbE0QGd6+1yOoYiXkhi66tuO5FN+n4PeMnvKcC
+          JXtFWme4W/9YnEk/3sbNOgAMPlhMhTsudzLiXtHd3g+xCmNs1pdEIInaNadrolWn
+          QTM0krUCcC6VLCri7ae/pDloglX4cBJ+EfqFC94T6wUICPd1P7zYsy8WwIQtPhLT
+          lbY8CHj9iMlxlUdwdiXTlifqHsPgTh3X5e9Vptd+wi0+vfjvrXd/8SuM1q8xdQvY
+          bZ27AlhgfQsVl9WQrk/47xd3g430G4cqSbyhLQIDAQABAoIBAFlSu153akIFhXtz
+          Ad7fbcxHLxs7WUCKKOevdTCyApgEqbWm5uazKqAIjqxytHuS65shqjz7C5M/Beti
+          z+x7Z73BFiDCZBgmLNZ1mhmF1niJcTdKcvXel4FvEZHv7OTX7AcC9XfIr9xKDrTZ
+          LLmtDqkR7UvDRiX44iMnxzOM+bkDsHVva00e3IoSiOsQ4DKQ1l/HFseVlPIaGzfZ
+          Z2q0myUrBzlOYE06VJluhexsrrVDi7KdIfR8UGpN4kC5R/vOnOi7ycd4tfsZe2Wb
+          CjbKMTNYRFnVTt6/SXAhhFu+kz0FftDXNTIOhikVB8ryZ5iyNXszYqiptUI9VUZB
+          mQLdPuECgYEA9odVxlPUgSMLhbE5vD57jbtB6Cswy5ztAuyCHMABM4U6pVvFDSNb
+          244y0ov0TzviaCZkb+0qrAM0ZSNItLQ1PmbeD0SnB4q/C8hDvVtpB+0SPBJMX8so
+          49n1Wr5dH0axGMLaZXGmQ4DPEW/t0dNbYpN1Sxgn6KZPprISXigBufkCgYEAyTNe
+          kY3vaJ6Nla1pBVUmiK7hu1G3Ddihy1w56upHbOnDvJySuVOM5HRPm2ISFwW38/b5
+          5+cGKWnmu7UhFi1d8Iz3Kmr6kpfRxEDtbrk5rkgKJmTtduxAzBH8CTZfxuYIC5xS
+          3fbcFpFYfrtE+3tjqlXJSOpLOuDqbA3uGwWFTdUCgYEAkSi9A8uGnAdDmJPzF/l+
+          jMTPGOKdl7auBAO41S7lRi3Ti1xO2d6RDuVa3YiU8TakqIi6qQDwGFrGtiqhe+2E
+          UFsHs9vLsfArb8eaw1uYq5c7HpHzsJASYp+LDcR7VpgsXRUWvZa+vI6S3oSWdu9J
+          pvCGpxHxJdcPnWrKz/AknBkCgYAnej/U+W9/LJUFSFgx5qo/6Wh7M6ZiPh5I45it
+          ojhPg3KXgHU9jco4TSYNi+mWwNV+NfiE6wyHdbMDI6ARVOd4uoAIv6M9NDLBeifc
+          MNXDf3kWXXlGe0afg+va9uNGCH6NoKeVy8kVWIFvpFj9qxE8K8bp2qbWL6lveDA+
+          9w9X3QKBgGtkQi9OI7TyrloZ5F6/0/LnOJMGd/+e2cJUN6Pa10ZAjQh12JZ5fK7i
+          Vwh5l0P5CGQsuC96n4xPELoBnbTdr+y17f0o+kAuSDAsXnDf/Jjr0y/+uzL6YYCg
+          VD1yNitgcQw6oHKdTbGn4jni3/VemzONOz0uTB+/K7WhW2J7faaJ
+          -----END RSA PRIVATE KEY-----
+        publicKey: "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwbpgqyoMLrvbxdnv/yiKqdBrDc6RCe8wGvO9mPF3/+wSiDFVz4X1ipbgUh3MfnvA2olcgLiIxsTRAZ8r7XI6hiJeSGLrq2123kU36fg94ye8pwIle0VaZ7hb/1icST/exs06AAw+WEyFOy53MuJe0d3e$"
+        knownHosts: "10.0.52.144 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPFZzQ6BmaswdhT8UWD5a/VYmZYrGv1qD3T+euf/gFjkPkeySYRIyM+Kg/UdHCHVBzc4aaFdBDmugHimZ4lbWpE="
+
+** Note the above values are all examples and are intentionally not operational keys.
+
+Then install Big Bang using Helm.
+
+    helm upgrade -i bigbang chart -n bigbang --create-namespace -f values.yaml
+
+** Note that the --create-namespace isn't needed if you created it earlier, but it doesn't hurt anything.
+
+You can watch to see if Flux is reconciling the projects by watching the progress.
+
+     watch kubectl get gitrepositories,kustomizations,hr,po -A
+
+You should see the diffent projects configure working through their reconciliation starting with "gatekeeper".
+

docs/airgap/developer/developer.md

+# K3D
+
+To test Airgap BigBang on k3d
+
+## Steps
+
+- Launch ec2 instance of size `c5.2xlarge` and ssh into the instance with at least 50GB storage.
+
+- Install [Docker](https://docs.docker.com/engine/install/ubuntu/)
+
+- Install [K3D](https://k3d.io/#installation)
+
+- Download `images.tar.gz`, `repositories.tar.gz` and `bigbang-version.tar.gz` from BigBang release.
+
+  ```bash
+  $ curl -O https://umbrella-bigbang-releases.s3-us-gov-west-1.amazonaws.com/umbrella/1.2.0/repositories.tar.gz
+  $ curl -O https://umbrella-bigbang-releases.s3-us-gov-west-1.amazonaws.com/umbrella/1.2.0/images.tar.gz
+  $ curl -O https://repo1.dso.mil/platform-one/big-bang/bigbang/-/archive/1.2.0/bigbang-1.2.0.tar.gz
+  ```
+
+  
+
+- Follow [Airgap Documentation](../README.md) to install Git server and Registry.
+
+- Once Git Server and Registry is up, setup k3d mirroring configuration  `registries.yaml`
+
+  ```yaml
+  mirrors:
+    registry.dso.mil:
+      endpoint:
+        - https://[$PRIVATEIP]:5443
+    registry1.dso.mil:
+      endpoint:
+        - https://[$PRIVATEIP]:5443
+    docker.io:
+      endpoint:
+        - https://[$PRIVATEIP]:5443
+  configs:
+    myregistry.com:5443:
+      tls:
+        ca_file: "/etc/ssl/certs/registry1.pem"
+  ```
+
+  
+
+- Launch k3d cluster
+
+  ```bash
+  $ PRIVATEIP=$( curl http://169.254.169.254/latest/meta-data/local-ipv4 )
+  $ k3d cluster create --api-port "${PRIVATEIP}:33989" -s 1 -a 2 -v "${HOME}/registries.yaml:/etc/rancher/k3s/registries.yaml" -v /etc/machine-id:/etc/machine-id -v "${HOME}/certs/myregistry.com.public.pem:/etc/ssl/certs/registry1.pem" --k3s-server-arg "--disable=traefik" --k3s-server-arg "--disable=metrics-server" --k3s-server-arg "--tls-san=$PRIVATEIP"  -p 80:80@loadbalancer -p 443:443@loadbalancer
+  ```
+
+  
+
+- Bock all egress with `iptables` except those going to instance IP before deploying bigbang  by running [k3d_airgap.sh](./scripts/k3d_airgap.sh)
+
+  ```bash
+  
+  ```
+
+  
+
+  ```bash
+  $ sudo ./k3d_airgap.sh
+  $ curl https://index.docker.io/ #shouldnt work
+  $ curl https://$PRIVATEIP:5443/v2/_catalog -k #show return list of images
+  curl https://$PRIVATEIP:5443/v2/repositories/rancher/library-busybox/tags
+  ```
+
+​		To permanently save the iptable rules across reboot, check out [link](https://unix.stackexchange.com/questions/52376/why-do-iptables-rules-disappear-when-restarting-my-debian-system)
+
+- Test that  mirroring is working
+
+```bash
+$ kubectl run -i --tty  test --image=registry1.dso.mil/rancher/library-busybox:1.31.1 --image-pull-policy='Always' -- sh
+$ kubectl run test --image=registry1.dso.mil/rancher/library-busybox:1.31.1 --image-pull-policy='Always' --restart=Never --command sleep infinity
+$ telnet default.kube-system.svc.cluster.local 443
+$ kubectl describe po test
+$ kubectl delete po test
+```
+
+- Test that cluster cannot pull outside private registry.
+
+```bash
+$ kubectl run test --image=nginx
+$ kubectl describe po test #should fail
+$ kubectl delete po test
+```
+
+
+
+- 
+- Proceed to [bigbang deployment process](../README.md#installing-big-bang) 
\ No newline at end of file

docs/airgap/developer/pipeline.md

+### Current Pipeline Outline and Notes
+
+<ol>
+<li><h4>.pre</h4>
+
+  <ol>
+    <li>
+      <h4><b>changelog</b></h4> Does a diff to lint what has changed for the logs
+    </li>
+    <li><h4><b>commits</b></h4> enforces the conventional commits stuff
+    </li>
+    <li>
+      <h4><b>pre vars</b></h4>
+      pre checks
+    </li>
+    <li>
+      <h4><b>version</b></h4>
+      gets various versions to build a complex version number for the build
+    </li>
+  </ol>
+</li>
+
+<li><h4><b>smoke tests</b></h4>
+  <ol>
+    <li><h4><b>clean install</b></h4>
+      Doesn't really effect airgap, this sets up things like cluster names and such
+    </li>
+    <li><h4><b>upgrade</b></h4>
+      Splits out testing and determines if there are breaking changes for testing of upgrades.
+    </li>
+  </ol>
+</li>
+
+<li><h4><b>network up</b></h4>
+  <ol>
+    <li><h4><b>airgap/network up</b></h4>
+      Creates a VPC and subnets for the cluster to be deployed in.
+    </li>
+    <li><h4><b>aws/airgap/package</b></h4>
+      Packages everything needed for the airgap install into a tar file. This leaves the repositories and images bundled in the Releases section for BB (https://repo1.dso.mil/platform-one/big-bang/bigbang/-/releases)
+    </li>
+  </ol>
+</li>
+
+<li><h4><b>airgap up</b></h4>
+  <ol>
+    <li><h4><b>aws/airgap/utility up</b></h4>
+      Sets up proxies using Route 53 to essentially fake out where Repo 1 and Registry 1 exist for the purposes of using an air gap registry and git repo.
+    </li>
+  </ol>
+</li>
+
+<li><h4><b>cluster up</b></h4>
+  <ol>
+    <li><h4><b>airgap/rke2/cluster up</b></h4>
+      Stands up an RKE2 cluster for BB in an airgapped network. ** Uses terraform ./gitlab-ci/jobs/rke2/dependencies/terraform/
+
+      Both this and the non-airgapped use the same image registry.dso.mil/platform-one/big-bang/pipeline-templates/pipeline-templates/k3d-builder:0.0.1
+    </li>
+  </ol>
+</li>
+
+<li><h4><b>bigbang up</b></h4>
+  <ol>
+    <li><h4><b>airgap/rke2/bigbang up</b></h4>
+      Stands up big bang
+    </li>
+  </ol>
+</li>
+
+<li><h4><b>test</b></h4>
+  <ol>
+    <li><h4><b>airgap/rke2/bigbang test</b></h4>
+      Runs some basic tests to make sure that Big Bang is up and working.
+    </li>
+  </ol>
+</li>
+
+<li><h4><b>bigbang down</b></h4>
+  <ol>
+    <li><h4><b>airgap/rke2/bigbang down</b></h4>
+      Tears down the Big Bang instance
+    </li>
+  </ol>
+</li>
+
+<li><h4><b>cluster down</b></h4>
+  <ol>
+    <li><h4><b>airgap/rke2/cluster down</b></h4></li>
+  </ol>
+</li>
+
+<li><h4><b>airgap down</b></h4>
+  <ol>
+    <li><h4><b>aws/airgap/package delete</b></h4></li>
+    <li><h4><b>aws/airgap/utility down</b></h4></li>
+  </ol>
+</li>
+
+<li><h4><b>network down</b></h4>
+  <ol>
+    <li><h4><b>airgap/network down</b></h4></li>
+  </ol>
+</li>
+</ol>
+

docs/airgap/scripts/README.md

+### The directory contains all the utility scripts and examples.
+

docs/airgap/scripts/airgap.sh

+#!/bin/sh
+
+PUBLICINTERFACE=$( route | grep '^default' | grep -o '[^ ]*$' )
+iptables -I DOCKER-USER -i ${PUBLICINTERFACE} -j DROP
+iptables -I DOCKER-USER -d 10.42.0.0/16 -j RETURN
+iptables -I DOCKER-USER -d 10.43.0.0/16 -j RETURN

docs/airgap/scripts/config.toml.tmpl

+version = 2
+root = "/var/lib/containerd"
+state = "/run/containerd"
+plugin_dir = ""
+disabled_plugins = []
+required_plugins = []
+oom_score = 0
+
+[grpc]
+  address = "/run/containerd/containerd.sock"
+  tcp_address = ""
+  tcp_tls_cert = ""
+  tcp_tls_key = ""
+  uid = 0
+  gid = 0
+  max_recv_message_size = 16777216
+  max_send_message_size = 16777216
+
+[ttrpc]
+  address = ""
+  uid = 0
+  gid = 0
+
+[debug]
+  address = ""
+  uid = 0
+  gid = 0
+  level = ""
+
+[metrics]
+  address = "{{ansible_host}}:1338"
+  grpc_histogram = false
+
+[cgroup]
+  path = ""
+
+[timeouts]
+  "io.containerd.timeout.shim.cleanup" = "5s"
+  "io.containerd.timeout.shim.load" = "5s"
+  "io.containerd.timeout.shim.shutdown" = "3s"
+  "io.containerd.timeout.task.state" = "2s"
+
+[plugins]
+  [plugins."io.containerd.gc.v1.scheduler"]
+    pause_threshold = 0.02
+    deletion_threshold = 0
+    mutation_threshold = 100
+    schedule_delay = "0s"
+    startup_delay = "100ms"
+  [plugins."io.containerd.grpc.v1.cri"]
+    disable_tcp_service = true
+    stream_server_address = "127.0.0.1"
+    stream_server_port = "0"
+    stream_idle_timeout = "4h0m0s"
+    enable_selinux = false
+    sandbox_image = "prodmicroservicesregistry.azurecr.io/k8s.gcr.io/pause:3.1"
+    stats_collect_period = 10
+    systemd_cgroup = false
+    enable_tls_streaming = false
+    max_container_log_line_size = 16384
+    disable_cgroup = false
+    disable_apparmor = false
+    restrict_oom_score_adj = false
+    max_concurrent_downloads = 3
+    disable_proc_mount = false
+    [plugins."io.containerd.grpc.v1.cri".containerd]
+      snapshotter = "overlayfs"
+      default_runtime_name = "runc"
+      no_pivot = false
+      [plugins."io.containerd.grpc.v1.cri".containerd.default_runtime]
+        runtime_type = ""
+        runtime_engine = ""
+        runtime_root = ""
+        privileged_without_host_devices = false
+      [plugins."io.containerd.grpc.v1.cri".containerd.untrusted_workload_runtime]
+        runtime_type = ""
+        runtime_engine = ""
+        runtime_root = ""
+        privileged_without_host_devices = false
+      [plugins."io.containerd.grpc.v1.cri".containerd.runtimes]
+        [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
+          runtime_type = "io.containerd.runc.v1"
+          runtime_engine = ""
+          runtime_root = ""
+          privileged_without_host_devices = false
+        [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia-container-runtime]
+          runtime_type = "io.containerd.runc.v1"
+          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia-container-runtime.options]
+            BinaryName = "/usr/bin/nvidia-container-runtime"
+    [plugins."io.containerd.grpc.v1.cri".cni]
+      bin_dir = "/opt/cni/bin"
+      conf_dir = "/etc/cni/net.d"
+      max_conf_num = 1
+      conf_template = ""
+    [plugins."io.containerd.grpc.v1.cri".registry]
+      [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
+        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.elastic.co"]
+          endpoint = ["{{ registry_endpoint }}"]
+        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
+          endpoint = ["{{ registry_endpoint }}","https://registry-1.docker.io"]
+        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."gcr.io"]
+          endpoint = ["{{ registry_endpoint }}"]
+        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."quay.io"]
+          endpoint = ["{{ registry_endpoint }}"]
+      [plugins."io.containerd.grpc.v1.cri".registry.configs]
+        [plugins."io.containerd.grpc.v1.cri".registry.configs."{{registry_server}}".auth]
+            username = "{{ registry_user }}"
+            password = "{{ registry_password }}"
+            auth = ""
+            identitytoken = ""
+    [plugins."io.containerd.grpc.v1.cri".x509_key_pair_streaming]
+      tls_cert_file = ""
+      tls_key_file = ""
+  [plugins."io.containerd.internal.v1.opt"]
+    path = "/opt/containerd"
+  [plugins."io.containerd.internal.v1.restart"]
+    interval = "10s"
+  [plugins."io.containerd.metadata.v1.bolt"]
+    content_sharing_policy = "shared"
+  [plugins."io.containerd.monitor.v1.cgroups"]
+    no_prometheus = false
+  [plugins."io.containerd.runtime.v1.linux"]
+    shim = "containerd-shim"
+    runtime = "runc"
+    runtime_root = ""
+    no_shim = false
+    shim_debug = false
+  [plugins."io.containerd.runtime.v2.task"]
+    platforms = ["linux/amd64"]
+  [plugins."io.containerd.service.v1.diff-service"]
+    default = ["walking"]
+  [plugins."io.containerd.snapshotter.v1.devmapper"]
+    root_path = ""
+    pool_name = ""
+    base_image_size = ""

docs/airgap/scripts/copy-containerd-config.yaml

+---
+- hosts: all
+  gather_facts: no
+  become: yes
+  vars:
+    registry_endpoint: http://localregistry/v2/
+    registry_server: localregistry
+    registry_user: someuser
+  vars_prompt:
+    - name: registry_password
+      prompt: Enter registry password
+      private: yes
+  tasks:
+  - name: Copy containerd config
+    template:
+      src: config.toml.tmpl
+      dest: /etc/containerd/config.toml
+      owner: root
+      group: root
+  - name: Restart containerd
+    service:
+      name: containerd
+      state: restarted
\ No newline at end of file

docs/airgap/scripts/deploy-images.sh

+#!/usr/bin/env bash
+
+set -e
+
+REGISTRY_PACKAGE_IMAGE="registry:package"
+REGISTRY_PACKAGE_TGZ="${REGISTRY_PACKAGE_IMAGE}.tar.gz"
+
+function purge_registry_containers {
+  echo "Stopping local registry containers"
+  docker stop registry &>/dev/null || true
+  docker rm registry &>/dev/null || true
+}
+
+function purge_registry_images {
+  echo "Removing local registry images"
+  docker image rm ${REGISTRY_PACKAGE_IMAGE} &>/dev/null || true
+}
+
+purge_registry_containers
+purge_registry_images
+
+echo "Loading local registry package tgz"
+docker load < ${REGISTRY_PACKAGE_TGZ}
+
+echo "Creating package registry container"
+docker run -d -p 5000:5000 --name registry ${REGISTRY_PACKAGE_IMAGE} >/dev/null
+
+echo "Showing package container registry catalog"
+sleep 1; curl -sX GET http://localhost:5000/v2/_catalog
\ No newline at end of file

docs/airgap/scripts/images.txt

+docker.io/library/busybox:1.32
+docker.io/rancher/coredns-coredns:1.8.0
+docker.io/rancher/klipper-lb:v0.1.2
+docker.io/rancher/library-busybox:1.31.1

docs/airgap/scripts/package_images.sh

+#!/usr/bin/env bash
+
+set -e
+
+IMAGES_TXT="images.txt"
+REGISTRY_IMAGE="registry:2"
+REGISTRY_PACKAGE_IMAGE="registry:package"
+REGISTRY_PACKAGE_TGZ="${REGISTRY_PACKAGE_IMAGE}.tar.gz"
+
+# $1 = image_original - original full image url from existing repository
+function get_image_sections {
+  image_full=$(echo ${1} | sed -n 's/^.*\/\(.*:.*\)$/\1/p')
+  image_base=$(echo ${image_full} | sed -n 's/\(^.*\):\(.*$\)/\1/p')
+  image_tag=$(echo ${image_full} | sed -n 's/\(^.*\):\(.*$\)/\2/p')
+  # [ -z "${image_full}" ] && { echo "Error: Unable to set image full variable"; exit 1; }
+  # [ -z "${image_base}" ] && { echo "Error: Unable to set image base variable"; exit 1; }
+  # [ -z "${image_tag}" ] && { echo "Error: Unable to set image tag variable"; exit 1; }
+}
+
+# $1 = image_base - image name and tag (nginx:latest)
+# $1 = image_tag - image tag only (latest)
+function verify_catalog_image {
+  echo "Verifying \"${1}\" exists in registry catalog with tag \"${2}\""
+  reg_tag=$(curl -sX GET http://localhost:5000/v2/${1}/tags/list | jq -r '.tags | .[0]')
+  if [ "${2}" != "${reg_tag}" ]; then
+    echo "Error: Unable to verify ${1} exists in catalog"
+    exit 1
+  fi
+}
+
+function purge_registry_containers {
+  echo "Stopping local registry containers"
+  docker stop registry &>/dev/null || true
+  docker rm registry &>/dev/null || true
+}
+
+function purge_registry_images {
+  echo "Removing local registry images"
+  docker image rm ${REGISTRY_IMAGE} &>/dev/null || true
+  docker image rm ${REGISTRY_PACKAGE_IMAGE} &>/dev/null || true
+}
+
+echo "Removing local registry package tgz"
+rm -rf ${REGISTRY_PACKAGE_TGZ}
+
+purge_registry_containers
+purge_registry_images
+
+echo "Creating initial registry container"
+docker run -d -p 5000:5000 --name registry ${REGISTRY_IMAGE} &>/dev/null
+
+echo "--"
+for image_original in $(sed '/^$/d' ${IMAGES_TXT}); do
+  get_image_sections ${image_original}
+  echo "Referencing \"${image_original}\""
+  echo "Uploading to registry as \"${image_base}\" with tag \"${image_tag}\""
+  docker pull ${image_original} >/dev/null
+  docker tag ${image_original} localhost:5000/${image_full} >/dev/null
+  docker push localhost:5000/${image_full} >/dev/null
+  verify_catalog_image ${image_base} ${image_tag}
+  echo "--"
+done
+
+# TODO - is a pass-through proxy needed? - https://docs.docker.com/registry/recipes/mirror/
+echo "Creating persistent package inside registry container"
+docker cp registry-config.yml registry:/etc/docker/registry/config.yml >/dev/null
+docker exec -it registry cp -r /var/lib/registry/ /var/lib/registry-package >/dev/null
+
+echo "Commiting initial registry image to package registry image"
+docker commit registry ${REGISTRY_PACKAGE_IMAGE} >/dev/null
+
+purge_registry_containers
+
+echo "Creating package registry container"
+docker run -d -p 5000:5000 --name registry ${REGISTRY_PACKAGE_IMAGE} &>/dev/null
+
+echo "--"
+for image_original in $(sed '/^$/d' ${IMAGES_TXT}); do
+  get_image_sections ${image_original}
+  verify_catalog_image ${image_base} ${image_tag}
+done
+echo "--"
+
+purge_registry_containers
+
+echo "Saving local registry package tgz"
+docker save ${REGISTRY_PACKAGE_IMAGE} | gzip --stdout > ${REGISTRY_PACKAGE_TGZ}
+
+purge_registry_images
\ No newline at end of file

docs/airgap/scripts/package_repos.sh

+#!/usr/bin/env bash
+
+set -x
+
+GITEA_IMAGE="gitea/gitea:1.13.2"
+
+GITEA_HTTP_METHOD="http"
+GITEA_URL="localhost:3000"
+GITEA_USERNAME="admin"
+GITEA_PASSWORD="password"
+
+curl -X POST "${GITEA_HTTP_METHOD}://${GITEA_USERNAME}:${GITEA_PASSWORD}@${GITEA_URL}/api/v1/user/repos" -H "accept: application/json" -H "content-type: application/json" -d \
+  "{\"name\":\"test-repo\", \"description\": \"Sample description\" }"
+  
\ No newline at end of file

docs/airgap/scripts/registry.sh

+#!/usr/bin/env bash
+
+# Installs/Configures:
+#  - Docker Registy Container with self-signed cert
+#
+# Tested on Ubuntu 14.04.1
+
+# Must be executed with elevated privilages
+if [ "$(id -u)" != "0" ]; then
+  printf "This script must be ran as root or sudo!\n"
+  exit 1
+fi
+
+# prompt helper function
+function prompt () {
+  if [ -z ${!1} ]; then
+    local response=""
+    while [[ ${response} = "" ]]; do
+      read -p "$2: " response
+    done
+    eval $1=${response}
+  fi
+}
+
+# collect required information
+# - C   Country
+# - ST  State
+# - L    Location
+# - O    Organization
+# - OU   Organizational Unit
+# - CN   Common Name
+echo -e "\nRequired information:"
+prompt BITS "Enter bit size for certs (Ex. 4096)"
+prompt DAYS "Enter number of days to sign the certs with (Ex. 3650)"
+prompt COUNTRY "Enter the 'Country' for the cert (Ex. US)"
+prompt STATE "Enter the 'State' for the cert (Ex. CO)"
+prompt LOCATION "Enter the 'Location' for the cert (Ex. ColoradoSprings)"
+prompt ORGANIZATION "Enter the 'Organization' for the cert (Ex. PlatformOne)"
+prompt OUNIT "Enter the 'Organizational Unit' for the cert (Ex. Bigbang)"
+prompt COMMON  "Enter the 'Common Name' for the cert (Must be a FQDN (at least one period character) E.g. myregistry.com)"
+
+PRIVATEIP=$( curl http://169.254.169.254/latest/meta-data/local-ipv4 )
+
+
+# ... Certs ...
+# ~~~~~~~~~~~~~
+
+# ... prep certs ...
+
+echo -e "\nGenerating certs ..."
+mkdir certs
+cd certs
+# Generate a root key
+openssl genrsa -out rootCA.key ${BITS}
+
+# Generate a root certificate
+openssl req -x509 -new -nodes -key rootCA.key -days ${DAYS}\
+    -subj "/C=${COUNTRY}/ST=${STATE}/L=${LOCATION}/O=${ORGANIZATION}/CN=${COMMON}" \
+    -out rootCA.crt
+
+# Generate key for host
+openssl genrsa -out ${COMMON}.key ${BITS}
+
+# Generate CSR
+openssl req -new -key ${COMMON}.key \
+    -subj "/C=${COUNTRY}/ST=${STATE}/L=${LOCATION}/O=${ORGANIZATION}/CN=${COMMON}" \
+    -out ${COMMON}.csr
+
+# Sign certificate request
+echo subjectAltName = IP:${PRIVATEIP} > extfile.cnf
+openssl x509 -req -in ${COMMON}.csr -CA rootCA.crt -CAkey rootCA.key -CAcreateserial -days ${DAYS} \
+-out ${COMMON}.crt -extfile extfile.cnf
+
+openssl rsa -in ${COMMON}.key -text > ${COMMON}.private.pem
+openssl x509 -inform PEM -in ${COMMON}.crt > ${COMMON}.public.pem
+
+mkdir -p /certs/${COMMON}
+cp rootCA.crt /certs/${COMMON}/ca.crt
+
+
+# ... launch registry ...
+# ~~~~~~~~~~~~~~~~~~~~~~~
+
+echo -e "\nLaunching our private registry ..."
+cd ..
+docker run -d -p 5443:5000 --restart=always --name bigbang_registry \
+    -v `pwd`/certs:/certs \
+    -v `pwd`/var/lib/registry:/var/lib/registry \
+    -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/${COMMON}.crt \
+    -e REGISTRY_HTTP_TLS_KEY=/certs/${COMMON}.key \
+    registry:2
+
+
+# Instructions
+echo -e "\nInstallation finished ...
+
+Notes
+=====
+
+To see images in the registry;
+
+=========================
+curl https://${PRIVATEIP}:5443/v2/_catalog -k
+=========================
+
+"
\ No newline at end of file

docs/airgap/scripts/registry_config.yaml

+version: 0.1
+log:
+  fields:
+    service: registry
+storage:
+  cache:
+    blobdescriptor: inmemory
+  filesystem:
+    rootdirectory: /var/lib/registry-package
+http:
+  addr: :5000
+  headers:
+    X-Content-Type-Options: [nosniff]
+health:
+  storagedriver:
+    enabled: true
+    interval: 10s
+    threshold: 3
\ No newline at end of file

docs/airgap/scripts/synker.md

+# Airgap Image Sync
+
+## Prerequisite
+
+- `images.tar.gz` from [Big Bang Releases](https://repo1.dso.mil/platform-one/big-bang/bigbang/-/releases)
+- `registry:2`, directions below to package dependency
+- 40gb disk space
+- docker
+
+## Usage
+
+Package registry to take airgap
+```
+docker save -o registry.tar registry:2
+```
+
+Load registry in airgap
+```
+docker load < registry.tar
+```
+
+Unpack images
+```
+tar -xvf images.tar.gz
+```
+
+Start a local registry based on the images we just unpacked
+```
+cd ./var/lib/registry
+docker run -p 25000:5000 -v $(pwd):/var/lib/registry registry:2
+# verify the registry mounted correctly
+curl http://localhost:25000/v2/_catalog -k
+# a list of Big Bang images should be displayed, if not check the volume mount of the registry
+```
+Configure `./synker.yaml`
+
+Example
+```
+destination:
+  registry:
+    # Hostname of the destination registry to push to
+    hostname: 10.0.0.10
+    # Port of the destination registry to push to
+    port: 5000
+```
+If using Harbor reference the project name
+```
+destination:
+  registry:
+    # Hostname of the destination registry to push to
+    hostname: harbor.domain.com/ironbank
+    # Port of the destination registry to push to
+    port: 443
+```
+If your destination repo requires credentials add them to ` ~/.docker/config.json`
+```
+{
+  "auths": {
+    "registry.dso.mil": {
+      "username": "gitlab -user",
+      "password": "",
+      "auth": "=="
+    },
+    "registry1.dso.mil": {
+      "auth": ""
+    },
+    "harbor.yourdomain.com": {
+      "username": "robot",
+      "password": "",
+      "auth": "base64(username:password)="
+    }
+  }
+}
+``` 
+
+**WARNING:** Verify your credentials with docker login before running synker. If your environment has login lockout after failed attempts synker could trigger a lockout if your credentials are incorrect.
+
+```
+./synker push
+```
+Verify the images were pushed to your registry

docs/airgap/scripts/values.yaml

+# -- Domain used for BigBang created exposed services, can be overridden by individual packages.
+hostname: bigbang.dev
+registryCredentials:
+  registry: 10.25.21.9
+  username: ""
+  password: ""
+  email: ""
+git:
+  # -- Existing secret to use for git credentials, must be in the appropriate format: https://toolkit.fluxcd.io/components/source/gitrepositories/#https-authentication
+  existingSecret: "ssh-credentials"
+
+flux:
+  interval: 1m
+  rollback:
+    cleanupOnFail: false
+
+
+eckoperator:
+  enabled: false
+  git:
+    repo: ssh://git@10.25.21.9/home/git/repos/eck-operator
+
+fluentbit:
+  enabled: false
+  git:
+    repo: ssh://git@10.25.21.9/home/git/repos/fluentbit
+logging:
+  enabled: false
+  git:
+    repo: ssh://git@10.25.21.9/home/git/repos/elasticsearch-kibana
+
+istio:
+  enabled: false
+  git:
+    repo: ssh://git@10.25.21.9/home/git/repos/istio-controlplane
+
+istiooperator:
+  enabled: false
+  git:
+    repo: ssh://git@10.25.21.9/home/git/repos/istio-operator
+
+clusterAuditor:
+  enabled: false
+  git:
+    repo: ssh://git@10.25.21.9/home/git/repos/cluster-auditor
+  values:
+    resources:
+      requests:
+        cpu: 100m
+        memory: 0.5Gi
+
+monitoring:
+  enabled: false
+  git:
+    repo: ssh://git@10.25.21.9/home/git/repos/monitoring
+
+gatekeeper:
+  enabled: true
+  git:
+    repo: ssh://git@10.25.21.9/home/git/repos/policy
+
+twistlock:
+  enabled: false
+  git:
+    repo: ssh://git@10.25.21.9/home/git/repos/twistlock
+
+# Explicitly enable all addons for CI
+addons:
+  argocd:
+    enabled: false
+    git:
+      repo: ssh://git@10.25.21.9/home/git/repos/argocd
+  authservice:
+    enabled: false
+    git:
+      repo: ssh://git@10.25.21.9/home/git/repos/authservice
+    chains:
+      minimal:
+        callback_uri: "https://minimal.bigbang.dev"
+  gitlab:
+    enabled: false
+    git:
+      repo: ssh://git@10.25.21.9/home/git/repos/gitlab
+    sso:
+      enabled: false
+  gitlabRunner:
+    enabled: false
+    git:
+      repo: ssh://git@10.25.21.9/home/git/repos/gitlab-runner
+  anchore:
+    enabled: false
+    git:
+      repo: ssh://git@10.25.21.9/home/git/repos/anchore-enterprise
+  sonarqube:
+    enabled: false
+    git:
+      repo: ssh://git@10.25.21.9/home/git/repos/sonarqube
+  minioOperator:
+    enabled: false
+    git:
+      repo: ssh://git@10.25.21.9/home/git/repos/minio-operator
+  minio:
+    enabled: false
+    git:
+      repo: ssh://git@10.25.21.9/home/git/repos/minio
+  haproxy:
+    enabled: false
+    git:
+      repo: ssh://git@10.25.21.9/home/git/repos/haproxy
+  mattermostoperator:
+    enabled: false
+    git:
+      repo: ssh://git@10.25.21.9/home/git/repos/mattermost-operator
+  mattermost:
+    enabled: false
+    git:
+      repo: ssh://git@10.25.21.9/home/git/repos/mattermost
\ No newline at end of file