Storage Documentation

docs/k8s-storage/README.md

+## Kubernetes Storage Options
+
+Use this data to assist in your CSI decision. However, when using a cloud provider we suggest you use their Kubernetes CSI.
+
+## Feature Matrix
+
+| Product | BB Compatible  | FOSS | In Ironbank  | RWX/RWM Support | Airgap Compatible | Cloud Agnostic |
+| --------- | --------- | --------- | --------- | --------- | --------- | --------- |
+Amazon EBS CSI    | **X** |  N/A  |  | **X** | AWS Dependent | No | 
+Azure Disk CSI    | Not Tested  |  N/A  |  | **X** | Azure Dependent | No | 
+Longhorn v1.1.0   | **X** | **X** |  | **X** | **X** - [Docs](https://longhorn.io/docs/1.1.0/advanced-resources/deploy/airgap/) | Yes, uses host storage | 
+OpenEBS (jiva)    | **X** | **X** |  | **X** **[Alpha](https://docs.openebs.io/docs/next/rwm.html)** | Manual Work Required | Yes, uses host storage |  
+Rook-Ceph         | **X** | **X** |  | **X** | Manual Work Required | Yes, uses host storage | 
+Portworx          | **X** |       |  | **X** | **X** - [Docs](https://docs.portworx.com/portworx-install-with-kubernetes/operate-and-maintain-on-kubernetes/pxcentral-onprem/install/px-central/) | Yes, uses host storage |
+
+## Benchmark Results
+
+Benchmarks were tested on AWS with GP2 ebs volumes using using FIO, see [example](./benchmark.yaml)
+
+| Product | Random Read/Write IOPS | Average Latency (usec) | Sequential Read/Write | Mixed Random Read/Write IOPS |
+| --------- | --------- | --------- | --------- | --------- |
+Amazon EBS CSI  | 2997/2996. BW: 128MiB/s / 128MiB/s | 1331.61 | 129MiB/s / 131MiB/s | 7203/2390
+Azure Disk CSI  |  |  |  | 
+Longhorn v1.1.0 | 6155/1551 BW: 230MiB/s / 96.3MiB/s | 1042.53 | 319MiB/s / 130MiB/s | 3804/1267
+OpenEBS (jiva) | 2183/770. BW: 76.8MiB/s / 45.8MiB/s | 2059.55 | 132MiB/s / 98.2MiB/s | 1590/533
+Rook-Ceph | 10.7k/3205. BW: 503MiB/s / 148MiB/s | 548.36/s | 496MiB/s / 154MiB/s | 6664/2228
+Portworx  2.6 | 3016/19.3k. BW: 74.5MiB/s / 85.1MiB/s | 1337.31 |  113MiB/s / 124MiB/s | 35.1k/11.1k
+
+## Amazon EBS CSI
+
+[Website/Docs](https://docs.aws.amazon.com/eks/latest/userguide/ebs-csi.html)
+
+### REQUIREMENTS
+
+- Must be using AWS
+### Notes
+
+- Super easy use, apply CSI and you done!
+
+## Azure Disk CSI
+
+[Website/Docs](https://docs.microsoft.com/en-us/azure/aks/azure-disk-csi)
+
+### REQUIREMENTS
+
+- Must be using Azure
+### Notes
+
+- Super easy use, apply CSI and you done!
+
+## Longhorn
+
+[Website/Docs](https://longhorn.io/)
+
+### REQUIREMENTS
+
+- RWX requires `nfs-common` to be installed on the nodes. [Longhorn RWX Docs](https://longhorn.io/docs/1.1.0/advanced-resources/rwx-workloads/)
+
+### Notes
+
+- 100% open source
+- Easiest to install
+- Documented airgap install process
+- GUI provides data and observability; replica status, cluster health status, backup status, and backup initiation/recovery.
+- Native backup to S3 or NFS
+
+## OpenEBS
+
+[Website/Docs](https://openebs.io/)
+
+### REQUIREMENTS
+
+- Blank, un-partitioned attached disk(s)
+- RWX is in Alpha and requires work. [OpenEBS RWX Docs](https://docs.openebs.io/docs/next/rwm.html)
+
+### Notes
+
+
+
+## Rook-Ceph
+
+[Website/Docs](https://rook.io/)
+
+### REQUIREMENTS
+
+- Blank, un-partitioned attached disk(s)
+
+### Notes
+
+- 100% open source
+- Very Fast
+
+## Portworx
+
+[Website/Docs](https://docs.portworx.com/portworx-install-with-kubernetes/)
+
+### REQUIREMENTS
+
+- Blank, un-partitioned attached disk(s)
+
+### Notes
+
+- Portworx Essentials is free **up to** 5nodes, 5TB Storage, 500 volumes
+- Portworx Enterprise and PX-Backup require paid licenses 
+- Best Mixed IOPS, average read/write performance
+- Install is very picky about the container runtime hostpath
+- Tested on Konvoy 1.6.1 due to Portworx issues when using RKE2

docs/k8s-storage/benchmark.yaml

+apiVersion: v1
+kind: Namespace
+metadata:
+  name: benchmark
+---
+kind: PersistentVolumeClaim
+apiVersion: v1
+metadata:
+  name: dbench
+  namespace: benchmark  
+spec:
+  accessModes:
+    - ReadWriteOnce
+  resources:
+    requests:
+      storage: 50Gi
+---
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: dbench
+  namespace: benchmark    
+spec:
+  template:
+    spec:
+      containers:
+      - name: dbench
+        image: sotoaster/dbench:latest
+        imagePullPolicy: IfNotPresent
+        env:
+          - name: DBENCH_MOUNTPOINT
+            value: /data
+          - name: FIO_SIZE
+            value: 25G
+        volumeMounts:
+        - name: dbench-pv
+          mountPath: /data
+      restartPolicy: Never
+      volumes:
+      - name: dbench-pv
+        persistentVolumeClaim:
+          claimName: dbench
+  backoffLimit: 4
\ No newline at end of file

docs/k8s-storage/rwx_test.yaml

+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: rwx-test
+spec:
+  accessModes:
+    - ReadWriteMany
+  resources:
+    requests:
+      storage: 1Gi
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: rwx-test
+  labels:
+    app: rwx-test
+spec:
+  replicas: 3
+  selector:
+    matchLabels:
+      app: rwx-test
+  strategy:
+    type: Recreate
+  template:
+    metadata:
+      labels:
+        app: rwx-test
+    spec:
+      containers:
+        - image: ubuntu:xenial
+          imagePullPolicy: Always
+          command: ["/bin/sh", "-c"]
+          args:
+            - sleep 30; touch /mnt/rwx-test/test.log; while true; do date >> /mnt/rwx-test/test.log; sleep 1; done;
+          name: rwx-test
+          stdin: true
+          tty: true
+          livenessProbe:
+            exec:
+              command:
+                - timeout
+                - "10"
+                - ls
+                - /mnt/rwx-test
+            initialDelaySeconds: 10
+            periodSeconds: 10
+            timeoutSeconds: 10
+          volumeMounts:
+            - mountPath: /mnt/rwx-test
+              name: rwx-test
+      restartPolicy: Always
+      volumes:
+        - name: rwx-test
+          persistentVolumeClaim:
+            claimName: rwx-test