diff --git a/docs/k8s-storage/README.md b/docs/k8s-storage/README.md new file mode 100644 index 0000000000000000000000000000000000000000..ad4285db68838969558faf644dad43d5a44d711f --- /dev/null +++ b/docs/k8s-storage/README.md @@ -0,0 +1,107 @@ +## Kubernetes Storage Options + +Use this data to assist in your CSI decision. However, when using a cloud provider we suggest you use their Kubernetes CSI. + +## Feature Matrix + +| Product | BB Compatible | FOSS | In Ironbank | RWX/RWM Support | Airgap Compatible | Cloud Agnostic | +| --------- | --------- | --------- | --------- | --------- | --------- | --------- | +Amazon EBS CSI | **X** | N/A | | **X** | AWS Dependent | No | +Azure Disk CSI | Not Tested | N/A | | **X** | Azure Dependent | No | +Longhorn v1.1.0 | **X** | **X** | | **X** | **X** - [Docs](https://longhorn.io/docs/1.1.0/advanced-resources/deploy/airgap/) | Yes, uses host storage | +OpenEBS (jiva) | **X** | **X** | | **X** **[Alpha](https://docs.openebs.io/docs/next/rwm.html)** | Manual Work Required | Yes, uses host storage | +Rook-Ceph | **X** | **X** | | **X** | Manual Work Required | Yes, uses host storage | +Portworx | **X** | | | **X** | **X** - [Docs](https://docs.portworx.com/portworx-install-with-kubernetes/operate-and-maintain-on-kubernetes/pxcentral-onprem/install/px-central/) | Yes, uses host storage | + +## Benchmark Results + +Benchmarks were tested on AWS with GP2 ebs volumes using using FIO, see [example](./benchmark.yaml) + +| Product | Random Read/Write IOPS | Average Latency (usec) | Sequential Read/Write | Mixed Random Read/Write IOPS | +| --------- | --------- | --------- | --------- | --------- | +Amazon EBS CSI | 2997/2996. BW: 128MiB/s / 128MiB/s | 1331.61 | 129MiB/s / 131MiB/s | 7203/2390 +Azure Disk CSI | | | | +Longhorn v1.1.0 | 6155/1551 BW: 230MiB/s / 96.3MiB/s | 1042.53 | 319MiB/s / 130MiB/s | 3804/1267 +OpenEBS (jiva) | 2183/770. BW: 76.8MiB/s / 45.8MiB/s | 2059.55 | 132MiB/s / 98.2MiB/s | 1590/533 +Rook-Ceph | 10.7k/3205. BW: 503MiB/s / 148MiB/s | 548.36/s | 496MiB/s / 154MiB/s | 6664/2228 +Portworx 2.6 | 3016/19.3k. BW: 74.5MiB/s / 85.1MiB/s | 1337.31 | 113MiB/s / 124MiB/s | 35.1k/11.1k + +## Amazon EBS CSI + +[Website/Docs](https://docs.aws.amazon.com/eks/latest/userguide/ebs-csi.html) + +### REQUIREMENTS + +- Must be using AWS +### Notes + +- Super easy use, apply CSI and you done! + +## Azure Disk CSI + +[Website/Docs](https://docs.microsoft.com/en-us/azure/aks/azure-disk-csi) + +### REQUIREMENTS + +- Must be using Azure +### Notes + +- Super easy use, apply CSI and you done! + +## Longhorn + +[Website/Docs](https://longhorn.io/) + +### REQUIREMENTS + +- RWX requires `nfs-common` to be installed on the nodes. [Longhorn RWX Docs](https://longhorn.io/docs/1.1.0/advanced-resources/rwx-workloads/) + +### Notes + +- 100% open source +- Easiest to install +- Documented airgap install process +- GUI provides data and observability; replica status, cluster health status, backup status, and backup initiation/recovery. +- Native backup to S3 or NFS + +## OpenEBS + +[Website/Docs](https://openebs.io/) + +### REQUIREMENTS + +- Blank, un-partitioned attached disk(s) +- RWX is in Alpha and requires work. [OpenEBS RWX Docs](https://docs.openebs.io/docs/next/rwm.html) + +### Notes + + + +## Rook-Ceph + +[Website/Docs](https://rook.io/) + +### REQUIREMENTS + +- Blank, un-partitioned attached disk(s) + +### Notes + +- 100% open source +- Very Fast + +## Portworx + +[Website/Docs](https://docs.portworx.com/portworx-install-with-kubernetes/) + +### REQUIREMENTS + +- Blank, un-partitioned attached disk(s) + +### Notes + +- Portworx Essentials is free **up to** 5nodes, 5TB Storage, 500 volumes +- Portworx Enterprise and PX-Backup require paid licenses +- Best Mixed IOPS, average read/write performance +- Install is very picky about the container runtime hostpath +- Tested on Konvoy 1.6.1 due to Portworx issues when using RKE2 diff --git a/docs/k8s-storage/benchmark.yaml b/docs/k8s-storage/benchmark.yaml new file mode 100644 index 0000000000000000000000000000000000000000..a7f625d713283f89432b65ed3650ed412a4ba9d2 --- /dev/null +++ b/docs/k8s-storage/benchmark.yaml @@ -0,0 +1,43 @@ +apiVersion: v1 +kind: Namespace +metadata: + name: benchmark +--- +kind: PersistentVolumeClaim +apiVersion: v1 +metadata: + name: dbench + namespace: benchmark +spec: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 50Gi +--- +apiVersion: batch/v1 +kind: Job +metadata: + name: dbench + namespace: benchmark +spec: + template: + spec: + containers: + - name: dbench + image: sotoaster/dbench:latest + imagePullPolicy: IfNotPresent + env: + - name: DBENCH_MOUNTPOINT + value: /data + - name: FIO_SIZE + value: 25G + volumeMounts: + - name: dbench-pv + mountPath: /data + restartPolicy: Never + volumes: + - name: dbench-pv + persistentVolumeClaim: + claimName: dbench + backoffLimit: 4 \ No newline at end of file diff --git a/docs/k8s-storage/rwx_test.yaml b/docs/k8s-storage/rwx_test.yaml new file mode 100644 index 0000000000000000000000000000000000000000..5a754267ee2a6a5fa8c51c055eef90aa30f53ca2 --- /dev/null +++ b/docs/k8s-storage/rwx_test.yaml @@ -0,0 +1,56 @@ +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: rwx-test +spec: + accessModes: + - ReadWriteMany + resources: + requests: + storage: 1Gi +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: rwx-test + labels: + app: rwx-test +spec: + replicas: 3 + selector: + matchLabels: + app: rwx-test + strategy: + type: Recreate + template: + metadata: + labels: + app: rwx-test + spec: + containers: + - image: ubuntu:xenial + imagePullPolicy: Always + command: ["/bin/sh", "-c"] + args: + - sleep 30; touch /mnt/rwx-test/test.log; while true; do date >> /mnt/rwx-test/test.log; sleep 1; done; + name: rwx-test + stdin: true + tty: true + livenessProbe: + exec: + command: + - timeout + - "10" + - ls + - /mnt/rwx-test + initialDelaySeconds: 10 + periodSeconds: 10 + timeoutSeconds: 10 + volumeMounts: + - mountPath: /mnt/rwx-test + name: rwx-test + restartPolicy: Always + volumes: + - name: rwx-test + persistentVolumeClaim: + claimName: rwx-test