Team Spike on Storage Class

Why

Multiple customers operating requesting the capability for CAS solutions for on prem deployments. The current recommendation is to leverage the csp csi drivers (ebs, azuredisks, etc...) or vsphere csi drivers. But there's still a gap that needs to be filled for deployments that don't fit that mold, or that still prefer to run their own storage solution.

What

Research spike to investigate three of the most popular open source solutions that exist:

• openebs
• rook
• longhorn
• portworx

The spike should include:

• product deployment (can be the recommended way, doesn't need to be bb standards)
• deploying bigbang on top of the cas soln
• measuring performance metrics during deployment and generic use
• demonstrating/testing backup capabilities on one or more BB packages
• testing a RWX deployment ( can be a simple as persistent busybox sharing files )

Goal

Collect information and build team competency around the existing CAS landscape to ultimately make a team ADR on what CAS solution BigBang will support and deploy.

Note: the goal is not to decide on a solution OR to build an ADR, simply to data collect.

EVALUATION

Details & Methods

Benchmark Results

Product	Random Read/Write IOPS	Average Latency (usec) Read/Write (lower is better)	Sequential Read/Write	Mixed Random Read/Write IOPS
Longhorn	6155/1551 BW: 230MiB/s / 96.3MiB/s	1042.53	319MiB/s / 130MiB/s	3804/1267
OpenEBS (jiva)	2183/770. BW: 76.8MiB/s / 45.8MiB/s	2059.55	132MiB/s / 98.2MiB/s	1590/533
Rook-Ceph	10700k/3205. BW: 503MiB/s / 148MiB/s	548.36/s	496MiB/s / 154MiB/s	6664/2228

OpenEBS

REQUIREMENTS

• Big Bang compatible
• Cloud Agnostic
  • Multiple options to support different clouds and on prem designs Docs
• Native Backup Tool
  • Recommends Valero
• Supports RWX
  • OpenEBS includes an alpha version of OpenEBS Dynamic NFS provisioner for RWX PVCs Docs
• Can be added Ironbank
  • 100% Opensource
• Airgap Ready
  • Requires manual work, no official docs or support

Notes:

• Extremely flexible with many options to support multiple environments and configurations.
• Full Bigbang install went without issue using OpenEBS with Jiva hostpath storage engine.
• Requires iSCSI initiator to be running on all nodes. Docs
• RWX solution leaves much to be desired.

Rook

REQUIREMENTS

• Big Bang compatible
• Cloud Agnostic
  • Yes, uses host storage drives; must be unmounted block storage: raw disks, drive partitions, LVM, or cloud block storage.
• Native Backup Tool
• Supports RWX
• Can be added Ironbank
  • 100% Opensource
• Airgap Ready
  • Requires manual work, no official docs or support

Notes:

• Can optionally install dashboard to view Rook-Ceph cluster stats and health.
• Very fast
• BB install went smooth without modifications needed

Longhorn

REQUIREMENTS

• Big Bang compatible
• Cloud Agnostic
  • Yes, uses host storage
• Native Backup Tool
  • Native support for NFS and S3 backups
• Supports RWX
  • Native support as of 1.1.0 Longhorn RWX Docs
• Can be added Ironbank
  • 100% Opensource
• Airgap Ready
  • Offical Longhorn Airgap Docs

Notes:

• Includes GUI for configuration and observability, view PVC replica states, backup state, health stats, and more.
• RWX is now native in 1.1.0, installs out of the box, works great, and does not require a NFS-Provisioner.
• BB install went smooth without modifications needed
• By far the easiest to install!

Portworx

REQUIREMENTS

• Big Bang compatible
• Cloud Agnostic
  • Yes, uses host storage drives; must be unmounted block storage: raw disks, drive partitions, LVM, or cloud block storage.
• Native Backup Tool
  • Requires "px-backup" license
• Supports RWX
• Can be added Ironbank
  • Would require Portworx coordination
• Airgap Ready
  • Offical Portworx Airgap Docs

Notes:

• Can install optional Lighthouse dashboard to monitor multiple Portworx clusters
• Requires License
• Not opensource
• Install requires Portworx account