UNCLASSIFIED - NO CUI

Skip to content
Snippets Groups Projects
Commit 5ed28f65 authored by Mark Sanchez's avatar Mark Sanchez Committed by Michael McLeroy
Browse files

Resolve "Diagram/workflow of all steps in CI jobs"

parent d59772af
No related branches found
No related tags found
1 merge request!867Resolve "Diagram/workflow of all steps in CI jobs"
...@@ -19,15 +19,6 @@ Table of Contents: ...@@ -19,15 +19,6 @@ Table of Contents:
- [DNS](#dns) - [DNS](#dns)
- [Secrets & Certificates](#secrets--certificates) - [Secrets & Certificates](#secrets--certificates)
- [Merge requests process](#merge-requests-process) - [Merge requests process](#merge-requests-process)
- [Pipeline Stages](#pipeline-stages)
- [Linting](#linting)
- [Smoke Testing](#smoke-testing)
- [Infrastructure Testing](#infrastructure-testing)
- [Network Creation](#network-creation)
- [Cluster(s) Creation](#clusters-creation)
- [Big Bang Installation](#big-bang-installation)
- [Big Bang Tests](#big-bang-tests)
- [Teardown](#teardown)
## Developers Guide ## Developers Guide
...@@ -98,76 +89,7 @@ Follow instructions in the [Big Bang encryption guide](docs/3_encryption.md) for ...@@ -98,76 +89,7 @@ Follow instructions in the [Big Bang encryption guide](docs/3_encryption.md) for
## Merge requests process ## Merge requests process
The following is meant to serve as an overview of the pipeline stages required to get a commit merged. The merge request process is provided as an overview of the pipeline stages required to get a commit merged.
### Pipeline Stages Follow instruction in [CI-Workflow](docs/developer/ci-workflow.md) for specific details on the pipeline stages.
The pipeline is split into several stages:
#### Linting
Several linting rules are first run to ensure yaml standards are met within the primary `./charts` folder.
This stage is ran on every commit, and is a requirement for merging.
#### Smoke Testing
For fast feedback testing, an ephemeral in cluster pipeline is created using [`k3d`](https://k3d.io) that lives for the lifetime of the gitlab ci job. Within that cluster, BigBang is deployed, and an initial set of smoke tests are performed against the deployment to ensure basic conformance.
This stage verifies several easy to check assumptions such as:
- does BigBang successfully install
- does BigBang successfully upgrade (from master)
- are endpoints routable
This stage also serves as a guide for local development, and care is taken to ensure all pipeline actions within this stage are repeatable locally.
This stage is ran on every commit, and is a requirement for merging.
#### Infrastructure Testing
Ultimately, BigBang is designed to deploy production ready workloads on real infrastructure. While local and ephemeral clusters are excellent for fast feedback during development, changes must ultimately be tested on real clusters on real infrastructure.
As part of BigBang's [charter](https://repo1.dso.mil/platform-one/big-bang/charter), it is expected work on any CNCF conformant kubernetes cluster, on multiple clouds, and on premise environments. By very definition, this means infrastructure testing is _slow_. To strive for a pipeline with a happy medium of providing fast feedback while still exhaustively testing against environments that closely mirror production, __infrastructure testing only occurs on manual actions on merge request commits.__
When you are comfortable your branch is ready to be merged, opening up an merge request will trigger the creation of a suite of infrastructure testing jobs which will require a manual action from a project maintainer (assuming previous linting and smoke tests have passed). Once the commit(s) are validated against the infrastructure tests, your changes are ready to be merged!
For _most_ of the infrastructure testing, `terraform` is chosen as the IAC tool of choice for infrastructure that BigBang owns, while the cluster creation process follows the vendors recommended installation process.
The infrastructure pipeline is designed to have _no_ human interaction, and are scoped to the lifecycle of the pipeline. This means a single pipeline is fully responsible for provisioning infrastructure, but just as important, deprovisioning infrastructure, ensuring resources are not orphaned.
More information on the full set of infrastructure tests are below:
##### Network Creation
For each cloud, a BigBang owned network will be created that conform with the appropriate set of tests about to be ran. For example, to validate that Big Bang deploys in a connected environment on AWS, a VPC, subnets, route tables, etc... are created, and the outputs are made available through terraform's remote `data` source.
##### Cluster(s) Creation
Several types of clusters are created within the previously provisioned network(s), and follow the vendors recommended iac approach.
For example, an `rke2` cluster is created that leverages the upstream [terraform modules](https://repo1.dso.mil/platform-one/distros/rancher-federal/rke2/rke2-aws-terraform), and an `eks` cluster is created with the upstream [terraform modules](https://docs.microsoft.com/en-us/azure/developer/terraform/create-k8s-cluster-with-tf-and-aks).
It is a hard requriement at this stage that every cluster outputs an admin scoped `kubeconfig` as a gitlab ci artifact. This artifact will be leveraged in the following stages for interacting with the created cluster.
##### Big Bang Installation
Given the kubeconfig created in the previous stage, BigBang is installed on the cluster using the same installation process used in the smoke tests.
Like any BigBang installation, several cluster requirements (TODO: doc these) must be met before BigBang is installed, and it is up to the vendor to ensure those requirements are met.
##### Big Bang Tests
Assuming BigBang has installed successfully, additional tests residing within the `./tests` folder of this repository are run against the deployed cluster. These tests range from automated UI testing, to internal kubernetes resource validation and verification.
TODO: Document these tests more once they are flushed out.
#### Teardown
Infrastructure teardown happens in the reverse sequence as to which they are deployed, and the pipeline will ensure these teardown jobs are _always_ ran, regardless of whether or not the previous jobs were successful.
Combined with terraform's declarative remote state, the "always on" teardown ensures no orphaned resources are left over once tests are run.
Within the teardown process, the commit scoped terraform workspace is also deleted to ensure the remote state remains clean.
For example, if an EKS cluster fails to provision, a full teardown of BigBang, EKS, and the network will be run, even though BigBang was never deployed. This will result in 2 failing jobs (EKS up and BigBang down), but will ensure that no infrastructure resources become orphaned.
## Gitlab-ci Workflow
The following is meant to serve as an overview of the pipeline stages required to get a commit merged. There are package, bigbang, and infrastructure pipelines.
### Table of Contents:
- [Generic Package Pipeline Stages](#generic-package-pipeline-stages)
- [Configuration Validation](#configuration-validation)
- [Package Tests](#package-tests)
- [BigBang Pipeline Stages](#bigbang-pipeline-stages)
- [Pre Vars](#pre-vars)
- [Smoke Tests](#smoke-tests)
- [Infrastructure Testing Pipeline Stages](#infrastructure-testing-pipeline-stages)
- [Network Creation](#network-creation)
- [Cluster Creation](#cluster-creation)
- [Big Bang Installation](#big-bang-installation)
- [Big Bang Tests](#big-bang-tests)
- [Teardown](#teardown)
### Generic Package Pipeline Stages
This pipeline is triggered by the following for individual bigbang packages:
- merge request events
- Note: Currently upgrade step only runs during MR events
- manual tag events
- commits to default branch
![Package Pipeline](imgs/Package_pipeline.png)
[Link to draw.io diagram file](diagrams/BB_gitlab_ci_diagram.drawio). This diagram file should be modified on draw.io and exported into this repository when the developer / ci workflow changes. It is provided here for ease of use.
#### Configuration Validation
This stage runs a `helm conftest` which is a plugin for testing helm charts with Open Policy Agent. It provides the following checks:
- confirms that the helm chart is valid (should fail similar to how a helm lint fails if there is bad yaml, etc)
- runs the helm chart against a set of rego policies - currently these tests will only raise warnings on "insecure" things and will allow pipeline to proceed.
#### Package Tests
This stage verifies several easy to check assumptions such as:
- does package successfully install
- does package successfully upgrade (from master)
- package specific tests
If required, the upgrade step can skipped when MR title starts with 'SKIP UPGRADE'
### BigBang Pipeline Stages
This pipeline is triggered by the following for individual bigbang packages:
- merge request events
- Note: Currently upgrade step only runs during MR events
- manual tag events
- commits to default branch
The pipeline is split into several stages:
![BB Pipeline](imgs/BB_pipelines.png)
[Link to draw.io diagram file](diagrams/BB_gitlab_ci_diagram.drawio). This diagram file should be modified on draw.io and exported into this repository when the developer / ci workflow changes. It is provided here for ease of use.
#### Pre Vars
This stage currently has one purpose at this point which is to generate a terraform var.
#### Smoke Tests
For fast feedback testing, an ephemeral in cluster pipeline is created using [`k3d`](https://k3d.io) that lives for the lifetime of the gitlab ci job. Within that cluster, BigBang is deployed, and an initial set of smoke tests are performed against the deployment to ensure basic conformance.
This stage verifies several easy to check assumptions such as:
- does BigBang successfully install
- does BigBang successfully upgrade (from master)
- are endpoints routable
This stage will fail if:
- script failures
- gitrepositories status condition != ready
- expected helm releases are not present
- helm releases fail or timeout
- kustomization secrets are not ready or timeout
- deployments status condition != ready
- jobs status condition != complete
- statefulsets/daemonsets not 100% ready (ex. 0/1)
- any virtual service endpoints are not accessible
This stage also serves as a guide for local development, and care is taken to ensure all pipeline actions within this stage are repeatable locally.
This stage is ran on every merge request event, and is a requirement for merging.
### Infrastructure Testing Pipeline Stages
Ultimately, BigBang is designed to deploy production ready workloads on real infrastructure. While local and ephemeral clusters are excellent for fast feedback during development, changes must ultimately be tested on real clusters on real infrastructure.
As part of BigBang's [charter](https://repo1.dso.mil/platform-one/big-bang/charter), it is expected work on any CNCF conformant kubernetes cluster, on multiple clouds, and on premise environments. By very definition, this means infrastructure testing is _slow_. To strive for a pipeline with a happy medium of providing fast feedback while still exhaustively testing against environments that closely mirror production, __infrastructure testing only occurs on manual actions on merge request commits.__
This requires adding `test-ci::infra` label to your MR. In addition, infrastructure testing pipeline is run nightly on a schedule.
Note: Due to the amount of resources and time required for this pipeline, the `test-ci::infra` label should be used sparringly. The scheduled nightly run will ideally catch issues if they are already in master. The `test-ci::infra` label should mainly be used when:
- your changes affect the infra ci
- your changes are large in scope and likely to behave differently on "real" clusters
When you are comfortable your branch is ready to be merged, opening up an merge request will trigger the creation of a suite of infrastructure testing jobs which will require a manual action from a project maintainer (assuming smoke tests have passed). Once the commit(s) are validated against the infrastructure tests, your changes are ready to be merged!
For _most_ of the infrastructure testing, `terraform` is chosen as the IAC tool of choice for infrastructure that BigBang owns, while the cluster creation process follows the vendors recommended installation process.
The infrastructure pipeline is designed to have _no_ human interaction, and is scoped to the lifecycle of the pipeline. This means a single pipeline is fully responsible for provisioning infrastructure, but just as important, deprovisioning infrastructure, ensuring resources are not orphaned.
More information on the full set of infrastructure tests are below:
![Infra Pipeline](imgs/Infra_test_pipelines.png)
[Link to draw.io diagram file](diagrams/BB_gitlab_ci_diagram.drawio). This diagram file should be modified on draw.io and exported into this repository when the developer / ci workflow changes. It is provided here for ease of use.
#### Network Creation
For each cloud, a BigBang owned network will be created that conform with the appropriate set of tests about to be ran. For example, to validate that Big Bang deploys in a connected environment on AWS, a VPC, subnets, route tables, etc... are created, and the outputs are made available through terraform's remote `data` source.
At this time the infrastructue testing pipeline is only utilizing internet-connect AWS govcloud.
#### Cluster Creation
The infrastructure pipeline is currently setup to standup an `rke2` cluster by default.
An `rke2` cluster is created that leverages the upstream [terraform modules](https://repo1.dso.mil/platform-one/distros/rancher-federal/rke2/rke2-aws-terraform)
It is a hard requriement at this stage that every cluster outputs an admin scoped `kubeconfig` as a gitlab ci artifact. This artifact will be leveraged in the following stages for interacting with the created cluster.
#### Big Bang Installation
Given the kubeconfig created in the previous stage, BigBang is installed on the cluster using the same installation process used in the smoke tests.
Like any BigBang installation, several cluster requirements (see [Pre-requisites](https://repo1.dso.mil/platform-one/big-bang/bigbang/-/tree/master/docs/guides/prerequisites)) must be met before BigBang is installed, and it is up to the vendor to ensure those requirements are met.
#### Big Bang Tests
Assuming BigBang has installed successfully, additional tests residing within the `./tests` folder of this repository are run against the deployed cluster.
Currently there are 3 test scripts that test the following:
- wait for resources to be ready, ensures everything goes to running at a kubernetes level
- curl VirtualService endpoints, to validate istio works + the UIs are up
- fetch a list of non-IB images (this test never fails but provides some contextual info)
#### Teardown
Infrastructure teardown happens in the reverse sequence as to which they are deployed, and the pipeline will ensure these teardown jobs are _always_ ran, regardless of whether or not the previous jobs were successful.
Combined with terraform's declarative remote state, the "always on" teardown ensures no orphaned resources are left over once tests are run.
Within the teardown process, the commit scoped terraform workspace is also deleted to ensure the remote state remains clean.
For example, if an RKE2 cluster fails to provision, a full teardown of BigBang, RKE2, and the network will be run, even though BigBang was never deployed. This will result in 2 failing jobs (RKE2 up and BigBang down), but will ensure that no infrastructure resources become orphaned.
<mxfile host="Electron" modified="2021-09-13T18:30:18.191Z" agent="5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) draw.io/12.9.9 Chrome/80.0.3987.163 Electron/8.2.1 Safari/537.36" etag="pn3BY3rUH7aw4UmqKB5J" version="12.9.9" type="device" pages="2"><diagram id="C5RBs43oDa-KdzZeNtuy" name="BB pipelines">7Vxbe5s4EP01/rb7kP0QV/sxtpO26WW7Tdu0jzLINrWMCAhf8pDfvhIXGxBxHMcYoiYPsRkLCebMmRkNQh1tMF+9DaA//UQchDuq4qw62rCjqsBSVPbBJetUAnp6IpkErpPKtoJr9w6lQiWVRq6DwkJDSgimrl8U2sTzkE0LMhgEZFlsNia4OKoPJ0gQXNsQi9Ib16HTRNpVra38HXIn02xkYPaSX+Ywa5zeSTiFDlnmRNpFRxsEhNDk23w1QJhrL9PLzfv1Df44M99e/Rfewu/9D98+/zhLOrt8yimbWwiQRw/u+lZ3Z4vVGvTsO+3D4Ap9+KJdnalaem90nSkMOUx/6SEJ6JRMiAfxxVbaD0jkOYh3q7CjbZuPhPhMCJjwN6J0nRoDjChhoimd4/RXtHLpz9z3X7yrf4z0aLhKe44P1tmBR4P1z/xB7ix+uD0tPsrOS+6P31TJHh5RZmarMJggukODoLcxBUYiROaIjc5ODBCG1F0UR4WpMU827TanfiEuux5VSZnXM1OapbzTMivMughJFNgoPWsLO/uSu4ytKDaGJxhGOtoC4ii9BcFQimawnLoUXfvQ5r8umTMpQj4JoOMyjQ8IJkF8vjaO/9hvYxfjTO4RD21QW6CAotUBuIlwpL3ooKjWbnqfy5xr6JqJbJpzC4L68wgWdP9URVtGIww8kE3Ic865U2aHI0zsWSK6dHHW8RHpVtnOqoVtwFQKZqFq+7GN6QKuc8183iB8eJxNLCmNc7lne6ArJYNLruCo1LdM2WNC3VacWMuOdtq+wUU5hrmf2kozZ3ksK92lwlyA8gMUC4JwR6gCj4eqMfFoaqZA5aELwzBMLSekAZltEkneepMVbpONfc1ECG8Ph60SIEAMWyADIR+29CNErYfdcEH71xR6TuQz4UxzeD6Po5CiQD4ojGIGsdH7KbCodhICFgOMoMdE772QQt5BCQN2q7SoaIjdice+20xbDDStzxXislnUefrD3HWcxNGj0L2Do7grrufUmbB+jX7HGPK+mG8PU9BqTOUsrQhEryKVq4ChtkROFWAYIh8TfsWXOFo1TITj69/oto0IYkjYINB3JyPoTTqqibnlj5hfMif825t3CM9Zg8jnMxT0t3QwWaBtMOkCTN9QyNMJaNuIqYsXiPi1kJAnKcobOwqwfLh09bbhYumCkl/z/hdZLAJlG6m5WJSlo7tSEGUcEO5oHTSGEeYXPwqgZ0873MYFrxwiFFsi8rkLgCOykNA1l1NJTWnaBQAxl/yeBEZJc8iSE1YtEQHzlEkkELPIwRTZMxIVGLN0Kf+wpyynQU1Pd+tPLVtADDG3PPd9vM6DkKQtLNPsv2aaLUHtNdWsSjWbByZzs/Kmms/MJPMV5B2Z5B4pZ+5R9WEp5/MY2BMYOHRDHyf16Ph8lgzENytfGANKySOqjRNPFTP1IcKI8mx70IqKbQ0wAKNtMOi9RvxfS32Z2t3Tl2kPAH0aX5ZdZhsfOdUwOS0+cjIa54zZfeVMmQr7cEZrkjOaWFDwEF2SYBbPhKRjjaa1jjavK/yeS7V9V2skDyQbo5pYotg8g5eRarrVOqpJ/wCldqrp+1LNaJRqYl1plD1glpFqJmgd1ZqZQMlENWNfqpmNUs0QqEbjEq5kFLOM1lFM+mXAtVPM3Jdi3UYpZu6IZszyPenI1u21jmzWSyFb7aTZuxj43LU0zyONWAzczrakJA1QWlfb0MSnS9vqkpwggNbNerOOcyDcA4OrHdmC/uVYNdQtQKBmlemmFg3pYpH13jqLMZi7IgfkwCDjfrbkMXuPMgeCdVIQxJrAPVDiB64yo1BaWWKK3uikL2Ho4nRRcmcEQHmRgSFicFJ3ZIgB4VvgTiYoYCOdiwvjzljba3uK7AizyKwqn/k1xkvr+pGLneoTPn1l/zAcxdtVxIMofyWFAcV2WcPbiG/L0H8C5hiN6ctAvFtMAfSKRavAqEDcqA3xigCkyez3NNMoZmEVnKtCoD7OqSICUsd/Q2kbAuLjNx7/NalRMMtTwgoUThr/jaosTGYErG7beFCRgUkdC3pq2xAQ67n3uswIAKV14VjcH+FelRoCtXXxWCzQCqrPvQ+a7juVQyFpfWChW1TUI3rIZM98E1RITHsl/T7wJqi4I5L6WE9Jgb62d0oNsbZ7zw415cyUu6RigYLeTaUnEOmkKZUlTu7+ACLp5VeqrQOJ1CttLSZ0VDOPLHFm+AfCp5VrH4fCJ3RUN3zitPJ6Tmb8tQP+KqP43oEUTrBbgu+E08rhBN98uh0YVx+d0Lr8OhhHv88qdsNMK5pxQbNiA4NM4riLsij0+cYIuUomYgbEPr+i2ygpYF4sULw/W9KeXXD+lJy4ovM9xptDL4L8jinkCzxQrYPZZD536eZFW3HzhycNu7epv5harloqoFTsRwqqivfmESz9F17cjn9r51H/sxr+MEfo3x/9l2Dp8ltB1V5mmz1dCtvSPt0M2OF2y+okSm13/tYu/gc=</diagram><diagram id="7LAne4oA9n-VVIV32Hwe" name="Package pipeline">7Vtbd5s4EP41Pmf3IT3IXIwfHcdpt2ez2zbppnmUQQbVAhEQdpyH/vaVQFxF4lzqS0nzEjToxnzzSTMjeaBPg7v3MYz8C+oiMhhq7t1APxsMh8AYmfyfkGykRNOlxIuxK2WV4BLfo6KilKbYRUmjIqOUMBw1hQ4NQ+SwhgzGMV03qy0oaY4aQQ8pgksHElV6jV3m51Lb1Cr5B4Q9vxgZaPJNAIvKUpD40KXrmkifDfRpTCnLn4K7KSJCe4VePmtM1+HN0KazK7j+eo1vo+lJ3tn5c5qUnxCjkL246+Gn2/vxhbbZTD7+o4ETG977tmyirSBJpb44DgvspTFkmIb5K+zKQqYFtilUG9M0dJHoHgz007WPGbqMoCPerrk1cZnPAiJfL2jIpHUArrpTj8BEQKvx54TFdFnCI2qXuhav5RxRzNBdC9QtGgElTNzAEQ0Qize8nezFkF8vTVu3pWWvKzsBBfh+zUaKdlCaplf2XKmfP0gEnoEGUFSMXG7Mskhj5lOPhpDMKulpBYJQVVXnb0ojqczviLGN1D1MGW0Cg+4w+1Z7vhFdvTNl6exO9pwVNkUh5J/7rV6otRLFqllW2jSQFB/1Ahy5YmgaO+gxBUoEGYw9xB6rOOq2jBgRbuur5vR+Os5DhXWPcEvbzi0vhi7mGpxSQuOsvb7I/gTvMCGFPKQh2iWfzCahhpamEMoedxBKH+6KUfpBGPVCdqDQnYgdjxfnhDrLXHSOyWuoumvKaU9knPlKwsmmnyjm0y5NDtjjd+Pa32jUsEBDM5s95p8jO6lsi6sdbmrVIlEheXjY0idojXP+xPqFwVemnc+gMvRSRS+3faPvu8lWvhyL7YMH1s23afymtgfjN5Ut9pLB0E0jLlzqrgg4SJowFPfPqx231hpN3YX369ZaChZTgqAILv4KEwZFBy0MuEpYU9E8CvFC/uxw1XDQ9FOhOMzDvIl8EWDXzZcwlOB7OM+6EnqW5sz7NU8H5pnoi69aiQRth0CMRi13aNThDnXgsDNnaKTgUOpfO0MRXztR6GCU9I4SVnv7PTgl7Eeg+ASdpUhq9A2FkXlsKIx/e0i7Dsif7CIdNCAvplnj4wdEAjFxlIjNY8Ifp5uI7y2JGD8UDkTixDgSfs4cJtwg+kZX2z42ugKw3ZHQFjEVuLloAVOSoRPD0PEHYqYWEV7FnPt8liee/kgQykBDUQbrnK7Qn70Dsu0QGvbBgVQTYF8jkcRSN71+eIItKpWJshoC1j49QaCrTPKRs6RpgzFrzMQ/x4eh9wbcwiMghqHgMokisqmDoHGHQny6dBL5ShcRugmQDOr7BFDbYzwCgNS8wm9Hob26HR4lS1Fyzxz7wa79dpkr2O632922sSe/XU1pnOEkInk+M2vPmZh9bP+2L6C1A+rDE0/Na5whgpjYp6Y9zbcC0E64Hh6GsQLDVYw9T7jLk85IqJC4eNUWJZGIsHLZCW98geLM7/iCblO+5/Gn2QplBwZ5fT7jepOauKPzJ4wXwDCFJFuLPLHm7XQwhwYBZqWXpUaRzxr2ybEMQQv2S0Qyw/YRf0ckA7pCGWtXtl4Qq2brZQJVu8qcsl7GlLZhNZHQVSS67lr8jJjyv5MEXZ7rHwMIlth2Aw0sFx1XyX7wIhAMxurdsV5g0DrgMTog2FVY3wmBmh/rPQRm+2hnj6mVTgzU1JbAQBPy/qJgGeZWFEb7REHNb2VMENcC+ovCaHRkKKjJkt6vR2NwZOuRevlCYKCLywD9RcFsoWB27Mx7ZYKap3hzTOjCYK9MUHMT2c7cZwws88gwUBMTb3Bf3udqNI7xzfeLxfr9l9VlcuX4YzSddIVq4mBDS5DTTwQAaJ8kGfsjwnxoUBsEq9nUvf53PZ1MPofpA9GykR2N95cJVitiLk9g98CEThjUiFlRfO0Olfz9TA2DvPYLz2FUNdXUYHaooZC98t54K19tgZZ2H7gnrt4/1/UtPeXHR0pPL7jk3YmeGmsfXZr7ySz+ZVPARse1ZmB0WK/+fBLzYvX72txqqp8p67P/AQ==</diagram></mxfile>
\ No newline at end of file
docs/developer/imgs/BB_pipelines.png

55.6 KiB

docs/developer/imgs/Infra_test_pipelines.png

22.7 KiB

docs/developer/imgs/Package_pipeline.png

61.5 KiB

0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment