diff --git a/CODEOWNERS.txt b/CODEOWNERS.txt deleted file mode 100644 index 3d0f6553fe4b783523e9f6dad9574685970aad90..0000000000000000000000000000000000000000 --- a/CODEOWNERS.txt +++ /dev/null @@ -1 +0,0 @@ -* @big-bang diff --git a/app/monitoring/prometheus/README.md b/app/monitoring/prometheus/README.md deleted file mode 100644 index c033ec537b18583f03d2cf32fba3f6a3b881bda1..0000000000000000000000000000000000000000 --- a/app/monitoring/prometheus/README.md +++ /dev/null @@ -1,15 +0,0 @@ -Twistlock Prometheus Monitoring is implemented as per the documentation - -https://docs.paloaltonetworks.com/prisma/prisma-cloud/prisma-cloud-admin-compute/audit/prometheus.html - - -1. Create ServiceMonitor for twistlock endpoint - -2. Create Role, RoleBinding for monitoring in twistlock namespace - -3. Create Secrets for metrics point authentication - -4. kubectl apply -k prometheus - - - diff --git a/app/monitoring/prometheus/kustomization.yaml b/app/monitoring/prometheus/kustomization.yaml deleted file mode 100644 index fedada9d42f511631fdb4c62bef12fb2f61cf821..0000000000000000000000000000000000000000 --- a/app/monitoring/prometheus/kustomization.yaml +++ /dev/null @@ -1,7 +0,0 @@ -namespace: vault - -resources: -- twistlockServiceMonitor.yaml -- role.yaml -- roleBinding.yaml -- twistlockSecret.yaml \ No newline at end of file diff --git a/CHANGELOG.md b/docs/CHANGELOG.md similarity index 100% rename from CHANGELOG.md rename to docs/CHANGELOG.md diff --git a/docs/CODEOWNERS.txt b/docs/CODEOWNERS.txt new file mode 100644 index 0000000000000000000000000000000000000000..eea3abbb979670341e8e8f16172ee4ce230f748e --- /dev/null +++ b/docs/CODEOWNERS.txt @@ -0,0 +1 @@ +* @thomas.burton @runyontr diff --git a/CONTRIBUTING.md b/docs/CONTRIBUTING.md similarity index 100% rename from CONTRIBUTING.md rename to docs/CONTRIBUTING.md diff --git a/docs/CYPRESS.md b/docs/CYPRESS.md new file mode 100644 index 0000000000000000000000000000000000000000..e6f2b2b91843d9807ff2766ebbbba9883999dc43 --- /dev/null +++ b/docs/CYPRESS.md @@ -0,0 +1,123 @@ +#Twistlock Cypress testing + +The current Cypress test for Twistlock is currently not working due to a bug in Cypress related to the Twistlock front end. The test is very basic at best an is looking for a web page, which it gets, but it is an error page. This should be resolved in future versions of Cypress. +##The Basics +Gitlab-ci forks testing across multiple pods within the deployed k8s cluster. When writing tests for your services, append any requirements to the patch.yml in the same folder. For example, if your test requires certificates, secrets, or configurations, create a kustomization patch for v1/pod with those yaml paths and mount them as either environment variables or files. These will be consumed by your cypress tests. When writing tests, realize that they will be copied to and run within a stock cypress container inside the k8s cluster. Please use external endpoints for frontend tests and internal service points for backend tests. + +Beware: if your tests take too long, kubectl exec will time out. If this becomes an issue, consider breaking your tests out so that kubectl exec is getting constant feedback from the remote test environment. + +*Do not add names to your patches.* + +The gitlab script will inception patch the kustomization patch with unique identifiers to secure the test environment from mispellings. + +## Environment Variables: + +The e2e step will scan available environment variables and match them against known system variables. Every gitlab defined environment variable will be added to the base kustomization cypress pod. This means that you may create tests that run in the kubernetes pod that have access to the same variables you'd have in a gitlab runner environment. + +*When creating environmental variables to be used in Cypress, you MUST use the prefix CYPRESS_ or it will not be available to the process.* + +*For validating environmental names, this is the regex used: /^[-._a-zA-Z][-._a-zA-Z0-9]*$/* + +## Samples: + +### Basic B Test +kustomization.yml: +```yaml +bases: +- ../../../../base + +patchesStrategicMerge: +- patch.yml +``` + +patch.yml: +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: cypress +``` + +### Frontend Patch to Disable Istio: +patch.yml: +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: cypress + annotations: + sidecar.istio.io/inject: "false" +``` + +### Namespace Patch: +kustomization.yml: +```yaml +namespace: name-spaaaaaace + +bases: +- ../../../../base + +patchesStrategicMerge: +- patch.yml +``` + +### Environment Vars Patch: +patch.yml: +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: cypress +spec: + containers: + - name: cypress + env: + - name: CYPRESS_MY_LITTLE_SECRET + valueFrom: + secretKeyRef: + name: mysecurity + key: fipsfops + - name: CYPRESS_MY_LITTLE_NOTSECRET + valueFrom: + configMapKeyRef: + name: spesculvalew + key: noimthevaluenotthisguyaboveme +``` + +### File Patch: +patch.yml: +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: cypress +spec: + containers: + - name: cypress + volumeMounts: + - name: config-volume + mountPath: /etc/config + volumes: + - name: config-volume + configMap: + name: special-config +``` + +### Resource Intensive Frontend + +Got a big front end project to drive? Ensure your cypress process has enough resources. This is just an example, tune to your needs. Additionally, consider using `{timeout: bigNumber}` options in your cypress steps. (bigNumber is not a real variable, substitute with your millisecond count.) + +patch.yml +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: cypress +spec: + containers: + - name: cypress + resources: + requests: + memory: "564Mi" + cpu: "1000m" +``` \ No newline at end of file diff --git a/docs/ELASTIC.md b/docs/ELASTIC.md index 25e78553aec1756a489fb5136f4f1874f1324f40..b656b4bdc2c2d91cf1cf1837a953806715333381 100644 --- a/docs/ELASTIC.md +++ b/docs/ELASTIC.md @@ -5,13 +5,13 @@ Follow the steps in either 'Install Defender' or 'Install Defender with Twistcli create an index pattern for fluentd if not already created -``` +```bash logstash-* ``` Build filter for twistlock namespace -``` +```json { "query": { "match_phrase": { @@ -23,7 +23,7 @@ Build filter for twistlock namespace There should be 4 pods in the twistlock namespace -``` +```bash kubectl get pods -n twistlock NAME READY STATUS RESTARTS AGE twistlock-console-random-number 1/1 Running 0 3h13m @@ -39,7 +39,7 @@ Here are some examples of a filter for specific containers twistlock-console -``` +```json { "query": { "match_phrase": { @@ -51,7 +51,7 @@ twistlock-console twistlock-defender -``` +```json { "query": { "match_phrase": { @@ -63,20 +63,19 @@ twistlock-defender In the KQL field you can text search within a source field such as twistlock-defender -``` +```bash kubernetes.labels.app: "twistlock-defender" ``` -``` +```bash kubernetes.namespace_name:twistlock kubernetes.labels.app:twistlock-defender stream:stdout log: F  ERRO 2020-07-14T19:13:25.646 defender.go:331  Failed to initialize GeoLite2 db: open /prisma-static-data/GeoLite2-Country.mmdb: no such file or directory docker.container_id:c0f14b6ba111ef0af3761484dd77a19a5a9f054a4853f757d303be838cad6e6a kubernetes.container_name:twistlock-defender kubernetes.pod_name:twistlock-defender-ds-dtdjv kubernetes.container_image:registry-auth.twistlock.com/tw_bbzc81abegfiqtnruvspkazws2ze0dby/twistlock/defender:defender_20_04_169 kubernetes.container_image_id:registry- ``` -``` +```bash kubernetes.container_name:twistlock-console ``` -``` -kubernetes.container_name:twistlock-console kubernetes.namespace_name:twistlock stream:stdout log: F  ERRO 2020-07-14T20:01:10.932 kubernetes_profile_resolver.go:38  Failed to fetch Istio resources in 863da02e-15f2-d3da-f74d-0256f77292ad: 1 error occurred: docker.container_id:8303db1aa9e2a694b5db5a454c07127944ee0a4799f3e15f190eaa0eec53ca63 kubernetes.pod_name:twistlock-console-7d77c954d-lnjxp kubernetes.container_image:registry.dsop.io/platform-one/apps/twistlock/console:20.04.169 kubernetes.container_image_id:registry.dsop.io/platform-one/apps/twistlock/console@sha256:db77c64af682161c52da2bbee5fb55f38c0bcd46cacdb4c1148f24d094f18a10 kubernetes.pod_id:c979ebe6-f636-41b8-bfff-eab27fd48692 -``` +```bash -``` +kubernetes.container_name:twistlock-console kubernetes.namespace_name:twistlock stream:stdout log: F  ERRO 2020-07-14T20:01:10.932 kubernetes_profile_resolver.go:38  Failed to fetch Istio resources in 863da02e-15f2-d3da-f74d-0256f77292ad: 1 error occurred: docker.container_id:8303db1aa9e2a694b5db5a454c07127944ee0a4799f3e15f190eaa0eec53ca63 kubernetes.pod_name:twistlock-console-7d77c954d-lnjxp kubernetes.container_image:registry.dsop.io/platform-one/apps/twistlock/console:20.04.169 kubernetes.container_image_id:registry.dsop.io/platform-one/apps/twistlock/console@sha256:db77c64af682161c52da2bbee5fb55f38c0bcd46cacdb4c1148f24d094f18a10 kubernetes.pod_id:c979ebe6-f636-41b8-bfff-eab27fd48692 +``` \ No newline at end of file diff --git a/docs/PROMETHEUS.md b/docs/PROMETHEUS.md index ed9c31850836b30f16de7f46b59403c028bbc472..6ba956a4b09f0c68277916d39c21ef7037b99187 100644 --- a/docs/PROMETHEUS.md +++ b/docs/PROMETHEUS.md @@ -18,9 +18,25 @@ NOTE: To enable prometheus metrics in twistlock: -``` +```bash cd app/monitoring/prometheus ``` -``` +```bash kubectl apply -k . +``` + +## ServiceMonitor + +Twistlock Prometheus Monitoring is implemented as per the documentation + +https://docs.paloaltonetworks.com/prisma/prisma-cloud/prisma-cloud-admin-compute/audit/prometheus.html + + +1. Create ServiceMonitor for twistlock endpoint + +2. Create Role, RoleBinding for monitoring in twistlock namespace + +3. Create Secrets for metrics point authentication. Secrets in this repo are examples and should not be used. The Secrets should be in the pipeline secrets and be encrypted using the pipeline processes. + +4. kubectl apply -k prometheus diff --git a/docs/README.md b/docs/README.md index e6aa719af5c4696ec8ce8aca4fed7ae8cc33b318..b32cfa5d43903fe9fc5586ef0d0b431b53e538c5 100644 --- a/docs/README.md +++ b/docs/README.md @@ -36,13 +36,13 @@ Twistlock monitors Docker for container deployment and Kubernetes for container Install kubectl -``` +```bash brew install kubectl ``` Install kustomize -``` +```bash brew install kustomize ``` @@ -50,14 +50,14 @@ brew install kustomize Clone repository -``` +```bash git clone https://repo1.dsop.io/platform-one/apps/twistlock.git cd twstlock ``` Apply kustomized manifest -``` +```bash kubectl -k ./ ``` @@ -67,7 +67,7 @@ The application needs a administrator, the license file needs to be installed, t The Variables required are as follows: -``` +```bash //Environment $ ADMIN_USER=Administrator $ ADMIN_PASSWORD=< my password> @@ -82,7 +82,7 @@ This process requires kubectl to be installed and able to communicate with the D Initially there is no users associated with twistlock console. Go to the external URL and add an Administrator account and a password. Alternatively, run the following script: -``` +```bash //Add Administrator if ! curl -k -H 'Content-Type: application/json' -X POST \ -d "{\"username\": \"$ADMIN_USER\", \"password\": \"$ADMIN_PASSWORD\"}" \ @@ -97,7 +97,7 @@ fi The License can be added directly from the TWISTLOCK_EXTERNAL_ROUTE. When first logging in the admin user will be prompted for a license. The following script will install the license: -``` +```bash //License if ! curl -k \ -u $ADMIN_USER:$ADMIN_PASSWORD \ @@ -117,28 +117,34 @@ Notes: curl has some difficulties with special characters. During the initial s Defender can be installed from console, script or by command line. The twistlock CLI is provided as a part of the installation. This can be found in the Manage/System/Download. After download ensure the file is made executable. -``` +```bash chmod +x twistcli ``` The "Bearer" token can be found in the twistlock application Manage/Authorization/User Certificates. Alternatively, run the following commands -``` +```bash //Windows twistcli: -curl --progress-bar -L -k --header "authorization: Bearer $TOKEN" https://twistlock.fences.dsop.io/api/v1/util/windows/twistcli.exe > twistcli.exe; +curl --progress-bar -L -k --header "authorization: Bearer$TOKEN" \ +https://twistlock.fences.dsop.io/api/v1/util/windows/twistcli.exe \ +> twistcli.exe; ``` -``` +```bash Linux twistcli: -curl --progress-bar -L -k --header "authorization: Bearer $TOKEN" https://twistlock.fences.dsop.io/api/v1/util/twistcli > twistcli; chmod a+x twistcli; +curl --progress-bar -L -k --header "authorization: Bearer $TOKEN" \ +https://twistlock.fences.dsop.io/api/v1/util/twistcli > twistcli; \ +chmod a+x twistcli; ``` -``` +```bash Mac OS twistcli: -curl --progress-bar -L -k --header "authorization: Bearer TOKEN" https://twistlock.fences.dsop.io/api/v1/util/osx/twistcli > twistcli; chmod a+x twistcli; +curl --progress-bar -L -k --header "authorization: Bearer TOKEN" \ +https://twistlock.fences.dsop.io/api/v1/util/osx/twistcli \ +> twistcli; chmod a+x twistcli; ``` #### Install Defender @@ -147,19 +153,26 @@ curl --progress-bar -L -k --header "authorization: Bearer TOKEN" https://twistlo The following command can be authenticated by TOKEN or Username/Password. -``` -./twistcli defender export kubernetes --namespace twistlock --privileged --cri --monitor-service-accounts --monitor-istio --user $ADMIN_USER --password $ADMIN_PASSWORD --address https://$TWISTLOCK_EXTERNAL_ROUTE --cluster-address twistlock-console:8084 +```bash +./twistcli defender export kubernetes \ +--namespace twistlock \ +--privileged --cri \ +--monitor-service-accounts \ +--monitor-istio --user $ADMIN_USER \ +--password $ADMIN_PASSWORD \ +--address https://$TWISTLOCK_EXTERNAL_ROUTE \ +--cluster-address twistlock-console:8084 ``` ##### Download the daemonset.yaml. The default Image is set to the Prisma server. The image should be hardened. To pull images from Platform 1. The image URL needs to be changed -##### Image: registry.dsop.io/platform-one/apps/twistlock/defender:20.04.163_ib +##### Image: Use the harbor image Note: The Console and Defender must use the same version. If your deploymnet is using 20.04.169 then edit the image accordingly. 2) Install Defender -``` +```bash kubectl apply -f defender.yaml ``` diff --git a/RELEASENOTES.md b/docs/RELEASENOTES.md similarity index 100% rename from RELEASENOTES.md rename to docs/RELEASENOTES.md diff --git a/twistlock_setup/saml_example.json.md b/docs/saml_example.json.md similarity index 100% rename from twistlock_setup/saml_example.json.md rename to docs/saml_example.json.md diff --git a/twistlock_setup/twistlock_setup.sh b/docs/twistlock_setup.md similarity index 51% rename from twistlock_setup/twistlock_setup.sh rename to docs/twistlock_setup.md index 0063a77add3f4957a52085959747707119fc7aaf..a0f3fb5368097cca9d0da443f70cedf921e8037e 100755 --- a/twistlock_setup/twistlock_setup.sh +++ b/docs/twistlock_setup.md @@ -1,13 +1,14 @@ -#!/bin/bash - -#The "Bearer" token can be found in the twistlock application Manage/Authorization/User Certificates. Alternatively, run the following commands +#Twistlock Setup notes +###The "Bearer" token can be found in the twistlock application Manage/Authorization/User Certificates. Alternatively, run the following commands +```html TWISTLOCK_CONSOLE_USER=Administrator TWISTLOCK_CONSOLE_PASSWORD=Passw0rd! # Don't use thes TWISTLOCK_EXTERNAL_ROUTE=twistlock.fences.dsop.io +``` +### set Twistlock console user/pass -# set Twistlock console user/pass - +```html if ! curl -k -H 'Content-Type: application/json' -X POST \ -d "{\"username\": \"$TWISTLOCK_CONSOLE_USER\", \"password\": \"$TWISTLOCK_CONSOLE_PASSWORD\"}" \ https://$TWISTLOCK_EXTERNAL_ROUTE/api/v1/signup; then @@ -15,9 +16,11 @@ if ! curl -k -H 'Content-Type: application/json' -X POST \ echo "Error creating Twistlock Console user $TWISTLOCK_CONSOLE_USER" exit 1 fi +``` -# Set Twistlock license. Using default user/pass +###Set Twistlock license. Using default user/pass +```html if ! curl -k \ -u $TWISTLOCK_CONSOLE_USER:$TWISTLOCK_CONSOLE_PASSWORD \ -H 'Content-Type: application/json' \ @@ -28,22 +31,29 @@ if ! curl -k \ echo "Error uploading Twistlock license to console" exit 1 fi - - - -curl -sSLk -u $TWISTLOCK_CONSOLE_USER:$TWISTLOCK_CONSOLE_PASSWORD https://$TWISTLOCK_EXTERNAL_ROUTE/api/v1/util/twistcli > twistcli +``` +```html +curl -sSLk -u $TWISTLOCK_CONSOLE_USER:$TWISTLOCK_CONSOLE_PASSWORD \ +https://$TWISTLOCK_EXTERNAL_ROUTE/api/v1/util/twistcli > twistcli chmod +x ./twistcli - -# Change the image tag to reflect Platform One registry: -# registry.dsop.io/platform-one/apps/twistlock/defender:20.04.169 - -./twistcli defender export kubernetes --namespace twistlock --privileged --cri --monitor-service-accounts --monitor-istio --user $TWISTLOCK_CONSOLE_USER --password $TWISTLOCK_CONSOLE_PASSWORD --address https://$TWISTLOCK_EXTERNAL_ROUTE --cluster-address twistlock-console:8084 - -# kubectl apply -f ./defender +``` +### Change the image tag to reflect Platform One registry: +There isn't a harbor image, The curretni iron Bank image is located: +##### registry.dsop.io/platform-one/apps/twistlock/defender:20.04.169 +```html +./twistcli defender export kubernetes \ +--namespace twistlock --privileged \ +--cri --monitor-service-accounts --monitor-istio \ +--user ``$TWISTLOCK_CONSOLE_USER`` --password ``$TWISTLOCK_CONSOLE_PASSWORD`` \ +--address https://``$TWISTLOCK_EXTERNAL_ROUTE`` --cluster-address\ +twistlock-console:8084 +``` +### kubectl apply -f ./defender +```html #setup logging to stdout if ! curl -k \ - -u $TWISTLOCK_CONSOLE_USER:$TWISTLOCK_CONSOLE_PASSWORD \ + -u ``$TWISTLOCK_CONSOLE_USER:$TWISTLOCK_CONSOLE_PASSWORD`` \ -H 'Content-Type: application/json' \ -X POST \ -d \ @@ -54,9 +64,10 @@ if ! curl -k \ "allProcEvents": true, } }' \ - https://$TWISTLOCK_EXTERNAL_ROUTE/api/v1/settings/logging; then + https://``$TWISTLOCK_EXTERNAL_ROUTE``/api/v1/settings/logging; then echo "Error editing syslog settings on console" exit 1 fi +``` diff --git a/app/console/kustomization.yaml b/manifests/console/kustomization.yaml similarity index 100% rename from app/console/kustomization.yaml rename to manifests/console/kustomization.yaml diff --git a/app/console/twistlock_console.yaml b/manifests/console/twistlock_console.yaml similarity index 100% rename from app/console/twistlock_console.yaml rename to manifests/console/twistlock_console.yaml diff --git a/manifests/kustomization.yaml b/manifests/kustomization.yaml new file mode 100644 index 0000000000000000000000000000000000000000..6120fe72c9bc16c9ab4b72493b3baface414d712 --- /dev/null +++ b/manifests/kustomization.yaml @@ -0,0 +1,8 @@ +namespace: twistlock + +# This needs to be patched in for the pipeline. The secrets must be added to the secrets folder which will not be in the public repo. +# bases: +# - monitoring/prometheus + +resources: + - console/twistlock_console.yaml diff --git a/manifests/monitoring/prometheus/kustomization.yaml b/manifests/monitoring/prometheus/kustomization.yaml new file mode 100644 index 0000000000000000000000000000000000000000..f25d1314022306a6f0449aa0f65658be3c4c5557 --- /dev/null +++ b/manifests/monitoring/prometheus/kustomization.yaml @@ -0,0 +1,8 @@ +namespace: twistlock + +resources: +- twistlockServiceMonitor.yaml +- role.yaml +- roleBinding.yaml +# This secrets file needs to be located in your pipeline secrets folder +# - twistlockSecret.yaml \ No newline at end of file diff --git a/app/monitoring/prometheus/role.yaml b/manifests/monitoring/prometheus/role.yaml similarity index 100% rename from app/monitoring/prometheus/role.yaml rename to manifests/monitoring/prometheus/role.yaml diff --git a/app/monitoring/prometheus/roleBinding.yaml b/manifests/monitoring/prometheus/roleBinding.yaml similarity index 100% rename from app/monitoring/prometheus/roleBinding.yaml rename to manifests/monitoring/prometheus/roleBinding.yaml diff --git a/app/monitoring/prometheus/twistlockServiceMonitor.yaml b/manifests/monitoring/prometheus/twistlockServiceMonitor.yaml similarity index 100% rename from app/monitoring/prometheus/twistlockServiceMonitor.yaml rename to manifests/monitoring/prometheus/twistlockServiceMonitor.yaml diff --git a/tests/cypress/health.spec.js b/tests/cypress/health.spec.js new file mode 100644 index 0000000000000000000000000000000000000000..ec523d4b2bc9332ffe9dcfdc9d67b7a09f4261d4 --- /dev/null +++ b/tests/cypress/health.spec.js @@ -0,0 +1,5 @@ +describe('Basic twistlock', function() { + it('Visits the twistlock sign in page', function() { + cy.visit(`https://twistlock.fences.dsop.io`) + }) + })