From 2a4c3e995eec94c19af5f85214a5daf3ffd34fdd Mon Sep 17 00:00:00 2001 From: Joshua Eason Date: Mon, 26 Oct 2020 14:38:45 +0000 Subject: [PATCH 01/27] Removing CC lines in issue templates --- .gitlab/issue_templates/Application - Archive.md | 3 +-- .gitlab/issue_templates/Application - Initial.md | 3 +-- .gitlab/issue_templates/Application - Update.md | 3 +-- .gitlab/issue_templates/Bug.md | 3 +-- .gitlab/issue_templates/Feature Request.md | 3 +-- .gitlab/issue_templates/New Findings.md | 3 +-- .gitlab/issue_templates/Pipeline Failure.md | 3 +-- README.md | 2 +- 8 files changed, 8 insertions(+), 15 deletions(-) diff --git a/.gitlab/issue_templates/Application - Archive.md b/.gitlab/issue_templates/Application - Archive.md index 9f3b5fe..a558faa 100644 --- a/.gitlab/issue_templates/Application - Archive.md +++ b/.gitlab/issue_templates/Application - Archive.md @@ -17,5 +17,4 @@ Requesting this application be archived due to one of the following reasons: - [ ] Iron Bank frontend no longer lists application as available or approved -/label ~"Container::Archive" -/cc @ironbank-notifications/archive \ No newline at end of file +/label ~"Container::Archive" \ No newline at end of file diff --git a/.gitlab/issue_templates/Application - Initial.md b/.gitlab/issue_templates/Application - Initial.md index 6594a05..b7acebd 100644 --- a/.gitlab/issue_templates/Application - Initial.md +++ b/.gitlab/issue_templates/Application - Initial.md @@ -28,5 +28,4 @@ Approval Process (container hardening team processes): -/label ~"Container::Initial" -/cc @ironbank-notifications/cht \ No newline at end of file +/label ~"Container::Initial" \ No newline at end of file diff --git a/.gitlab/issue_templates/Application - Update.md b/.gitlab/issue_templates/Application - Update.md index caebb3e..d95cb35 100644 --- a/.gitlab/issue_templates/Application - Update.md +++ b/.gitlab/issue_templates/Application - Update.md @@ -31,5 +31,4 @@ Approval Process: -/label ~"Container::Update" -/cc @ironbank-notifications/updates \ No newline at end of file +/label ~"Container::Update" \ No newline at end of file diff --git a/.gitlab/issue_templates/Bug.md b/.gitlab/issue_templates/Bug.md index 1427a0c..2030c24 100644 --- a/.gitlab/issue_templates/Bug.md +++ b/.gitlab/issue_templates/Bug.md @@ -33,5 +33,4 @@ logs, and code as it's very hard to read otherwise.) - [ ] Bug has been identified and corrected within the container -/label ~Bug -/cc @ironbank-notifications/bug \ No newline at end of file +/label ~Bug \ No newline at end of file diff --git a/.gitlab/issue_templates/Feature Request.md b/.gitlab/issue_templates/Feature Request.md index a0e2f19..94aea9a 100644 --- a/.gitlab/issue_templates/Feature Request.md +++ b/.gitlab/issue_templates/Feature Request.md @@ -28,5 +28,4 @@ - [ ] Feature has been implemented -/label ~Feature -/cc @ironbank-notifications/feature \ No newline at end of file +/label ~Feature \ No newline at end of file diff --git a/.gitlab/issue_templates/New Findings.md b/.gitlab/issue_templates/New Findings.md index 068d029..1fd613d 100644 --- a/.gitlab/issue_templates/New Findings.md +++ b/.gitlab/issue_templates/New Findings.md @@ -16,5 +16,4 @@ Approval Process: -/label ~"Container::New Findings" -/cc @ironbank-notifications/security \ No newline at end of file +/label ~"Container::New Findings" \ No newline at end of file diff --git a/.gitlab/issue_templates/Pipeline Failure.md b/.gitlab/issue_templates/Pipeline Failure.md index 28b82a9..dd6ab98 100644 --- a/.gitlab/issue_templates/Pipeline Failure.md +++ b/.gitlab/issue_templates/Pipeline Failure.md @@ -27,5 +27,4 @@ - [ ] Pipeline failure has been resolved -/label ~Pipeline -/cc @ironbank-notifications/pipelines \ No newline at end of file +/label ~Pipeline \ No newline at end of file diff --git a/README.md b/README.md index f2a2b88..5dc6fa6 100644 --- a/README.md +++ b/README.md @@ -1,3 +1,3 @@ -# master-project-template +# Project template for all Iron Bank container repositories. \ No newline at end of file -- GitLab From fb4c993c0c29da108d218f708cc200aa95af9514 Mon Sep 17 00:00:00 2001 From: Al Fontaine Date: Mon, 26 Oct 2020 15:35:38 +0000 Subject: [PATCH 02/27] Update Application - Update.md --- .gitlab/issue_templates/Application - Update.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/.gitlab/issue_templates/Application - Update.md b/.gitlab/issue_templates/Application - Update.md index d95cb35..15567be 100644 --- a/.gitlab/issue_templates/Application - Update.md +++ b/.gitlab/issue_templates/Application - Update.md @@ -19,6 +19,9 @@ Hardening: - [ ] Container version has been updated in greylist file - [ ] Branch has been merged into `development` +No new findings: +- [ ] There are no new findings in this update. Skip the Justifications and Approval Process steps and apply the label ~"Approval". + Justifications: - [ ] All findings have been justified per the above documentation - [ ] Justifications have been provided to the container hardening team -- GitLab From cdc6ae3b1b07606657897ebd0c639dc170b02b62 Mon Sep 17 00:00:00 2001 From: Jinoy Parekh Date: Tue, 10 Nov 2020 22:07:05 +0000 Subject: [PATCH 03/27] Add LICENSE --- LICENSE | 201 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 201 insertions(+) create mode 100644 LICENSE diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..9c8f3ea --- /dev/null +++ b/LICENSE @@ -0,0 +1,201 @@ + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "{}" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright {yyyy} {name of copyright owner} + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. \ No newline at end of file -- GitLab From 680c4986ef66ac4d4e9caee8bd6f3f7fd1f60e68 Mon Sep 17 00:00:00 2001 From: Jinoy Parekh Date: Tue, 10 Nov 2020 22:09:51 +0000 Subject: [PATCH 04/27] Update README.md --- README.md | 679 +++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 677 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 5dc6fa6..37a8be8 100644 --- a/README.md +++ b/README.md @@ -1,3 +1,678 @@ -# +# AKHQ (previously known as KafkaHQ) -Project template for all Iron Bank container repositories. \ No newline at end of file +![Last Version](https://img.shields.io/github/tag-pre/tchiotludo/akhq.svg) +![License](https://img.shields.io/github/license/tchiotludo/akhq) +![Docker Pull](https://img.shields.io/docker/pulls/tchiotludo/akhq.svg) +![Github Downloads](https://img.shields.io/github/downloads/tchiotludo/akhq/total) +![Github Start](https://img.shields.io/github/stars/tchiotludo/akhq.svg) +![Main](https://github.com/tchiotludo/akhq/workflows/Main/badge.svg) +[![Artifact HUB](https://img.shields.io/endpoint?url=https://artifacthub.io/badge/repository/akhq)](https://artifacthub.io/packages/search?repo=akhq) + +> Kafka GUI for [Apache Kafka](http://kafka.apache.org/) to manage topics, topics data, consumers group, schema registry, connect and more... + +

+ AKHQ for Kafka logo

+ AKHQ for Kafka preview +

+ +## Contents + +- [Features](#features) +- [Quick Preview](#quick-preview) +- [Installation](#installation) + - [Docker](#docker) + - [Stand Alone](#stand-alone) + - [Kubernetes using Helm](#running-in-kubernetes-using-a-helm-chart) +- [Configuration](#configuration) + - [JVM.options file](#run-with-another-jvmoptions-file) + - [Kafka cluster](#kafka-cluster-configuration) + - [AKHQ](#akhq-configuration) + - [Security](#security) + - [Server](#server) + - [Micronaut](#micronaut-configuration) +- [Api](#api) +- [Monitoring Endpoint](#monitoring-endpoint) +- [Development Environment](#development-environment) +- [Schema references](#schema-references) +- [Who's using AKHQ](#whos-using-akhq) + + +## Features + +- **General** + - Works with modern Kafka cluster (1.0+) + - Connection on standard or ssl, sasl cluster + - Multi cluster +- **Topics** + - List + - Configurations view + - Partitions view + - ACLS view + - Consumers groups assignments view + - Node leader & assignments view + - Create a topic + - Configure a topic + - Delete a topic +- **Browse Topic datas** + - View data, offset, key, timestamp & headers + - Automatic deserializarion of avro message encoded with schema registry + - Configurations view + - Logs view + - Delete a record + - Empty a Topic (Delete all the record from one topic) + - Sort view + - Filter per partitions + - Filter with a starting time + - Filter data with a search string +- **Consumer Groups** (only with kafka internal storage, not with old Zookeeper) + - List with lag, topics assignments + - Partitions view & lag + - ACLS view + - Node leader & assignments view + - Display active and pending consumers groups + - Delete a consumer group + - Update consumer group offsets to start / end / timestamp +- **Schema Registry** + - List schema + - Create / Update / Delete a schema + - View and delete individual schema version +- **Connect** + - List connect definition + - Create / Update / Delete a definition + - Pause / Resume / Restart a definition or a task +- **Nodes** + - List + - Configurations view + - Logs view + - Configure a node +- **ACLS** + - List principals + - List principals topic & group acls +- **Authentification and Roles** + - Read only mode + - BasicHttp with roles per user + - User groups configuration + - Filter topics with regexp for current groups + - Ldap configuration to match AKHQ groups/roles + +## New React UI + +Since this is a major rework, the new UI can have some issues, so please [report any issue](https://github.com/tchiotludo/akhq/issues), thanks! + +## Quick preview +* Download [docker-compose.yml](https://raw.githubusercontent.com/tchiotludo/akhq/master/docker-compose.yml) file +* run `docker-compose pull` to be sure to have the last version of AKHQ +* run `docker-compose up` +* go to [http://localhost:8080](http://localhost:8080) + +It will start a Kafka node, a Zookeeper node, a Schema Registry, a Connect, fill with some sample data, start a consumer +group and a kafka stream & start AKHQ. + +## Installation + +First you need a [configuration files](#configuration) in order to configure AKHQ connections to Kafka Brokers. + +### Docker + +```sh +docker run -d \ + -p 8080:8080 \ + -v /tmp/application.yml:/app/application.yml \ + tchiotludo/akhq +``` +* With `-v /tmp/application.yml` must be an absolute path to configuration file +* Go to + + +### Stand Alone +* Install Java 11 +* Download the latest jar on [release page](https://github.com/tchiotludo/akhq/releases) +* Create an [configuration files](#configuration) +* Launch the application with `java -Dmicronaut.config.files=/path/to/application.yml -jar akhq.jar` +* Go to + + +### Running in Kubernetes (using a Helm Chart) + +### Using Helm repository + +* Add the AKHQ helm charts repository: +```sh +helm repo add akhq https://akhq.io/ +``` +* Install or upgrade +```sh +helm upgrade --install akhq akhq/akhq +``` +#### Requirements + +* Chart version >=0.1.1 requires Kubernetes version >=1.14 +* Chart version 0.1.0 works on previous Kubernetes versions +```sh +helm install akhq akhq/akhq --version 0.1.0 +``` + +### Using git +* Clone the repository: +```sh +git clone https://github.com/tchiotludo/akhq && cd akhq/deploy/helm/akhq +``` +* Update helm values located in [deploy/helm/values.yaml](helm/akhq/values.yaml) + * `configuration` values will contains all related configuration that you can find in [application.example.yml](application.example.yml) and will be store in a `ConfigMap` + * `secrets` values will contains all sensitive configurations (with credentials) that you can find in [application.example.yml](application.example.yml) and will be store in `Secret` + * Both values will be merged at startup +* Apply the chart: +```sh +helm install --name=akhq-release-name . +``` + + +## Configuration +Configuration file can by default be provided in either Java properties, YAML, JSON or Groovy files. YML Configuration +file example can be found here :[application.example.yml](application.example.yml) + +### Pass custom Java opts + +By default, the docker container will allow a custom jvn options setting the environnments vars `JAVA_OPTS`. +For example, if you want to change the default timezome, just add `-e "JAVA_OPTS=-Duser.timezone=Europe/Paris"` + +### Run with another jvm.options file + +By default, the docker container will run with a [jvm.options](docker/app/jvm.options) file, you can override it with +your own with an Environment Variable. With the `JVM_OPTS_FILE` environment variable, you can override the jvm.options file by passing +the path of your file instead. + +Override the `JVM_OPTS_FILE` with docker run: + +```sh +docker run -d \ + --env JVM_OPTS_FILE={{path-of-your-jvm.options-file}} + -p 8080:8080 \ + -v /tmp/application.yml:/app/application.yml \ + tchiotludo/akhq +``` + +Override the `JVM_OPTS_FILE` with docker-compose: + +```yaml +version: '3.7' +services: + akhq: + image: tchiotludo/akhq-jvm:dev + environment: + JVM_OPTS_FILE: /app/jvm.options + ports: + - "8080:8080" + volumes: + - /tmp/application.yml:/app/application.yml +``` + +If you do not override the `JVM_OPTS_FILE`, the docker container will take the defaults one instead. + +### Kafka cluster configuration +* `akhq.connections` is a key value configuration with : + * `key`: must be an url friendly (letter, number, _, -, ... dot are not allowed here) string the identify your cluster (`my-cluster-1` and `my-cluster-2` is the example above) + * `properties`: all the configurations found on [Kafka consumer documentation](https://kafka.apache.org/documentation/#consumerconfigs). Most important is `bootstrap.servers` that is a list of host:port of your Kafka brokers. + * `schema-registry`: *(optional)* + * `url`: the schema registry url + * `basic-auth-username`: schema registry basic auth username + * `basic-auth-password`: schema registry basic auth password + * `properties`: all the configurations for registry client, especially ssl configuration + * `connect`: *(optional list, define each connector as a element of a list)* + * `name`: connect name + * `url`: connect url + * `basic-auth-username`: connect basic auth username + * `basic-auth-password`: connect basic auth password + * `ssl-trust-store`: /app/truststore.jks + * `ssl-trust-store-password`: trust-store-password + * `ssl-key-store`: /app/truststore.jks + * `ssl-key-store-password`: key-store-password + +#### SSL Kafka Cluster with basic auth +Configuration example for kafka cluster secured by ssl for saas provider like aiven (full https & basic auth): + +You need to generate a jks & p12 file from pem, cert files give by saas provider. +```bash +openssl pkcs12 -export -inkey service.key -in service.cert -out client.keystore.p12 -name service_key +keytool -import -file ca.pem -alias CA -keystore client.truststore.jks +``` + +Configurations will look like this example: + +```yaml +akhq: + connections: + ssl-dev: + properties: + bootstrap.servers: "{{host}}.aivencloud.com:12835" + security.protocol: SSL + ssl.truststore.location: {{path}}/avnadmin.truststore.jks + ssl.truststore.password: {{password}} + ssl.keystore.type: "PKCS12" + ssl.keystore.location: {{path}}/avnadmin.keystore.p12 + ssl.keystore.password: {{password}} + ssl.key.password: {{password}} + schema-registry: + url: "https://{{host}}.aivencloud.com:12838" + basic-auth-username: avnadmin + basic-auth-password: {{password}} + properties: {} + connect: + - name: connect-1 + url: "https://{{host}}.aivencloud.com:{{port}}" + basic-auth-username: avnadmin + basic-auth-password: {{password}} +``` + +### AKHQ configuration + +#### Pagination +* `akhq.pagination.page-size` number of topics per page (default : 25) + +#### Topic List +* `akhq.topic.default-view` is default list view (ALL, HIDE_INTERNAL, HIDE_INTERNAL_STREAM, HIDE_STREAM) +* `akhq.topic.internal-regexps` is list of regexp to be considered as internal (internal topic can't be deleted or updated) +* `akhq.topic.stream-regexps` is list of regexp to be considered as internal stream topic + +#### Topic creation default values + +These parameters are the default values used in the topic creation page. + +* `akhq.topic.retention` Default retention in ms +* `akhq.topic.replication` Default number of replica to use +* `akhq.topic.partition` Default number of partition + +#### Topic Data +* `akhq.topic-data.sort`: default sort order (OLDEST, NEWEST) (default: OLDEST) +* `akhq.topic-data.size`: max record per page (default: 50) +* `akhq.topic-data.poll-timeout`: The time, in milliseconds, spent waiting in poll if data is not available in the + buffer (default: 1000). + + +### Security +* `akhq.security.default-group`: Default group for all the user even unlogged user. +By default, the default group is `admin` and allow you all read / write access on the whole app. + +By default, security & roles is enabled by default but anonymous user have full access. You can completely disabled +security with `micronaut.security.enabled: false`. + +If you need a read-only application, simply add this to your configuration files : +```yaml +akhq: + security: + default-group: reader +``` + + + +#### Auth + +##### JWT + +AKHQ uses JWT tokens to perform authentication. +Please generate a secret that is at least 256 bits and change the config like this: + +```yaml +micronaut: + security: + enabled: true + token: + jwt: + signatures: + secret: + generator: + secret: +``` + +##### Groups + +Groups allow you to limit user + +Define groups with specific roles for your users +* `akhq.security.default-group`: Default group for all the user even unlogged user + +* `akhq.security.groups`: Groups list definition + * `- name: group-name` Group identifier + * `roles`: Roles list for the group + * `attributes.topics-filter-regexp`: Regexp to filter topics available for current group + * `attributes.connects-filter-regexp`: Regexp to filter Connect tasks available for current group + + +3 defaults group are available : +- `admin` with all right +- `reader` with only read acces on all AKHQ +- `no-roles` without any roles, that force user to login + +##### Basic Auth +* `akhq.security.basic-auth`: List user & password with affected roles + * `- username: actual-username`: Login of the current user as a yaml key (may be anything email, login, ...) + * `password`: Password in sha256 (default) or bcrypt. The password can be converted + * For default SHA256, with command `echo -n "password" | sha256sum` or Ansible filter `{{ 'password' | hash('sha256') }}` + * For BCrypt, with Ansible filter `{{ 'password' | password_hash('blowfish') }}` + * `passwordHash`: Password hashing algorithm, either `SHA256` or `BCRYPT` + * `groups`: Groups for current user + +> Take care that basic auth will use session store in server **memory**. If your instance is behind a reverse proxy or a +> loadbalancer, you will need to forward the session cookie named `SESSION` and / or use +> [sesssion stickiness](https://en.wikipedia.org/wiki/Load_balancing_(computing)#Persistence) + +Configure basic-auth connection in AKHQ +```yaml +akhq.security: + basic-auth: + - username: admin + password: "$2a$" + passwordHash: BCRYPT + groups: + - admin + - username: reader + password: "" + groups: + - reader +``` + +##### LDAP +Configure how the ldap groups will be matched in AKHQ groups +* `akhq.security.ldap.groups`: Ldap groups list + * `- name: ldap-group-name`: Ldap group name (same name as in ldap) + * `groups`: AKHQ group list to be used for current ldap group + +Example using [online ldap test server](https://www.forumsys.com/tutorials/integration-how-to/ldap/online-ldap-test-server/) + +Configure ldap connection in micronaut +```yaml +micronaut: + security: + ldap: + default: + enabled: true + context: + server: 'ldap://ldap.forumsys.com:389' + managerDn: 'cn=read-only-admin,dc=example,dc=com' + managerPassword: 'password' + search: + base: "dc=example,dc=com" + groups: + enabled: true + base: "dc=example,dc=com" +``` + +If you want to enable anonymous auth to your LDAP server you can pass : +```yaml +managerDn: '' +managerPassword: '' +``` + +Debuging ldap connection can be done with +```bash +curl -i -X POST -H "Content-Type: application/json" \ + -d '{ "configuredLevel": "TRACE" }' \ + http://localhost:8080/loggers/io.micronaut.configuration.security +``` + + +Configure AKHQ groups and Ldap groups and users +```yaml +akhq: + security: + groups: + - name: topic-reader # Group name + roles: # roles for the group + - topic/read + attributes: + # Regexp to filter topic available for group + topics-filter-regexp: "test\\.reader.*" + connects-filter-regexp: "^test.*$" + - name: topic-writer # Group name + roles: + - topic/read + - topic/insert + - topic/delete + - topic/config/update + attributes: + topics-filter-regexp: "test.*" + connects-filter-regexp: "^test.*$" + ldap: + groups: + - name: mathematicians + groups: + - topic-reader + - name: scientists + groups: + - topic-reader + - topic-writer + users: + - username: franz + groups: + - topic-reader + - topic-writer + +``` + +### OIDC +To enable OIDC in the application, you'll first have to enable OIDC in micronaut: + +```yaml +micronaut: + security: + oauth2: + enabled: true + clients: + google: + client-id: "" + client-secret: "" + openid: + issuer: "" +``` + +To further tell AKHQ to display OIDC options on the login page and customize claim mapping, configure OIDC in the AKHQ config: + +```yaml +akhq: + security: + oidc: + enabled: true + providers: + google: + label: "Login with Google" + username-field: preferred_username + groups-field: roles + default-group: topic-reader + groups: + - name: mathematicians + groups: + - topic-reader + - name: scientists + groups: + - topic-reader + - topic-writer + users: + - username: franz + groups: + - topic-reader + - topic-writer +``` + +The username field can be any string field, the roles field has to be a JSON array. + +### Server +* `micronaut.server.context-path`: if behind a reverse proxy, path to akhq with trailing slash (optional). Example: + akhq is behind a reverse proxy with url , set base-path: "/akhq/". Not needed if you're + behind a reverse proxy with subdomain + +### Kafka admin / producer / consumer default properties +* `akhq.clients-defaults.{{admin|producer|consumer}}.properties`: default configuration for admin producer or + consumer. All properties from [Kafka documentation](https://kafka.apache.org/documentation/) is available. + +### Micronaut configuration +> Since AKHQ is based on [Micronaut](https://micronaut.io/), you can customize configurations (server port, ssl, ...) with [Micronaut configuration](https://docs.micronaut.io/snapshot/guide/configurationreference.html#io.micronaut.http.server.HttpServerConfiguration). +> More information can be found on [Micronaut documentation](https://docs.micronaut.io/snapshot/guide/index.html#config) + +### Docker +AKHQ docker image support 3 environment variables to handle configuraiton : +* `AKHQ_CONFIGURATION`: a string that contains the full configuration in yml that will be written on + /app/configuration.yml on container. +* `MICRONAUT_APPLICATION_JSON`: a string that contains the full configuration in JSON format +* `MICRONAUT_CONFIG_FILES`: a path to to a configuration file on container. Default path is `/app/application.yml` + +#### How to mount configuration file + +Take care when you mount configuration files to not remove akhq files located on /app. +You need to explicitely mount the `/app/application.yml` and not mount the `/app` directory. +This will remove the AKHQ binnaries and give you this error: ` +/usr/local/bin/docker-entrypoint.sh: 9: exec: ./akhq: not found` + +```yaml +volumeMounts: +- mountPath: /app/application.yml + subPath: application.yml + name: config + readOnly: true + +``` + +## Api +An **experimental** api is available that allow you to fetch all the exposed on AKHQ through api. + +Take care that this api is **experimental** and **will** change in a future release. +Some endpoint expose too many datas and is slow to fetch, and we will remove +some properties in a future in order to be fast. + +Example: List topic endpoint expose log dir, consumer groups, offsets. Fetching all of theses +is slow for now and we will remove these in a future. + +You can discover the api endpoint here : +* `/api`: a [RapiDoc](https://mrin9.github.io/RapiDoc/) webpage that document all the endpoints. +* `/swagger/akhq.yml`: a full [OpenApi](https://www.openapis.org/) specifications files + +## Monitoring endpoint +Several monitoring endpoint is enabled by default. You can disabled it or restrict access only for authenticated users +following micronaut configuration below. + +* `/info` [Info Endpoint](https://docs.micronaut.io/snapshot/guide/index.html#infoEndpoint) with git status + informations. +* `/health` [Health Endpoint](https://docs.micronaut.io/snapshot/guide/index.html#healthEndpoint) +* `/loggers` [Loggers Endpoint](https://docs.micronaut.io/snapshot/guide/index.html#loggersEndpoint) +* `/metrics` [Metrics Endpoint](https://docs.micronaut.io/snapshot/guide/index.html#metricsEndpoint) +* `/prometheus` [Prometheus Endpoint](https://micronaut-projects.github.io/micronaut-micrometer/latest/guide/) + +## Debugging AKHQ performance issues + +You can debug all query duration from AKHQ with this commands +```bash +curl -i -X POST -H "Content-Type: application/json" \ + -d '{ "configuredLevel": "TRACE" }' \ + http://localhost:8080/loggers/org.akhq +``` + +## Development Environment + +### Early dev image + +You can have access to last feature / bug fix with docker dev image automatically build on tag `dev` +```bash +docker pull tchiotludo/akhq:dev +``` + +The dev jar is not publish on GitHub, you have 2 solutions to have the `dev` jar : + +Get it from docker image +```bash +docker pull tchiotludo/akhq:dev +docker run --rm --name=akhq -it tchiotludo/akhq:dev +docker cp akhq:/app/akhq.jar . +``` +Or build it with a `./gradlew shadowJar`, the jar will be located here `build/libs/akhq-*.jar` + + +### Development Server + +A docker-compose is provide to start a development environnement. +Just install docker & docker-compose, clone the repository and issue a simple `docker-compose -f docker-compose-dev.yml up` to start a dev server. +Dev server is a java server & webpack-dev-server with live reload. + +The configuration for the dev server is in `application.dev.yml`. + +## Schema references + +Since Confluent 5.5.0, Avro schemas can now be reused by others schemas through schema references. This feature allows to define a schema once and use it as a record type inside one or more schemas. + +When registering new Avro schemas with AKHQ UI, it is now possible to pass a slightly more complex object with a `schema` and a `references` field. + +To register a new schema without references, no need to change anything: + +```json +{ + "name": "Schema1", + "namespace": "org.akhq", + "type": "record", + "fields": [ + { + "name": "description", + "type": "string" + } + ] +} +``` + +To register a new schema with a reference to an already registered schema: + +```json +{ + "schema": { + "name": "Schema2", + "namespace": "org.akhq", + "type": "record", + "fields": [ + { + "name": "name", + "type": "string" + }, + { + "name": "schema1", + "type": "Schema1" + } + ] + }, + "references": [ + { + "name": "Schema1", + "subject": "SCHEMA_1", + "version": 1 + } + ] +} +```` + +Documentation on Confluent 5.5 and schema references can be found [here](https://docs.confluent.io/5.5.0/schema-registry/serdes-develop/index.html). + + +## Who's using AKHQ +* [Adeo](https://www.adeo.com/) +* [Auchan Retail](https://www.auchan-retail.com/) +* [Bell](https://www.bell.ca) +* [BMW Group](https://www.bmwgroup.com) +* [Boulanger](https://www.boulanger.com/) +* [GetYourGuide](https://www.getyourguide.com) +* [Klarna](https://www.klarna.com) +* [La Redoute](https://laredoute.io/) +* [Leroy Merlin](https://www.leroymerlin.fr/) +* [NEXT Technologies](https://www.nextapp.co/) +* [Nuxeo](https://www.nuxeo.com/) +* [Pipedrive](https://www.pipedrive.com) +* [BARMER](https://www.barmer.de/) +* [TVG](https://www.tvg.com) + + +## Credits + +Many thanks to: + +* [JetBrains](https://www.jetbrains.com/?from=AKHQ) for their free OpenSource license. +* Apache, Apache Kafka, Kafka, and associated open source project names are trademarks of the Apache Software Foundation. AKHQ is not affiliated with, endorsed by, or otherwise associated with the Apache Software. + +[![Jetbrains](https://user-images.githubusercontent.com/2064609/55432917-6df7fc00-5594-11e9-90c4-5133fbb6d4da.png)](https://www.jetbrains.com/?from=AKHQ) + + +## License +Apache 2.0 © [tchiotludo](https://github.com/tchiotludo) \ No newline at end of file -- GitLab From 99e43541b6e211f9f7b5cc99fe5ba066953f3ce6 Mon Sep 17 00:00:00 2001 From: Jinoy Parekh Date: Tue, 10 Nov 2020 22:11:37 +0000 Subject: [PATCH 05/27] Update README.md --- README.md | 16 ---------------- 1 file changed, 16 deletions(-) diff --git a/README.md b/README.md index 37a8be8..15402c5 100644 --- a/README.md +++ b/README.md @@ -1,20 +1,4 @@ # AKHQ (previously known as KafkaHQ) - -![Last Version](https://img.shields.io/github/tag-pre/tchiotludo/akhq.svg) -![License](https://img.shields.io/github/license/tchiotludo/akhq) -![Docker Pull](https://img.shields.io/docker/pulls/tchiotludo/akhq.svg) -![Github Downloads](https://img.shields.io/github/downloads/tchiotludo/akhq/total) -![Github Start](https://img.shields.io/github/stars/tchiotludo/akhq.svg) -![Main](https://github.com/tchiotludo/akhq/workflows/Main/badge.svg) -[![Artifact HUB](https://img.shields.io/endpoint?url=https://artifacthub.io/badge/repository/akhq)](https://artifacthub.io/packages/search?repo=akhq) - -> Kafka GUI for [Apache Kafka](http://kafka.apache.org/) to manage topics, topics data, consumers group, schema registry, connect and more... - -

- AKHQ for Kafka logo

- AKHQ for Kafka preview -

- ## Contents - [Features](#features) -- GitLab From 9d559c8dc56d3407c8a4200091a2042959256a92 Mon Sep 17 00:00:00 2001 From: Jinoy Parekh Date: Tue, 10 Nov 2020 22:15:17 +0000 Subject: [PATCH 06/27] Add Jfile --- Jenkinsfile | 2 ++ 1 file changed, 2 insertions(+) create mode 100644 Jenkinsfile diff --git a/Jenkinsfile b/Jenkinsfile new file mode 100644 index 0000000..18f06b0 --- /dev/null +++ b/Jenkinsfile @@ -0,0 +1,2 @@ +@Library('DCCSCR@master') _ +dccscrPipeline(version: "0.16.0") -- GitLab From a58df1ce311de846b4d57939fcd82c4a3d1ba81b Mon Sep 17 00:00:00 2001 From: Jinoy Parekh Date: Tue, 10 Nov 2020 20:28:36 -0600 Subject: [PATCH 07/27] Updated Dfile --- Dockerfile | 31 +++++++++++++++++++++++++++++++ download.yaml | 3 +++ 2 files changed, 34 insertions(+) create mode 100644 Dockerfile create mode 100644 download.yaml diff --git a/Dockerfile b/Dockerfile new file mode 100644 index 0000000..ac739ef --- /dev/null +++ b/Dockerfile @@ -0,0 +1,31 @@ +ARG BASE_REGISTRY=registry1.dsop.io/ironbank +ARG BASE_IMAGE=redhat/openjdk/openjdk11 +ARG BASE_TAG=1.11 + +FROM tchiotludo/akhq:0.16.0 as base + +FROM ${BASE_REGISTRY}/${BASE_IMAGE}:${BASE_TAG} + +ENV LANG=C.UTF-8 + +USER 0 + +LABEL name="kafka dashboard" \ + maintainer="jparekh@vivsoft.io" \ + vendor="tchiotludo" \ + version="0.16.0" \ + release="1" \ + summary="" \ + description="" + +WORKDIR /app + +COPY --from=base /app /app + +ENV MICRONAUT_CONFIG_FILES=/app/application.yml + +ENTRYPOINT ["docker-entrypoint.sh"] + +CMD ["./akhq"] + +HEALTHCHECK NONE \ No newline at end of file diff --git a/download.yaml b/download.yaml new file mode 100644 index 0000000..4565625 --- /dev/null +++ b/download.yaml @@ -0,0 +1,3 @@ +resources: + - url: "docker://docker.io/tchiotludo/akhq@sha256:44e00630f95b0a42c716f635b94cd53a7185aa6626502839f8afa8d8b35bf576" + tag: "tchiotludo/akhq:0.16.0" \ No newline at end of file -- GitLab From fffa9e65657cb58dde341055a7043a6ac4de1b36 Mon Sep 17 00:00:00 2001 From: Jinoy Parekh Date: Wed, 11 Nov 2020 03:00:21 +0000 Subject: [PATCH 08/27] Update Dockerfile with summary/desc. --- Dockerfile | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Dockerfile b/Dockerfile index ac739ef..511c3c9 100644 --- a/Dockerfile +++ b/Dockerfile @@ -15,8 +15,8 @@ LABEL name="kafka dashboard" \ vendor="tchiotludo" \ version="0.16.0" \ release="1" \ - summary="" \ - description="" + summary="Kafka dashboard" \ + description="Kafka GUI for Apache Kafka to manage topics, topics data, consumers group and schema registry" WORKDIR /app -- GitLab From 26c5a61c51f8680d2bce6dbe922f1f154a7ea30a Mon Sep 17 00:00:00 2001 From: Jinoy Parekh Date: Tue, 17 Nov 2020 20:42:30 -0600 Subject: [PATCH 09/27] Updated Dfile --- Dockerfile | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/Dockerfile b/Dockerfile index 511c3c9..33897e1 100644 --- a/Dockerfile +++ b/Dockerfile @@ -8,8 +8,6 @@ FROM ${BASE_REGISTRY}/${BASE_IMAGE}:${BASE_TAG} ENV LANG=C.UTF-8 -USER 0 - LABEL name="kafka dashboard" \ maintainer="jparekh@vivsoft.io" \ vendor="tchiotludo" \ @@ -18,6 +16,12 @@ LABEL name="kafka dashboard" \ summary="Kafka dashboard" \ description="Kafka GUI for Apache Kafka to manage topics, topics data, consumers group and schema registry" +USER 0 + +RUN dnf upgrade -y && \ + dnf clean all && \ + rm -rf /var/cache/dnf + WORKDIR /app COPY --from=base /app /app @@ -28,4 +32,6 @@ ENTRYPOINT ["docker-entrypoint.sh"] CMD ["./akhq"] -HEALTHCHECK NONE \ No newline at end of file +USER 1001 + +HEALTHCHECK NONE -- GitLab From 11e1717221b2fa8a61232a4099f6dcfd5626b944 Mon Sep 17 00:00:00 2001 From: Jinoy Parekh Date: Thu, 19 Nov 2020 22:45:53 -0600 Subject: [PATCH 10/27] Added scripts and updatd Dfile --- Dockerfile | 18 +++-- scripts/application.yml | 172 ++++++++++++++++++++++++++++++++++++++++ scripts/entrypoint.sh | 9 +++ 3 files changed, 194 insertions(+), 5 deletions(-) create mode 100644 scripts/application.yml create mode 100644 scripts/entrypoint.sh diff --git a/Dockerfile b/Dockerfile index 33897e1..66df061 100644 --- a/Dockerfile +++ b/Dockerfile @@ -18,17 +18,25 @@ LABEL name="kafka dashboard" \ USER 0 -RUN dnf upgrade -y && \ - dnf clean all && \ - rm -rf /var/cache/dnf - WORKDIR /app COPY --from=base /app /app +COPY scripts/application.yml /app/application.yml + +COPY scripts/entrypoint.sh /app/entrypoint.sh + ENV MICRONAUT_CONFIG_FILES=/app/application.yml -ENTRYPOINT ["docker-entrypoint.sh"] +RUN useradd -u 1001 akhq + +RUN chown -R akhq /app + +RUN dnf upgrade -y && \ + dnf clean all && \ + rm -rf /var/cache/dnf + +ENTRYPOINT ["entrypoint.sh"] CMD ["./akhq"] diff --git a/scripts/application.yml b/scripts/application.yml new file mode 100644 index 0000000..cf0394d --- /dev/null +++ b/scripts/application.yml @@ -0,0 +1,172 @@ +micronaut: + application: + name: akhq + io: + watch: + paths: src/main + restart: false # enabled dev server with env vars MICRONAUT_IO_WATCH_RESTART=true + server: + thread-selection: AUTO + router: + static-resources: + react: + paths: classpath:ui + mapping: "/ui/**" + static: + paths: classpath:static + mapping: "/static/**" + swagger: + paths: classpath:META-INF/swagger + mapping: "/swagger/**" + security: + enabled: false + authentication: cookie + endpoints: + login: + path: "/login" + logout: + path: "/logout" + get-allowed: true + token: + jwt: + enabled: true + cookie: + enabled: true + cookie-same-site: strict + signatures: + secret: + generator: + secret: "pleasechangeme!pleasechangeme!pleasechangeme!pleasechangeme!pleasechangeme!pleasechangeme!pleasechangeme!pleasechangeme!pleasechangeme!pleasechangeme!pleasechangeme!pleasechangeme!pleasechangeme!pleasechangeme!pleasechangeme!pleasechangeme!" + + redirect: + login-success: "${micronaut.server.context-path:}/ui" + forbidden: + url: "${micronaut.server.context-path:}/ui/login/forbidden" + unauthorized: + url: "${micronaut.server.context-path:}/ui/login/unauthorized" + login-failure: "${micronaut.server.context-path:}/ui/login/failed" + logout: "${micronaut.server.context-path:}/ui" + intercept-url-map: + - pattern: "${micronaut.server.context-path:}/ui/**" + access: "isAnonymous()" + - pattern: "${micronaut.server.context-path:}/static/**" + access: "isAnonymous()" + - pattern: "${micronaut.server.context-path:}/swagger/**" + access: "isAnonymous()" + oauth2: + login-uri: "${micronaut.server.context-path:}/oauth/login{/provider}" + callback-uri: "${micronaut.server.context-path:}/oauth/callback{/provider}" + caches: + kafka-wrapper: + record-stats: true + expire-after-write: 0s + +jackson: + serialization: + writeDatesAsTimestamps: false + +endpoints: + health: + enabled: true + sensitive: false + details-visible: anonymous + info: + enabled: true + sensitive: false + metrics: + enabled: true + sensitive: false + export: + prometheus: + enabled: true + step: PT1M + descriptions: true + prometheus: + enabled: true + sensitive: false + caches: + enabled: true + sensitive: false + +akhq: + server: + access-log: + enabled: true + name: org.akhq.log.access + format: "[Date: {}] [Duration: {} ms] [Url: {} {}] [Status: {}] [Ip: {}] [User: {}]" + filters: + - "((?!/health).)*" + + clients-defaults: + consumer: + properties: + max.poll.records: 50 + isolation.level: read_committed + # group.id: Akhq + enable.auto.commit: "false" + default.api.timeout.ms: 15000 + + pagination: + page-size: 25 + threads: 16 + + topic: + default-view: HIDE_INTERNAL + replication: 1 + retention: 86400000 + partition: 1 + internal-regexps: + - "^_.*$" + - "^.*_schemas$" + - "^.*connect-config$" + - "^.*connect-offsets$1" + - "^.*connect-status$" + stream-regexps: + - "^.*-changelog$" + - "^.*-repartition$" + - "^.*-rekey$" + + topic-data: + sort: OLDEST + size: 50 + poll-timeout: 1000 + + security: + default-group: admin + groups: + - name: admin + roles: + - topic/read + - topic/insert + - topic/delete + - topic/config/update + - node/read + - node/config/update + - topic/data/read + - topic/data/insert + - topic/data/delete + - group/read + - group/delete + - group/offsets/update + - registry/read + - registry/insert + - registry/update + - registry/delete + - registry/version/delete + - acls/read + - connect/read + - connect/insert + - connect/update + - connect/delete + - connect/state/update + - name: reader + roles: + - topic/read + - node/read + - topic/data/read + - group/read + - registry/read + - acls/read + - connect/read + - name: no-roles + roles: [] diff --git a/scripts/entrypoint.sh b/scripts/entrypoint.sh new file mode 100644 index 0000000..52228a6 --- /dev/null +++ b/scripts/entrypoint.sh @@ -0,0 +1,9 @@ +#!/usr/bin/env sh + +set -e + +if [ "${AKHQ_CONFIGURATION}" ]; then + echo "${AKHQ_CONFIGURATION}" > /app/application.yml +fi + +exec "$@" -- GitLab From 4b26486e6daaa4aa69bdc4a9aef21677d3cec63f Mon Sep 17 00:00:00 2001 From: Jinoy Parekh Date: Mon, 7 Dec 2020 20:43:34 -0600 Subject: [PATCH 11/27] Updated Dfile and reorg config/app.yml --- Dockerfile | 6 ++---- {scripts => config}/application.yml | 0 2 files changed, 2 insertions(+), 4 deletions(-) rename {scripts => config}/application.yml (100%) diff --git a/Dockerfile b/Dockerfile index 66df061..194b8c0 100644 --- a/Dockerfile +++ b/Dockerfile @@ -22,15 +22,13 @@ WORKDIR /app COPY --from=base /app /app -COPY scripts/application.yml /app/application.yml +COPY config/application.yml /app/application.yml COPY scripts/entrypoint.sh /app/entrypoint.sh ENV MICRONAUT_CONFIG_FILES=/app/application.yml -RUN useradd -u 1001 akhq - -RUN chown -R akhq /app +RUN chown -R 1001 /app RUN dnf upgrade -y && \ dnf clean all && \ diff --git a/scripts/application.yml b/config/application.yml similarity index 100% rename from scripts/application.yml rename to config/application.yml -- GitLab From 1420e26ef740254262b96a8649b50ba292f53300 Mon Sep 17 00:00:00 2001 From: Jinoy Parekh Date: Mon, 7 Dec 2020 21:50:09 -0600 Subject: [PATCH 12/27] Update Dfile --- Dockerfile | 9 +++++++-- scripts/akhq | 11 +++++++++++ scripts/entrypoint.sh | 9 --------- scripts/jvm.options | 34 ++++++++++++++++++++++++++++++++++ 4 files changed, 52 insertions(+), 11 deletions(-) create mode 100644 scripts/akhq delete mode 100644 scripts/entrypoint.sh create mode 100644 scripts/jvm.options diff --git a/Dockerfile b/Dockerfile index 194b8c0..684ba36 100644 --- a/Dockerfile +++ b/Dockerfile @@ -20,10 +20,14 @@ USER 0 WORKDIR /app -COPY --from=base /app /app +COPY --from=base /app/akhq.jar /app/akhq.jar COPY config/application.yml /app/application.yml +COPY scripts/akhq /app/akhq + +COPY scripts/jvm.options /app/jvm.options + COPY scripts/entrypoint.sh /app/entrypoint.sh ENV MICRONAUT_CONFIG_FILES=/app/application.yml @@ -32,7 +36,8 @@ RUN chown -R 1001 /app RUN dnf upgrade -y && \ dnf clean all && \ - rm -rf /var/cache/dnf + rm -rf /var/cache/dnf && \ + chmod +x /app/akhq ENTRYPOINT ["entrypoint.sh"] diff --git a/scripts/akhq b/scripts/akhq new file mode 100644 index 0000000..1ec024e --- /dev/null +++ b/scripts/akhq @@ -0,0 +1,11 @@ +#!/usr/bin/env sh + +# Read user-defined JVM options from jvm.options file +JVM_HOME="/usr/lib/jvm/java-11-openjdk-11.0.9.11-2.el8_3.x86_64/bin/java" +JVM_OPTS_FILE=${JVM_OPTS_FILE:-/app/jvm.options} +for JVM_OPT in `grep "^-" ${JVM_OPTS_FILE}` +do + JAVA_OPTS="${JAVA_OPTS} ${JVM_OPT}" +done + +${JVM_HOME} ${JAVA_OPTS} -jar /app/akhq.jar diff --git a/scripts/entrypoint.sh b/scripts/entrypoint.sh deleted file mode 100644 index 52228a6..0000000 --- a/scripts/entrypoint.sh +++ /dev/null @@ -1,9 +0,0 @@ -#!/usr/bin/env sh - -set -e - -if [ "${AKHQ_CONFIGURATION}" ]; then - echo "${AKHQ_CONFIGURATION}" > /app/application.yml -fi - -exec "$@" diff --git a/scripts/jvm.options b/scripts/jvm.options new file mode 100644 index 0000000..c046099 --- /dev/null +++ b/scripts/jvm.options @@ -0,0 +1,34 @@ +########################################################################### +# jvm.options # +# # +# - all flags defined here will be used to startup the JVM # +# - one flag should be specified per line # +# - lines that do not start with '-' will be ignored # +# - only static flags are accepted (no variables or parameters) # +# - dynamic flags will be appended to these on cassandra-env # +########################################################################### + +# Server Hotspot JVM +-server + +# ensure UTF-8 encoding by default (e.g. filenames) +-Dfile.encoding=UTF-8 + +# set to headless, just in case +-Djava.awt.headless=true + +# generate a heap dump when an allocation from the Java heap fails +# heap dumps are created in the working directory of the JVM +-XX:+HeapDumpOnOutOfMemoryError +-XX:HeapDumpPath=/tmp/heapdump.log + +# Do not rely on the system configuration +-Dfile.encoding=UTF-8 +-Duser.timezone=UTC + +# Jmx Remote +-Dcom.sun.management.jmxremote +-Dcom.sun.management.jmxremote.port=8686 +-Dcom.sun.management.jmxremote.local.only=false +-Dcom.sun.management.jmxremote.authenticate=false +-Dcom.sun.management.jmxremote.ssl=false -- GitLab From 0e922a06c157be0a3b3d6159907c69a6e96fe1f0 Mon Sep 17 00:00:00 2001 From: Jinoy Parekh Date: Mon, 7 Dec 2020 22:00:18 -0600 Subject: [PATCH 13/27] Updated scripts/** --- scripts/entrypoint.sh | 9 +++++++++ 1 file changed, 9 insertions(+) create mode 100644 scripts/entrypoint.sh diff --git a/scripts/entrypoint.sh b/scripts/entrypoint.sh new file mode 100644 index 0000000..52228a6 --- /dev/null +++ b/scripts/entrypoint.sh @@ -0,0 +1,9 @@ +#!/usr/bin/env sh + +set -e + +if [ "${AKHQ_CONFIGURATION}" ]; then + echo "${AKHQ_CONFIGURATION}" > /app/application.yml +fi + +exec "$@" -- GitLab From f67ef507ab189e8d35774d1e7efeffed97a31436 Mon Sep 17 00:00:00 2001 From: ironbank-bot Date: Thu, 10 Dec 2020 01:22:04 +0000 Subject: [PATCH 14/27] Migrate to hardening_manifest.yaml --- Dockerfile | 7 ------ Jenkinsfile | 2 -- download.yaml | 3 --- hardening_manifest.yaml | 55 +++++++++++++++++++++++++++++++++++++++++ 4 files changed, 55 insertions(+), 12 deletions(-) delete mode 100644 Jenkinsfile delete mode 100644 download.yaml create mode 100644 hardening_manifest.yaml diff --git a/Dockerfile b/Dockerfile index 684ba36..817a515 100644 --- a/Dockerfile +++ b/Dockerfile @@ -8,13 +8,6 @@ FROM ${BASE_REGISTRY}/${BASE_IMAGE}:${BASE_TAG} ENV LANG=C.UTF-8 -LABEL name="kafka dashboard" \ - maintainer="jparekh@vivsoft.io" \ - vendor="tchiotludo" \ - version="0.16.0" \ - release="1" \ - summary="Kafka dashboard" \ - description="Kafka GUI for Apache Kafka to manage topics, topics data, consumers group and schema registry" USER 0 diff --git a/Jenkinsfile b/Jenkinsfile deleted file mode 100644 index 18f06b0..0000000 --- a/Jenkinsfile +++ /dev/null @@ -1,2 +0,0 @@ -@Library('DCCSCR@master') _ -dccscrPipeline(version: "0.16.0") diff --git a/download.yaml b/download.yaml deleted file mode 100644 index 4565625..0000000 --- a/download.yaml +++ /dev/null @@ -1,3 +0,0 @@ -resources: - - url: "docker://docker.io/tchiotludo/akhq@sha256:44e00630f95b0a42c716f635b94cd53a7185aa6626502839f8afa8d8b35bf576" - tag: "tchiotludo/akhq:0.16.0" \ No newline at end of file diff --git a/hardening_manifest.yaml b/hardening_manifest.yaml new file mode 100644 index 0000000..44174da --- /dev/null +++ b/hardening_manifest.yaml @@ -0,0 +1,55 @@ +--- +apiVersion: v1 + +# The repository name in registry1, excluding /ironbank/ +name: "opensource/apache/kafka-dashboard" + +# List of tags to push for the repository in registry1 +# The most specific version should be the first tag and will be shown +# on ironbank.dsop.io +tags: +- "0.16.0" +- "latest" + +# Build args passed to Dockerfile ARGs +args: + BASE_IMAGE: "redhat/openjdk/openjdk11" + BASE_TAG: "1.11" + +# Docker image labels +labels: + org.opencontainers.image.title: "kafka-dashboard" + ## Human-readable description of the software packaged in the image + # org.opencontainers.image.description: "FIXME" + ## License(s) under which contained software is distributed + # org.opencontainers.image.licenses: "FIXME" + ## URL to find more information on the image + # org.opencontainers.image.url: "FIXME" + ## Name of the distributing entity, organization or individual + # org.opencontainers.image.vendor: "FIXME" + org.opencontainers.image.version: "0.16.0" + ## Keywords to help with search (ex. "cicd,gitops,golang") + # mil.dso.ironbank.image.keywords: "FIXME" + ## This value can be "opensource" or "commercial" + # mil.dso.ironbank.image.type: "FIXME" + ## Product the image belongs to for grouping multiple images + # mil.dso.ironbank.product.name: "FIXME" + +# List of resources to make available to the offline build context +resources: +- tag: tchiotludo/akhq:0.16.0 + url: docker://docker.io/tchiotludo/akhq@sha256:44e00630f95b0a42c716f635b94cd53a7185aa6626502839f8afa8d8b35bf576 + +# List of project maintainers +# FIXME: Fill in the following details for the current container owner in the whitelist +# FIXME: Include any other vendor information if applicable +maintainers: +- email: "jparekh@vivsoft.io" +# # The name of the current container owner +# name: "FIXME" +# # The gitlab username of the current container owner +# username: "FIXME" +# cht_member: true # FIXME: Uncomment if the maintainer is a member of CHT +# - name: "FIXME" +# username: "FIXME" +# email: "FIXME" -- GitLab From 1b0885a4e4c2b7c20cd794e310de6b3fe2ab9e5f Mon Sep 17 00:00:00 2001 From: Olga O Date: Fri, 8 Jan 2021 12:56:55 -0600 Subject: [PATCH 15/27] updated hardening_manifest.yaml --- hardening_manifest.yaml | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/hardening_manifest.yaml b/hardening_manifest.yaml index 44174da..ac24524 100644 --- a/hardening_manifest.yaml +++ b/hardening_manifest.yaml @@ -20,20 +20,20 @@ args: labels: org.opencontainers.image.title: "kafka-dashboard" ## Human-readable description of the software packaged in the image - # org.opencontainers.image.description: "FIXME" + org.opencontainers.image.description: "Kafka GUI for Apache Kafka to manage topics, topics data, consumers group and schema registry" ## License(s) under which contained software is distributed - # org.opencontainers.image.licenses: "FIXME" + org.opencontainers.image.licenses: "Apache License 2.0" ## URL to find more information on the image - # org.opencontainers.image.url: "FIXME" + org.opencontainers.image.url: "https://hub.docker.com/r/tchiotludo/akhq" ## Name of the distributing entity, organization or individual - # org.opencontainers.image.vendor: "FIXME" + org.opencontainers.image.vendor: "opensource" org.opencontainers.image.version: "0.16.0" ## Keywords to help with search (ex. "cicd,gitops,golang") - # mil.dso.ironbank.image.keywords: "FIXME" + mil.dso.ironbank.image.keywords: "kafka-dashboard" ## This value can be "opensource" or "commercial" - # mil.dso.ironbank.image.type: "FIXME" + mil.dso.ironbank.image.type: "opensource" ## Product the image belongs to for grouping multiple images - # mil.dso.ironbank.product.name: "FIXME" + mil.dso.ironbank.product.name: "kafka-dashboard" # List of resources to make available to the offline build context resources: @@ -46,10 +46,10 @@ resources: maintainers: - email: "jparekh@vivsoft.io" # # The name of the current container owner -# name: "FIXME" + name: "Jinoy Parekh" # # The gitlab username of the current container owner -# username: "FIXME" -# cht_member: true # FIXME: Uncomment if the maintainer is a member of CHT + username: "jparekh" + cht_member: true # FIXME: Uncomment if the maintainer is a member of CHT # - name: "FIXME" # username: "FIXME" # email: "FIXME" -- GitLab From 836f6faa5d90738ef0447efcd036e9865c053088 Mon Sep 17 00:00:00 2001 From: Jinoy Parekh Date: Wed, 3 Feb 2021 21:54:23 -0600 Subject: [PATCH 16/27] Updated Dfile & hardening_manifest.yaml --- Dockerfile | 13 +++---------- hardening_manifest.yaml | 15 ++++----------- renovate.json | 41 +++++++++++++++++++++++++++++++++++++++++ scripts/akhq | 3 +-- 4 files changed, 49 insertions(+), 23 deletions(-) create mode 100644 renovate.json diff --git a/Dockerfile b/Dockerfile index 817a515..6bba93b 100644 --- a/Dockerfile +++ b/Dockerfile @@ -6,22 +6,15 @@ FROM tchiotludo/akhq:0.16.0 as base FROM ${BASE_REGISTRY}/${BASE_IMAGE}:${BASE_TAG} -ENV LANG=C.UTF-8 - - USER 0 WORKDIR /app -COPY --from=base /app/akhq.jar /app/akhq.jar - -COPY config/application.yml /app/application.yml - -COPY scripts/akhq /app/akhq +COPY --from=base /app/akhq.jar . -COPY scripts/jvm.options /app/jvm.options +COPY config/* . -COPY scripts/entrypoint.sh /app/entrypoint.sh +COPY scripts/* . ENV MICRONAUT_CONFIG_FILES=/app/application.yml diff --git a/hardening_manifest.yaml b/hardening_manifest.yaml index ac24524..41f861d 100644 --- a/hardening_manifest.yaml +++ b/hardening_manifest.yaml @@ -29,7 +29,7 @@ labels: org.opencontainers.image.vendor: "opensource" org.opencontainers.image.version: "0.16.0" ## Keywords to help with search (ex. "cicd,gitops,golang") - mil.dso.ironbank.image.keywords: "kafka-dashboard" + mil.dso.ironbank.image.keywords: "kafka-dashboard,dataflow,processing" ## This value can be "opensource" or "commercial" mil.dso.ironbank.image.type: "opensource" ## Product the image belongs to for grouping multiple images @@ -41,15 +41,8 @@ resources: url: docker://docker.io/tchiotludo/akhq@sha256:44e00630f95b0a42c716f635b94cd53a7185aa6626502839f8afa8d8b35bf576 # List of project maintainers -# FIXME: Fill in the following details for the current container owner in the whitelist -# FIXME: Include any other vendor information if applicable maintainers: -- email: "jparekh@vivsoft.io" -# # The name of the current container owner - name: "Jinoy Parekh" -# # The gitlab username of the current container owner +- name: "Jinoy Parekh" username: "jparekh" - cht_member: true # FIXME: Uncomment if the maintainer is a member of CHT -# - name: "FIXME" -# username: "FIXME" -# email: "FIXME" + email: "jparekh@vivsoft.io" + cht_member: true diff --git a/renovate.json b/renovate.json new file mode 100644 index 0000000..fc9226c --- /dev/null +++ b/renovate.json @@ -0,0 +1,41 @@ +{ + "assignees": [ + "@jparekh" + ], + "baseBranches": [ + "development" + ], + "regexManagers": [ + { + "fileMatch": [ + "^Dockerfile$" + ], + "matchStrings": [ + "version=\"(?.*?)\"" + ], + "depNameTemplate": "kafka-dashboard", + "datasourceTemplate": "docker" + }, + { + "fileMatch": [ + "^hardening_manifest.yaml$" + ], + "matchStrings": [ + "org\\.opencontainers\\.image\\.version:\\s+\"(?.+?)\"" + ], + "depNameTemplate": "kafka-dashboard", + "datasourceTemplate": "docker" + }, + { + "fileMatch": [ + "^hardening_manifest.yaml$" + ], + "matchStrings": [ + "tags:\\s+-\\s+\"(?.+?)\"" + ], + "depNameTemplate": "kafka-dashboard", + "datasourceTemplate": "docker" + } + ] +} + diff --git a/scripts/akhq b/scripts/akhq index 1ec024e..3a4b946 100644 --- a/scripts/akhq +++ b/scripts/akhq @@ -1,11 +1,10 @@ #!/usr/bin/env sh # Read user-defined JVM options from jvm.options file -JVM_HOME="/usr/lib/jvm/java-11-openjdk-11.0.9.11-2.el8_3.x86_64/bin/java" JVM_OPTS_FILE=${JVM_OPTS_FILE:-/app/jvm.options} for JVM_OPT in `grep "^-" ${JVM_OPTS_FILE}` do JAVA_OPTS="${JAVA_OPTS} ${JVM_OPT}" done -${JVM_HOME} ${JAVA_OPTS} -jar /app/akhq.jar +java ${JAVA_OPTS} -jar /app/akhq.jar -- GitLab From 8dffd7b4d4e2e3ed20fb25afd30b55716ac725eb Mon Sep 17 00:00:00 2001 From: renovate Date: Sat, 13 Feb 2021 01:02:39 +0000 Subject: [PATCH 17/27] Update tchiotludo/akhq Docker tag to v0.17.0 --- Dockerfile | 2 +- hardening_manifest.yaml | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/Dockerfile b/Dockerfile index 6bba93b..0591d03 100644 --- a/Dockerfile +++ b/Dockerfile @@ -2,7 +2,7 @@ ARG BASE_REGISTRY=registry1.dsop.io/ironbank ARG BASE_IMAGE=redhat/openjdk/openjdk11 ARG BASE_TAG=1.11 -FROM tchiotludo/akhq:0.16.0 as base +FROM tchiotludo/akhq:0.17.0 as base FROM ${BASE_REGISTRY}/${BASE_IMAGE}:${BASE_TAG} diff --git a/hardening_manifest.yaml b/hardening_manifest.yaml index 41f861d..31208b1 100644 --- a/hardening_manifest.yaml +++ b/hardening_manifest.yaml @@ -37,8 +37,8 @@ labels: # List of resources to make available to the offline build context resources: -- tag: tchiotludo/akhq:0.16.0 - url: docker://docker.io/tchiotludo/akhq@sha256:44e00630f95b0a42c716f635b94cd53a7185aa6626502839f8afa8d8b35bf576 +- tag: tchiotludo/akhq:0.17.0 + url: docker://docker.io/tchiotludo/akhq@sha256:22d80b063232ca4d770be4e4c28b732c693827228652131125ab95271608268c # List of project maintainers maintainers: -- GitLab From 47326aac6d9cf990275c6b9f6dedcef91ab5fc19 Mon Sep 17 00:00:00 2001 From: Chris Byrd Date: Wed, 10 Mar 2021 13:29:09 -0700 Subject: [PATCH 18/27] reinit --- Dockerfile | 13 +- config/akhq-override | 10 ++ config/application.yml | 361 ++++++++++++++++++++++++---------------- hardening_manifest.yaml | 20 +-- 4 files changed, 248 insertions(+), 156 deletions(-) create mode 100644 config/akhq-override diff --git a/Dockerfile b/Dockerfile index 0591d03..0d7ab8d 100644 --- a/Dockerfile +++ b/Dockerfile @@ -1,3 +1,4 @@ + ARG BASE_REGISTRY=registry1.dsop.io/ironbank ARG BASE_IMAGE=redhat/openjdk/openjdk11 ARG BASE_TAG=1.11 @@ -6,15 +7,15 @@ FROM tchiotludo/akhq:0.17.0 as base FROM ${BASE_REGISTRY}/${BASE_IMAGE}:${BASE_TAG} -USER 0 - WORKDIR /app -COPY --from=base /app/akhq.jar . +USER 0 -COPY config/* . +COPY --from=base /app . +COPY --from=base /usr/local/bin/ /usr/local/bin/ +COPY ./config/akhq-override . -COPY scripts/* . +RUN cat akhq-override > akhq ENV MICRONAUT_CONFIG_FILES=/app/application.yml @@ -25,7 +26,7 @@ RUN dnf upgrade -y && \ rm -rf /var/cache/dnf && \ chmod +x /app/akhq -ENTRYPOINT ["entrypoint.sh"] +ENTRYPOINT ["docker-entrypoint.sh"] CMD ["./akhq"] diff --git a/config/akhq-override b/config/akhq-override new file mode 100644 index 0000000..5930e6b --- /dev/null +++ b/config/akhq-override @@ -0,0 +1,10 @@ +#!/usr/bin/env sh + +# Read user-defined JVM options from jvm.options file +JVM_OPTS_FILE=${JVM_OPTS_FILE:-/app/jvm.options} +for JVM_OPT in `grep "^-" ${JVM_OPTS_FILE}` +do + JAVA_OPTS="${JAVA_OPTS} ${JVM_OPT}" +done + +java ${JAVA_OPTS} -cp /app/akhq.jar:${CLASSPATH} org.akhq.App \ No newline at end of file diff --git a/config/application.yml b/config/application.yml index cf0394d..7c7bc55 100644 --- a/config/application.yml +++ b/config/application.yml @@ -1,172 +1,253 @@ micronaut: - application: - name: akhq - io: - watch: - paths: src/main - restart: false # enabled dev server with env vars MICRONAUT_IO_WATCH_RESTART=true - server: - thread-selection: AUTO - router: - static-resources: - react: - paths: classpath:ui - mapping: "/ui/**" - static: - paths: classpath:static - mapping: "/static/**" - swagger: - paths: classpath:META-INF/swagger - mapping: "/swagger/**" security: - enabled: false - authentication: cookie - endpoints: - login: - path: "/login" - logout: - path: "/logout" - get-allowed: true - token: - jwt: + enabled: true + # Ldap authentificaton configuration + ldap: + default: enabled: true - cookie: + context: + server: 'ldap://ldap.forumsys.com:389' + managerDn: 'cn=read-only-admin,dc=example,dc=com' + managerPassword: 'password' + search: + base: "dc=example,dc=com" + groups: enabled: true - cookie-same-site: strict + base: "dc=example,dc=com" + # OIDC authentification configuration + oauth2: + enabled: true + clients: + oidc: + client-id: "" + client-secret: "" + openid: + issuer: "" + token: + jwt: signatures: secret: generator: - secret: "pleasechangeme!pleasechangeme!pleasechangeme!pleasechangeme!pleasechangeme!pleasechangeme!pleasechangeme!pleasechangeme!pleasechangeme!pleasechangeme!pleasechangeme!pleasechangeme!pleasechangeme!pleasechangeme!pleasechangeme!pleasechangeme!" - - redirect: - login-success: "${micronaut.server.context-path:}/ui" - forbidden: - url: "${micronaut.server.context-path:}/ui/login/forbidden" - unauthorized: - url: "${micronaut.server.context-path:}/ui/login/unauthorized" - login-failure: "${micronaut.server.context-path:}/ui/login/failed" - logout: "${micronaut.server.context-path:}/ui" - intercept-url-map: - - pattern: "${micronaut.server.context-path:}/ui/**" - access: "isAnonymous()" - - pattern: "${micronaut.server.context-path:}/static/**" - access: "isAnonymous()" - - pattern: "${micronaut.server.context-path:}/swagger/**" - access: "isAnonymous()" - oauth2: - login-uri: "${micronaut.server.context-path:}/oauth/login{/provider}" - callback-uri: "${micronaut.server.context-path:}/oauth/callback{/provider}" - caches: - kafka-wrapper: - record-stats: true - expire-after-write: 0s - -jackson: - serialization: - writeDatesAsTimestamps: false - -endpoints: - health: - enabled: true - sensitive: false - details-visible: anonymous - info: - enabled: true - sensitive: false - metrics: - enabled: true - sensitive: false - export: - prometheus: - enabled: true - step: PT1M - descriptions: true - prometheus: - enabled: true - sensitive: false - caches: - enabled: true - sensitive: false + secret: pleasechangeme + server: + context-path: "" # if behind a reverse proxy, path to akhq without trailing slash (optional). Example: akhq is + # behind a reverse proxy with url http://my-server/akhq, set base-path: "/akhq". + # Not needed if you're behind a reverse proxy with subdomain http://akhq.my-server/ akhq: server: - access-log: - enabled: true - name: org.akhq.log.access - format: "[Date: {}] [Duration: {} ms] [Url: {} {}] [Status: {}] [Ip: {}] [User: {}]" - filters: - - "((?!/health).)*" + access-log: # Access log configuration (optional) + enabled: true # true by default + name: org.akhq.log.access # Logger name + format: "[Date: {}] [Duration: {} ms] [Url: {} {}] [Status: {}] [Ip: {}] [User: {}]" # Logger format + # default kafka properties for each clients, available for admin / producer / consumer (optional) clients-defaults: consumer: properties: - max.poll.records: 50 isolation.level: read_committed - # group.id: Akhq - enable.auto.commit: "false" - default.api.timeout.ms: 15000 + + # list of kafka cluster available for akhq + connections: + my-cluster-plain-text: # url friendly name for the cluster (letter, number, _, -, ... dot are not allowed here) + properties: # standard kafka properties (optional) + bootstrap.servers: "kafka:9092" + schema-registry: + url: "http://schema-registry:8085" # schema registry url (optional) + type: "confluent" # schema registry type (optional). Supported types are "confluent" (default) or "tibco" + # Basic Auth user / pass + basic-auth-username: basic-auth-user + basic-auth-password: basic-auth-pass + properties: # standard kafka properties (optional) + ssl.protocol: TLS + connect: + - name: connect-1 + url: "http://connect:8083" + # Basic Auth user / pass (optional) + basic-auth-username: basic-auth-user + basic-auth-password: basic-auth-pass + # ssl store configuration (optional) + ssl-trust-store: /app/truststore.jks + ssl-trust-store-password: trust-store-password + ssl-key-store: /app/truststore.jks + ssl-key-store-password: key-store-password + - name: connect-2 + url: "http://connect:8084" + # Basic Auth user / pass (optional) + basic-auth-username: basic-auth-user + basic-auth-password: basic-auth-pass + # ssl store configuration (optional) + ssl-trust-store: /app/truststore.jks + ssl-trust-store-password: trust-store-password + ssl-key-store: /app/truststore.jks + ssl-key-store-password: key-store-password + deserialization: + protobuf: + # (optional) if descriptor-file properties are used + descriptors-folder: "/app/protobuf_desc" + topics-mapping: + - topic-regex: "album.*" + descriptor-file-base64: "Cs4BCgthbGJ1bS5wcm90bxIXY29tLm5ldGNyYWNrZXIucHJvdG9idWYidwoFQWxidW0SFAoFdGl0bGUYASABKAlSBXRpdGxlEhYKBmFydGlzdBgCIAMoCVIGYXJ0aXN0EiEKDHJlbGVhc2VfeWVhchgDIAEoBVILcmVsZWFzZVllYXISHQoKc29uZ190aXRsZRgEIAMoCVIJc29uZ1RpdGxlQiUKF2NvbS5uZXRjcmFja2VyLnByb3RvYnVmQgpBbGJ1bVByb3RvYgZwcm90bzM=" + value-message-type: "Album" + - topic-regex: "film.*" + descriptor-file-base64: "CuEBCgpmaWxtLnByb3RvEhRjb20uY29tcGFueS5wcm90b2J1ZiKRAQoERmlsbRISCgRuYW1lGAEgASgJUgRuYW1lEhoKCHByb2R1Y2VyGAIgASgJUghwcm9kdWNlchIhCgxyZWxlYXNlX3llYXIYAyABKAVSC3JlbGVhc2VZZWFyEhoKCGR1cmF0aW9uGAQgASgFUghkdXJhdGlvbhIaCghzdGFycmluZxgFIAMoCVIIc3RhcnJpbmdCIQoUY29tLmNvbXBhbnkucHJvdG9idWZCCUZpbG1Qcm90b2IGcHJvdG8z" + value-message-type: "Film" + - topic-regex: "test.*" + descriptor-file: "other.desc" + key-message-type: "Row" + value-message-type: "Envelope" + # Ui Cluster Options (optional) + ui-options: + topic: + default-view: ALL # default list view (ALL, HIDE_INTERNAL, HIDE_INTERNAL_STREAM, HIDE_STREAM). Overrides default + skip-consumer-groups: false # Skip loading consumer group information when showing topics. Overrides default + skip-last-record: true # Skip loading last record date information when showing topics. Overrides default + topic-data: + sort: NEWEST # default sort order (OLDEST, NEWEST) (default: OLDEST). Overrides default + + my-cluster-ssl: + properties: + bootstrap.servers: "kafka:9093" + security.protocol: SSL + ssl.truststore.location: /app/truststore.jks + ssl.truststore.password: password + ssl.keystore.location: /app/keystore.jks + ssl.keystore.password: password + ssl.key.password: password + + my-cluster-sasl: + properties: + bootstrap.servers: "kafka:9094" + security.protocol: SASL_SSL + sasl.mechanism: SCRAM-SHA-256 + sasl.jaas.config: org.apache.kafka.common.security.scram.ScramLoginModule required username="admin" password="password"; + ssl.truststore.location: /app/truststore.jks + ssl.truststore.password: password + ssl.keystore.location: /app/keystore.jks + ssl.keystore.password: password + ssl.key.password: password pagination: - page-size: 25 - threads: 16 + page-size: 25 # number of elements per page (default : 25) + threads: 16 # Number of parallel threads to resolve page + # Topic list display options (optional) topic: - default-view: HIDE_INTERNAL - replication: 1 - retention: 86400000 - partition: 1 - internal-regexps: + retention: 172800000 # default retention period when creating topic + partition: 3 # default number of partition when creating topic + replication: 3 # default number of replicas when creating topic + internal-regexps: # list of regexp to be considered as internal (internal topic can't be deleted or updated) - "^_.*$" - "^.*_schemas$" - "^.*connect-config$" - "^.*connect-offsets$1" - "^.*connect-status$" - stream-regexps: + stream-regexps: # list of regexp to be considered as internal stream topic - "^.*-changelog$" - "^.*-repartition$" - "^.*-rekey$" + skip-consumer-groups: false # Skip loading consumer group information when showing topics + skip-last-record: false # Skip loading last record date information when showing topics + # Topic display data options (optional) topic-data: - sort: OLDEST - size: 50 - poll-timeout: 1000 + size: 50 # max record per page (default: 50) + poll-timeout: 1000 # The time, in milliseconds, spent waiting in poll if data is not available in the buffer. + # Ui Global Options (optional) + ui-options: + topic: + default-view: ALL # default list view (ALL, HIDE_INTERNAL, HIDE_INTERNAL_STREAM, HIDE_STREAM). Overrides default + skip-consumer-groups: false # Skip loading consumer group information when showing topics. Overrides default + skip-last-record: true # Skip loading last record date information when showing topics. Overrides default + topic-data: + sort: NEWEST # default sort order (OLDEST, NEWEST) (default: OLDEST). Overrides default + + # Auth & Roles (optional) security: - default-group: admin + default-group: admin # Default groups for all the user even unlogged user + # Groups definition groups: - - name: admin - roles: - - topic/read - - topic/insert - - topic/delete - - topic/config/update - - node/read - - node/config/update - - topic/data/read - - topic/data/insert - - topic/data/delete - - group/read - - group/delete - - group/offsets/update - - registry/read - - registry/insert - - registry/update - - registry/delete - - registry/version/delete - - acls/read - - connect/read - - connect/insert - - connect/update - - connect/delete - - connect/state/update - - name: reader - roles: - - topic/read - - node/read - - topic/data/read - - group/read - - registry/read - - acls/read - - connect/read - - name: no-roles - roles: [] + admin: # unique key + name: admin # Group name + roles: # roles for the group + - topic/read + - topic/insert + - topic/delete + - topic/config/update + - node/read + - node/config/update + - topic/data/read + - topic/data/insert + - topic/data/delete + - group/read + - group/delete + - group/offsets/update + - registry/read + - registry/insert + - registry/update + - registry/delete + - registry/version/delete + - acls/read + - connect/read + - connect/insert + - connect/update + - connect/delete + - connect/state/update + attributes: + # Regexp to filter topic available for group + topics-filter-regexp: "test.*" + # Regexp to filter connect configs visible for group + connects-filter-regexp: "^test.*$" + # Regexp to filter consumer groups visible for group + consumer-groups-filter-regexp: "consumer.*" + topic-reader: # unique key + name: topic-reader # Other group + roles: + - topic/read + attributes: + topics-filter-regexp: "test\\.reader.*" + + # Basic auth configuration + basic-auth: + - username: user # Username + password: pass # Password in sha256 + groups: # Groups for the user + - admin + - topic-reader + + # Ldap Groups configuration (when using ldap) + ldap: + default-group: topic-reader + groups: + - name: group-ldap-1 + groups: # Akhq groups list + - topic-reader + - name: group-ldap-2 + groups: + - admin + users: + - username: riemann # ldap user id + groups: # Akhq groups list + - topic-reader + - username: einstein + groups: + - admin + + # OIDC configuration + oidc: + enabled: true + providers: + oidc: + label: "Login with OIDC" + username-field: preferred_username + groups-field: roles + default-group: topic-reader + groups: + - name: oidc-admin-group + groups: + - admin + users: + - username: einstein + groups: + - admin diff --git a/hardening_manifest.yaml b/hardening_manifest.yaml index 31208b1..82269d3 100644 --- a/hardening_manifest.yaml +++ b/hardening_manifest.yaml @@ -2,13 +2,13 @@ apiVersion: v1 # The repository name in registry1, excluding /ironbank/ -name: "opensource/apache/kafka-dashboard" +name: "opensource/tchiotludo/akhq" # List of tags to push for the repository in registry1 # The most specific version should be the first tag and will be shown # on ironbank.dsop.io tags: -- "0.16.0" +- "0.17.0" - "latest" # Build args passed to Dockerfile ARGs @@ -18,22 +18,22 @@ args: # Docker image labels labels: - org.opencontainers.image.title: "kafka-dashboard" + org.opencontainers.image.title: "akhq" ## Human-readable description of the software packaged in the image - org.opencontainers.image.description: "Kafka GUI for Apache Kafka to manage topics, topics data, consumers group and schema registry" + org.opencontainers.image.description: "AKHQ was previously Kafka GUI for Apache Kafka to manage topics, topics data, consumers group and schema registry" ## License(s) under which contained software is distributed org.opencontainers.image.licenses: "Apache License 2.0" ## URL to find more information on the image org.opencontainers.image.url: "https://hub.docker.com/r/tchiotludo/akhq" ## Name of the distributing entity, organization or individual org.opencontainers.image.vendor: "opensource" - org.opencontainers.image.version: "0.16.0" + org.opencontainers.image.version: "0.17.0" ## Keywords to help with search (ex. "cicd,gitops,golang") - mil.dso.ironbank.image.keywords: "kafka-dashboard,dataflow,processing" + mil.dso.ironbank.image.keywords: "kafka-dashboard,dataflow,processing,akhq" ## This value can be "opensource" or "commercial" mil.dso.ironbank.image.type: "opensource" ## Product the image belongs to for grouping multiple images - mil.dso.ironbank.product.name: "kafka-dashboard" + mil.dso.ironbank.product.name: "akhq" # List of resources to make available to the offline build context resources: @@ -42,7 +42,7 @@ resources: # List of project maintainers maintainers: -- name: "Jinoy Parekh" - username: "jparekh" - email: "jparekh@vivsoft.io" +- name: "Chris Byrd" + username: "crbcos" + email: "crbcos@parsons.com" cht_member: true -- GitLab From 6d89f2695e083bd9737388ee0e347a666bc5e996 Mon Sep 17 00:00:00 2001 From: Chris Byrd Date: Wed, 10 Mar 2021 14:13:46 -0700 Subject: [PATCH 19/27] restructure - add script dir --- Dockerfile | 5 ++--- config/akhq-override | 10 ---------- scripts/akhq | 2 +- scripts/entrypoint.sh | 9 --------- scripts/jvm.options | 34 ---------------------------------- 5 files changed, 3 insertions(+), 57 deletions(-) delete mode 100644 config/akhq-override delete mode 100644 scripts/entrypoint.sh delete mode 100644 scripts/jvm.options diff --git a/Dockerfile b/Dockerfile index 0d7ab8d..0ec7c31 100644 --- a/Dockerfile +++ b/Dockerfile @@ -13,9 +13,8 @@ USER 0 COPY --from=base /app . COPY --from=base /usr/local/bin/ /usr/local/bin/ -COPY ./config/akhq-override . - -RUN cat akhq-override > akhq +COPY ./config . +COPY ./scripts/akhq . ENV MICRONAUT_CONFIG_FILES=/app/application.yml diff --git a/config/akhq-override b/config/akhq-override deleted file mode 100644 index 5930e6b..0000000 --- a/config/akhq-override +++ /dev/null @@ -1,10 +0,0 @@ -#!/usr/bin/env sh - -# Read user-defined JVM options from jvm.options file -JVM_OPTS_FILE=${JVM_OPTS_FILE:-/app/jvm.options} -for JVM_OPT in `grep "^-" ${JVM_OPTS_FILE}` -do - JAVA_OPTS="${JAVA_OPTS} ${JVM_OPT}" -done - -java ${JAVA_OPTS} -cp /app/akhq.jar:${CLASSPATH} org.akhq.App \ No newline at end of file diff --git a/scripts/akhq b/scripts/akhq index 3a4b946..5930e6b 100644 --- a/scripts/akhq +++ b/scripts/akhq @@ -7,4 +7,4 @@ do JAVA_OPTS="${JAVA_OPTS} ${JVM_OPT}" done -java ${JAVA_OPTS} -jar /app/akhq.jar +java ${JAVA_OPTS} -cp /app/akhq.jar:${CLASSPATH} org.akhq.App \ No newline at end of file diff --git a/scripts/entrypoint.sh b/scripts/entrypoint.sh deleted file mode 100644 index 52228a6..0000000 --- a/scripts/entrypoint.sh +++ /dev/null @@ -1,9 +0,0 @@ -#!/usr/bin/env sh - -set -e - -if [ "${AKHQ_CONFIGURATION}" ]; then - echo "${AKHQ_CONFIGURATION}" > /app/application.yml -fi - -exec "$@" diff --git a/scripts/jvm.options b/scripts/jvm.options deleted file mode 100644 index c046099..0000000 --- a/scripts/jvm.options +++ /dev/null @@ -1,34 +0,0 @@ -########################################################################### -# jvm.options # -# # -# - all flags defined here will be used to startup the JVM # -# - one flag should be specified per line # -# - lines that do not start with '-' will be ignored # -# - only static flags are accepted (no variables or parameters) # -# - dynamic flags will be appended to these on cassandra-env # -########################################################################### - -# Server Hotspot JVM --server - -# ensure UTF-8 encoding by default (e.g. filenames) --Dfile.encoding=UTF-8 - -# set to headless, just in case --Djava.awt.headless=true - -# generate a heap dump when an allocation from the Java heap fails -# heap dumps are created in the working directory of the JVM --XX:+HeapDumpOnOutOfMemoryError --XX:HeapDumpPath=/tmp/heapdump.log - -# Do not rely on the system configuration --Dfile.encoding=UTF-8 --Duser.timezone=UTC - -# Jmx Remote --Dcom.sun.management.jmxremote --Dcom.sun.management.jmxremote.port=8686 --Dcom.sun.management.jmxremote.local.only=false --Dcom.sun.management.jmxremote.authenticate=false --Dcom.sun.management.jmxremote.ssl=false -- GitLab From 042f4d848f9d38a22665e4e93a72bc729f37bcf9 Mon Sep 17 00:00:00 2001 From: Chris Byrd Date: Tue, 23 Mar 2021 14:38:29 -0600 Subject: [PATCH 20/27] tweak boilerplate config --- config/application.yml | 54 +++++++++++++++++++++--------------------- 1 file changed, 27 insertions(+), 27 deletions(-) diff --git a/config/application.yml b/config/application.yml index 7c7bc55..943710e 100644 --- a/config/application.yml +++ b/config/application.yml @@ -8,7 +8,7 @@ micronaut: context: server: 'ldap://ldap.forumsys.com:389' managerDn: 'cn=read-only-admin,dc=example,dc=com' - managerPassword: 'password' + managerPassword: search: base: "dc=example,dc=com" groups: @@ -28,7 +28,7 @@ micronaut: signatures: secret: generator: - secret: pleasechangeme + secret: server: context-path: "" # if behind a reverse proxy, path to akhq without trailing slash (optional). Example: akhq is @@ -56,41 +56,41 @@ akhq: url: "http://schema-registry:8085" # schema registry url (optional) type: "confluent" # schema registry type (optional). Supported types are "confluent" (default) or "tibco" # Basic Auth user / pass - basic-auth-username: basic-auth-user - basic-auth-password: basic-auth-pass + basic-auth-username: ${UN} + basic-auth-password: ${PW} properties: # standard kafka properties (optional) ssl.protocol: TLS connect: - name: connect-1 url: "http://connect:8083" # Basic Auth user / pass (optional) - basic-auth-username: basic-auth-user - basic-auth-password: basic-auth-pass + basic-auth-username: ${UN} + basic-auth-password: ${PW} # ssl store configuration (optional) ssl-trust-store: /app/truststore.jks - ssl-trust-store-password: trust-store-password + ssl-trust-store-password: ${PW} ssl-key-store: /app/truststore.jks - ssl-key-store-password: key-store-password + ssl-key-store-password: ${PW} - name: connect-2 url: "http://connect:8084" # Basic Auth user / pass (optional) - basic-auth-username: basic-auth-user - basic-auth-password: basic-auth-pass + basic-auth-username: ${UN} + basic-auth-password: ${PW} # ssl store configuration (optional) ssl-trust-store: /app/truststore.jks - ssl-trust-store-password: trust-store-password + ssl-trust-store-password: ${PW} ssl-key-store: /app/truststore.jks - ssl-key-store-password: key-store-password + ssl-key-store-password: ${PW} deserialization: protobuf: # (optional) if descriptor-file properties are used descriptors-folder: "/app/protobuf_desc" topics-mapping: - topic-regex: "album.*" - descriptor-file-base64: "Cs4BCgthbGJ1bS5wcm90bxIXY29tLm5ldGNyYWNrZXIucHJvdG9idWYidwoFQWxidW0SFAoFdGl0bGUYASABKAlSBXRpdGxlEhYKBmFydGlzdBgCIAMoCVIGYXJ0aXN0EiEKDHJlbGVhc2VfeWVhchgDIAEoBVILcmVsZWFzZVllYXISHQoKc29uZ190aXRsZRgEIAMoCVIJc29uZ1RpdGxlQiUKF2NvbS5uZXRjcmFja2VyLnByb3RvYnVmQgpBbGJ1bVByb3RvYgZwcm90bzM=" + descriptor-file-base64: "" #Base64 value-message-type: "Album" - topic-regex: "film.*" - descriptor-file-base64: "CuEBCgpmaWxtLnByb3RvEhRjb20uY29tcGFueS5wcm90b2J1ZiKRAQoERmlsbRISCgRuYW1lGAEgASgJUgRuYW1lEhoKCHByb2R1Y2VyGAIgASgJUghwcm9kdWNlchIhCgxyZWxlYXNlX3llYXIYAyABKAVSC3JlbGVhc2VZZWFyEhoKCGR1cmF0aW9uGAQgASgFUghkdXJhdGlvbhIaCghzdGFycmluZxgFIAMoCVIIc3RhcnJpbmdCIQoUY29tLmNvbXBhbnkucHJvdG9idWZCCUZpbG1Qcm90b2IGcHJvdG8z" + descriptor-file-base64: "" #Base64 value-message-type: "Film" - topic-regex: "test.*" descriptor-file: "other.desc" @@ -110,22 +110,22 @@ akhq: bootstrap.servers: "kafka:9093" security.protocol: SSL ssl.truststore.location: /app/truststore.jks - ssl.truststore.password: password + ssl.truststore.password: ${PW} ssl.keystore.location: /app/keystore.jks - ssl.keystore.password: password - ssl.key.password: password + ssl.keystore.password: ${PW} + ssl.key.password: ${PW} my-cluster-sasl: properties: bootstrap.servers: "kafka:9094" security.protocol: SASL_SSL sasl.mechanism: SCRAM-SHA-256 - sasl.jaas.config: org.apache.kafka.common.security.scram.ScramLoginModule required username="admin" password="password"; + sasl.jaas.config: org.apache.kafka.common.security.scram.ScramLoginModule required username="${UN}" password="${PW}"; ssl.truststore.location: /app/truststore.jks - ssl.truststore.password: password + ssl.truststore.password: ${PW} ssl.keystore.location: /app/keystore.jks - ssl.keystore.password: password - ssl.key.password: password + ssl.keystore.password: ${PW} + ssl.key.password: ${PW} pagination: page-size: 25 # number of elements per page (default : 25) @@ -210,8 +210,8 @@ akhq: # Basic auth configuration basic-auth: - - username: user # Username - password: pass # Password in sha256 + - username: ${UN} # Username + password: ${PW} # Password in sha256 groups: # Groups for the user - admin - topic-reader @@ -227,10 +227,10 @@ akhq: groups: - admin users: - - username: riemann # ldap user id + - username: ${UN} # ldap user id groups: # Akhq groups list - topic-reader - - username: einstein + - username: ${UN} groups: - admin @@ -240,7 +240,7 @@ akhq: providers: oidc: label: "Login with OIDC" - username-field: preferred_username + username-field: ${UN} groups-field: roles default-group: topic-reader groups: @@ -248,6 +248,6 @@ akhq: groups: - admin users: - - username: einstein + - username: ${UN} groups: - admin -- GitLab From 9bc9d251d785178d67d03822998aa23907f17296 Mon Sep 17 00:00:00 2001 From: Zachary Sanders Date: Fri, 21 May 2021 14:58:44 +0000 Subject: [PATCH 21/27] renovate-automerge --- renovate.json | 16 ++++------------ 1 file changed, 4 insertions(+), 12 deletions(-) diff --git a/renovate.json b/renovate.json index fc9226c..e5953d1 100644 --- a/renovate.json +++ b/renovate.json @@ -5,17 +5,9 @@ "baseBranches": [ "development" ], + "automerge": true, + "gitLabAutomerge": true, "regexManagers": [ - { - "fileMatch": [ - "^Dockerfile$" - ], - "matchStrings": [ - "version=\"(?.*?)\"" - ], - "depNameTemplate": "kafka-dashboard", - "datasourceTemplate": "docker" - }, { "fileMatch": [ "^hardening_manifest.yaml$" @@ -23,7 +15,7 @@ "matchStrings": [ "org\\.opencontainers\\.image\\.version:\\s+\"(?.+?)\"" ], - "depNameTemplate": "kafka-dashboard", + "depNameTemplate": "tchiotludo/akhq", "datasourceTemplate": "docker" }, { @@ -33,7 +25,7 @@ "matchStrings": [ "tags:\\s+-\\s+\"(?.+?)\"" ], - "depNameTemplate": "kafka-dashboard", + "depNameTemplate": "tchiotludo/akhq", "datasourceTemplate": "docker" } ] -- GitLab From 7d79a8897a2330f5f167482ca4813e85bd542029 Mon Sep 17 00:00:00 2001 From: "ortiz.jacob" Date: Tue, 1 Jun 2021 15:42:59 -0400 Subject: [PATCH 22/27] update dockerfile --- Dockerfile | 5 ++--- scripts/docker-entrypoint.sh | 9 +++++++++ 2 files changed, 11 insertions(+), 3 deletions(-) create mode 100644 scripts/docker-entrypoint.sh diff --git a/Dockerfile b/Dockerfile index 0ec7c31..84dba48 100644 --- a/Dockerfile +++ b/Dockerfile @@ -1,6 +1,5 @@ - -ARG BASE_REGISTRY=registry1.dsop.io/ironbank -ARG BASE_IMAGE=redhat/openjdk/openjdk11 +ARG BASE_REGISTRY=registry1.dso.mil +ARG BASE_IMAGE=ironbank/redhat/openjdk/openjdk11 ARG BASE_TAG=1.11 FROM tchiotludo/akhq:0.17.0 as base diff --git a/scripts/docker-entrypoint.sh b/scripts/docker-entrypoint.sh new file mode 100644 index 0000000..ef7ff08 --- /dev/null +++ b/scripts/docker-entrypoint.sh @@ -0,0 +1,9 @@ + #!/usr/bin/env sh + +set -e + +if [ "${AKHQ_CONFIGURATION}" ]; then + echo "${AKHQ_CONFIGURATION}" > /app/application.yml +fi + +exec "$@" \ No newline at end of file -- GitLab From 914309f33f2e98d902647082452f39f79c858352 Mon Sep 17 00:00:00 2001 From: "ortiz.jacob" Date: Tue, 1 Jun 2021 15:46:54 -0400 Subject: [PATCH 23/27] update dockerfile --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 15402c5..ee049db 100644 --- a/README.md +++ b/README.md @@ -650,7 +650,7 @@ Documentation on Confluent 5.5 and schema references can be found [here](https:/ ## Credits -Many thanks to: +Many thanks to: * [JetBrains](https://www.jetbrains.com/?from=AKHQ) for their free OpenSource license. * Apache, Apache Kafka, Kafka, and associated open source project names are trademarks of the Apache Software Foundation. AKHQ is not affiliated with, endorsed by, or otherwise associated with the Apache Software. -- GitLab From 72caf3f5f287d48915c239fa3e4d794b158da320 Mon Sep 17 00:00:00 2001 From: renovate Date: Thu, 1 Jul 2021 01:09:41 +0000 Subject: [PATCH 24/27] Update tchiotludo/akhq Docker tag to v0.18.0 --- Dockerfile | 2 +- hardening_manifest.yaml | 8 ++++---- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/Dockerfile b/Dockerfile index 84dba48..be68af3 100644 --- a/Dockerfile +++ b/Dockerfile @@ -2,7 +2,7 @@ ARG BASE_REGISTRY=registry1.dso.mil ARG BASE_IMAGE=ironbank/redhat/openjdk/openjdk11 ARG BASE_TAG=1.11 -FROM tchiotludo/akhq:0.17.0 as base +FROM tchiotludo/akhq:0.18.0 as base FROM ${BASE_REGISTRY}/${BASE_IMAGE}:${BASE_TAG} diff --git a/hardening_manifest.yaml b/hardening_manifest.yaml index 82269d3..81d50f3 100644 --- a/hardening_manifest.yaml +++ b/hardening_manifest.yaml @@ -8,7 +8,7 @@ name: "opensource/tchiotludo/akhq" # The most specific version should be the first tag and will be shown # on ironbank.dsop.io tags: -- "0.17.0" +- "0.18.0" - "latest" # Build args passed to Dockerfile ARGs @@ -27,7 +27,7 @@ labels: org.opencontainers.image.url: "https://hub.docker.com/r/tchiotludo/akhq" ## Name of the distributing entity, organization or individual org.opencontainers.image.vendor: "opensource" - org.opencontainers.image.version: "0.17.0" + org.opencontainers.image.version: "0.18.0" ## Keywords to help with search (ex. "cicd,gitops,golang") mil.dso.ironbank.image.keywords: "kafka-dashboard,dataflow,processing,akhq" ## This value can be "opensource" or "commercial" @@ -37,8 +37,8 @@ labels: # List of resources to make available to the offline build context resources: -- tag: tchiotludo/akhq:0.17.0 - url: docker://docker.io/tchiotludo/akhq@sha256:22d80b063232ca4d770be4e4c28b732c693827228652131125ab95271608268c +- tag: tchiotludo/akhq:0.18.0 + url: docker://docker.io/tchiotludo/akhq@sha256:c977e6dfe9d3a290eb16530e3b8ad0c251076a5997cb9867b635d1727799cbf5 # List of project maintainers maintainers: -- GitLab From 18900953e9cdd250e6570156f028a9ba2943c90b Mon Sep 17 00:00:00 2001 From: Jacob Rohlman Date: Fri, 2 Jul 2021 10:27:54 -0500 Subject: [PATCH 25/27] updating container --- hardening_manifest.yaml | 6 +++--- renovate.json | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/hardening_manifest.yaml b/hardening_manifest.yaml index 81d50f3..063fcd2 100644 --- a/hardening_manifest.yaml +++ b/hardening_manifest.yaml @@ -42,7 +42,7 @@ resources: # List of project maintainers maintainers: -- name: "Chris Byrd" - username: "crbcos" - email: "crbcos@parsons.com" +- name: "Jacob Rohlman" + username: "jacob.rohlman" + email: "jacob.rohlman@us.af.mil" cht_member: true diff --git a/renovate.json b/renovate.json index e5953d1..6bc1eb2 100644 --- a/renovate.json +++ b/renovate.json @@ -1,6 +1,6 @@ { "assignees": [ - "@jparekh" + "@jacob.rohlman" ], "baseBranches": [ "development" -- GitLab From 5507002062c67117b87903323a5de97559b0da19 Mon Sep 17 00:00:00 2001 From: alfontaine Date: Thu, 8 Jul 2021 10:17:04 -0600 Subject: [PATCH 26/27] conflicts --- .gitlab/issue_templates/Access Request.md | 3 ++ .../issue_templates/Application - Archive.md | 3 ++ .../issue_templates/Application - Initial.md | 54 +++++++++++++++++-- .../issue_templates/Application - Update.md | 51 ++++++++++++++++-- .gitlab/issue_templates/Bug.md | 5 ++ .gitlab/issue_templates/Feature Request.md | 5 ++ .../issue_templates/Leadership Question.md | 5 ++ .gitlab/issue_templates/New Findings.md | 8 +++ .../issue_templates/Onboarding Question.md | 5 ++ .gitlab/issue_templates/Pipeline Failure.md | 6 +++ 10 files changed, 137 insertions(+), 8 deletions(-) diff --git a/.gitlab/issue_templates/Access Request.md b/.gitlab/issue_templates/Access Request.md index 1a7b224..2e3e8b7 100644 --- a/.gitlab/issue_templates/Access Request.md +++ b/.gitlab/issue_templates/Access Request.md @@ -13,4 +13,7 @@ The access level should be: - [ ] All accounts have been provided the necessary accesses + + + /label ~"Access" ~"To Do" \ No newline at end of file diff --git a/.gitlab/issue_templates/Application - Archive.md b/.gitlab/issue_templates/Application - Archive.md index a558faa..0304276 100644 --- a/.gitlab/issue_templates/Application - Archive.md +++ b/.gitlab/issue_templates/Application - Archive.md @@ -17,4 +17,7 @@ Requesting this application be archived due to one of the following reasons: - [ ] Iron Bank frontend no longer lists application as available or approved + + + /label ~"Container::Archive" \ No newline at end of file diff --git a/.gitlab/issue_templates/Application - Initial.md b/.gitlab/issue_templates/Application - Initial.md index b7acebd..7ddab91 100644 --- a/.gitlab/issue_templates/Application - Initial.md +++ b/.gitlab/issue_templates/Application - Initial.md @@ -7,25 +7,69 @@ Requesting application to be hardened. This is only for initial hardening of a c Current version: (State the current version of the application as you see it) -Under support: (Is the updated version within the same major version of the application or is this a new major version?) +## Communication + +All communication should occur through this issue. This ensures that all information is documented in a centralized location and also ensures that all of the assignees are notified of updates. It is imperative that all required parties are listed as assignees of this issue and that individuals are not removed. Please do not remove anyone from the assignee list. + +If you need to contact the Container Hardening team, please identify your assigned point of contact. You can find your point of contact by: +1. They should be listed as an assignee on this ticket +2. They should be listed in the `hardening_manifest.yaml` file under the `maintainers` section with a field of `cht_member: true` + +If you have no assignee, feel free to tag Container Hardening leadership in your issue by commenting on this issue with your questions/concerns and then add `/cc @ironbank-notifications/leadership`. Gitlab will automatically notify all Container Hardening leadership to look at this issue and respond. + + +## Responsibilities + +If this application is owned by a Contributor or Vendor (identifed as `Owner::Contributor` and `Owner::Vendor` respectively), then it is your responsibility to drive this issue through completion. This means that the Container Hardening team is not here to help push any deadlines/timeframes you may have with other customers or DoD agencies. If you have issues with the activity, you may notify Container Hardening leadership above. Do not change the ownership labels. ## Definition of Done + Hardening: -- [ ] Container builds successfully -- [ ] Greylist file has been created (requires a member from container hardening) +- [ ] Hardening manifest is created and adheres to the schema (https://repo1.dsop.io/ironbank-tools/ironbank-pipeline/-/blob/master/schema/hardening_manifest.schema.json) +- [ ] Container builds successfully through the Gitlab CI pipeline - [ ] Branch has been merged into `development` +- [ ] Project is configured for automatic renovate updates (if possible) Justifications: - [ ] All findings have been justified per the above documentation -- [ ] Justifications have been provided to the container hardening team +- [ ] Justifications have been attached to this issue +- [ ] Apply the label `Approval` to indicate this container is ready for the approval phase + +Note: The justifications must be provided in a timely fashion. Failure to do so could result in new findings being identified which may start this process over. -Approval Process (container hardening team processes): +Approval Process (Container Hardening Team processes): - [ ] Peer review from Container Hardening Team - [ ] Findings Approver has reviewed and approved all justifications - [ ] Approval request has been sent to Authorizing Official - [ ] Approval request has been processed by Authorizing Official +Note: If the above approval process is kicked back for any reason, the `Approval` label will be removed and the issue will be sent back to `Open`. Any comments will be listed in this issue for you to address. Once they have been addressed, you may re-add the `Approval` label. + +## Post Approval + +### Continuous Monitoring + +Once a container is approved, the `Approved` label will be applied to this issue and it will be closed. You will be able to find your applications on http://ironbank.dsop.io and https://registry1.dsop.io. + +In addition to the above, your application will now be subscribed to continuous monitoring. This means that any new findings discovered as part of this will need justifications. To satisfy this process, any new findings will trigger a new Gitlab issue in this project with the label `Container::New Findings`. All members listed in the `maintainers` section of the `hardening_manifest.yaml` file will automatically be assigned. It is your responsibility as a Contributor or Vendor to monitor for this and provide justifications in a timely fashion. This newly created issue will have all the instructions necessary to complete the process. Failure to provide justifications could result in the revocation of the application's approval status. + +### Updates + +It is imperative that application updates be submitted as quickly as possible. We do not want applications to become stale. To help with this process, Ironbank recommends using a tool called [Renovate](https://github.com/renovatebot/renovate). This requires a `renovate.json` file to be placed in your project and can automate the creation of issues and merge requests. + +If not using Renovate, it will be up to you as a Contributor or Vendor to keep this image up-to-date at all times. When you wish to submit an application update, you must create a new issue in this project using the `Application - Update` template and associate it with the corresponding merge request. If you submit a merge request alone, work will not proceed until a related issue is created. These issues are tracked using the label `Container::Update`. + +Additionally, it is imperative that all updates must be followed through to completion. Simply submitting an application update but not following through on justifications and approvals does not suffice and risk your application's approval status being revoked. + +### Bugs + +Occassionally, users may file bug reports for your application. It is your responsibility to monitor for these since they are created inside your project repository. Assignees will automatically be populated by the `members` section of the `hardening_manifest.yaml` file and will have the label `Bug`. + + + + + /label ~"Container::Initial" \ No newline at end of file diff --git a/.gitlab/issue_templates/Application - Update.md b/.gitlab/issue_templates/Application - Update.md index 15567be..569e75d 100644 --- a/.gitlab/issue_templates/Application - Update.md +++ b/.gitlab/issue_templates/Application - Update.md @@ -13,18 +13,38 @@ Updated version: (State the version you would like the application updated to) Under support: (Is the updated version within the same major version of the application or is this a new major version?) +## Communication + +All communication should occur through this issue. This ensures that all information is documented in a centralized location and also ensures that all of the assignees are notified of updates. It is imperative that all required parties are listed as assignees of this issue and that individuals are not removed. Please do not remove anyone from the assignee list. + +If you need to contact the Container Hardening team, please identify your assigned point of contact. You can find your point of contact by: +1. They should be listed as an assignee on this ticket +2. They should be listed in the `hardening_manifest.yaml` file under the `maintainers` section with a field of `cht_member: true` + +If you have no assignee, feel free to tag Container Hardening leadership in your issue by commenting on this issue with your questions/concerns and then add `/cc @ironbank-notifications/leadership`. Gitlab will automatically notify all Container Hardening leadership to look at this issue and respond. + + +## Responsibilities + +If this application is owned by a Contributor or Vendor (identifed as `Owner::Contributor` and `Owner::Vendor` respectively), then it is your responsibility to drive this issue through completion. This means that the Container Hardening team is not here to help push any deadlines/timeframes you may have with other customers or DoD agencies. If you have issues with the activity, you may notify Container Hardening leadership above. Do not change the ownership labels. + + ## Definition of Done Hardening: -- [ ] Container builds successfully -- [ ] Container version has been updated in greylist file +- [ ] Hardening manifest has been updated and adheres to the schema (https://repo1.dsop.io/ironbank-tools/ironbank-pipeline/-/blob/master/schema/hardening_manifest.schema.json) +- [ ] Container builds successfully throughthe Gitlab CI pipeline - [ ] Branch has been merged into `development` +- [ ] Project is configured for automatic renovate updates (if possible) No new findings: -- [ ] There are no new findings in this update. Skip the Justifications and Approval Process steps and apply the label ~"Approval". +- [ ] There are no new findings in this update. Skip the Justifications and Approval Process steps and apply the label `Approval` Justifications: - [ ] All findings have been justified per the above documentation - [ ] Justifications have been provided to the container hardening team +- [ ] Skip the Justifications and Approval Process steps and apply the label `Approval` + +Note: The justifications must be provided in a timely fashion. Failure to do so could result in new findings being identified which may start this process over. Approval Process: - [ ] Peer review from Container Hardening Team @@ -32,6 +52,31 @@ Approval Process: - [ ] Approval request has been sent to Authorizing Official - [ ] Approval request has been processed by Authorizing Official +Note: If the above approval process is kicked back for any reason, the `Approval` label will be removed and the issue will be sent back to `Open`. Any comments will be listed in this issue for you to address. Once they have been addressed, you may re-add the `Approval` label. + + +## Post Approval + +### Continuous Monitoring + +Once a container is approved, the `Approved` label will be applied to this issue and it will be closed. You will be able to find your applications on http://ironbank.dsop.io and https://registry1.dsop.io. + +In addition to the above, your application will now be subscribed to continuous monitoring. This means that any new findings discovered as part of this will need justifications. To satisfy this process, any new findings will trigger a new Gitlab issue in this project with the label `Container::New Findings`. All members listed in the `maintainers` section of the `hardening_manifest.yaml` file will automatically be assigned. It is your responsibility as a Contributor or Vendor to monitor for this and provide justifications in a timely fashion. This newly created issue will have all the instructions necessary to complete the process. Failure to provide justifications could result in the revocation of the application's approval status. + +### Updates + +It is imperative that application updates be submitted as quickly as possible. We do not want applications to become stale. To help with this process, Ironbank recommends using a tool called [Renovate](https://github.com/renovatebot/renovate). This requires a `renovate.json` file to be placed in your project and can automate the creation of issues and merge requests. + +If not using Renovate, it will be up to you as a Contributor or Vendor to keep this image up-to-date at all times. When you wish to submit an application update, you must create a new issue in this project using the `Application - Update` template and associate it with the corresponding merge request. If you submit a merge request alone, work will not proceed until a related issue is created. These issues are tracked using the label `Container::Update`. + +Additionally, it is imperative that all updates must be followed through to completion. Simply submitting an application update but not following through on justifications and approvals does not suffice and risk your application's approval status being revoked. + +### Bugs + +Occassionally, users may file bug reports for your application. It is your responsibility to monitor for these since they are created inside your project repository. Assignees will automatically be populated by the `members` section of the `hardening_manifest.yaml` file and will have the label `Bug`. + + + /label ~"Container::Update" \ No newline at end of file diff --git a/.gitlab/issue_templates/Bug.md b/.gitlab/issue_templates/Bug.md index 2030c24..069eaf0 100644 --- a/.gitlab/issue_templates/Bug.md +++ b/.gitlab/issue_templates/Bug.md @@ -33,4 +33,9 @@ logs, and code as it's very hard to read otherwise.) - [ ] Bug has been identified and corrected within the container + + + + + /label ~Bug \ No newline at end of file diff --git a/.gitlab/issue_templates/Feature Request.md b/.gitlab/issue_templates/Feature Request.md index 94aea9a..aad0671 100644 --- a/.gitlab/issue_templates/Feature Request.md +++ b/.gitlab/issue_templates/Feature Request.md @@ -28,4 +28,9 @@ - [ ] Feature has been implemented + + + + + /label ~Feature \ No newline at end of file diff --git a/.gitlab/issue_templates/Leadership Question.md b/.gitlab/issue_templates/Leadership Question.md index 4674f82..b2cf9e5 100644 --- a/.gitlab/issue_templates/Leadership Question.md +++ b/.gitlab/issue_templates/Leadership Question.md @@ -3,5 +3,10 @@ (Detailed description of the question you'd like to ask the leadership team) + + + + + /label ~"Question::Leadership" ~"To Do" /cc @ironbank-notifications/leadership \ No newline at end of file diff --git a/.gitlab/issue_templates/New Findings.md b/.gitlab/issue_templates/New Findings.md index 1fd613d..867f832 100644 --- a/.gitlab/issue_templates/New Findings.md +++ b/.gitlab/issue_templates/New Findings.md @@ -8,12 +8,20 @@ Container has new findings discovered during continuous monitoring. Justifications: - [ ] All findings have been justified - [ ] Justifications have been provided to the container hardening team +- [ ] `Approval` label has been applied + +Note: The justifications must be provided in a timely fashion. Failure to do so could result in new findings being identified which may start this process over. Approval Process: - [ ] Findings Approver has reviewed and approved all justifications - [ ] Approval request has been sent to Authorizing Official - [ ] Approval request has been processed by Authorizing Official +Note: If the above approval process is kicked back for any reason, the `Approval` label will be removed and the issue will be sent back to `Open`. Any comments will be listed in this issue for you to address. Once they have been addressed, you may re-add the `Approval` label. + + + + /label ~"Container::New Findings" \ No newline at end of file diff --git a/.gitlab/issue_templates/Onboarding Question.md b/.gitlab/issue_templates/Onboarding Question.md index 77dea11..ae8011e 100644 --- a/.gitlab/issue_templates/Onboarding Question.md +++ b/.gitlab/issue_templates/Onboarding Question.md @@ -3,5 +3,10 @@ (Detailed description of the question you'd like to ask the onboarding team) + + + + + /label ~"Question::Onboarding" ~"To Do" /cc @ironbank-notifications/onboarding \ No newline at end of file diff --git a/.gitlab/issue_templates/Pipeline Failure.md b/.gitlab/issue_templates/Pipeline Failure.md index dd6ab98..36aa982 100644 --- a/.gitlab/issue_templates/Pipeline Failure.md +++ b/.gitlab/issue_templates/Pipeline Failure.md @@ -27,4 +27,10 @@ - [ ] Pipeline failure has been resolved + + + + + + /label ~Pipeline \ No newline at end of file -- GitLab From 18083444a0bee1bf7bd3b8d5ec0aa10787f6c909 Mon Sep 17 00:00:00 2001 From: Al Fontaine Date: Thu, 8 Jul 2021 16:21:26 +0000 Subject: [PATCH 27/27] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index ee049db..94540e0 100644 --- a/README.md +++ b/README.md @@ -659,4 +659,4 @@ Many thanks to: ## License -Apache 2.0 © [tchiotludo](https://github.com/tchiotludo) \ No newline at end of file +Apache 2.0 © [tchiotludo](https://github.com/tchiotludo) -- GitLab