From cdc6ae3b1b07606657897ebd0c639dc170b02b62 Mon Sep 17 00:00:00 2001 From: Jinoy Parekh Date: Tue, 10 Nov 2020 22:07:05 +0000 Subject: [PATCH 1/6] Add LICENSE --- LICENSE | 201 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 201 insertions(+) create mode 100644 LICENSE diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..9c8f3ea --- /dev/null +++ b/LICENSE @@ -0,0 +1,201 @@ + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "{}" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright {yyyy} {name of copyright owner} + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. \ No newline at end of file -- GitLab From 680c4986ef66ac4d4e9caee8bd6f3f7fd1f60e68 Mon Sep 17 00:00:00 2001 From: Jinoy Parekh Date: Tue, 10 Nov 2020 22:09:51 +0000 Subject: [PATCH 2/6] Update README.md --- README.md | 679 +++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 677 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 5dc6fa6..37a8be8 100644 --- a/README.md +++ b/README.md @@ -1,3 +1,678 @@ -# +# AKHQ (previously known as KafkaHQ) -Project template for all Iron Bank container repositories. \ No newline at end of file +![Last Version](https://img.shields.io/github/tag-pre/tchiotludo/akhq.svg) +![License](https://img.shields.io/github/license/tchiotludo/akhq) +![Docker Pull](https://img.shields.io/docker/pulls/tchiotludo/akhq.svg) +![Github Downloads](https://img.shields.io/github/downloads/tchiotludo/akhq/total) +![Github Start](https://img.shields.io/github/stars/tchiotludo/akhq.svg) +![Main](https://github.com/tchiotludo/akhq/workflows/Main/badge.svg) +[![Artifact HUB](https://img.shields.io/endpoint?url=https://artifacthub.io/badge/repository/akhq)](https://artifacthub.io/packages/search?repo=akhq) + +> Kafka GUI for [Apache Kafka](http://kafka.apache.org/) to manage topics, topics data, consumers group, schema registry, connect and more... + +

+ AKHQ for Kafka logo

+ AKHQ for Kafka preview +

+ +## Contents + +- [Features](#features) +- [Quick Preview](#quick-preview) +- [Installation](#installation) + - [Docker](#docker) + - [Stand Alone](#stand-alone) + - [Kubernetes using Helm](#running-in-kubernetes-using-a-helm-chart) +- [Configuration](#configuration) + - [JVM.options file](#run-with-another-jvmoptions-file) + - [Kafka cluster](#kafka-cluster-configuration) + - [AKHQ](#akhq-configuration) + - [Security](#security) + - [Server](#server) + - [Micronaut](#micronaut-configuration) +- [Api](#api) +- [Monitoring Endpoint](#monitoring-endpoint) +- [Development Environment](#development-environment) +- [Schema references](#schema-references) +- [Who's using AKHQ](#whos-using-akhq) + + +## Features + +- **General** + - Works with modern Kafka cluster (1.0+) + - Connection on standard or ssl, sasl cluster + - Multi cluster +- **Topics** + - List + - Configurations view + - Partitions view + - ACLS view + - Consumers groups assignments view + - Node leader & assignments view + - Create a topic + - Configure a topic + - Delete a topic +- **Browse Topic datas** + - View data, offset, key, timestamp & headers + - Automatic deserializarion of avro message encoded with schema registry + - Configurations view + - Logs view + - Delete a record + - Empty a Topic (Delete all the record from one topic) + - Sort view + - Filter per partitions + - Filter with a starting time + - Filter data with a search string +- **Consumer Groups** (only with kafka internal storage, not with old Zookeeper) + - List with lag, topics assignments + - Partitions view & lag + - ACLS view + - Node leader & assignments view + - Display active and pending consumers groups + - Delete a consumer group + - Update consumer group offsets to start / end / timestamp +- **Schema Registry** + - List schema + - Create / Update / Delete a schema + - View and delete individual schema version +- **Connect** + - List connect definition + - Create / Update / Delete a definition + - Pause / Resume / Restart a definition or a task +- **Nodes** + - List + - Configurations view + - Logs view + - Configure a node +- **ACLS** + - List principals + - List principals topic & group acls +- **Authentification and Roles** + - Read only mode + - BasicHttp with roles per user + - User groups configuration + - Filter topics with regexp for current groups + - Ldap configuration to match AKHQ groups/roles + +## New React UI + +Since this is a major rework, the new UI can have some issues, so please [report any issue](https://github.com/tchiotludo/akhq/issues), thanks! + +## Quick preview +* Download [docker-compose.yml](https://raw.githubusercontent.com/tchiotludo/akhq/master/docker-compose.yml) file +* run `docker-compose pull` to be sure to have the last version of AKHQ +* run `docker-compose up` +* go to [http://localhost:8080](http://localhost:8080) + +It will start a Kafka node, a Zookeeper node, a Schema Registry, a Connect, fill with some sample data, start a consumer +group and a kafka stream & start AKHQ. + +## Installation + +First you need a [configuration files](#configuration) in order to configure AKHQ connections to Kafka Brokers. + +### Docker + +```sh +docker run -d \ + -p 8080:8080 \ + -v /tmp/application.yml:/app/application.yml \ + tchiotludo/akhq +``` +* With `-v /tmp/application.yml` must be an absolute path to configuration file +* Go to + + +### Stand Alone +* Install Java 11 +* Download the latest jar on [release page](https://github.com/tchiotludo/akhq/releases) +* Create an [configuration files](#configuration) +* Launch the application with `java -Dmicronaut.config.files=/path/to/application.yml -jar akhq.jar` +* Go to + + +### Running in Kubernetes (using a Helm Chart) + +### Using Helm repository + +* Add the AKHQ helm charts repository: +```sh +helm repo add akhq https://akhq.io/ +``` +* Install or upgrade +```sh +helm upgrade --install akhq akhq/akhq +``` +#### Requirements + +* Chart version >=0.1.1 requires Kubernetes version >=1.14 +* Chart version 0.1.0 works on previous Kubernetes versions +```sh +helm install akhq akhq/akhq --version 0.1.0 +``` + +### Using git +* Clone the repository: +```sh +git clone https://github.com/tchiotludo/akhq && cd akhq/deploy/helm/akhq +``` +* Update helm values located in [deploy/helm/values.yaml](helm/akhq/values.yaml) + * `configuration` values will contains all related configuration that you can find in [application.example.yml](application.example.yml) and will be store in a `ConfigMap` + * `secrets` values will contains all sensitive configurations (with credentials) that you can find in [application.example.yml](application.example.yml) and will be store in `Secret` + * Both values will be merged at startup +* Apply the chart: +```sh +helm install --name=akhq-release-name . +``` + + +## Configuration +Configuration file can by default be provided in either Java properties, YAML, JSON or Groovy files. YML Configuration +file example can be found here :[application.example.yml](application.example.yml) + +### Pass custom Java opts + +By default, the docker container will allow a custom jvn options setting the environnments vars `JAVA_OPTS`. +For example, if you want to change the default timezome, just add `-e "JAVA_OPTS=-Duser.timezone=Europe/Paris"` + +### Run with another jvm.options file + +By default, the docker container will run with a [jvm.options](docker/app/jvm.options) file, you can override it with +your own with an Environment Variable. With the `JVM_OPTS_FILE` environment variable, you can override the jvm.options file by passing +the path of your file instead. + +Override the `JVM_OPTS_FILE` with docker run: + +```sh +docker run -d \ + --env JVM_OPTS_FILE={{path-of-your-jvm.options-file}} + -p 8080:8080 \ + -v /tmp/application.yml:/app/application.yml \ + tchiotludo/akhq +``` + +Override the `JVM_OPTS_FILE` with docker-compose: + +```yaml +version: '3.7' +services: + akhq: + image: tchiotludo/akhq-jvm:dev + environment: + JVM_OPTS_FILE: /app/jvm.options + ports: + - "8080:8080" + volumes: + - /tmp/application.yml:/app/application.yml +``` + +If you do not override the `JVM_OPTS_FILE`, the docker container will take the defaults one instead. + +### Kafka cluster configuration +* `akhq.connections` is a key value configuration with : + * `key`: must be an url friendly (letter, number, _, -, ... dot are not allowed here) string the identify your cluster (`my-cluster-1` and `my-cluster-2` is the example above) + * `properties`: all the configurations found on [Kafka consumer documentation](https://kafka.apache.org/documentation/#consumerconfigs). Most important is `bootstrap.servers` that is a list of host:port of your Kafka brokers. + * `schema-registry`: *(optional)* + * `url`: the schema registry url + * `basic-auth-username`: schema registry basic auth username + * `basic-auth-password`: schema registry basic auth password + * `properties`: all the configurations for registry client, especially ssl configuration + * `connect`: *(optional list, define each connector as a element of a list)* + * `name`: connect name + * `url`: connect url + * `basic-auth-username`: connect basic auth username + * `basic-auth-password`: connect basic auth password + * `ssl-trust-store`: /app/truststore.jks + * `ssl-trust-store-password`: trust-store-password + * `ssl-key-store`: /app/truststore.jks + * `ssl-key-store-password`: key-store-password + +#### SSL Kafka Cluster with basic auth +Configuration example for kafka cluster secured by ssl for saas provider like aiven (full https & basic auth): + +You need to generate a jks & p12 file from pem, cert files give by saas provider. +```bash +openssl pkcs12 -export -inkey service.key -in service.cert -out client.keystore.p12 -name service_key +keytool -import -file ca.pem -alias CA -keystore client.truststore.jks +``` + +Configurations will look like this example: + +```yaml +akhq: + connections: + ssl-dev: + properties: + bootstrap.servers: "{{host}}.aivencloud.com:12835" + security.protocol: SSL + ssl.truststore.location: {{path}}/avnadmin.truststore.jks + ssl.truststore.password: {{password}} + ssl.keystore.type: "PKCS12" + ssl.keystore.location: {{path}}/avnadmin.keystore.p12 + ssl.keystore.password: {{password}} + ssl.key.password: {{password}} + schema-registry: + url: "https://{{host}}.aivencloud.com:12838" + basic-auth-username: avnadmin + basic-auth-password: {{password}} + properties: {} + connect: + - name: connect-1 + url: "https://{{host}}.aivencloud.com:{{port}}" + basic-auth-username: avnadmin + basic-auth-password: {{password}} +``` + +### AKHQ configuration + +#### Pagination +* `akhq.pagination.page-size` number of topics per page (default : 25) + +#### Topic List +* `akhq.topic.default-view` is default list view (ALL, HIDE_INTERNAL, HIDE_INTERNAL_STREAM, HIDE_STREAM) +* `akhq.topic.internal-regexps` is list of regexp to be considered as internal (internal topic can't be deleted or updated) +* `akhq.topic.stream-regexps` is list of regexp to be considered as internal stream topic + +#### Topic creation default values + +These parameters are the default values used in the topic creation page. + +* `akhq.topic.retention` Default retention in ms +* `akhq.topic.replication` Default number of replica to use +* `akhq.topic.partition` Default number of partition + +#### Topic Data +* `akhq.topic-data.sort`: default sort order (OLDEST, NEWEST) (default: OLDEST) +* `akhq.topic-data.size`: max record per page (default: 50) +* `akhq.topic-data.poll-timeout`: The time, in milliseconds, spent waiting in poll if data is not available in the + buffer (default: 1000). + + +### Security +* `akhq.security.default-group`: Default group for all the user even unlogged user. +By default, the default group is `admin` and allow you all read / write access on the whole app. + +By default, security & roles is enabled by default but anonymous user have full access. You can completely disabled +security with `micronaut.security.enabled: false`. + +If you need a read-only application, simply add this to your configuration files : +```yaml +akhq: + security: + default-group: reader +``` + + + +#### Auth + +##### JWT + +AKHQ uses JWT tokens to perform authentication. +Please generate a secret that is at least 256 bits and change the config like this: + +```yaml +micronaut: + security: + enabled: true + token: + jwt: + signatures: + secret: + generator: + secret: +``` + +##### Groups + +Groups allow you to limit user + +Define groups with specific roles for your users +* `akhq.security.default-group`: Default group for all the user even unlogged user + +* `akhq.security.groups`: Groups list definition + * `- name: group-name` Group identifier + * `roles`: Roles list for the group + * `attributes.topics-filter-regexp`: Regexp to filter topics available for current group + * `attributes.connects-filter-regexp`: Regexp to filter Connect tasks available for current group + + +3 defaults group are available : +- `admin` with all right +- `reader` with only read acces on all AKHQ +- `no-roles` without any roles, that force user to login + +##### Basic Auth +* `akhq.security.basic-auth`: List user & password with affected roles + * `- username: actual-username`: Login of the current user as a yaml key (may be anything email, login, ...) + * `password`: Password in sha256 (default) or bcrypt. The password can be converted + * For default SHA256, with command `echo -n "password" | sha256sum` or Ansible filter `{{ 'password' | hash('sha256') }}` + * For BCrypt, with Ansible filter `{{ 'password' | password_hash('blowfish') }}` + * `passwordHash`: Password hashing algorithm, either `SHA256` or `BCRYPT` + * `groups`: Groups for current user + +> Take care that basic auth will use session store in server **memory**. If your instance is behind a reverse proxy or a +> loadbalancer, you will need to forward the session cookie named `SESSION` and / or use +> [sesssion stickiness](https://en.wikipedia.org/wiki/Load_balancing_(computing)#Persistence) + +Configure basic-auth connection in AKHQ +```yaml +akhq.security: + basic-auth: + - username: admin + password: "$2a$" + passwordHash: BCRYPT + groups: + - admin + - username: reader + password: "" + groups: + - reader +``` + +##### LDAP +Configure how the ldap groups will be matched in AKHQ groups +* `akhq.security.ldap.groups`: Ldap groups list + * `- name: ldap-group-name`: Ldap group name (same name as in ldap) + * `groups`: AKHQ group list to be used for current ldap group + +Example using [online ldap test server](https://www.forumsys.com/tutorials/integration-how-to/ldap/online-ldap-test-server/) + +Configure ldap connection in micronaut +```yaml +micronaut: + security: + ldap: + default: + enabled: true + context: + server: 'ldap://ldap.forumsys.com:389' + managerDn: 'cn=read-only-admin,dc=example,dc=com' + managerPassword: 'password' + search: + base: "dc=example,dc=com" + groups: + enabled: true + base: "dc=example,dc=com" +``` + +If you want to enable anonymous auth to your LDAP server you can pass : +```yaml +managerDn: '' +managerPassword: '' +``` + +Debuging ldap connection can be done with +```bash +curl -i -X POST -H "Content-Type: application/json" \ + -d '{ "configuredLevel": "TRACE" }' \ + http://localhost:8080/loggers/io.micronaut.configuration.security +``` + + +Configure AKHQ groups and Ldap groups and users +```yaml +akhq: + security: + groups: + - name: topic-reader # Group name + roles: # roles for the group + - topic/read + attributes: + # Regexp to filter topic available for group + topics-filter-regexp: "test\\.reader.*" + connects-filter-regexp: "^test.*$" + - name: topic-writer # Group name + roles: + - topic/read + - topic/insert + - topic/delete + - topic/config/update + attributes: + topics-filter-regexp: "test.*" + connects-filter-regexp: "^test.*$" + ldap: + groups: + - name: mathematicians + groups: + - topic-reader + - name: scientists + groups: + - topic-reader + - topic-writer + users: + - username: franz + groups: + - topic-reader + - topic-writer + +``` + +### OIDC +To enable OIDC in the application, you'll first have to enable OIDC in micronaut: + +```yaml +micronaut: + security: + oauth2: + enabled: true + clients: + google: + client-id: "" + client-secret: "" + openid: + issuer: "" +``` + +To further tell AKHQ to display OIDC options on the login page and customize claim mapping, configure OIDC in the AKHQ config: + +```yaml +akhq: + security: + oidc: + enabled: true + providers: + google: + label: "Login with Google" + username-field: preferred_username + groups-field: roles + default-group: topic-reader + groups: + - name: mathematicians + groups: + - topic-reader + - name: scientists + groups: + - topic-reader + - topic-writer + users: + - username: franz + groups: + - topic-reader + - topic-writer +``` + +The username field can be any string field, the roles field has to be a JSON array. + +### Server +* `micronaut.server.context-path`: if behind a reverse proxy, path to akhq with trailing slash (optional). Example: + akhq is behind a reverse proxy with url , set base-path: "/akhq/". Not needed if you're + behind a reverse proxy with subdomain + +### Kafka admin / producer / consumer default properties +* `akhq.clients-defaults.{{admin|producer|consumer}}.properties`: default configuration for admin producer or + consumer. All properties from [Kafka documentation](https://kafka.apache.org/documentation/) is available. + +### Micronaut configuration +> Since AKHQ is based on [Micronaut](https://micronaut.io/), you can customize configurations (server port, ssl, ...) with [Micronaut configuration](https://docs.micronaut.io/snapshot/guide/configurationreference.html#io.micronaut.http.server.HttpServerConfiguration). +> More information can be found on [Micronaut documentation](https://docs.micronaut.io/snapshot/guide/index.html#config) + +### Docker +AKHQ docker image support 3 environment variables to handle configuraiton : +* `AKHQ_CONFIGURATION`: a string that contains the full configuration in yml that will be written on + /app/configuration.yml on container. +* `MICRONAUT_APPLICATION_JSON`: a string that contains the full configuration in JSON format +* `MICRONAUT_CONFIG_FILES`: a path to to a configuration file on container. Default path is `/app/application.yml` + +#### How to mount configuration file + +Take care when you mount configuration files to not remove akhq files located on /app. +You need to explicitely mount the `/app/application.yml` and not mount the `/app` directory. +This will remove the AKHQ binnaries and give you this error: ` +/usr/local/bin/docker-entrypoint.sh: 9: exec: ./akhq: not found` + +```yaml +volumeMounts: +- mountPath: /app/application.yml + subPath: application.yml + name: config + readOnly: true + +``` + +## Api +An **experimental** api is available that allow you to fetch all the exposed on AKHQ through api. + +Take care that this api is **experimental** and **will** change in a future release. +Some endpoint expose too many datas and is slow to fetch, and we will remove +some properties in a future in order to be fast. + +Example: List topic endpoint expose log dir, consumer groups, offsets. Fetching all of theses +is slow for now and we will remove these in a future. + +You can discover the api endpoint here : +* `/api`: a [RapiDoc](https://mrin9.github.io/RapiDoc/) webpage that document all the endpoints. +* `/swagger/akhq.yml`: a full [OpenApi](https://www.openapis.org/) specifications files + +## Monitoring endpoint +Several monitoring endpoint is enabled by default. You can disabled it or restrict access only for authenticated users +following micronaut configuration below. + +* `/info` [Info Endpoint](https://docs.micronaut.io/snapshot/guide/index.html#infoEndpoint) with git status + informations. +* `/health` [Health Endpoint](https://docs.micronaut.io/snapshot/guide/index.html#healthEndpoint) +* `/loggers` [Loggers Endpoint](https://docs.micronaut.io/snapshot/guide/index.html#loggersEndpoint) +* `/metrics` [Metrics Endpoint](https://docs.micronaut.io/snapshot/guide/index.html#metricsEndpoint) +* `/prometheus` [Prometheus Endpoint](https://micronaut-projects.github.io/micronaut-micrometer/latest/guide/) + +## Debugging AKHQ performance issues + +You can debug all query duration from AKHQ with this commands +```bash +curl -i -X POST -H "Content-Type: application/json" \ + -d '{ "configuredLevel": "TRACE" }' \ + http://localhost:8080/loggers/org.akhq +``` + +## Development Environment + +### Early dev image + +You can have access to last feature / bug fix with docker dev image automatically build on tag `dev` +```bash +docker pull tchiotludo/akhq:dev +``` + +The dev jar is not publish on GitHub, you have 2 solutions to have the `dev` jar : + +Get it from docker image +```bash +docker pull tchiotludo/akhq:dev +docker run --rm --name=akhq -it tchiotludo/akhq:dev +docker cp akhq:/app/akhq.jar . +``` +Or build it with a `./gradlew shadowJar`, the jar will be located here `build/libs/akhq-*.jar` + + +### Development Server + +A docker-compose is provide to start a development environnement. +Just install docker & docker-compose, clone the repository and issue a simple `docker-compose -f docker-compose-dev.yml up` to start a dev server. +Dev server is a java server & webpack-dev-server with live reload. + +The configuration for the dev server is in `application.dev.yml`. + +## Schema references + +Since Confluent 5.5.0, Avro schemas can now be reused by others schemas through schema references. This feature allows to define a schema once and use it as a record type inside one or more schemas. + +When registering new Avro schemas with AKHQ UI, it is now possible to pass a slightly more complex object with a `schema` and a `references` field. + +To register a new schema without references, no need to change anything: + +```json +{ + "name": "Schema1", + "namespace": "org.akhq", + "type": "record", + "fields": [ + { + "name": "description", + "type": "string" + } + ] +} +``` + +To register a new schema with a reference to an already registered schema: + +```json +{ + "schema": { + "name": "Schema2", + "namespace": "org.akhq", + "type": "record", + "fields": [ + { + "name": "name", + "type": "string" + }, + { + "name": "schema1", + "type": "Schema1" + } + ] + }, + "references": [ + { + "name": "Schema1", + "subject": "SCHEMA_1", + "version": 1 + } + ] +} +```` + +Documentation on Confluent 5.5 and schema references can be found [here](https://docs.confluent.io/5.5.0/schema-registry/serdes-develop/index.html). + + +## Who's using AKHQ +* [Adeo](https://www.adeo.com/) +* [Auchan Retail](https://www.auchan-retail.com/) +* [Bell](https://www.bell.ca) +* [BMW Group](https://www.bmwgroup.com) +* [Boulanger](https://www.boulanger.com/) +* [GetYourGuide](https://www.getyourguide.com) +* [Klarna](https://www.klarna.com) +* [La Redoute](https://laredoute.io/) +* [Leroy Merlin](https://www.leroymerlin.fr/) +* [NEXT Technologies](https://www.nextapp.co/) +* [Nuxeo](https://www.nuxeo.com/) +* [Pipedrive](https://www.pipedrive.com) +* [BARMER](https://www.barmer.de/) +* [TVG](https://www.tvg.com) + + +## Credits + +Many thanks to: + +* [JetBrains](https://www.jetbrains.com/?from=AKHQ) for their free OpenSource license. +* Apache, Apache Kafka, Kafka, and associated open source project names are trademarks of the Apache Software Foundation. AKHQ is not affiliated with, endorsed by, or otherwise associated with the Apache Software. + +[![Jetbrains](https://user-images.githubusercontent.com/2064609/55432917-6df7fc00-5594-11e9-90c4-5133fbb6d4da.png)](https://www.jetbrains.com/?from=AKHQ) + + +## License +Apache 2.0 © [tchiotludo](https://github.com/tchiotludo) \ No newline at end of file -- GitLab From 99e43541b6e211f9f7b5cc99fe5ba066953f3ce6 Mon Sep 17 00:00:00 2001 From: Jinoy Parekh Date: Tue, 10 Nov 2020 22:11:37 +0000 Subject: [PATCH 3/6] Update README.md --- README.md | 16 ---------------- 1 file changed, 16 deletions(-) diff --git a/README.md b/README.md index 37a8be8..15402c5 100644 --- a/README.md +++ b/README.md @@ -1,20 +1,4 @@ # AKHQ (previously known as KafkaHQ) - -![Last Version](https://img.shields.io/github/tag-pre/tchiotludo/akhq.svg) -![License](https://img.shields.io/github/license/tchiotludo/akhq) -![Docker Pull](https://img.shields.io/docker/pulls/tchiotludo/akhq.svg) -![Github Downloads](https://img.shields.io/github/downloads/tchiotludo/akhq/total) -![Github Start](https://img.shields.io/github/stars/tchiotludo/akhq.svg) -![Main](https://github.com/tchiotludo/akhq/workflows/Main/badge.svg) -[![Artifact HUB](https://img.shields.io/endpoint?url=https://artifacthub.io/badge/repository/akhq)](https://artifacthub.io/packages/search?repo=akhq) - -> Kafka GUI for [Apache Kafka](http://kafka.apache.org/) to manage topics, topics data, consumers group, schema registry, connect and more... - -

- AKHQ for Kafka logo

- AKHQ for Kafka preview -

- ## Contents - [Features](#features) -- GitLab From 9d559c8dc56d3407c8a4200091a2042959256a92 Mon Sep 17 00:00:00 2001 From: Jinoy Parekh Date: Tue, 10 Nov 2020 22:15:17 +0000 Subject: [PATCH 4/6] Add Jfile --- Jenkinsfile | 2 ++ 1 file changed, 2 insertions(+) create mode 100644 Jenkinsfile diff --git a/Jenkinsfile b/Jenkinsfile new file mode 100644 index 0000000..18f06b0 --- /dev/null +++ b/Jenkinsfile @@ -0,0 +1,2 @@ +@Library('DCCSCR@master') _ +dccscrPipeline(version: "0.16.0") -- GitLab From a58df1ce311de846b4d57939fcd82c4a3d1ba81b Mon Sep 17 00:00:00 2001 From: Jinoy Parekh Date: Tue, 10 Nov 2020 20:28:36 -0600 Subject: [PATCH 5/6] Updated Dfile --- Dockerfile | 31 +++++++++++++++++++++++++++++++ download.yaml | 3 +++ 2 files changed, 34 insertions(+) create mode 100644 Dockerfile create mode 100644 download.yaml diff --git a/Dockerfile b/Dockerfile new file mode 100644 index 0000000..ac739ef --- /dev/null +++ b/Dockerfile @@ -0,0 +1,31 @@ +ARG BASE_REGISTRY=registry1.dsop.io/ironbank +ARG BASE_IMAGE=redhat/openjdk/openjdk11 +ARG BASE_TAG=1.11 + +FROM tchiotludo/akhq:0.16.0 as base + +FROM ${BASE_REGISTRY}/${BASE_IMAGE}:${BASE_TAG} + +ENV LANG=C.UTF-8 + +USER 0 + +LABEL name="kafka dashboard" \ + maintainer="jparekh@vivsoft.io" \ + vendor="tchiotludo" \ + version="0.16.0" \ + release="1" \ + summary="" \ + description="" + +WORKDIR /app + +COPY --from=base /app /app + +ENV MICRONAUT_CONFIG_FILES=/app/application.yml + +ENTRYPOINT ["docker-entrypoint.sh"] + +CMD ["./akhq"] + +HEALTHCHECK NONE \ No newline at end of file diff --git a/download.yaml b/download.yaml new file mode 100644 index 0000000..4565625 --- /dev/null +++ b/download.yaml @@ -0,0 +1,3 @@ +resources: + - url: "docker://docker.io/tchiotludo/akhq@sha256:44e00630f95b0a42c716f635b94cd53a7185aa6626502839f8afa8d8b35bf576" + tag: "tchiotludo/akhq:0.16.0" \ No newline at end of file -- GitLab From fffa9e65657cb58dde341055a7043a6ac4de1b36 Mon Sep 17 00:00:00 2001 From: Jinoy Parekh Date: Wed, 11 Nov 2020 03:00:21 +0000 Subject: [PATCH 6/6] Update Dockerfile with summary/desc. --- Dockerfile | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Dockerfile b/Dockerfile index ac739ef..511c3c9 100644 --- a/Dockerfile +++ b/Dockerfile @@ -15,8 +15,8 @@ LABEL name="kafka dashboard" \ vendor="tchiotludo" \ version="0.16.0" \ release="1" \ - summary="" \ - description="" + summary="Kafka dashboard" \ + description="Kafka GUI for Apache Kafka to manage topics, topics data, consumers group and schema registry" WORKDIR /app -- GitLab