UNCLASSIFIED - NO CUI

Kiali-Operator Fails to Start in BB 2.7.1

Big Bang Version: 2.7.1

I am using the customer template to deploy Big Bang. The kiali-operator does not start successfully. It logs errors - "Failed to get ansible-runner stdout" - see below for the complete error logs. When I change the git tag only for kiali to "1.70.0-bb.0", kiali starts working.

Working configmap entry for kiali:

kiali:
  enabled: true
  git:    
    tag: "1.70.0-bb.0"
  helmRepo:    
    tag: "1.70.0-bb.0"
  sso:
    enabled: true
    client_id: "xxx"
    client_secret: "xxx"

Non-working configmap entry for kiali:

kiali:
  enabled: true
  sso:
    enabled: true
    client_id: "xxx"
    client_secret: "xxx"

Error logs for non-working kiali for BB 2.7.1:

{"level":"error","ts":"2023-08-03T13:18:09Z","logger":"reconciler","msg":"Failed to get ansible-runner stdout","job":"1519298202041714372","name":"flux-system","namespace":"","error":"open /tmp/ansible-operator/runner/v1/Namespace/flux-system/artifacts/1519298202041714372/stdout: no such file or directory","stacktrace":"github.com/operator-framework/operator-sdk/internal/ansible/controller.(*AnsibleOperatorReconciler).Reconcile\n\t/workspace/internal/ansible/controller/reconcile.go:226\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:122\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:323\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:235"}
{"level":"error","ts":"2023-08-03T13:18:09Z","msg":"Reconciler error","controller":"namespace-controller","object":{"name":"flux-system"},"namespace":"","name":"flux-system","reconcileID":"5f7295ae-a08d-4c43-ae93-6b167f258dbb","error":"open /tmp/ansible-operator/runner/v1/Namespace/flux-system/artifacts/1519298202041714372/stdout: no such file or directory","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:235"}
{"level":"error","ts":"2023-08-03T13:18:09Z","logger":"runner","msg":"Traceback (most recent call last):\n  File \"/usr/local/lib/python3.9/site-packages/ansible_runner/__main__.py\", line 874, in main\n    res = run(**run_options)\n  File \"/usr/local/lib/python3.9/site-packages/ansible_runner/interface.py\", line 210, in run\n    r.run()\n  File \"/usr/local/lib/python3.9/site-packages/ansible_runner/runner.py\", line 118, in run\n    self.status_callback('starting')\n  File \"/usr/local/lib/python3.9/site-packages/ansible_runner/runner.py\", line 106, in status_callback\n    ansible_runner.plugins[plugin].status_handler(self.config, status_data)\n  File \"/usr/local/lib/python3.9/site-packages/ansible_runner_http/events.py\", line 35, in status_handler\n    status = send_request(plugin_config['runner_url'],\n  File \"/usr/local/lib/python3.9/site-packages/ansible_runner_http/events.py\", line 18, in send_request\n    return session.post(url_actual, headers=headers, json=(data))\n  File \"/usr/local/lib/python3.9/site-packages/requests/sessions.py\", line 637, in post\n    return self.request(\"POST\", url, data=data, json=json, **kwargs)\n  File \"/usr/local/lib/python3.9/site-packages/requests/sessions.py\", line 589, in request\n    resp = self.send(prep, **send_kwargs)\n  File \"/usr/local/lib/python3.9/site-packages/requests/sessions.py\", line 703, in send\n    r = adapter.send(request, **kwargs)\n  File \"/usr/local/lib/python3.9/site-packages/requests/adapters.py\", line 486, in send\n    resp = conn.urlopen(\n  File \"/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py\", line 790, in urlopen\n    response = self._make_request(\n  File \"/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py\", line 496, in _make_request\n    conn.request(\nTypeError: request() got an unexpected keyword argument 'chunked'\n","job":"6130218263336997435","name":"keycloak","namespace":"","error":"exit status 1","stacktrace":"github.com/operator-framework/operator-sdk/internal/ansible/runner.(*runner).Run.func1\n\t/workspace/internal/ansible/runner/runner.go:269"}
{"level":"error","ts":"2023-08-03T13:18:09Z","logger":"reconciler","msg":"Unable to mark error to run reconciliation","job":"6130218263336997435","name":"keycloak","namespace":"","error":"namespaces \"keycloak\" is forbidden: User \"system:serviceaccount:kiali:kiali-kiali-kiali-operator\" cannot update resource \"namespaces/status\" in API group \"\" in the namespace \"keycloak\"","stacktrace":"github.com/operator-framework/operator-sdk/internal/ansible/controller.(*AnsibleOperatorReconciler).Reconcile\n\t/workspace/internal/ansible/controller/reconcile.go:224\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:122\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:323\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:235"}
{"level":"error","ts":"2023-08-03T13:18:09Z","logger":"reconciler","msg":"Failed to get ansible-runner stdout","job":"6130218263336997435","name":"keycloak","namespace":"","error":"open /tmp/ansible-operator/runner/v1/Namespace/keycloak/artifacts/6130218263336997435/stdout: no such file or directory","stacktrace":"github.com/operator-framework/operator-sdk/internal/ansible/controller.(*AnsibleOperatorReconciler).Reconcile\n\t/workspace/internal/ansible/controller/reconcile.go:226\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:122\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:323\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:235"}
{"level":"error","ts":"2023-08-03T13:18:09Z","msg":"Reconciler error","controller":"namespace-controller","object":{"name":"keycloak"},"namespace":"","name":"keycloak","reconcileID":"836fb995-db69-4544-bc8b-3cfbf2126ca7","error":"open /tmp/ansible-operator/runner/v1/Namespace/keycloak/artifacts/6130218263336997435/stdout: no such file or directory","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:235"}
{"level":"error","ts":"2023-08-03T13:18:09Z","logger":"runner","msg":"Traceback (most recent call last):\n  File \"/usr/local/lib/python3.9/site-packages/ansible_runner/__main__.py\", line 874, in main\n    res = run(**run_options)\n  File \"/usr/local/lib/python3.9/site-packages/ansible_runner/interface.py\", line 210, in run\n    r.run()\n  File \"/usr/local/lib/python3.9/site-packages/ansible_runner/runner.py\", line 118, in run\n    self.status_callback('starting')\n  File \"/usr/local/lib/python3.9/site-packages/ansible_runner/runner.py\", line 106, in status_callback\n    ansible_runner.plugins[plugin].status_handler(self.config, status_data)\n  File \"/usr/local/lib/python3.9/site-packages/ansible_runner_http/events.py\", line 35, in status_handler\n    status = send_request(plugin_config['runner_url'],\n  File \"/usr/local/lib/python3.9/site-packages/ansible_runner_http/events.py\", line 18, in send_request\n    return session.post(url_actual, headers=headers, json=(data))\n  File \"/usr/local/lib/python3.9/site-packages/requests/sessions.py\", line 637, in post\n    return self.request(\"POST\", url, data=data, json=json, **kwargs)\n  File \"/usr/local/lib/python3.9/site-packages/requests/sessions.py\", line 589, in request\n    resp = self.send(prep, **send_kwargs)\n  File \"/usr/local/lib/python3.9/site-packages/requests/sessions.py\", line 703, in send\n    r = adapter.send(request, **kwargs)\n  File \"/usr/local/lib/python3.9/site-packages/requests/adapters.py\", line 486, in send\n    resp = conn.urlopen(\n  File \"/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py\", line 790, in urlopen\n    response = self._make_request(\n  File \"/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py\", line 496, in _make_request\n    conn.request(\nTypeError: request() got an unexpected keyword argument 'chunked'\n","job":"5828395720349580836","name":"logging","namespace":"","error":"exit status 1","stacktrace":"github.com/operator-framework/operator-sdk/internal/ansible/runner.(*runner).Run.func1\n\t/workspace/internal/ansible/runner/runner.go:269"}
{"level":"error","ts":"2023-08-03T13:18:09Z","logger":"reconciler","msg":"Unable to mark error to run reconciliation","job":"5828395720349580836","name":"logging","namespace":"","error":"namespaces \"logging\" is forbidden: User \"system:serviceaccount:kiali:kiali-kiali-kiali-operator\" cannot update resource \"namespaces/status\" in API group \"\" in the namespace \"logging\"","stacktrace":"github.com/operator-framework/operator-sdk/internal/ansible/controller.(*AnsibleOperatorReconciler).Reconcile\n\t/workspace/internal/ansible/controller/reconcile.go:224\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:122\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:323\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:235"}
{"level":"error","ts":"2023-08-03T13:18:09Z","logger":"reconciler","msg":"Failed to get ansible-runner stdout","job":"5828395720349580836","name":"logging","namespace":"","error":"open /tmp/ansible-operator/runner/v1/Namespace/logging/artifacts/5828395720349580836/stdout: no such file or directory","stacktrace":"github.com/operator-framework/operator-sdk/internal/ansible/controller.(*AnsibleOperatorReconciler).Reconcile\n\t/workspace/internal/ansible/controller/reconcile.go:226\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:122\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:323\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:235"}
{"level":"error","ts":"2023-08-03T13:18:09Z","msg":"Reconciler error","controller":"namespace-controller","object":{"name":"logging"},"namespace":"","name":"logging","reconcileID":"4e38f003-db8b-4a38-9840-c54d340b266d","error":"open /tmp/ansible-operator/runner/v1/Namespace/logging/artifacts/5828395720349580836/stdout: no such file or directory","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:235"}

UNCLASSIFIED - NO CUI