UNCLASSIFIED - NO CUI

Skip to content

SKIP UPDATE CHECK Handle pipe escaping differently for strings and objects

Wesley Griffing requested to merge escape-pipe-in-objects into master

General MR

Summary

Recently, an update to loki's README.md broke the nightly doc compiler pipeline. After investigation, I discovered that | characters need to be escaped differently for objects than for strings. The error occurs due to this line.

This change preserves the existing behavior for all types other than objects. For objects, pipes will be escaped with 2 backslashes so that the resulting output is compatible with docs_compiler's usage of json.loads.

While investigating, I found 2 locations where a | character was present within a multi-line string (as opposed to just being at the start). I went alphabetically through the packages and stopped at keycloak as I believe I found enough samples to be certain that this MR won't be overly broad.

  1. The loki values.yaml that caused the pipeline failure.
    • In loki, the enterprise entry contains the config as part of an object.
  2. In keycloak: https://repo1.dso.mil/big-bang/product/packages/keycloak/-/blob/main/chart/values.yaml?ref_type=heads#L168
    • In keycloak, affinity is treated as a string.

Relevant logs/screenshots

Link to failing job: https://repo1.dso.mil/big-bang/team/tools/bb-docs-compiler/-/jobs/34866065

Error message:

──────────────────────────────  loki@6.5.2-bb.0  ───────────────────────────────
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/builds/big-bang/team/tools/scheduled-build/docs_compiler/cli.py", line 278, in compiler
    compile(bb, ref)
  File "/builds/big-bang/team/tools/scheduled-build/docs_compiler/cli.py", line 142, in compile
    write_values_md(f"docs/packages/{pkg}/values.md", values_table, pkg)
  File "/builds/big-bang/team/tools/scheduled-build/docs_compiler/utils.py", line 73, in write_values_md
    yaml.dump(json.loads(data["Default"]), indent=2).split(r"\n")
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
               ^^^^^^^^^^^^^^^^^^^^^^
json.decoder.JSONDecodeError: Invalid \escape: line 1 column 254 (char 253)

Linked Issue

loki#140 (closed)

Upgrade Notices

N/A

Testing

I used this version of _templates.gotmpl to regenerate the README.md for both loki and keycloak.

In the case of loki, the output changed (expected behavior) and was compatible with docs_compiler (verified by running docs_compiler locally with a pair of branches -- 1 umbrella, 1 loki -- where the changed README.md was committed).

In the case of keycloak, the output did not change (expected behavior).

Edited by Wesley Griffing

Merge request reports