Bug: missing property in confluence.cfg.xml.j2 causes clustering/backups to break

Summary

struts.multipart.saveDir appears to be a new property that is required for various scheduled jobs to work in Confluence. It appears that this property is required for new nodes to be able to join the cluster. Additionally, this breaks the, although not recommended but still convenient, built-in backup function Confluence provides. It also breaks other various scheduled jobs that need to save files such as index snapshotting.

If I had to guess, Atlassian has replaced webwork.multipart.saveDir in favor of the new property.

Further testing indicates this mainly affects clustering for clusters that already have some index history. In other words, small indexes from a brand new cluster don't seem to have this issue as they appear to go down a separate startup path.

New nodes in established clusters will attempt to use a index snapshot stored in the shared-home. However, if this property isn't defined, index shapshots are not available, or at the very least, the new node is forced to use an old index snapshot assuming one existed prior to the update that required the new property. Not having or using an old index snapshot appears to be what prevents new nodes from joining the cluster.

Steps to reproduce

(How one can reproduce the issue - this is very important)

1. Start Confluence
2. Pod generates confluence.cfg.xml from template
3. In the admin panel, go to Backup and restore
4. Start a site backup

What is the current bug behavior?

Backup fails due to property not being set

What is the expected correct behavior?

Backup succeeds and is available to be used

Relevant logs and/or screenshots

com.atlassian.confluence.backuprestore.exception.BackupRestoreException: java.lang.IllegalStateException: struts.multipart.saveDir property has not been defined
    at com.atlassian.confluence.impl.backuprestore.backup.SiteBackupService.doBackupSynchronously(SiteBackupService.java:122)
    at com.atlassian.confluence.impl.backuprestore.BackupRestoreJobRunner.runJobSynchronously(BackupRestoreJobRunner.java:54)
    at com.atlassian.confluence.impl.backuprestore.ConfluenceBackupRestoreManager.runJobAndUpdateJobProperties(ConfluenceBackupRestoreManager.java:277)
    at com.atlassian.confluence.impl.backuprestore.ConfluenceBackupRestoreManager.processJobsFromTheQueue(ConfluenceBackupRestoreManager.java:259)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.IllegalStateException: struts.multipart.saveDir property has not been defined
    at com.atlassian.confluence.impl.filestore.ConfluenceFileStoreDirectories.resolveBootstrapPathProperty(ConfluenceFileStoreDirectories.java:44)
    at com.atlassian.confluence.impl.filestore.ConfluenceFileStoreDirectories.getTempDirectory(ConfluenceFileStoreDirectories.java:33)
    at com.atlassian.confluence.impl.backuprestore.backup.container.BackupContainerWriterFactoryImpl.createBackupContainerWriter(BackupContainerWriterFactoryImpl.java:72)
    at com.atlassian.confluence.impl.backuprestore.backup.SiteBackupService.doBackupSynchronously(SiteBackupService.java:96)
    ... 8 more

Possible fixes

Add <property name="struts.multipart.saveDir">${localHome}/temp</property> just below https://repo1.dso.mil/dsop/atlassian/confluence-data-center/confluence-node/-/blob/27c0e3b43e9746a78e127b8be23b53455e4700e2/scripts/config/confluence.cfg.xml.j2#L11

Workarounds

1. Ensure Confluence doesn't overwrite the file on startup by making sure the atl_force_cfg_update environment variable is not supplied or set to "False".
2. Add <property name="struts.multipart.saveDir">${localHome}/temp</property> to your confluence.cfg.xml in your local home
3. Restart the pod
4. Repeat for all Confluence pods

Tasks

• Bug has been identified and corrected within the container

Please read the Iron Bank Documentation for more info