Replica Count isn't used

added 1 deleted label

added BB Automated Release Process statusdoing labels

removed BB Automated Release Process label

I believe this is an issue with the way Mattermost prioritizes size over replicas. From the operator spec:

Size defines the size of the Mattermost. This is typically
specified in number of users. This will override replica and resource
requests/limits appropriately for the provided number of users.
values of resources. Accepted values are: 100users, 1000users, 5000users,
10000users, and 250000users. If replicas and resource requests/limits
are not specified, and Size is not provided the configuration for
5000users will be applied. Setting ''Replicas'', ''Scheduling.Resources'',
''FileStore.Replicas'', ''FileStore.Resource'', ''Database.Replicas'',
or ''Database.Resources'' will override the values set by Size.
Setting new Size will override previous values regardless if set
by Size or manually.

If I'm reading this correctly, the replica count you are seeing is because we have a default of 100 users for that size value:

https://repo1.dso.mil/platform-one/big-bang/apps/collaboration-tools/mattermost/-/blob/main/chart/values.yaml#L42

I believe the size/users field of the spec is required (I will double check this though). If that's the case, I'm not sure if there is much we can do the get around this besides removing the replicaCount line since it has no effect. I'll do more testing to double check. Thoughts?

Just tested and confirmed a couple things:

users/size is NOT required
if no users/size is provided the replica count works as desired

Is it safe to assume we prefer to specify replica count over Mattermost deciding that based on user count? If so I can null users in the default values @runyontr

@megamind reassigning this to you since you have the best background on Mattermost scaling and such. Right now we are setting a default of 100users for Mattermost (and a replicaCount of 3 which is overridden by the users). We're trying to determine what the best defaults/config options are for customers. Currently someone could change the users config but changing explicit replica config does nothing unless users is nulled out.

Looking for your recommendation on the best way to handle this for Big Bang going forward.

assigned to @megamind and unassigned @micah.nagel

removed statusdoing label

Yeah we stopped using the arbitrary users counts and instead did explicit sizing because the values were too generic when we were surging to tens of thousands of users. User sizing does indeed override and you should do one or the other. I'd actually say remove the user count completely as there are many cases we would want to tune this. Also worth noting that more than one replica will require an E20 license, https://mattermost.com/pricing-self-managed/#pricing-grid-block_5fa2028808529.

Got it, thanks. We can make sure to restructure this to get rid of user sizing (at least by default) and switch to using replicas explicitly.

Interesting you mention the license requirement, Mattermost lets you deploy 3 replicas w/o a license it seems, although maybe they aren't function right:

{"level":"error","ts":1618582207.6445131,"caller":"cluster/cluster.go:245","msg":"This server is not licensed to run in High Availability mode."}

Definitely will try to tackle this in a future sprint to make sure we provide the right values to customers.

unassigned @megamind

added 1 deleted label

added kindbug priority8 labels

Although a customer can still work around this (and we have documented that in the architecture doc), this still reads as a bug to me. Hopefully work this next sprint.