Flagging material for review on KSAT change
What problem(s) will it solve?
Right now if a KSAT or requirement changes the mapped content still shows it as meeting the requirement. This could potentially lead to content which doesn't satisfy the requirement being presented as meeting the requirement (false positive).
Who is the intended user(s)?
KSAT/requirements owners/managers - When they make changes to requirements they want to know it will drive changes to content. Right now if we update a requirement (and don't change the ID) it immediately creates a false positive. This might be concerning to requirements owners as they don't understand how Content maintainers will detect their changes and then act on them. This might require them to manually tell content maintainers about the change and rely on this communication to drive work.
Given a KSAT manager is editing a requirements.json (e.g. abilities.json) When They commit a change to a KSAT record in that file Then A pipeline job should detect associated training & eval content and produce a report of affected training/eval items. And Issues should be created for each affected item to "review impact of "
Content and Eval maintainers - Cares about this because they need to know what to work on and don't want to be blamed if content is out of date or doesn't actually meet the requirements.
Leadership - Cares about this because they are ultimately responsible if content goes wrong and not having a mechanism to detect this problem is a risk to them. Implementing this feature would increase the confidence in the overall system and how it helps us to do our jobs more effectively and efficiently.
Lay out the expected user experience
- Requirements owners would make a change to their requirement.
- Upon acceptance of this change the system should detect that a change to a requirement has happened and all associated content (training and eval) should have a change in their state from "good" to "needs review". This should be reflected in the UI/visualizations and status reports.
- Content owners (training and eval) would see that their material status has changed per above and add that task to their backlog/todo list.
- Content owners complete their reviews (which may or may not require updates to content) and submit an MR to clear the flag. The flag would be cleared once the MR was accepted. NOTE: Results of the review could be different but aren't necessarily relevant to this ticket (e.g. delete, update, make new)
- Finally, the UI/visualizations and reports would update to reflect the cleared flags.
Proposal
When a KSAT changes we want to be able to flag material as requiring a review so that we can re-validate that it still meets the requirement.
Option 1: Create a flag - Create a "Needs review" kay:value pair in the association between KSAT/Requirements and Mapped content. This flag could be toggled by a script or automation if the KSAT/Req changes. Then when a review has been completed a MR could toggle it back.
Option 2: Clear the mapping - Another option is to remove the mapping upon a change to the requirement. This could be accomplished a number of different ways:
-
Hash - If mapping is done using a hash of key aspects of the KSAT/Requirement (such as hash (ID, Desc, Prof) then anytime those elements change the hash would change and the mapping would error out.
-
Workflow - We could update directions and state that changes to KSAT's MUST create new KSAT IDs. This would many be handled during KSAT creation.
Option 3 - An issue is opened for each mapped training and Eval item. This issue is created automatically on detection of the change and would capture the requirement to review the materials mapping.
Given a KSAT manager is editing a requirements.json (e.g. abilities.json)
When They commit a change to a KSAT record in that file
Then A pipeline job should detect associated training & eval content and produce a report of affected training/eval items.
And Issues should be created for each affected item to "review impact of "
Other options: ? (Any suggestions welcome)
Further details
Application - Would apply to both mapped eval items as well as training content as each need to be reviewed to ensure they are still accurately mapped.
Permissions and Security
Should not affect permissions and/or Security
Documentation
- We'll need to describe this as a feedback loop for requirements owners and content owners in their documentation.
- Need to update the developer docs to describe the job(s).
- We should also update the CYT overview brief to show that the control/risk has been satisfied (elimination of False Positive).
Availability & Testing
Change description on MTTL --> Should set flag for all Change Topic on MTTL --> Should set flag for all Change proficiency in Workrole mapping --> Should set flag for that workrole needs more test cases probably
What does success look like, and how can we measure that?
Hard to Measure
- Confidence that the system detects changes is improved
Easier to measure
- New reporting for the status of "Needs Review"
Define both the success metrics and acceptance criteria. Note that success metrics indicate the desired business outcomes, while acceptance criteria indicate when the solution is working correctly. If there is no way to measure success, link to an issue that will implement a way to measure this.