Training Surveys for KSAT proficiency and Coverage
Goal
In order to continuously validate the effectiveness of training courses (both commercial and/or organic) I'd like to generate surveys to help people check off if a course satisfies a KSA and if so, at what proficiency level. This is used to measure the effectiveness of training by assessing if the training is teaching the material at the expected level.
This requires someone to first identify which KSATs a particular training course or lesson satisfies. So if a course claims to teach a KSAT it should be put on that courses survey.
The general workflow of a course flow: (@A-10, @caroline.fowler )
-
1a. ID KSAT's to be taught in a particular training course -
1b. ID Training Course -
2. Generate Survey <-- Generate from info in 1a. -
3. Attend training and populate survey -
4. Submit survey as part of training trip report or end of lesson survey -
5. Analysis by vendor and CYT for continuous review and QA
NOTE: Either 1a or 1b can be done first... but both should be done prior to going to training
Completing the Survey
Attendees should not be given the proficiency codes the course is supposed to meet as this may encourage just "pencil whipping" vs actually determining at what level the course taught at. The survey should show proficiency codes appropriate to the type of KSAT.
- T#### : Description: When (day, lesson #, etc.) was this task taught? _____________
- K#### : Description : N/A, A, B, C, D (select one)
- S/A#### : Description : 1a, 2b, 3c, 4d (or break them out # and letter)
***We might want to use plain english in the survey vs 1a/2b/etc. as students/attendees won't be knowledgable about those meanings. Would rather say... "Can you identify these facts from a list of options OR write an essay about why this is important" for knowledge and for performance "1a - I can do this work with help/supervision vs 4 - I'm prepared to teach others how to do this after this class. ***
Conditions: Do not display proficiency on the survey this could influence students to just "go with it" rather than determining the level.
Analysis
Big decisions, Jira vs Gitlab vs google forms vs other...
- Need a way to gather the results and conduct analysis.
- e.g. 4 students said it was a B, 1 student said it was a C.
- Being able to do this by instance would help track changes (better or worse)
- Last time mostly B, now C (or vice versa)
- Do NOT want to do this manually
@caroline.fowler For example if we use Jira, should we use the training request workflow to generate and collect this information via screens? Or do we use it just to point people at surveys on Gitlab (or somewhere lese like google forms?) We'd need to figure out how to get MTTL info back and forth between Jira and Gitlab... doable just want to plan it out before we commit to an analysis platform.
Use cases
1. Gather initial information
We might think a course teaches KSAT's but not know at what level. This allows the attendees to inform us as they go through it.
2. Validate appropriate level
The course is supposed to teach KSAT's at known levels and we want to gather information about whether this is true or not.
Checking for honest effort
We should also have the ability to insert known good information to validate that students are putting in good info. For example if we know a course DOESN'T teach a particular item, we should ask for the proficiency. It SHOULD be "not present" but they might pencil whip it and say something else.
Used to gather information from Vendors
Can we use this survey as part of RFI's to collect what they think their course covers