UNCLASSIFIED - NO CUI

Skip to content

Changes

Page history
migrate old wiki to new repo authored by Jason Thomas's avatar Jason Thomas
Show whitespace changes
Inline Side-by-side
Home.md 0 → 100644
View page @ 0dcf18eb
## Table of Contents
- [Introduction](#introduction)
- [Data file Directory Structure](#data-file-directory-structure)
- [Change Management Workflow](#change-management-workflow)
- [Knowledge, Skill, Ability, and Task Defined](#knowledge-skill-ability-and-task-definition)
- [KSAT Relationships in the MTTL](#ksat-relationships-in-the-mttl)
- [KSAT Proficiency Requirements](#ksat-proficiency-requirements)
- [Knowledge Skill Ability and Task files](#knowledge-skill-ability-and-task-json-files)
- [WORK-ROLE and SPECIALIZATIONS JSON files](#work-roles-and-specializations-json-files)
- [PROF (proficiency) JSON file](#prof-json-file)
- [Rel-Link JSON file](#rel-link-json-file)
- [Pipeline Workflow](#pipeline-workflow)
- [TRN EVL Pipeline In-Depth](#trn-evl-pipeline-in-depth)
# Introduction
This repository design is for the automation of training and evaluation requirements, traceability, and metrics. Below are the requirements of our organization's Master Training Tasks List (MTTL).
1. Must contain all necessary Knowledge, Skills, Abilities, and Tasks (KSA&Ts) for the organizations work roles/specializations
2. We must be able to link every KSA&T to "one or many" work roles/specializations
3. The proficiency of each KSA&T as they relate to their respective work role/specialization.
4. Should be able to trace the training required to accomplish a specific KSA&T at a specific proficiency level
5. Should be able to trace the evaluation material to accomplish a specific KSA&T at a specific proficiency level
6. Individuals should be able to submit a change management request to add/remove/modify a work role/specialization KSA&T.
We have chosen not to use a traditional database for a few reasons: One, traditional databases do not provide the foundational tools necessary to operate in the "[Agile](https://www.guru99.com/agile-scrum-extreme-testing.html)" paradigm. Git (gitlab) provides the infrastructure for change management, distribution of work, and more. Two, gaining access to a Relational/Non-Relational database on our organization's enterprise network takes too much time. Limiting the necessary requirements to operate the tool also allows for potentially fast exportation of the code base to other networks/organizations. Three, the data relationships we are making are simplistic and a traditional database is not necessary to maintain the data as our requirements grow. We will only have a finite number of requirements ever. Gitlab comes with many different tools that we use and we will discuss these further in this document.
In short, we are using [Git](https://git-scm.com/) and [Gitlab](https://gitlab.com/) for change and project management, [Gitlab continuous integration (CI) pipelines](https://about.gitlab.com/stages-devops-lifecycle/continuous-integration/) to join the data and build the presentation layer, and [Gitlab Pages](https://about.gitlab.com/stages-devops-lifecycle/pages/) to host the presentation layer to view the information. To do this we must first organize the data in a logical manner to promote scalability/sustainability. The items below will be necessary to maintain this concept.
1. The MTTL repo will not contain an MTTL file. Instead, the MTTL file will be an artifact that is produced in the repo's CI/CD pipeline.
2. The meat of what makes the MTTL repo is the [knowledge, skill, ability, and task files](#knowledge-skill-ability-and-task-json-files), and the [WORK-ROLES.json and SPECIALIZATIONS.json files](#work-roles-and-specializations-json-files) that contain a list of all work-roles and specializations.
- The [knowledge, skill, ability, and task files](#knowledge-skill-ability-and-task-json-files) will be dictionaries of all of the knowledge, skills, abilities, tasks that are required for the organization. These items will only have the KSA&T to KSA&T relationship information, topic information, requirement source information, and available training and evaluation information. Proficiency is work role/specialization specific and will be located inside the TTL files respectively.
- The [WORK-ROLES.json and SPECIALIZATIONS.json files](#work-roles-and-specializations-json-files) will be dictionaries of work role/specialization specific information which will contain work role/specialization repository URLs that will contain rel-link information to populate training and evaluation coverage of the KSAs
- The [PROF.json file](#prof-json-file) (or proficiency JSON file) is a dictionary of proficiency standards from different organizations. This file will be used as a "Rosetta Stone" to easily compile an MTTL with different proficiency standards if an organization requires it.
## Data file Directory Structure
```sh
MTTL Repo
├─ requirements
│ ├─ KNOWLEDGE.json
│ ├─ SKILLS.json
│ ├─ ABILTIES.json
│ └─ TASKS.json
├─ .gitlab-ci.yml
├─ WORK-ROLES.json
├─ SPECIALIZATIONS.json
└─ PROF.json
```
## Change Management Workflow
```mermaid
graph TD;
START-->ISSUE([Request Change via Issue])
ISSUE-->CHANGE([Make change on dedicated branch])
CHANGE-->IF_PASS{CI/CD}
IF_PASS-- Failed -->CHANGE
IF_PASS-- Passed -->IF_APPROVE{Approved}
IF_APPROVE-- No -->CHANGE
IF_APPROVE-- Yes -->FINISH[Merged to Master]
```
## Knowledge Skill Ability and Task Definition
*The following are definitions for Knowledge, Skill, and Ability according to National Initiative for Cybersecurity Education (NICE) Cybersecurity Workforce Framework (NCWF).*
**Knowledge** is a body of information applied directly to the performance of a function.
**Skill** is often defined as an observable competence to perform a learned psychomotor act.
Skills in the psychomotor domain describe the ability to physically manipulate a tool or
an instrument like a hand or a hammer. Skills needed for cybersecurity rely less on physical
manipulation of tools and instruments and more on applying tools, frameworks, processes.
A skill can be measured objectively via test. (e.g. Did they find a key, compile correctly, pass unit tests, etc.)
**Ability** is competence to perform an observable behavior or a behavior that results in an
observable product.
An ability requires a subjective determination to properly evaluate (e.g. OJT, Scenario, etc.)
*US Cyber Command's Cyberspace Standards Analyst Team (CSAT) defined a task as:*
**Task** A Succession of work activities used to produce a distinguishable and discernable output that can be independently used or consumed. Tasks should align to the roles and responsibilities associated with a work role, function, or position. Task statements should:
- Describe activities (not KSAs)
- Have a certain beginning and end
- Represent activities performed by an individual
- Stated as not too broad nor specific
- Have an identifiable output
## KSAT Relationships in the MTTL
Many, but not all, of our KSAs will be related to one another in a parent to child relationship. The Rel-Link JSON files (defined below) used for each KSA category (excluding Tasks) have a 'parent' key where this relationship is defined. Use the following guidelines when creating these relationships.
| Item | Allowable Parent(s) |
| ------ | ------ |
| Knowledge | Skill, Ability, Task |
| Skill | Ability, Task |
| Ability | Task |
| Task | None |
## KSAT Proficiency Requirements
The following are proficiency codes defined by the JCT&CS
| Knowledge Levels | Definition |
|-----|--------|
| A | An 'A' indicates the individual must be able to identify basic facts and terms about the subject.|
| B | A 'B' indicates the individual must be able to identify relationships of basic facts and state general principles about the subject.|
| C | A 'C' indicates the individual must be able to analyze facts and principles and draw conclusions about the subject.|
| D | A 'D' indicates the individual must be able to evaluate conditions and make proper decisions about the subject.|
| Skill/Ability Levels | Definition |
|-----|--------|
| 1 | A '1' indicates the individual must be familiar with this competency and be generally capable of independently handling simple tasks or assignments.|
| 2 | A '2' indicates the individual must be capable of independently handling some complex tasks or assignments related to this competency, but may need direction and guidance on others.|
| 3 | A '3' indicates the individual must be capable of independently handling a wide variety of complex and or high profile tasks or assignments related to this competency. Must be an authority in this area and or often sought out by others for advice or to teach/mentor others on highly complex or challenging tasks or assignments related to this competency.|
Each KSAT has a required proficiency defined for the associated work role. These proficiency codes are captured and mapped to each KSAT in the work role's TTL JSON files describe below. Furthermore, when creating training or evaluation content that satisfies one of more KSATs, there is an associated Rel-Link JSON File. Although several KSATs may be satisfied in the content related to this file, the proficiency code used should be for the core of the content.
For example, if an evaluation is created to develop a class that implements a singleton design pattern and singleton design has a KSA proficiency of 2, but the problem also addresses a KSA of creating functions which has a proficiency code of 3, the overall proficiency should be 2 to address the core topic of singleton.
## Knowledge Skill Ability and Task JSON Files
This section will describe the knowledge, skill, ability, and task JSON files. These files will contain every possible KSA&T for every work role/specialization in the organization. Each KSA&T will also contain KSA&T to KSA&T dependency information and general training and evaluation material information.
Field information
- "KSAT_ID": This is a unique identifier of the KSA
- "description": this is the actual KSA
- "parent": this is a list of parent KSAs that directly relate to this KSA
- "topic": this is the subject of the material needed to learn
- "requirement_src": this is a list of all documents or individuals that have started this KSA a requirement
- "requirement_owner": this is an individual or organization that is responsible for setting this requirement
- "comments": this is any other information that needs to be captured for better understanding of the requirement, etc.
- "training": this is a list of training, proficiency of specified training, domain the training is located on, and a link to the training that relate the requirement set in the description
- "eval": this is a list of evaluation questions, proficiency of specified questions, domain the eval is located on, and a link to the eval question study material that relate to the requirement description
```json
{
"KSAT_ID": {
"description": "",
"parents": [],
"topic": "",
"requirement_src": [],
"requirement_owner": "",
"comments": "",
"training": [
["TRN_ID", "NAME", "PROF_ID", "NETWORK", "url"],
...
],
"eval": [
["EVL_ID", "NAME", "PROF_ID", "NETWORK", "url"],
...
],
"work-roles": {
"WR_SPEC_ID": "PROF_ID",
...
},
"specializations": {
"WR_SPEC_ID": "PROF_ID",
...
}
},
...
}
```
## WORK-ROLES and SPECIALIZATIONS JSON Files
This section will describe the WORK-ROLES.json and SPECIALIZATIONS.json files. These files currently house information about the evaluation and/or training repository URLs. The URLs will be used to populate the 'training' and 'eval' rel-link information during the 'insert-rel-link-artifacts.py' script. Mandatory field information below.
Field Information
- "WR_SPEC_ID": unique identifier of the work-role/specialization (kabob case)
- "trn-rel-link-urls": urls for training repositories
- "evl-rel-link-urls": urls for eval repositories
- "follow-ons": list of WR_SPEC_IDs that can happen next
```json
{
"WR_SPEC_ID": {
"trn-rel-link-urls": [],
"evl-rel-link-urls": [],
"follow-ons": [
"WR_SPEC_ID,
...
]
},
...
}
```
## PROF JSON File
This section describes the PROF.json file. This is a necessary file for relating multiple proficiency sources across different organizations. This will enable the MTTL to be compiled with different proficiency codes if an organization requires it.
Field Information
- "PROF_ID": will be a single proficiency identifier that the 90COS uses internally
- The rest will be the proficiency source and the proficiency code that source uses the is indicative of the 90th's internal code
```json
{
"PROF_ID": {
"JCT&CS": "",
"ACC": "",
...
},
...
}
```
## Rel-Link JSON File
This section describes the *.rel-link.json file; rel is short for relation. There will be many of these files inside the training/eval repositories that will describe what KSAT and proficiency the actual material adheres to. There will be a job in the training or evaluation repository that will join these files into a single artifact. When the MTTL repository is linked to one of these repositories, it will fetch the joined artifact file and map the "training/eval" fields in the KSAT files (as long as the KSAT_IDs are correct). These fields will then be used for metrics.
The job that will join your *.rel-link.json files will be provided to you and is described in the [TRN EVL Pipeline In-Depth](#trn-evl-pipeline-in-depth) section. Below is a template for the separated *.rel-link.json files that will be present in your repository.
On the job training (OJT) can also be specified in your material repo using a file named *obj*.rel-link.json (just make sure 'ojt' is somewhere in the name). All KSATs specified here will automatically be mapped to the 'eval' list with the name of 'OJT'.
NOTE: Think about how you want to maintain this information as the material evolves. You can use as many of these files to link requirements to your material as you need. For example, if you choose to use a single *.rel-link.json file, you will need to update this single file every time the material changes.
Field Information
- "TRN_EVL_ID": this will be an identifier of the training or evaluation that this material is a part of. NOTE: this ID should be the same ID used in the TRN.json or EVL.json respectively.
- "NETWORK": this will be a string specifying the network a member will need to be on to view this material. Example, 'Commercial', 'NIPR', 'TAZ', etc.
- "KSATs": will be a list of lists. Each list will contain the KSAT_ID, PROF_ID the material is taught at.
NOTE: the name of the *.rel-link.json file should be indicative of the material it represents and will be used in the linking process.
```json
{
"TRN_EVL_ID": "",
"NETWORK": "",
"KSATs": [
["KSAT_ID", "PROF_ID"],
...
]
}
```
## Pipeline Workflow
```mermaid
graph TD;
START-->STAGE_VALIDATE[Stage: Validate]
STAGE_VALIDATE-- Job -->VALIDATE_K([Knowledge])
STAGE_VALIDATE-- Job -->VALIDATE_S([Skills])
STAGE_VALIDATE-- Job -->VALIDATE_A([Abilities])
STAGE_VALIDATE-- Job -->VALIDATE_T([Tasks])
STAGE_VALIDATE-- Job -->INSERT_REL_LINKS([Rel-Links])
STAGE_VALIDATE-- Job -->VALIDATE_WR_SPEC([KSAT Work-Roles/Specs])
STAGE_VALIDATE-- Job -->VALIDATE_CHILDREN([KSAT Children])
VALIDATE_K-->STAGE_PRE_BUILD[Stage: Pre-Build]
VALIDATE_S-->STAGE_PRE_BUILD[Stage: Pre-Build]
VALIDATE_A-->STAGE_PRE_BUILD[Stage: Pre-Build]
VALIDATE_T-->STAGE_PRE_BUILD[Stage: Pre-Build]
VALIDATE_WR_SPEC-->STAGE_PRE_BUILD[Stage: Pre-Build]
INSERT_REL_LINKS-->STAGE_PRE_BUILD[Stage: Pre-Build]
VALIDATE_CHILDREN-->STAGE_PRE_BUILD[Stage: Pre-Build]
STAGE_PRE_BUILD-- Job -->PRE_BUILD_PREP([Build Prep])
PRE_BUILD_PREP-->STAGE_BUILD[Stage: Build]
STAGE_BUILD-- Job -->BUILD_MTTL([MTTL Dataset])
STAGE_BUILD-- Job -->BUILD_TTL([TTL Dataset])
STAGE_BUILD-- Job -->BUILD_EXTRAS([Extra Datasets])
STAGE_BUILD-- Job -->BUILD_METRICS([MTTL Metrics])
BUILD_MTTL-->STAGE_TEST[Stage: Test]
BUILD_TTL-->STAGE_TEST[Stage: Test]
BUILD_EXTRAS-->STAGE_TEST[Stage: Test]
BUILD_METRICS-->STAGE_TEST[Stage: Test]
STAGE_TEST-- Job -->TEST_DUPS([Test Duplicate KSATs])
TEST_DUPS-->STAGE_DEPLOY[Stage: Deploy]
STAGE_DEPLOY-- Job -->DEPLOY_PAGES([Deploy to GitLab Static Pages])
DEPLOY_PAGES-->FINISH>User can see MTTL changes]
```
## TRN EVL Pipeline In-Depth
A single job is provided to you that is used to create artifacts for the MTTL repository; they are described below.
1. The **SaveJson** job. This job is responsible for joining the ```*.rel-link.json``` files together into a single artifact. This artifact will contain more fields than the ones specified in the [Rel-Link JSON file](#rel-link-json-file) section. These fields include, ```'REL_PATH'```, and ```'NAME'```.
- Add the following to your gitlab CI/CD ```.gitlab-ci.yml``` pipeline definition file and the ```data.rel-link.json``` artifact be automatically created.
```yml
SaveJsons:
stage: deploy
image: 90cos/generic_cyt_ci
script:
- git clone https://gitlab.com/90COS/public/scripts.git
- python3 scripts/gen_mttl_metric_data.py
artifacts:
paths:
- data.rel-link.json
only:
- master
```
* Note: To see full functionality, use the script's --help option.
\ No newline at end of file

UNCLASSIFIED - NO CUI