Skip to content

Commit bbc9ea5

Browse files
committed
updating testing document with test strategy
1 parent 2d96bc9 commit bbc9ea5

File tree

1 file changed

+24
-4
lines changed

1 file changed

+24
-4
lines changed

docs/contributing/testing.md

Lines changed: 24 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,8 @@
11
Testing
22
=======
33

4+
5+
46
Pipenv
57
------
68

@@ -69,16 +71,34 @@ invoke test.security
6971
invoke unit.nose
7072
```
7173

72-
Local Unit Testing and Integration Testing: Quick and Easy
73-
----------------------------------------------------------
74+
Local Unit Testing and Integration Testing:
75+
------------------------------------------
76+
77+
### Strategy to write new unit tests
78+
79+
* File mapping: Unit test files map directly to the .py files in the Policy Sentry package
80+
* Folder mapping: Folder structure of the test files maps directly to the folder structure in the Policy Sentry package
81+
* GitHub issue mapping: Add unit tests that correspond directly to those GitHub issues to show that the fix work specifically for the issue raised. prefix test name with issue number.
82+
For example: [test_gh_211_write_with_empty_access_level_lists](https://github.com/salesforce/policy_sentry/blob/0b3398a365f4cf0901bd779c6156207688e7858b/test/writing/test_write_policy_library_usage.py#L269) is prefixed with issue number 111.
83+
* Adaptable to IAM Definition changes:
84+
* For tests that are dependent on current state of the IAM definition (aka the IAM Database, iam_definition.json), avoid doing equals comparision of the results with the expected_results.
85+
* Over time, as IAM definiton changes, chances are that such tests would fail.
86+
* Example: let's say that hypothetically the secretsmanager service had 4 actions at the "write" access level that could be restricted to the secret resource type. If I wrote the tests so that they tested the value of the generated policy, that works for the initial tests. But if AWS adds two new actions that can be restricted to that access level and resource type, then the tests would fail.
87+
* This only really applies for where the expected test values would be dependent on the current state of the IAM definition (aka the IAM Database, iam_definition.json) and the actions in there. So, validating the output of write-policy related commands, and query related commands.
88+
* After I realized that, I started writing my tests so that the expected output that showed what actions you'd want would be in the comments instead of the expected_results, so the human knows what's going on. And then I would just check that the size of the array holding the actions is greater than or equal to the size that it is currently at.
89+
So, in the secretsmanager example above, the test would probably just check that the size of the actions is greater than or equal to 4.
90+
91+
92+
93+
### Quick and Easy way to run tests
94+
95+
Just run this from the root of the repository:
7496

7597
We highly suggest that you run all the tests before pushing a
7698
significant commit. It would be painful to copy/paste all of those lines
7799
above - so we've compiled a test script in the `utils`
78100
folder.
79101

80-
Just run this from the root of the repository:
81-
82102
```bash
83103
./utils/run_tests.sh
84104
```

0 commit comments

Comments
 (0)