General Reinforcement Learning Questions

[E-Cylab]

Hello,

Sorry to take from your time for these basic questions, but some clarification would be greatly appreciated !

1. What are the actions that Red team agents can make ? ( particularly RL red agents ), are those actions correlated to CVEs, or Mitre att&ck techniques ?

2. Is it possible to train an RL red agent against RL blue agent ?, RL red agent against heuristic blue agent ? (i've seen the examples folder, but i am not sure if all of these are already implemented)

3. Does each simulation / emulation environment has configurable state / observation stated ? (POMDP for the Red agent, and MDP for the Blue agent for example ?)

Thank you in advance for your help, and amazing tool !!!

Best regards,
Elli

[Kim-Hammar]

Hi Elli,

Thank you for the feedback.

1. The red actions are listed in Table 5 here: https://kth.diva-portal.org/smash/get/diva2:1912164/FULLTEXT01.pdf You can also add new actions and run arbitrary red commands. See here: https://github.com/Limmen/csle/blob/master/simulation-system/libs/csle-common/src/csle_common/dao/emulation_action/attacker/emulation_attacker_action_id.py

2. Yes it is possible and there are examples in the examples folder.

3. Yes, but they require certain engineering expertise to configure effectively, especially the emulation.
   Best,
   Kim

[E-Cylab]

Hello Kim,

Thank you very much for the reply !

I wanted to share just a small remark in regards to the installation of the CSLE environment : during manual installation it is important to change the ip address in the file "csle/management-system/csle-mgmt-webapp/server/server.py" to the node ip. If not done the flask API won't work.

Going back to my first question, i noticed that the red action space is 0/1 how can the agent choose from all the actions described in the table in you thesis ?

Also a technical question, is it possible to run training jobs on multiple servers in parallel ? and how is running a training job on multiple GPUs managed ?

Thank you in advance for your help !

Best regards,
Elli

[Kim-Hammar]

Thanks, I will add that to the installation documentation.

There are several simulation-environments, each has its own action space for the attacker, you can also define your own. The action space can be chosen freely. For example in the stopping game with 0/1 action space, the action =1 for the attacker is a high-level action that means "attack" and when that happens the attacker will randomly execute actions from the list in Table 5.

It is supported to run jobs on different GPUs and servers. If you run a PyTorch agent you can specify the GPU inside the torch agent and it will work out of the box. If you want to run on a different server you can simply execute the Python code that starts the job on the corresponding server.

Best ,

Kim

[E-Cylab]

Thank you !

Best regards,
Elli