Skip to content

Releases: Kaszanas/SC2InfoExtractorGo

2.1.3 SC2InfoExtractorGo Release

24 Mar 02:32
@Kaszanas Kaszanas
2.1.3
d289247
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Choose a tag to compare

View all tags
2.1.3 SC2InfoExtractorGo Release Latest
Latest

What's Changed

Full Changelog: 2.1.2...2.1.3

DOI

SC2InfoExtractorGo

A tool meant to allow for quick data extraction from StarCraft 2 replay files ".SC2Replay".

Example Usage (Docker)

Note

This steps assumes that you have Docker installed on your machine and working.
Additionally, you will either need to pull the docker image from DockerHub or build it locally as described in Build Docker Images.

The easiest way to run this tool is to use the provided Docker image release. Please note that the maps directory can be mounted to see what map files are downloaded, and what map files were pre-seeded with the Docker Image. Other command line options are described in CLI Options.

Commands below showcases the exampe usage of the tool.
Steps:

  1. Pull the Docker image from DockerHub or Build Docker Images:
docker pull kaszanas/sc2infoextractorgo:latest
  1. Place your .SC2Replay files in the directory which you want to mount as the input directory. The default directory for replays within the container is /app/replays/input.
  2. Pick one of the following commands.
    • Run the docker container and mount the entire directory:
    docker run -it --rm \
-v </path/to/your/directory>:/app \
sc2infoextractorgo:latest -help
    • Run the Docker container and mount specific paths.
      Replace the <path> with the appropriate paths on your system:
    docker run -it --rm \
-v </path/to/your/replays>:/app/replays/input \
-v </path/to/your/output>:/app/replays/output \
-v </path/to/your/logs>:/app/logs \
-v </path/to/your/maps>:/app/maps \
sc2infoextractorgo:latest -help
  3. When you execute the processing command, please note that you need to pass the appropriate paths to the tool that are withing the container. The default paths are:
    • Input directory: /app/replays/input
    • Output directory: /app/replays/output
    • Logs directory: /app/logs
    • Maps directory: /app/maps
    • Additionally the tool outputs some auxiliary files that can be used for debugging purposes. These are created in the main directory of the container /app/.

Note

If you wish to see other files that are generated by the tool, you will have to mount the appropriate directories. The tool generates logs in the logs directory.

Example Usage (Local)

Note

These steps assume that you downloaded the release binary or you will have to compile the tool from source.

  1. Place .SC2Replay files in ./replays/input
  2. Run SC2InfoExtractorGo.exe as follows:
SC2InfoExtractorGo.exe -input ./replays/input -output ./replays/output
  1. Verify the output in ./replays/output
  2. If The output packages do not contain any processed replays, proceed to verify ./logs/.

CLI Options

To see the full list of available options, run the tool with the -help flag:

SC2InfoExtractorGo.exe -help

The following flags are available:

  -help
        Show command usage
  -game_mode_filter int
        Specifies which game mode should be included from the processed files
        in a format of a binary flag: AllGameModes: 0b11111111 (default 0b11111111) (default 255)
  -input string
        Input directory where .SC2Replay files are held. (default "./replays/input")
  -log_dir string
        Specifies directory which will hold the logging information. (default "./logs/")
  -log_level int
        Specifies a log level from 1-7:
        Panic - 1, Fatal - 2,
        Error - 3, Warn - 4,
        Info - 5, Debug - 6,
        Trace - 7 (default 4)
  -maps_directory string
        Directory where the maps will be downloaded as a result of the replay processing. (default "./maps/")
  -max_procs int
        Specifies the number of logic cores of a processor that will be used for processing (default runtime.NumCPU()). (default 24)
  -number_of_packages int
        Provide a number of zip packages to be created and compressed
        into a zip archive. Please remember that this number needs to be lower
        than the number of processed files. If set to 0, will omit the
        zip packaging and output .json directly to drive. (default 1)
  -only_map_download
        Flag specifying if the tool is supposed to only download
        the maps and not process the replays.
  -output string
        Output directory where compressed zip packages will be saved. (default "./replays/output")
  -perform_chat_anonymization
        Flag, specifying if the chat anonymization should be performed.
  -perform_cleanup
        Flag specifying if the tool is supposed to perform the cleaning
        functions within the processing pipeline.
  -perform_filtering
        Flag, specifying if the pipeline ought to verify different hard coded game modes.
        If set to false completely bypasses the filtering.
  -perform_integrity_checks
        Flag specifying if the software is supposed to check the hardcoded
        integrity checks for the provided replays
  -perform_player_anonymization
        Flag specifying if the tool is supposed to perform player anonymization
        functions within the processing pipeline.
        If set to true please remember to download and run
        an anonymization server: https://doi.org/10.5281/zenodo.5138313
  -perform_validity_checks
        Flag, specifying if the tool is supposed to use hardcoded validity checks
        and verify if the replay file variables are within 'common sense' ranges.
  -skip_map_download
        Flag specifying if the tool is supposed to skip the map download
  -with_cpu_profiler string
        Set path to the file where pprof cpu profiler will save its information.
        If this is empty no profiling is performed.

Dataset Preparation

If You have a pack of replays with nested directories and You would like to automatically flatten the directory structure, We have published a tool that can be used for that, please see SC2DatasetPreparator: https://doi.org/10.5281/zenodo.5296664

Anonymization (Optional)

In order to anonymize the replays please make sure to download and run our open-source implementation of an anonymization server the SC2AnonServerPy: https://doi.org/10.5281/zenodo.5138313

This is required because of the multiprocessing nature of our code that needs to perform synchronization with an existing database of unique toons (player IDs) that are mapped to arbitrary incrementing integer.

Map Translation Support

Existing implementation downloads the maps from the Blizzard servers. This is to normalize the map names to English language. When there is no internet connection available, our tool should fallback to reading the map names from the files placed in the ./maps directory.

Filtering Capabilities

Currently the software supports some game mode filtering capabilities which can be used with -game_mode flag.
The flag itself is a binary flag where 0b11111111 is all game modes which is the default.

Other ways to set the flag:

  • 0b00000001: 1v1 Ranked Games
  • 0b00000010: 2v2 Ranked Games
  • 0b00000100: 3v3 Ranked Games
  • 0b00001000: 4v4 Ranked Games
  • 0b00010000: 1v1 Custom Games
  • 0b00100000: 2v2 Custom Games
  • 0b01000000: 3v3 Custom Games
  • 0b10000000: 4v4 Custom Games

License / Dual Licensing

This repository is licensed under GNU GPL v3 license. If You would like to acquire a different license please contact me directly.

Contributors

Kaszanas
Assets 3
Loading

2.1.2 SC2InfoExtractorGo Release

13 Mar 00:23
@Kaszanas Kaszanas
2.1.2
e78b0f1
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Choose a tag to compare

View all tags
2.1.2 SC2InfoExtractorGo Release

What's Changed

  • [PR] Anonymization hotfix, working anonymization by @Kaszanas in #92
  • [PR] Anonymization Hotfix 2.1.2 by @Kaszanas in #93
  • [PR] Hotfix: Added Checks for Empty Map Names by @Kaszanas in #94
  • [PR] Hotfix 2.1.2: Checking empty map names by @Kaszanas in #95

Full Changelog: 2.1.1...2.1.2

DOI

SC2InfoExtractorGo

A tool meant to allow for quick data extraction from StarCraft 2 replay files ".SC2Replay".

Example Usage (Docker)

Note

This steps assumes that you have Docker installed on your machine and working.
Additionally, you will either need to pull the docker image from DockerHub or build it locally as described in Build Docker Images.

The easiest way to run this tool is to use the provided Docker image release. Please note that the maps directory can be mounted to see what map files are downloaded, and what map files were pre-seeded with the Docker Image. Other command line options are described in CLI Options.

Commands below showcases the exampe usage of the tool.
Steps:

  1. Pull the Docker image from DockerHub or Build Docker Images:
docker pull kaszanas/sc2infoextractorgo:latest
  1. Place your .SC2Replay files in the directory which you want to mount as the input directory. The default directory for replays within the container is /app/replays/input.
  2. Pick one of the following commands.
    • Run the docker container and mount the entire directory:
    docker run -it --rm \
-v </path/to/your/directory>:/app \
sc2infoextractorgo:latest -help
    • Run the Docker container and mount specific paths.
      Replace the <path> with the appropriate paths on your system:
    docker run -it --rm \
-v </path/to/your/replays>:/app/replays/input \
-v </path/to/your/output>:/app/replays/output \
-v </path/to/your/logs>:/app/logs \
-v </path/to/your/maps>:/app/maps \
sc2infoextractorgo:latest -help
  3. When you execute the processing command, please note that you need to pass the appropriate paths to the tool that are withing the container. The default paths are:
    • Input directory: /app/replays/input
    • Output directory: /app/replays/output
    • Logs directory: /app/logs
    • Maps directory: /app/maps
    • Additionally the tool outputs some auxiliary files that can be used for debugging purposes. These are created in the main directory of the container /app/.

Note

If you wish to see other files that are generated by the tool, you will have to mount the appropriate directories. The tool generates logs in the logs directory.

Example Usage (Local)

Note

These steps assume that you downloaded the release binary or you will have to compile the tool from source.

  1. Place .SC2Replay files in ./replays/input
  2. Run SC2InfoExtractorGo.exe as follows:
SC2InfoExtractorGo.exe -input ./replays/input -output ./replays/output
  1. Verify the output in ./replays/output
  2. If The output packages do not contain any processed replays, proceed to verify ./logs/.

CLI Options

To see the full list of available options, run the tool with the -help flag:

SC2InfoExtractorGo.exe -help

The following flags are available:

  -help
        Show command usage
  -game_mode_filter int
        Specifies which game mode should be included from the processed files
        in a format of a binary flag: AllGameModes: 0b11111111 (default 0b11111111) (default 255)
  -input string
        Input directory where .SC2Replay files are held. (default "./replays/input")
  -log_dir string
        Specifies directory which will hold the logging information. (default "./logs/")
  -log_level int
        Specifies a log level from 1-7:
        Panic - 1, Fatal - 2,
        Error - 3, Warn - 4,
        Info - 5, Debug - 6,
        Trace - 7 (default 4)
  -maps_directory string
        Directory where the maps will be downloaded as a result of the replay processing. (default "./maps/")
  -max_procs int
        Specifies the number of logic cores of a processor that will be used for processing (default runtime.NumCPU()). (default 24)
  -number_of_packages int
        Provide a number of zip packages to be created and compressed
        into a zip archive. Please remember that this number needs to be lower
        than the number of processed files. If set to 0, will ommit the
        zip packaging and output .json directly to drive. (default 1)
  -only_map_download
        Flag specifying if the tool is supposed to only download
        the maps and not process the replays.
  -output string
        Output directory where compressed zip packages will be saved. (default "./replays/output")
  -perform_chat_anonymization
        Flag, specifying if the chat anonymization should be performed.
  -perform_cleanup
        Flag specifying if the tool is supposed to perform the cleaning
                        functions within the processing pipeline.
  -perform_filtering
        Flag, specifying if the pipeline ought to verify different hard coded game modes.
        If set to false completely bypasses the filtering.
  -perform_integrity_checks
        Flag specifying if the software is supposed to check the hardcoded
        integrity checks for the provided replays
  -perform_player_anonymization
        Flag specifying if the tool is supposed to perform player anonymization
        functions within the processing pipeline.
        If set to true please remember to download and run
        an anonymization server: https://doi.org/10.5281/zenodo.5138313
  -perform_validity_checks
        Flag, specifying if the tool is supposed to use hardcoded validity checks
        and verify if the replay file variables are within 'common sense' ranges.
  -skip_map_download
        Flag specifying if the tool is supposed to skip the map download
  -with_cpu_profiler string
        Set path to the file where pprof cpu profiler will save its information.
        If this is empty no profiling is performed.

Dataset Preparation

If You have a pack of replays with nested directories and You would like to automatically flatten the directory structure, We have published a tool that can be used for that, please see SC2DatasetPreparator: https://doi.org/10.5281/zenodo.5296664

Anonymization (Optional)

In order to anonymize the replays please make sure to download and run our open-source implementation of an anonymization server the SC2AnonServerPy: https://doi.org/10.5281/zenodo.5138313

This is required because of the multiprocessing nature of our code that needs to perform synchronization with an existing database of unique toons (player IDs) that are mapped to arbitrary incrementing integer.

Map Translation Support

Existing implementation downloads the maps from the Blizzard servers. This is to normalize the map names to English language. When there is no internet connection available, our tool should fallback to reading the map names from the files placed in the ./maps directory.

Filtering Capabilities

Currently the software supports some game mode filtering capabilities which can be used with -game_mode flag.
The flag itself is a binary flag where 0b11111111 is all game modes which is the default.

Other ways to set the flag:

  • 0b00000001: 1v1 Ranked Games
  • 0b00000010: 2v2 Ranked Games
  • 0b00000100: 3v3 Ranked Games
  • 0b00001000: 4v4 Ranked Games
  • 0b00010000: 1v1 Custom Games
  • 0b00100000: 2v2 Custom Games
  • 0b01000000: 3v3 Custom Games
  • 0b10000000: 4v4 Custom Games

License / Dual Licensing

This repository is licensed under GNU GPL v3 license. If You would like to acquire a different license please contact me directly.

Cite Us!

@software{BialeckiExtractor2021,
  author    = {Białecki, Andrzej and
               Białecki, Piotr and
               Krupiński, Leszek},
  title     = {{Kaszanas/SC2InfoExtractorGo: 1.2.0 
               SC2InfoExtractorGo Release}},
  month     = {jun},
  year      = {2022},
  publisher = {Zenodo},
  version   = {1.2.0},
  doi       = {10.5281/zenodo.5296788},
  url       = {https://doi.org/10.5281/zenodo.5296788}
}

Contributors

Kaszanas
Loading

2.1.1 SC2InfoExtractorGo Release

04 Mar 17:55
@Kaszanas Kaszanas
2.1.1
42d0fa6
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Choose a tag to compare

View all tags
2.1.1 SC2InfoExtractorGo Release

What's Changed

Full Changelog: 2.1.0...2.1.1

DOI

SC2InfoExtractorGo

A tool meant to allow for quick data extraction from StarCraft 2 replay files ".SC2Replay".

Example Usage (Docker)

Note

This steps assumes that you have Docker installed on your machine and working.
Additionally, you will either need to pull the docker image from DockerHub or build it locally as described in Build Docker Images.

The easiest way to run this tool is to use the provided Docker image release. Please note that the maps directory can be mounted to see what map files are downloaded, and what map files were pre-seeded with the Docker Image. Other command line options are described in CLI Options.

Commands below showcases the exampe usage of the tool.
Steps:

  1. Pull the Docker image from DockerHub or Build Docker Images:
docker pull kaszanas/sc2infoextractorgo:latest
  1. Place your .SC2Replay files in the directory which you want to mount as the input directory. The default directory for replays within the container is /app/replays/input.
  2. Pick one of the following commands.
    • Run the docker container and mount the entire directory:
    docker run -it --rm \
-v </path/to/your/directory>:/app \
sc2infoextractorgo:latest -help
    • Run the Docker container and mount specific paths.
      Replace the <path> with the appropriate paths on your system:
    docker run -it --rm \
-v </path/to/your/replays>:/app/replays/input \
-v </path/to/your/output>:/app/replays/output \
-v </path/to/your/logs>:/app/logs \
-v </path/to/your/maps>:/app/maps \
sc2infoextractorgo:latest -help
  3. When you execute the processing command, please note that you need to pass the appropriate paths to the tool that are withing the container. The default paths are:
    • Input directory: /app/replays/input
    • Output directory: /app/replays/output
    • Logs directory: /app/logs
    • Maps directory: /app/maps
    • Additionally the tool outputs some auxiliary files that can be used for debugging purposes. These are created in the main directory of the container /app/.

Note

If you wish to see other files that are generated by the tool, you will have to mount the appropriate directories. The tool generates logs in the logs directory.

Example Usage (Local)

Note

These steps assume that you downloaded the release binary or you will have to compile the tool from source.

  1. Place .SC2Replay files in ./replays/input
  2. Run SC2InfoExtractorGo.exe as follows:
SC2InfoExtractorGo.exe -input ./replays/input -output ./replays/output
  1. Verify the output in ./replays/output
  2. If The output packages do not contain any processed replays, proceed to verify ./logs/.

CLI Options

To see the full list of available options, run the tool with the -help flag:

SC2InfoExtractorGo.exe -help

The following flags are available:

  -help
        Show command usage
  -game_mode_filter int
        Specifies which game mode should be included from the processed files
        in a format of a binary flag: AllGameModes: 0b11111111 (default 0b11111111) (default 255)
  -input string
        Input directory where .SC2Replay files are held. (default "./replays/input")
  -log_dir string
        Specifies directory which will hold the logging information. (default "./logs/")
  -log_level int
        Specifies a log level from 1-7:
        Panic - 1, Fatal - 2,
        Error - 3, Warn - 4,
        Info - 5, Debug - 6,
        Trace - 7 (default 4)
  -maps_directory string
        Directory where the maps will be downloaded as a result of the replay processing. (default "./maps/")
  -max_procs int
        Specifies the number of logic cores of a processor that will be used for processing (default runtime.NumCPU()). (default 24)
  -number_of_packages int
        Provide a number of zip packages to be created and compressed
        into a zip archive. Please remember that this number needs to be lower
        than the number of processed files. If set to 0, will ommit the
        zip packaging and output .json directly to drive. (default 1)
  -only_map_download
        Flag specifying if the tool is supposed to only download
        the maps and not process the replays.
  -output string
        Output directory where compressed zip packages will be saved. (default "./replays/output")
  -perform_chat_anonymization
        Flag, specifying if the chat anonymization should be performed.
  -perform_cleanup
        Flag specifying if the tool is supposed to perform the cleaning
                        functions within the processing pipeline.
  -perform_filtering
        Flag, specifying if the pipeline ought to verify different hard coded game modes.
        If set to false completely bypasses the filtering.
  -perform_integrity_checks
        Flag specifying if the software is supposed to check the hardcoded
        integrity checks for the provided replays
  -perform_player_anonymization
        Flag specifying if the tool is supposed to perform player anonymization
        functions within the processing pipeline.
        If set to true please remember to download and run
        an anonymization server: https://doi.org/10.5281/zenodo.5138313
  -perform_validity_checks
        Flag, specifying if the tool is supposed to use hardcoded validity checks
        and verify if the replay file variables are within 'common sense' ranges.
  -skip_map_download
        Flag specifying if the tool is supposed to skip the map download
  -with_cpu_profiler string
        Set path to the file where pprof cpu profiler will save its information.
        If this is empty no profiling is performed.

Dataset Preparation

If You have a pack of replays with nested directories and You would like to automatically flatten the directory structure, We have published a tool that can be used for that, please see SC2DatasetPreparator: https://doi.org/10.5281/zenodo.5296664

Anonymization (Optional)

In order to anonymize the replays please make sure to download and run our open-source implementation of an anonymization server the SC2AnonServerPy: https://doi.org/10.5281/zenodo.5138313

This is required because of the multiprocessing nature of our code that needs to perform synchronization with an existing database of unique toons (player IDs) that are mapped to arbitrary incrementing integer.

Map Translation Support

Existing implementation downloads the maps from the Blizzard servers. This is to normalize the map names to English language. When there is no internet connection available, our tool should fallback to reading the map names from the files placed in the ./maps directory.

Filtering Capabilities

Currently the software supports some game mode filtering capabilities which can be used with -game_mode flag.
The flag itself is a binary flag where 0b11111111 is all game modes which is the default.

Other ways to set the flag:

  • 0b00000001: 1v1 Ranked Games
  • 0b00000010: 2v2 Ranked Games
  • 0b00000100: 3v3 Ranked Games
  • 0b00001000: 4v4 Ranked Games
  • 0b00010000: 1v1 Custom Games
  • 0b00100000: 2v2 Custom Games
  • 0b01000000: 3v3 Custom Games
  • 0b10000000: 4v4 Custom Games

License / Dual Licensing

This repository is licensed under GNU GPL v3 license. If You would like to acquire a different license please contact me directly.

Contributors

Kaszanas
Loading

2.1.0 SC2InfoExtractorGo Release

01 Mar 17:25
@Kaszanas Kaszanas
2.1.0
dcfbd30
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Choose a tag to compare

View all tags
2.1.0 SC2InfoExtractorGo Release

What's Changed

Full Changelog: 2.0.0...2.1.0

SC2InfoExtractorGo

A tool meant to allow for quick data extraction from StarCraft 2 replay files ".SC2Replay".

Example Usage (Docker)

Note

This steps assumes that you have Docker installed on your machine and working.
Additionally, you will either need to pull the docker image from DockerHub or build it locally as described in Build Docker Images.

The easiest way to run this tool is to use the provided Docker image release. Please note that the maps directory can be mounted to see what map files are downloaded, and what map files were pre-seeded with the Docker Image. Other command line options are described in CLI Options.

Commands below showcases the exampe usage of the tool.
Steps:

  1. Pull the Docker image from DockerHub or Build Docker Images:
docker pull kaszanas/sc2infoextractorgo:latest
  1. Place your .SC2Replay files in the directory which you want to mount as the input directory. The default directory for replays within the container is /app/replays/input.
  2. Pick one of the following commands.
    • Run the docker container and mount the entire directory:
    docker run -it --rm \
-v </path/to/your/directory>:/app \
sc2infoextractorgo:latest -help
    • Run the Docker container and mount specific paths.
      Replace the <path> with the appropriate paths on your system:
    docker run -it --rm \
-v </path/to/your/replays>:/app/replays/input \
-v </path/to/your/output>:/app/replays/output \
-v </path/to/your/logs>:/app/logs \
-v </path/to/your/maps>:/app/maps \
sc2infoextractorgo:latest -help
  3. When you execute the processing command, please note that you need to pass the appropriate paths to the tool that are withing the container. The default paths are:
    • Input directory: /app/replays/input
    • Output directory: /app/replays/output
    • Logs directory: /app/logs
    • Maps directory: /app/maps
    • Additionally the tool outputs some auxiliary files that can be used for debugging purposes. These are created in the main directory of the container /app/.

Note

If you wish to see other files that are generated by the tool, you will have to mount the appropriate directories. The tool generates logs in the logs directory.

Example Usage (Local)

Note

These steps assume that you downloaded the release binary or you will have to compile the tool from source.

  1. Place .SC2Replay files in ./replays/input
  2. Run SC2InfoExtractorGo.exe as follows:
SC2InfoExtractorGo.exe -input ./replays/input -output ./replays/output
  1. Verify the output in ./replays/output
  2. If The output packages do not contain any processed replays, proceed to verify ./logs/.

CLI Options

To see the full list of available options, run the tool with the -help flag:

SC2InfoExtractorGo.exe -help

The following flags are available:

  -help
        Show command usage
  -game_mode_filter int
        Specifies which game mode should be included from the processed files
        in a format of a binary flag: AllGameModes: 0b11111111 (default 0b11111111) (default 255)
  -input string
        Input directory where .SC2Replay files are held. (default "./replays/input")
  -log_dir string
        Specifies directory which will hold the logging information. (default "./logs/")
  -log_level int
        Specifies a log level from 1-7:
        Panic - 1, Fatal - 2,
        Error - 3, Warn - 4,
        Info - 5, Debug - 6,
        Trace - 7 (default 4)
  -maps_directory string
        Directory where the maps will be downloaded as a result of the replay processing. (default "./maps/")
  -max_procs int
        Specifies the number of logic cores of a processor that will be used for processing (default runtime.NumCPU()). (default 24)
  -number_of_packages int
        Provide a number of zip packages to be created and compressed
        into a zip archive. Please remember that this number needs to be lower
        than the number of processed files. If set to 0, will ommit the
        zip packaging and output .json directly to drive. (default 1)
  -only_map_download
        Flag specifying if the tool is supposed to only download
        the maps and not process the replays.
  -output string
        Output directory where compressed zip packages will be saved. (default "./replays/output")
  -perform_chat_anonymization
        Flag, specifying if the chat anonymization should be performed.
  -perform_cleanup
        Flag specifying if the tool is supposed to perform the cleaning
                        functions within the processing pipeline.
  -perform_filtering
        Flag, specifying if the pipeline ought to verify different hard coded game modes.
        If set to false completely bypasses the filtering.
  -perform_integrity_checks
        Flag specifying if the software is supposed to check the hardcoded
        integrity checks for the provided replays
  -perform_player_anonymization
        Flag specifying if the tool is supposed to perform player anonymization
        functions within the processing pipeline.
        If set to true please remember to download and run
        an anonymization server: https://doi.org/10.5281/zenodo.5138313
  -perform_validity_checks
        Flag, specifying if the tool is supposed to use hardcoded validity checks
        and verify if the replay file variables are within 'common sense' ranges.
  -skip_map_download
        Flag specifying if the tool is supposed to skip the map download
  -with_cpu_profiler string
        Set path to the file where pprof cpu profiler will save its information.
        If this is empty no profiling is performed.

Dataset Preparation

If You have a pack of replays with nested directories and You would like to automatically flatten the directory structure, We have published a tool that can be used for that, please see SC2DatasetPreparator: https://doi.org/10.5281/zenodo.5296664

Anonymization (Optional)

In order to anonymize the replays please make sure to download and run our open-source implementation of an anonymization server the SC2AnonServerPy: https://doi.org/10.5281/zenodo.5138313

This is required because of the multiprocessing nature of our code that needs to perform synchronization with an existing database of unique toons (player IDs) that are mapped to arbitrary incrementing integer.

Map Translation Support

Existing implementation downloads the maps from the Blizzard servers. This is to normalize the map names to English language. When there is no internet connection available, our tool should fallback to reading the map names from the files placed in the ./maps directory.

Filtering Capabilities

Currently the software supports some game mode filtering capabilities which can be used with -game_mode flag.
The flag itself is a binary flag where 0b11111111 is all game modes which is the default.

Other ways to set the flag:

  • 0b00000001: 1v1 Ranked Games
  • 0b00000010: 2v2 Ranked Games
  • 0b00000100: 3v3 Ranked Games
  • 0b00001000: 4v4 Ranked Games
  • 0b00010000: 1v1 Custom Games
  • 0b00100000: 2v2 Custom Games
  • 0b01000000: 3v3 Custom Games
  • 0b10000000: 4v4 Custom Games

License / Dual Licensing

This repository is licensed under GNU GPL v3 license. If You would like to acquire a different license please contact me directly.

Contributors

Kaszanas
Loading

2.0.0 SC2InfoExtractorGo Release

19 Nov 23:33
@Kaszanas Kaszanas
2.0.0
d9f2812

Choose a tag to compare

View all tags
2.0.0 SC2InfoExtractorGo Release

Changes

SC2InfoExtractorGo

A tool meant to allow for quick data extraction from StarCraft 2 replay files ".SC2Replay".

Example Usage (Docker)

Note

This steps assumes that you have Docker installed on your machine and working.
Additionally, you will either need to pull the docker image from DockerHub or build it locally as described in Build Docker Images.

The easiest way to run this tool is to use the provided Docker image release. Please note that the maps directory can be mounted to see what map files are downloaded, and what map files were pre-seeded with the Docker Image. Other command line options are described in CLI Options.

Commands below showcases the exampe usage of the tool.
Steps:

  1. Pull the Docker image from DockerHub or Build Docker Images:
docker pull kaszanas/sc2infoextractorgo:latest
  1. Place your .SC2Replay files in the directory which you want to mount as the input directory. The default directory for replays within the container is /app/replays/input.
  2. Pick one of the following commands.
    • Run the docker container and mount the entire directory:
    docker run -it --rm \
-v </path/to/your/directory>:/app \
sc2infoextractorgo:latest -help
    • Run the Docker container and mount specific paths.
      Replace the <path> with the appropriate paths on your system:
    docker run -it --rm \
-v </path/to/your/replays>:/app/replays/input \
-v </path/to/your/output>:/app/replays/output \
-v </path/to/your/logs>:/app/logs \
-v </path/to/your/maps>:/app/maps \
sc2infoextractorgo:latest -help
  3. When you execute the processing command, please note that you need to pass the appropriate paths to the tool that are withing the container. The default paths are:
    • Input directory: /app/replays/input
    • Output directory: /app/replays/output
    • Logs directory: /app/logs
    • Maps directory: /app/maps
    • Additionally the tool outputs some auxiliary files that can be used for debugging purposes. These are created in the main directory of the container /app/.

Note

If you wish to see other files that are generated by the tool, you will have to mount the appropriate directories. The tool generates logs in the logs directory.

Example Usage (Local)

Note

These steps assume that you downloaded the release binary or you will have to compile the tool from source.

  1. Place .SC2Replay files in ./replays/input
  2. Run SC2InfoExtractorGo.exe as follows:
SC2InfoExtractorGo.exe -input ./replays/input -output ./replays/output
  1. Verify the output in ./replays/output
  2. If The output packages do not contain any processed replays, proceed to verify ./logs/.

CLI Options

To see the full list of available options, run the tool with the -help flag:

SC2InfoExtractorGo.exe -help

The following flags are available:

  -help
    Show command usage
  -game_mode_filter int
    Specifies which game mode should be included from the processed files
    in a format of a binary flag: AllGameModes: 0b11111111 (default 0b11111111) (default 255)
  -input string
    Input directory where .SC2Replay files are held. (default "./replays/input")
  -log_dir string
    Specifies directory which will hold the logging information. (default "./logs/")
  -log_level int
    Specifies a log level from 1-7:
    Panic - 1, Fatal - 2,
    Error - 3, Warn - 4,
    Info - 5, Debug - 6,
    Trace - 7 (default 4)
  -maps_directory string
        Directory where the maps will be downloaded as a result of the
        replay processing. (default "./maps/")
  -max_procs int
    Specifies the number of logic cores of a processor that will be used for processing (default runtime.NumCPU()). (default 24)
  -number_of_packages int
    Provide a number of zip packages to be created and compressed
    into a zip archive. Please remember that this number needs to be lower
    than the number of processed files. If set to 0, will ommit the
    zip packaging and output .json directly to drive. (default 1)
  -only_map_download
    Flag specifying if the tool is supposed to only download
    the maps and not process the replays.
  -output string
    Output directory where compressed zip packages will be saved. (default "./replays/output")
  -perform_chat_anonymization
    Flag, specifying if the chat anonymization should be performed.
  -perform_cleanup
    Flag specifying if the tool is supposed to perform the cleaning
    functions within the processing pipeline.
  -perform_filtering
    Flag, specifying if the pipeline ought to verify different hard coded game modes.
    If set to false completely bypasses the filtering.
  -perform_integrity_checks
    Flag specifying if the software is supposed to check the hardcoded
    integrity checks for the provided replays
  -perform_player_anonymization
    Flag specifying if the tool is supposed to perform player anonymization
    functions within the processing pipeline.
    If set to true please remember to download and run
    an anonymization server: https://doi.org/10.5281/zenodo.5138313
  -perform_validity_checks
    Flag, specifying if the tool is supposed to use hardcoded validity checks
    and verify if the replay file variables are within 'common sense' ranges.
  -with_cpu_profiler string
    Set path to the file where pprof cpu profiler will save its information.
    If this is empty no profiling is performed.

Dataset Preparation

If You have a pack of replays with nested directories and You would like to automatically flatten the directory structure, We have published a tool that can be used for that, please see SC2DatasetPreparator: https://doi.org/10.5281/zenodo.5296664

Anonymization (Optional)

In order to anonymize the replays please make sure to download and run our open-source implementation of an anonymization server the SC2AnonServerPy: https://doi.org/10.5281/zenodo.5138313

This is required because of the multiprocessing nature of our code that needs to perform synchronization with an existing database of unique toons (player IDs) that are mapped to arbitrary incrementing integer.

Map Translation Support

Existing implementation downloads the maps from the Blizzard servers. This is to normalize the map names to English language. When there is no internet connection available, our tool should fallback to reading the map names from the files placed in the ./maps directory.

Filtering Capabilities

Currently the software supports some game mode filtering capabilities which can be used with -game_mode flag.
The flag itself is a binary flag where 0b11111111 is all game modes which is the default.

Other ways to set the flag:

  • 0b00000001: 1v1 Ranked Games
  • 0b00000010: 2v2 Ranked Games
  • 0b00000100: 3v3 Ranked Games
  • 0b00001000: 4v4 Ranked Games
  • 0b00010000: 1v1 Custom Games
  • 0b00100000: 2v2 Custom Games
  • 0b01000000: 3v3 Custom Games
  • 0b10000000: 4v4 Custom Games

License / Dual Licensing

This repository is licensed under GNU GPL v3 license. If You would like to acquire a different license please contact me directly.

Cite Us!

@software{BialeckiExtractor2021,
  author    = {Białecki, Andrzej and
               Białecki, Piotr and
               Krupiński, Leszek},
  title     = {{Kaszanas/SC2InfoExtractorGo: 1.2.0 
               SC2InfoExtractorGo Release}},
  month     = {jun},
  year      = {2022},
  publisher = {Zenodo},
  version   = {1.2.0},
  doi       = {10.5281/zenodo.5296788},
  url       = {https://doi.org/10.5281/zenodo.5296788}
}
Loading

1.2.1 SC2InfoExtractorGo Release

08 Jun 23:00
@Kaszanas Kaszanas
1.2.1
7f97d45

Choose a tag to compare

View all tags
1.2.1 SC2InfoExtractorGo Release

SC2InfoExtractorGo

This tool is meant to allow for quick data extraction from SC2 replay files ".SC2Replay".

Usage

In order to use this tool please call SC2InfoExtractorGo.exe and set the choosen flags listed below:

  -input string
    	Input directory where .SC2Replay files are held. (default "./DEMOS/Input")
  -output string
    	Output directory where compressed zip packages will be saved. (default "./DEMOS/Output")
  -perform_filtering
    	Specifies if the pipeline ought to verify different hard coded game modes. If set to false completely bypasses the filtering.
  -game_mode_filter int
    	Specifies which game mode should be included from the processed files in a format of a binary flag: AllGameModes: 0b11111111 (default 255)
  -localized_maps_file string
    	Specifies a path to localization file containing {'ForeignName': 'EnglishName'} of maps. If this flag is not set and the default is unavailable, map translation will be ommited. (default "./operation_files/output.json")
  -perform_integrity_checks
    	If the software is supposed to check the hardcoded integrity checks for the provided replays
  -perform_validity_checks
    	Provide if the tool is supposed to use hardcoded validity checks and verify if the replay file variables are within 'common sense' ranges.
  -perform_cleanup
    	Provide if the tool is supposed to perform the cleaning functions within the processing pipeline.
  -perform_chat_anonymization
    	Specifies if the chat anonymization should be performed.
  -perform_player_anonymization
    	Specifies if the tool is supposed to perform player anonymization functions within the processing pipeline. If set to true please remember to download and run an anonymization server: https://doi.org/10.5281/zenodo.5138313
  -max_procs int
    	Specifies the number of logic cores of a processor that will be used for processing. (default 24)
  -number_of_packages int
    	Provide a number of zip packages to be created and compressed into a zip archive. Please remember that this number needs to be lower than the number of processed files. If set to 0, will ommit the zip packaging and output .json directly to drive. (default 1)
  -with_cpu_profiler string
    	Set path to the file where pprof cpu profiler will save its information. If this is empty no profiling is performed.
  -log_dir string
    	Specifies directory which will hold the logging information. (default "./logs/")
  -log_level int
    	Specifies a log level from 1-7. Panic - 1, Fatal - 2, Error - 3, Warn - 4, Info - 5, Debug - 6, Trace - 7 (default 4)

Minimal Example

  1. Place .SC2Replay files in ./DEMOS/Input
  2. Run SC2InfoExtractorGo.exe with default flags.
  3. Verify the output in ./DEMOS/Output
  4. If The output packages do not contain any processed replays, proceed to verify ./logs/.

Dataset Preparation

If You have a pack of replays with nested directories and You would like to automatically flatten the directory structure, We have published a tool that can be used for that, please see SC2DatasetPreparator: https://doi.org/10.5281/zenodo.5296664

Two scripts contained within that software can:

  1. Flatten directory structure of Your collected replays by looking for .SC2Replay files.
  2. Running Python multiprocessing package on multiple directories containing replays by calling them with -with-multiprocessing=false flag. This allows to have one package per directory created and speed up the processing.

Anonymization

In order to anonymize the replays please make sure to download and run our open-source implementation of an anonymization server the SC2AnonServerPy: https://doi.org/10.5281/zenodo.5138313

This is required because of the multiprocessing nature of our code that needs to perform synchronization with an existing database of unique toons (player IDs) that are mapped to arbitrary incrementing integer.

Map Translation Support

If the provided file output.json does not support map names that You require, You will have to either find it online or within the game documents. We are close to publishing a tool that will allow You to download StarCraft maps.

If You already have a set of StarCraft II maps that You would like to create a .json file to be included within Our software, use Your own solution or SC2MapLocaleExtractor: https://doi.org/10.5281/zenodo.4733264

Filtering Capabilities

Currently the software supports some game mode filtering capabilities which can be used with -game_mode flag.
The flag itself is a binary flag where 0b11111111 is all game modes which is the default.

Other ways to set the flag:

  • 0b00000001: 1v1 Ranked Games
  • 0b00000010: 2v2 Ranked Games
  • 0b00000100: 3v3 Ranked Games
  • 0b00001000: 4v4 Ranked Games
  • 0b00010000: 1v1 Custom Games
  • 0b00100000: 2v2 Custom Games
  • 0b01000000: 3v3 Custom Games
  • 0b10000000: 4v4 Custom Games

Build from source

Our working solution was built by using go build command on 64 bit version of Windows 10

License / Dual Licensing

This repository is licensed under GNU GPL v3 license. If You would like to acquire a different license please contact me directly.

Testing

The software contains defined unit tests. In order to run them please use the following command with the default provided .SC2Replay files:

go test ./...

If You have Your own replay files please either create directories containing different sets of replays to be tested and edit the TEST_INPUT_REPLAYPACK_DIR variable in ./dataproc/dataproc_pipeline_test.go or provide them all in the ./test_files/test_replays directory as input. If the TEST_INPUT_REPLAYPACK_DIR is not set then the test is skipped. If You decide that it is suitable to skip certain directories while testing on replaypacks please include the directory names in TEST_BYPASS_THESE variable in ./dataproc/dataproc_pipeline_test.go.

If You wish to run the pipeline tests against a very big set of replays please keep in mind that the default Golang test timeout is set to 10 minutes. We have found that the processing of 47 tournaments from 2016 until 2021 takes about 240 minutes to complete. Example command:

go test ./dataproc -v -run TestPipelineWrapperMultiple -timeout 240m

Loading

1.2.0 SC2InfoExtractorGo Release

08 Jun 20:42
@Kaszanas Kaszanas
1.2.0
16d6e7f
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Verified
Learn about vigilant mode.

Choose a tag to compare

View all tags
1.2.0 SC2InfoExtractorGo Release

SC2InfoExtractorGo

This tool is meant to allow for quick data extraction from SC2 replay files ".SC2Replay".

Usage

In order to use this tool please call SC2InfoExtractorGo.exe and set the choosen flags listed below:

  -input string
    	Input directory where .SC2Replay files are held. (default "./DEMOS/Input")
  -output string
    	Output directory where compressed zip packages will be saved. (default "./DEMOS/Output")
  -perform_filtering
    	Specifies if the pipeline ought to verify different hard coded game modes. If set to false completely bypasses the filtering.
  -game_mode_filter int
    	Specifies which game mode should be included from the processed files in a format of a binary flag: AllGameModes: 0b11111111 (default 255)
  -localized_maps_file string
    	Specifies a path to localization file containing {'ForeignName': 'EnglishName'} of maps. If this flag is not set and the default is unavailable, map translation will be ommited. (default "./operation_files/output.json")
  -perform_integrity_checks
    	If the software is supposed to check the hardcoded integrity checks for the provided replays
  -perform_validity_checks
    	Provide if the tool is supposed to use hardcoded validity checks and verify if the replay file variables are within 'common sense' ranges.
  -perform_cleanup
    	Provide if the tool is supposed to perform the cleaning functions within the processing pipeline.
  -perform_chat_anonymization
    	Specifies if the chat anonymization should be performed.
  -perform_player_anonymization
    	Specifies if the tool is supposed to perform player anonymization functions within the processing pipeline. If set to true please remember to download and run an anonymization server: https://doi.org/10.5281/zenodo.5138313
  -max_procs int
    	Specifies the number of logic cores of a processor that will be used for processing. (default 24)
  -number_of_packages int
    	Provide a number of zip packages to be created and compressed into a zip archive. Please remember that this number needs to be lower than the number of processed files. If set to 0, will ommit the zip packaging and output .json directly to drive. (default 1)
  -with_cpu_profiler string
    	Set path to the file where pprof cpu profiler will save its information. If this is empty no profiling is performed.
  -log_dir string
    	Specifies directory which will hold the logging information. (default "./logs/")
  -log_level int
    	Specifies a log level from 1-7. Panic - 1, Fatal - 2, Error - 3, Warn - 4, Info - 5, Debug - 6, Trace - 7 (default 4)

Minimal Example

  1. Place .SC2Replay files in ./DEMOS/Input
  2. Run SC2InfoExtractorGo.exe with default flags.
  3. Verify the output in ./DEMOS/Output
  4. If The output packages do not contain any processed replays, proceed to verify ./logs/.

Dataset Preparation

If You have a pack of replays with nested directories and You would like to automatically flatten the directory structure, We have published a tool that can be used for that, please see SC2DatasetPreparator: https://doi.org/10.5281/zenodo.5296664

Two scripts contained within that software can:

  1. Flatten directory structure of Your collected replays by looking for .SC2Replay files.
  2. Running Python multiprocessing package on multiple directories containing replays by calling them with -with-multiprocessing=false flag. This allows to have one package per directory created and speed up the processing.

Anonymization

In order to anonymize the replays please make sure to download and run our open-source implementation of an anonymization server the SC2AnonServerPy: https://doi.org/10.5281/zenodo.5138313

This is required because of the multiprocessing nature of our code that needs to perform synchronization with an existing database of unique toons (player IDs) that are mapped to arbitrary incrementing integer.

Map Translation Support

If the provided file output.json does not support map names that You require, You will have to either find it online or within the game documents. We are close to publishing a tool that will allow You to download StarCraft maps.

If You already have a set of StarCraft II maps that You would like to create a .json file to be included within Our software, use Your own solution or SC2MapLocaleExtractor: https://doi.org/10.5281/zenodo.4733264

Filtering Capabilities

Currently the software supports some game mode filtering capabilities which can be used with -game_mode flag.
The flag itself is a binary flag where 0b11111111 is all game modes which is the default.

Other ways to set the flag:

  • 0b00000001: 1v1 Ranked Games
  • 0b00000010: 2v2 Ranked Games
  • 0b00000100: 3v3 Ranked Games
  • 0b00001000: 4v4 Ranked Games
  • 0b00010000: 1v1 Custom Games
  • 0b00100000: 2v2 Custom Games
  • 0b01000000: 3v3 Custom Games
  • 0b10000000: 4v4 Custom Games

Build from source

Our working solution was built by using go build command on 64 bit version of Windows 10

License / Dual Licensing

This repository is licensed under GNU GPL v3 license. If You would like to acquire a different license please contact me directly.

Testing

The software contains defined unit tests. In order to run them please use the following command with the default provided .SC2Replay files:

go test ./...

If You have Your own replay files please either create directories containing different sets of replays to be tested and edit the TEST_INPUT_REPLAYPACK_DIR variable in ./dataproc/dataproc_pipeline_test.go or provide them all in the ./test_files/test_replays directory as input. If the TEST_INPUT_REPLAYPACK_DIR is not set then the test is skipped. If You decide that it is suitable to skip certain directories while testing on replaypacks please include the directory names in TEST_BYPASS_THESE variable in ./dataproc/dataproc_pipeline_test.go.

If You wish to run the pipeline tests against a very big set of replays please keep in mind that the default Golang test timeout is set to 10 minutes. We have found that the processing of 47 tournaments from 2016 until 2021 takes about 240 minutes to complete. Example command:

go test ./dataproc -v -run TestPipelineWrapperMultiple -timeout 240m

Loading

1.1.0 SC2InfoExtractorGo Release

28 Sep 13:10
@Kaszanas Kaszanas
1.1.0
192e724

Choose a tag to compare

View all tags
1.1.0 SC2InfoExtractorGo Release

SC2InfoExtractorGo

This tool is meant to allow for quick data extraction from SC2 replay files ".SC2Replay".

Usage

In order to use this tool please call SC2InfoExtractorGo.exe and set the choosen flags listed below:

  -input string
    	Input directory where .SC2Replay files are held. (default "./DEMOS/Input")
  -output string
    	Output directory where compressed zip packages will be saved. (default "./DEMOS/Output")
  -perform_filtering
    	Specifies if the pipeline ought to verify different hard coded game modes. If set to false completely bypasses the filtering.
  -game_mode_filter int
    	Specifies which game mode should be included from the processed files in a format of a binary flag: AllGameModes: 0b11111111 (default 255)
  -localized_maps_file string
    	Specifies a path to localization file containing {'ForeignName': 'EnglishName'} of maps. If this flag is not set and the default is unavailable, map translation will be ommited. (default "./operation_files/output.json")
  -perform_integrity_checks
    	If the software is supposed to check the hardcoded integrity checks for the provided replays
  -perform_validity_checks
    	Provide if the tool is supposed to use hardcoded validity checks and verify if the replay file variables are within 'common sense' ranges.
  -perform_cleanup
    	Provide if the tool is supposed to perform the cleaning functions within the processing pipeline.
  -perform_chat_anonymization
    	Specifies if the chat anonymization should be performed.
  -perform_player_anonymization
    	Specifies if the tool is supposed to perform player anonymization functions within the processing pipeline. If set to true please remember to download and run an anonymization server: https://doi.org/10.5281/zenodo.5138313
  -max_procs int
    	Specifies the number of logic cores of a processor that will be used for processing. (default 24)
  -number_of_packages int
    	Provide a number of zip packages to be created and compressed into a zip archive. Please remember that this number needs to be lower than the number of processed files. If set to 0, will ommit the zip packaging and output .json directly to drive. (default 1)
  -with_cpu_profiler string
    	Set path to the file where pprof cpu profiler will save its information. If this is empty no profiling is performed.
  -log_dir string
    	Specifies directory which will hold the logging information. (default "./logs/")
  -log_level int
    	Specifies a log level from 1-7. Panic - 1, Fatal - 2, Error - 3, Warn - 4, Info - 5, Debug - 6, Trace - 7 (default 4)

Minimal Example

  1. Place .SC2Replay files in ./DEMOS/Input
  2. Run SC2InfoExtractorGo.exe with default flags.
  3. Verify the output in ./DEMOS/Output
  4. If The output packages do not contain any processed replays, proceed to verify ./logs/.

Dataset Preparation

If You have a pack of replays with nested directories and You would like to automatically flatten the directory structure, We have published a tool that can be used for that, please see SC2DatasetPreparator: https://doi.org/10.5281/zenodo.5296664

Two scripts contained within that software can:

  1. Flatten directory structure of Your collected replays by looking for .SC2Replay files.
  2. Running Python multiprocessing package on multiple directories containing replays by calling them with -with-multiprocessing=false flag. This allows to have one package per directory created and speed up the processing.

Anonymization

In order to anonymize the replays please make sure to download and run our open-source implementation of an anonymization server the SC2AnonServerPy: https://doi.org/10.5281/zenodo.5138313

This is required because of the multiprocessing nature of our code that needs to perform synchronization with an existing database of unique toons (player IDs) that are mapped to arbitrary incrementing integer.

Map Translation Support

If the provided file output.json does not support map names that You require, You will have to either find it online or within the game documents. We are close to publishing a tool that will allow You to download StarCraft maps.

If You already have a set of StarCraft II maps that You would like to create a .json file to be included within Our software, use Your own solution or SC2MapLocaleExtractor: https://doi.org/10.5281/zenodo.4733264

Filtering Capabilities

Currently the software supports some game mode filtering capabilities which can be used with -game_mode flag.
The flag itself is a binary flag where 0b11111111 is all game modes which is the default.

Other ways to set the flag:

  • 0b00000001: 1v1 Ranked Games
  • 0b00000010: 2v2 Ranked Games
  • 0b00000100: 3v3 Ranked Games
  • 0b00001000: 4v4 Ranked Games
  • 0b00010000: 1v1 Custom Games
  • 0b00100000: 2v2 Custom Games
  • 0b01000000: 3v3 Custom Games
  • 0b10000000: 4v4 Custom Games

Build from source

Our working solution was built by using go build command on 64 bit version of Windows 10

License / Dual Licensing

This repository is licensed under GNU GPL v3 license. If You would like to acquire a different license please contact me directly.

Loading

1.0.0 SC2InfoExtractorGo Release

13 Sep 20:24
@Kaszanas Kaszanas
1.0.0
f24d7a3

Choose a tag to compare

View all tags
1.0.0 SC2InfoExtractorGo Release

SC2InfoExtractorGo

This tool is meant to allow for quick data extraction from SC2 replay files ".SC2Replay".

Usage

In order to use this tool please call SC2InfoExtractorGo.exe and set the choosen flags listed below:

  -input string
    	Input directory where .SC2Replay files are held. (default "./DEMOS/Input")
  -output string
    	Output directory where compressed zip packages will be saved. (default "./DEMOS/Output")
  -game_mode int
    	Provide which game mode should be included from the processed files in a format of a binary flag: AllGameModes: 0b11111111 (default 255)
  -max_procs int
    	Specifies the number of logic cores of a processor that will be used for processing. (default 24)
  -number_of_packages int
    	Provide a number of zip packages to be created and compressed into a zip archive. Please remember that this number needs to be lower than the number of processed files. If set to 0, will ommit the zip packaging and save output .json directly to drive. (default 1)
  -localized_maps_file string
    	Specifies a path to localization file containing {'ForeignName': 'EnglishName'} of maps. If this flag is not set and the default is unavailable, map translation will be ommited. (default "./operation_files/output.json")
  -perform_player_anonymization
    	Specifies if the tool is supposed to perform player anonymization functions within the processing pipeline. If set to true please remember to download and run an anonymization server: https://doi.org/10.5281/zenodo.5138313
  -perform_chat_anonymization
    	Specifies if the chat anonymization should be performed.
  -perform_cleanup
    	Provide if the tool is supposed to perform the cleaning functions within the processing pipeline.
  -perform_integrity_checks
    	If the software is supposed to check the hardcoded integrity checks for the provided replays
  -perform_validity_checks
    	Provide if the tool is supposed to use hardcoded validity checks and verify if the replay file variables are within 'common sense' ranges.
  -log_dir string
    	Specifies directory which will hold the logging information. (default "./logs/")
  -log_level int
    	Specifies a log level from 1-7. Panic - 1, Fatal - 2, Error - 3, Warn - 4, Info - 5, Debug - 6, Trace - 7 (default 4)
  -with_cpu_profiler string
    	Set path to the file where pprof cpu profiler will save its information. If this is empty no profiling is performed.

Minimal Example

  1. Place .SC2Replay files in ./DEMOS/Input
  2. Run SC2InfoExtractorGo.exe with default flags.
  3. Verify the output in ./DEMOS/Output
  4. If The output packages do not contain any processed replays, proceed to verify ./logs/.

Dataset Preparation

If You have a pack of replays with nested directories and You would like to automatically flatten the directory structure, We have published a tool that can be used for that, please see SC2DatasetPreparator: https://doi.org/10.5281/zenodo.5296664

Two scripts contained within that software can:

  1. Flatten directory structure of Your collected replays by looking for .SC2Replay files.
  2. Running Python multiprocessing package on multiple directories containing replays by calling them with -with-multiprocessing=false flag. This allows to have one package per directory created and speed up the processing.

Anonymization

In order to anonymize the replays please make sure to download and run our open-source implementation of an anonymization server the SC2AnonServerPy: https://doi.org/10.5281/zenodo.5138313

This is required because of the multiprocessing nature of our code that needs to perform synchronization with an existing database unique toons (player IDs) that are mapped to arbitrary incrementing integer.

Map Translation Support

If the provided file output.json does not support map names that You require, You will have to either find it online or within the game documents. We are close to publishing a tool that will allow You to download StarCraft maps.

If You already have a set of StarCraft II maps that You would like to create a .json file to be included within Our software, use Your own solution or SC2MapLocaleExtractor: https://doi.org/10.5281/zenodo.4733264

Filtering Capabilities

Currently the software supports some game mode filtering capabilities which can be used with -game_mode flag.
The flag itself is a binary flag where 0b11111111 is all game modes which is the default.

Other ways to set the flag:

  • 0b00000001: 1v1 Ranked Games
  • 0b00000010: 2v2 Ranked Games
  • 0b00000100: 3v3 Ranked Games
  • 0b00001000: 4v4 Ranked Games
  • 0b00010000: 1v1 Custom Games
  • 0b00100000: 2v2 Custom Games
  • 0b01000000: 3v3 Custom Games
  • 0b10000000: 4v4 Custom Games

Build from source

Our working solution was built by using go build command on 64 bit version of Windows 10

License / Dual Licensing

This repository is licensed under GNU GPL v3 license. If You would like to acquire a different license please contact me directly.

Loading

0.5.0 SC2InfoExtractorGo Release

27 Aug 22:39
@Kaszanas Kaszanas
0.5.0
b55d88d

Choose a tag to compare

View all tags
0.5.0 SC2InfoExtractorGo Release

SC2InfoExtractorGo

This tool is meant to allow for quick data extraction from SC2 replay files ".SC2Replay".

Usage

In order to use this tool please call SC2InfoExtractorGo.exe and set the choosen flags listed below:

  -compression_method int
    	Provide a compression method number, default is 8 'Deflate', other compression methods need to be registered manually in code. (default 8)
  -game_mode int
    	Provide which game mode should be included from the processed files in a format of a binary flag: AllGameModes: 0b1111111111 (default 1023)
  -input string
    	Input directory where .SC2Replay files are held. (default "./DEMOS/Input")
  -integrity_check
    	If the software is supposed to check the hardcoded integrity checks for the provided replays (default true)
  -localize_maps
    	Set to false if You want to keep the original (possibly foreign) map names. (default true)
  -localized_maps_file string
    	Specify a path to localization file containing {'ForeignName': 'EnglishName'} of maps. (default "./operation_files/output.json")
  -log_dir string
    	Provide directory which will hold the logging information. (default "./logs/")
  -log_level int
    	Provide a log level from 1-7. Panic - 1, Fatal - 2, Error - 3, Warn - 4, Info - 5, Debug - 6, Trace - 7 (default 4)
  -number_of_packages int
    	Provide a number of packages to be created and compressed into a zip archive. Please remember that this number needs to be lower than the number of processed files. (default 1)
  -output string
    	Output directory where compressed zip packages will be saved. (default "./DEMOS/Output")
  -perform_anonymization
    	Provide if the tool is supposed to perform the anonymization functions within the processing pipeline. (default true)
  -perform_cleanup
    	Provide if the tool is supposed to perform the cleaning functions within the processing pipeline. (default true)
  -validity_check
    	Provide if the tool is supposed to use hardcoded validity checks and verify if the replay file variables are within 'common sense' ranges. (default true)
  -with_cpu_profiler string
    	Set path to the file where pprof cpu profiler will save its information. If this is empty no profiling is performed.
  -with_multiprocessing
    	Provide if the processing is supposed to be perform with maximum amount of available cores. If set to false, the program will use one core. (default true)

Build from source

Our working solution was built by using go build command on 64 bit version of Windows 10

License

This repository is licensed under GNU GPL v3 license. If You would like to acquire a different license please contact me directly.

Loading