Replication package

This repository contains the source code and data used in the paper "Estimating Deep Learning energy consumption based on model architecture and training environment".

Training data

We do not share the Chesslive dataset. However, you can use the Caltech101 and Stanford Dogs datasets to train the Inception V3 model. To use this datasets with the rest of the models you can extend each of the models' base class located in src/models/dl and adding it to the model_factory.

Collected data

All the data collected and produced during the study can be found in the data.zip file in the Releases section.

The folder is expected to be extracted at the root of the project and the metrics collected can be found inside the data/metrics folder. The data is organized in the following structure:

.
├── auxiliary
├── raw
├── interim
└── processed

The auxiliary folder contains a the list of raw measurements that have been processed. These are used to speed up the processing of new raw data. The raw folder contains the raw measurements collected during the experiment. The interim folder contains the processed data that is used to generate the final dataset. The processed folder contains the final data used to perform the analysis.

Replicating the experiments

We do not provide a Docker container for the experiments since it is not possible to collect energy measurements inside a container. Nevertheless, we provide all the raw data and the source code used to run the experiments.

If you want to reproduce the experiments, you can use the provided scripts and configuration files.

Set up the environment

To set up the environment, you need to create a virtual environment and install the required dependencies. We use uv to manage the dependencies. To install the dependencies, you can run the following command:

uv sync

If you want to use any other dependency manager, you can look at the pyproject.toml file for the required dependencies. Alternatively, you can use the requirements.txt file to install the dependencies using pip:

pip install -r requirements.txt

MLflow configuration

We use MLflow to keep track of the different experiments. By default, its usage is disabled. If you want to use MLflow, you need to:

• Configure your own tracking server.

• Activate MLflow logging in the corresponding experiment_#.yaml configuration file.

• Create a .env file in the project root with the following structure:

MLFLOW_TRACKING_URI = https://url/to/your/tracking/server
MLFLOW_TRACKING_USERNAME = tracking_server_username
MLFLOW_TRACKING_PASSWORD = tracking_server_password

If you do not need users credentials for MLflow leave the fields empty.

Resources configuration

This package uses a resources configuration to manage the GPU memory limit allowed and the use of cache. We do not share this file since each machine has its own hardware specifications. You will need to create a resources.yaml file inside the config folder with the following structure:

GPU:
  MEM_LIMIT: 2048

USE_CACHE: true

The GPU memory limit must be specified in Megabytes. If you do not want to set a GPU memory limit, leave the field empty.

WARNING! The memory limit value is just an example. Do not take it as a reference.

Running the experiment

Once the environment is set up, you can run the experiment by executing the following command:

$ python -m run_experiments [-h] [--experiment-name EXPERIMENT_NAME] {desktop,server} {experiment_1.yaml,experiment_2.yaml,experiment_3.yaml}

positional arguments:
  {desktop,server}         The environment to run the profiling in.
  {experiment_1.yaml,experiment_2.yaml,experiment_3.yaml}
                        The name of the configuration file to use.

options:
  -h, --help            show this help message and exit
  --experiment-name EXPERIMENT_NAME
                        The name of the MLflow experiment.

The raw measurements for each architecture will be saved in the data/metrics/raw/{desktop, server}/architecture_name folder. If MLflow is enabled, the measurements will also be saved in the MLflow tracking server, together with the trained models. If not, the trained models will be saved in the models folder and the training history will be saved with the raw measurements as performance-%Y%m%dT%H%M%S.csv.

You can also train a single model by executing the following command:

$ python -m runner [-h] [--warmup] {desktop,server} {experiment_1.yaml,experiment_2.yaml,experiment_3.yaml} {single-run} ...

positional arguments:
  {desktop,server}         The type of training environment.
  {experiment_1.yaml,experiment_2.yaml,experiment_3.yaml}
                        The configuration file to use.
  {single-run}          sub-command help
    single-run          Single training help

options:
  -h, --help            show this help message and exit
  --warmup              Warmup the GPU.

The training history and the model will be saved following the same rules as the profiling script.

Data analysis

The data analysis is done using Jupyter Notebooks and R markdown notebooks. You can find the analysis inside the notebooks folder. All the plots generated are saved in the reports/figures folder.

Running the data analysis from a Docker container

We provide a Docker image you can use to reproduce the data analysis. You can get it from Docker Hub:

docker pull santidr/dl-training-energy-estimation:1.0

The image uses the appuser user to run the container and all the necessary files are located at /home/appuser/workspace. This user does not have root privileges, so you may need to adjust file permissions if you want to install additional packages.

To create and start the container for the first time, you can use the following command:

docker run -it santidr/dl-training-energy-estimation:1.0

Running the data analysis from source

If you prefer to run the data analysis directly on your machine, you can do so by following the steps in the Replicating the experiments section and the following additional steps.

Install R dependencies

To run the data analysis, you will need to install R and the required packages. You can use the install_packages.R script to install the required packages.

To run the script, you can execute the following command:

Rscript install_packages.R

Install required fonts

This project uses the Computer Modern Unicode font for the plots. You can install it in Ubuntu-based systems by running the following command:

sudo apt install fonts-cmu

License

The software under this project is licensed under the terms of the Apache 2.0 license. See the LICENSE file for more information.

The data used in this project is licensed under the terms of the CC BY 4.0 license. See the LICENSE in the data/LICENSE file for more information.