Skip to content

README.md

Latest commit

 

History

History
126 lines (86 loc) · 4.36 KB

README.md

File metadata and controls

126 lines (86 loc) · 4.36 KB

Building ya & ymake from sources

This directory contains files required to build a build system.

1st way: build using toolchain-registry

Only linux-x86_64 is supported for bootstrapping. Note that the build process consumes some 100Gb of disk space and takes over 1 hour to complete depending on the machine used.

mkdir $HOME/tr_ya_bootstrap
# or docker
sudo podman build --network=host -f tr.Dockerfile --tag tr_ya_bootstrap
sudo podman run --name tr_ya_bootstrap_container --network=host --mount type=bind,source=$HOME/tr_ya_bootstrap,target=/result tr_ya_bootstrap

After the process is successfully complete you will see ya-bin, ymake and other tools in ~/tr_ya_bootstrap. Note that they will be owned by root if you run podman in rootful mode.

You can then do a clean up

sudo podman container rm tr_ya_bootstrap_container
sudo podman image rm tr_ya_bootstrap

2nd way: execute a premade graph

Dependencies

For successful build it is required to have docker or podman and python3.9+ installed in system.

Launching bootstrap

Bootstrap is launched with run_bootstrap.py script

It has several flags, as follows:

  • --source-root -- Path to the repository root. Required
  • --result-root -- Path to the directory, where results should be put. Required
  • --all-platforms-build -- This flag will also launch the second stage of bootstrap.
  • --cleanup -- Remove all containers and images created by this image.

How does it work?

Before running the script checkout to the commit when the graph was last updated. Blame a line with URL variable in run_bootstrap.py to find out.

Bootstrap can be described in two stages

Stage one

This stage's execution results in ya and ymake for linux.

This stage can be launched separately with stage1.sh Call syntax is as follows

./stage1.sh <source_root> <build_root> <results_root>

Here:

  • source_root - Path to repository root with ya sources;
  • build_root - Path to the directory, where all build results will be stored;
  • results_root - Path to the directory, where final results will be stored.

This script uses system libraries, thus tools built, for example, with Ubuntu 24.04, will not work on earlier Ubuntu's

This script invokes graph_executor.py, which downloads build graph from S3 and executes it, then moves ya, ymake and libiconv.so from build_root to $results_root/stage1.

It is called by:

python3 graph_executor.py <source_root> <build_root> <threads> [<path/to/local/graph>]

The last argument is optional and sohuld be used only in debug purposes.

Important! Python 3.9 or newer is required for this script.

Stage two

This stage is optional and uses tooling built in the stage one to build all targets, namely:

  • devtools/ya/bin
  • devtools/local_cache/toolscache/server
  • devtools/ymake/bin
  • devtools/yexport/bin

This stage is orchestrated by stage2.sh. It has similar to stage1.sh invocation syntax, but it builds all targets from this repositories for platforms from platform_list

Build results will be stored in $result_root/stage2/<arch> for main tools and in $result_root/stage2/additional/<arch> for additional ones (now it is yexport only)

Docker images

To ensure platform independency we use docker images built from two Dockerfiles.

These images each execute their own stage of bootstrap described earlier

  • stage1.Dockerfile is a Ubuntu 24.04-based image with everything needed for successful graph execution.
  • stage2.Dockerfile is the basic Ubuntu 24.04 image. Its launch allows you to get all the tools distributed in this repository.

How to build an image

An image is built by default as follows:

docker build -f <path/to/dockerfile> . -t ya-bootstrap-stage<StageNo>:latest

How to run an image

These images are run as follows:

docker run \
    --name test \
    --mount type=bind,source=<path/to/repo>,target=/source_root,readonly \
    --mount type=bind,source=<path/to/result>,target=/result \
    ya-bootstrap-stage<StageNo>:latest

How graph is distributed

Graph is obtained with gen_graph/gg2 script and uploaded to S3.

This allows to avoid large diffs compared to committing graph straight into a repo.

The path to repository root is required in source_root variable.

The invocation is as follows:

source_root=path/to/repo ./gen_graph/gg2 > fname