Commit 639e387
committed
Optimize _create_zip_file with C++ wrapper to reduce analysis-phase memory
Context:
The _create_zip_file function in py_executable.bzl was causing significant
memory and CPU usage during Bazel's analysis phase when building at scale.
The root cause was calling to_list() on runfiles depsets containing thousands
of files, forcing O(N) memory allocation during analysis.
Intent:
Replace Starlark depset materialization with a lightweight C++ wrapper that
processes file lists at execution time. This achieves O(1) analysis-phase
memory while maintaining native execution performance.
Changes:
- Add py_executable_zip_gen.cc: C++ tool using std::filesystem for path
manipulation and Bazel's runfiles library to locate zipper
- Update BUILD.bazel: Replace sh_binary with cc_binary for py_executable_zip_gen
- Update py_executable.bzl: Simplify _create_zip_file to use wrapper's API,
eliminate to_list() calls, pass runfiles_without_exe to avoid filtering
- Update py_repositories.bzl: Add dependency on rules_shell
Performance Impact:
- Analysis phase: O(N) → O(1) memory (primary goal achieved)
- Execution phase: +0.1% overhead (within measurement noise)
- One-time cost: 1-2s C++ compilation per workspace
- Build artifacts: Byte-for-byte identical to main branch (same SHA256)
Testing:
- Verified identical build artifacts via SHA256 hash comparison
- Benchmarked incremental builds: 9.869s (main) vs 9.879s (C++ wrapper)
- Tested with py_binary and py_test targets
- Confirmed std::filesystem works with default Bazel C++ toolchain1 parent f92ad71 commit 639e387
File tree
4 files changed
+300
-54
lines changed- python/private
4 files changed
+300
-54
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
829 | 829 | | |
830 | 830 | | |
831 | 831 | | |
| 832 | + | |
| 833 | + | |
| 834 | + | |
| 835 | + | |
| 836 | + | |
| 837 | + | |
| 838 | + | |
| 839 | + | |
| 840 | + | |
| 841 | + | |
832 | 842 | | |
833 | 843 | | |
834 | 844 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
236 | 236 | | |
237 | 237 | | |
238 | 238 | | |
239 | | - | |
| 239 | + | |
240 | 240 | | |
241 | 241 | | |
242 | 242 | | |
| |||
377 | 377 | | |
378 | 378 | | |
379 | 379 | | |
| 380 | + | |
380 | 381 | | |
381 | 382 | | |
382 | 383 | | |
383 | | - | |
384 | 384 | | |
385 | | - | |
| 385 | + | |
386 | 386 | | |
387 | 387 | | |
388 | 388 | | |
| |||
803 | 803 | | |
804 | 804 | | |
805 | 805 | | |
806 | | - | |
| 806 | + | |
| 807 | + | |
807 | 808 | | |
808 | | - | |
809 | 809 | | |
810 | 810 | | |
811 | | - | |
812 | | - | |
813 | | - | |
814 | | - | |
815 | | - | |
816 | | - | |
817 | | - | |
818 | | - | |
819 | | - | |
820 | | - | |
821 | | - | |
822 | | - | |
823 | | - | |
824 | | - | |
825 | | - | |
826 | | - | |
827 | | - | |
828 | | - | |
829 | | - | |
830 | | - | |
831 | | - | |
832 | | - | |
| 811 | + | |
| 812 | + | |
| 813 | + | |
833 | 814 | | |
834 | | - | |
| 815 | + | |
| 816 | + | |
| 817 | + | |
| 818 | + | |
| 819 | + | |
| 820 | + | |
| 821 | + | |
| 822 | + | |
835 | 823 | | |
836 | 824 | | |
837 | 825 | | |
| |||
844 | 832 | | |
845 | 833 | | |
846 | 834 | | |
847 | | - | |
848 | | - | |
849 | | - | |
850 | | - | |
| 835 | + | |
| 836 | + | |
851 | 837 | | |
852 | 838 | | |
853 | | - | |
854 | | - | |
855 | | - | |
856 | | - | |
857 | | - | |
858 | | - | |
859 | | - | |
860 | | - | |
861 | | - | |
862 | | - | |
| 839 | + | |
| 840 | + | |
863 | 841 | | |
864 | 842 | | |
865 | 843 | | |
866 | | - | |
867 | | - | |
| 844 | + | |
| 845 | + | |
868 | 846 | | |
869 | 847 | | |
870 | 848 | | |
871 | 849 | | |
872 | 850 | | |
873 | 851 | | |
874 | | - | |
875 | | - | |
876 | | - | |
877 | | - | |
878 | | - | |
879 | | - | |
880 | | - | |
881 | | - | |
882 | | - | |
883 | | - | |
| 852 | + | |
| 853 | + | |
| 854 | + | |
| 855 | + | |
| 856 | + | |
| 857 | + | |
| 858 | + | |
884 | 859 | | |
885 | 860 | | |
886 | 861 | | |
| |||
0 commit comments