Skip to content

Commit c6ee9cf

Browse files
[fix](compile) auto-regenerate hadoop-deps/lib when missing due to Ma… (#60964)
…ven cache Problem: - When Maven build cache is hit, the dependency:copy-dependencies plugin execution is skipped - This causes target/lib directory to be missing in hadoop-deps module - Results in an incomplete BE output package without Hadoop dependencies Solution: - Add automatic regeneration logic in build.sh copy phase - When lib directory is missing and BUILD_BE_JAVA_EXTENSIONS=1, execute 'mvn dependency:copy-dependencies' to regenerate it - Preserves Maven cache benefits while ensuring complete output Impact: - Fixes intermittent missing hadoop_hdfs jars in BE output - No impact on normal build flow (only triggers when needed) - Maintains build cache acceleration for other plugins Related PR: #60819
1 parent bd2210a commit c6ee9cf

File tree

1 file changed

+7
-0
lines changed

1 file changed

+7
-0
lines changed

build.sh

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -970,6 +970,13 @@ EOF
970970
mkdir "${BE_HADOOP_HDFS_DIR}"
971971
HADOOP_DEPS_JAR_DIR="${DORIS_HOME}/fe/be-java-extensions/${HADOOP_DEPS_NAME}/target"
972972
echo "HADOOP_DEPS_JAR_DIR: ${HADOOP_DEPS_JAR_DIR}"
973+
if [[ "${BUILD_BE_JAVA_EXTENSIONS}" -eq 1 && ! -d "${HADOOP_DEPS_JAR_DIR}/lib" ]]; then
974+
echo "WARN: lib directory missing (likely due to Maven cache). Regenerating..."
975+
pushd "${DORIS_HOME}/fe/be-java-extensions/${HADOOP_DEPS_NAME}"
976+
"${MVN_CMD}" dependency:copy-dependencies -DskipTests -Dcheckstyle.skip=true
977+
mv target/dependency target/lib
978+
popd
979+
fi
973980
if [[ -f "${HADOOP_DEPS_JAR_DIR}/${HADOOP_DEPS_NAME}.jar" ]]; then
974981
echo "Copy Be Extensions hadoop deps jar to ${BE_HADOOP_HDFS_DIR}"
975982
cp "${HADOOP_DEPS_JAR_DIR}/${HADOOP_DEPS_NAME}.jar" "${BE_HADOOP_HDFS_DIR}"

0 commit comments

Comments
 (0)