TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices #427

abstractdog · 2025-09-06T12:12:50Z

This PR contains a Zookeeper-based Framework service and a major refactor in this area.

Pieces of this PR:

include curator dependency
remove amUuid from DAGAppMaster
harmonized id/opaque id to externalId in terms of variable and env constant name
FrameworkService base interface: marker interface for framework-level services, implementations are framework-specific (currently yarn or zookeeper)
- ClientFrameworkService: interface for client-side framework services; users of this class can acquire tez clients for the actual framework (yarn/zookeeper)
- ServerFrameworkService: interface for server-side framework services; users of this class can acquire server-side logic for the actual framework (yarn/zookeeper)
AMExtensions: a class that encapsulates all the pluggable logic within DAGAppMaster
update unit tests to provide an AMExtensions to prevent NPE while mocking

okumin · 2025-09-11T02:32:40Z

tez-api/src/main/java/org/apache/tez/client/registry/zookeeper/ZkAMRegistryClient.java

+  * Curator/Zookeeper impl of AMRegistryClient
+*/
+@InterfaceAudience.Public
+public class ZkAMRegistryClient extends AMRegistryClient {


Just a question. What is the duty of ZooKeeper here? When I read this patch a while ago, I presumed ZooKeeper would be used for two reasons. My memory might be wrong.

Generate a unique application ID

Service discovery for application masters

@okumin: both:

ZkAMRegistry.generateNewId takes care of generating applicationId (backed by zookeeper)

it also for service discovery (AM registers itself, client receives updates about that)

I'm happy to see that you're interested in the area!
please be aware that it's under refactor, I'll let you know, when it's ready

Thanks. Background: I wonder if it can be independent of ZooKeeper technically. The reliability and fault-tolerance of ZK are beneficial in many environments, but in other cases, high availability is not necessary. Trino does not provide fault-tolerance with its coordinator, and some users appreciate the ease of use.

absolutely! be aware that these patches have been working for years in downstream distributions. This PR is about contributing them back, so apparently, it's ZK the long-term plan is to improve abstractions and make it more pluggable, and replace ZK with Kubernetes etcd, for example, but the default (yarn-based) behavior doesn't utilize an AM registry at all
can you please elaborate on how Trino's lack of fault tolerance comes to this picture? I mean, this ZK-based registry is more about AM service discovery, not fault tolerance

Hi folks, sorry, I'd like to chime in with a question here. :)

From the description of TEZ-3991, this feature seems to allow the AM to manage its own lifecycle, while clients such as HiveServer2 only need to discover available AMs via ZooKeeper (ZK) and submit tasks.

Based on my current limited understanding, the benefits of this feature should include reducing the burden on HS2 and potentially making task submission more efficient.

However, I have a few questions:

Is this feature primarily beneficial for LLAP mode? Since LLAP requires long-running Tez AMs, and IMO this feature seems more useful for long-running tasks.

Could this functionality serve as preliminary groundwork for decoupling the Tez framework from YARN? For example, could it pave the way for Tez to run directly on Kubernetes without relying on YARN for resource management?

Thanks.

be aware that these patches have been working for years in downstream distributions

Sounds excellent.

can you please elaborate on how Trino's lack of fault tolerance comes to this picture?

Trino must have SPOF, but still attracts users. Hive on Tez (with LLAP) without fault tolerance is easier to set up than Hive on Tez with fault tolerance. I expect the easy quick start should attract users.

tez-yetus · 2025-12-02T12:02:29Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 30s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+0 🆗	xmllint	0m 1s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 14 new or modified test files.
			_ master Compile Tests _
+0 🆗	mvndep	2m 38s		Maven dependency ordering for branch
+1 💚	mvninstall	9m 51s		master passed
+1 💚	compile	5m 23s		master passed
+1 💚	checkstyle	3m 14s		master passed
+1 💚	javadoc	4m 3s		master passed
+0 🆗	spotbugs	2m 9s		tez-api in master has 610 extant spotbugs warnings.
+0 🆗	spotbugs	0m 47s		tez-common in master has 13 extant spotbugs warnings.
+0 🆗	spotbugs	1m 10s		tez-runtime-library in master has 241 extant spotbugs warnings.
+0 🆗	spotbugs	0m 45s		tez-examples in master has 2 extant spotbugs warnings.
+0 🆗	spotbugs	1m 44s		tez-dag in master has 785 extant spotbugs warnings.
+0 🆗	spotbugs	6m 51s		root in master has 2066 extant spotbugs warnings.
-0 ⚠️	patch	7m 34s		Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 17s		Maven dependency ordering for patch
+1 💚	mvninstall	6m 21s		the patch passed
+1 💚	codespell	0m 50s		No new issues.
+1 💚	compile	5m 19s		the patch passed
+1 💚	javac	5m 19s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 26s		tez-api: The patch generated 0 new + 118 unchanged - 3 fixed = 118 total (was 121)
+1 💚	checkstyle	0m 23s		The patch passed checkstyle in tez-common
+1 💚	checkstyle	0m 27s		The patch passed checkstyle in tez-runtime-library
+1 💚	checkstyle	0m 23s		tez-examples: The patch generated 0 new + 10 unchanged - 1 fixed = 10 total (was 11)
-0 ⚠️	checkstyle	0m 33s	/results-checkstyle-tez-dag.txt	tez-dag: The patch generated 2 new + 610 unchanged - 2 fixed = 612 total (was 612)
-0 ⚠️	checkstyle	0m 45s	/results-checkstyle-root.txt	root: The patch generated 2 new + 740 unchanged - 6 fixed = 742 total (was 746)
+1 💚	javadoc	3m 59s		the patch passed
+1 💚	spotbugs	14m 13s		the patch passed
			_ Other Tests _
+1 💚	unit	2m 36s		tez-api in the patch passed.
+1 💚	unit	0m 42s		tez-common in the patch passed.
+1 💚	unit	6m 9s		tez-runtime-library in the patch passed.
+1 💚	unit	0m 32s		tez-examples in the patch passed.
+1 💚	unit	6m 7s		tez-dag in the patch passed.
+1 💚	unit	73m 45s		root in the patch passed.
+1 💚	asflicense	2m 46s		The patch does not generate ASF License warnings.
		170m 36s

Subsystem	Report/Notes
Docker	ClientAPI=1.52 ServerAPI=1.52 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/36/artifact/out/Dockerfile
GITHUB PR	#427
Optional Tests	dupname asflicense javac javadoc unit codespell detsecrets xmllint compile spotbugs checkstyle
uname	Linux e3f0c21725bf 5.15.0-161-generic #171-Ubuntu SMP Sat Oct 11 08:17:01 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	/home/jenkins/jenkins-home/workspace/tez-multibranch_PR-427/src/.yetus/personality.sh
git revision	master / `3938781`
Default Java	Ubuntu-21.0.8+9-Ubuntu-0ubuntu124.04.1
Test Results	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/36/testReport/
Max. process+thread count	1379 (vs. ulimit of 5500)
modules	C: tez-api tez-common tez-runtime-library tez-examples tez-dag . U: .
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/36/console
versions	git=2.43.0 maven=3.8.7 spotbugs=4.9.3 codespell=2.0.0
Powered by	Apache Yetus 0.15.1 https://yetus.apache.org

This message was automatically generated.

ayushtkn

Minor comments. Rest LGTM

ayushtkn · 2025-12-11T08:15:05Z

tez-api/src/main/java/org/apache/tez/client/registry/AMRecord.java

    if (other instanceof AMRecord otherRecord) {
      return appId.equals(otherRecord.appId)
-          && host.equals(otherRecord.host)
+          && hostName.equals(otherRecord.hostName)


can this lead to NPE, we don't have any default here

this.hostName = serviceRecord.get(HOST_NAME_RECORD_KEY);

this is initialized from DAGAppMaster:
https://github.com/apache/tez/pull/427/files#diff-54ba4a2af15261379079ed5a9c1f9eea52da7bdd3f1109fe7b96d4abf07f6173R677-R678

AMRecord amRecord = amRegistry.createAmRecord(appId, rpcServerAddress.getHostName(), rpcServerAddress.getAddress().getHostAddress(), rpcServerAddress.getPort(), computeName);

hostName is not supposed to be null, so if it's null, it's already a fatal issue, which is revealed by NPE, I don't think we make this better

ayushtkn · 2025-12-11T08:20:27Z

tez-api/src/main/java/org/apache/tez/client/registry/AMRecord.java


+  @Override
+  public String toString() {
+    return toServiceRecord().attributes().toString();


toString calls toServiceRecord, So, it can lead to initializing the serviceRecord = new ServiceRecord();, are we ok with it. This could lead to issues later, like someone logged it or so, before setting the values and then we have the service record cached as well

I think there is no "before setting the values" scenario given only final fields, I'm assuming the the logging must happen with the AMRecord instance when the constructor is definitely finished

ayushtkn · 2025-12-11T08:21:32Z

tez-api/src/main/java/org/apache/tez/client/registry/AMRegistry.java

+  AMRecord createAmRecord(ApplicationId appId, String hostName, String hostIp, int port,
+                                          String computeName);
+
+  void close();


AutoCloseable already defines close, do we need to define again here?

ayushtkn · 2025-12-11T08:22:59Z

tez-api/src/main/java/org/apache/tez/client/registry/AMRegistry.java

+  void add(AMRecord server) throws Exception;
+
+  void remove(AMRecord server) throws Exception;


should it be server or record

record is better, fixing this

ayushtkn · 2025-12-11T08:23:47Z

tez-api/src/main/java/org/apache/tez/client/registry/AMRegistryClient.java

+ * <p>Listeners may be registered to receive notifications when AM records
+ * appear or are removed.</p>
+ */
+public abstract class AMRegistryClient implements Closeable {


can we do implements AutoCloseable?

we can, but it doesn't make the code better: this class is usually used as field, not local variable in try-catch, this applies to production as well as testing code, so I think Closeable might be better here

ayushtkn · 2025-12-11T08:41:35Z

tez-api/src/main/java/org/apache/tez/client/registry/zookeeper/ZkFrameworkClient.java

+  @Override
+  public void start() {
+    try {
+      amRegistryClient.start();


we should return if isRunning is true here

ayushtkn · 2025-12-11T08:43:52Z

tez-api/src/main/java/org/apache/tez/client/registry/zookeeper/ZkFrameworkClient.java

+
+public class ZkFrameworkClient extends FrameworkClient {
+
+  private AMRecord amRecord;


should this be volatile

it should, also, tracking amHost and amPort separately doesn't make sense, let's just rely on a single volatile record field

ayushtkn · 2025-12-11T08:45:57Z

tez-api/src/main/java/org/apache/tez/frameworkplugins/FrameworkUtils.java

+      } else if (modeInEnv != null) {
+        return getByMode(interfaze, modeInEnv);
+      } else if (defaultClazz != null) {
+        return (T) defaultClazz.newInstance();


I think this is deprecated, does this work

defaultClazz.getDeclaredConstructor().newInstance();

ayushtkn · 2025-12-11T08:46:26Z

tez-api/src/main/java/org/apache/tez/frameworkplugins/FrameworkUtils.java

+    String modeInConf = conf != null ? conf.get(TezConfiguration.TEZ_FRAMEWORK_MODE) : null;
+    String modeInEnv = System.getenv(TezConstants.TEZ_FRAMEWORK_MODE);
+    try {
+      if (modeInConf != null) {


should it be StringUtils.isNotEmpty()?

ack, much cleaner and takes care of empty string too

ayushtkn · 2025-12-11T08:48:54Z

tez-api/src/test/java/org/apache/tez/client/TestTezClient.java

+            .thenReturn(YarnApplicationState.RUNNING);
+
+    //Client 1 start
+    client.start();


missing close?

…s - initial patch

…s - checkstyle, spotbugs, javadoc improvements, refactor, test fixes

abstractdog · 2026-01-09T12:02:03Z

@ayushtkn : address your comments, good catches, thanks!

okumin

Thanks for porting the nice feature 👍

okumin · 2026-01-12T07:56:14Z

tez-dag/src/main/java/org/apache/tez/dag/app/DAGAppMaster.java

+
    if (sessionToken == null) {
-      throw new RuntimeException("Could not find session token in AM Credentials");
+      sessionToken = TokenCache.getSessionToken(amCredentials);


When does this double-check help?

YARN -> When sessionToken is null, TokenCache.getSessionToken(amCredentials) is probably always null

ZK -> sessionToken is not always null

right, so the extra attempt to fetch it from session token doesn't make sense, let me change this to:

if (sessionToken == null) { throw new RuntimeException("Could not find session token in AM Credentials"); }

okumin · 2026-01-12T08:08:21Z

.../main/java/org/apache/tez/frameworkplugins/zookeeper/ZkStandaloneClientFrameworkService.java

+import org.apache.tez.frameworkplugins.ClientFrameworkService;
+
+public class ZkStandaloneClientFrameworkService implements ClientFrameworkService {
+  @Override public FrameworkClient newFrameworkClient() {


Suggested change

@Override public FrameworkClient newFrameworkClient() {

@Override

public FrameworkClient newFrameworkClient() {

good catch, thanks!

tez-yetus · 2026-01-12T09:48:12Z

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/40/console in case of problems.

tez-yetus · 2026-01-12T12:46:31Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	20m 46s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	shelldocs	0m 1s		Shelldocs was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 14 new or modified test files.
			_ master Compile Tests _
+0 🆗	mvndep	2m 15s		Maven dependency ordering for branch
+1 💚	mvninstall	9m 32s		master passed
+1 💚	compile	5m 24s		master passed
+1 💚	checkstyle	3m 17s		master passed
+1 💚	javadoc	4m 5s		master passed
-1 ❌	spotbugs	1m 15s	/branch-spotbugs-tez-api.txt	tez-api in master failed.
-1 ❌	spotbugs	0m 29s	/branch-spotbugs-tez-common.txt	tez-common in master failed.
-1 ❌	spotbugs	0m 38s	/branch-spotbugs-tez-runtime-library.txt	tez-runtime-library in master failed.
-1 ❌	spotbugs	0m 31s	/branch-spotbugs-tez-examples.txt	tez-examples in master failed.
-1 ❌	spotbugs	0m 43s	/branch-spotbugs-tez-dag.txt	tez-dag in master failed.
-1 ❌	spotbugs	0m 24s	/branch-spotbugs-root.txt	root in master failed.
-0 ⚠️	patch	8m 59s		Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 19s		Maven dependency ordering for patch
+1 💚	mvninstall	6m 20s		the patch passed
+1 💚	codespell	0m 51s		No new issues.
+1 💚	compile	5m 25s		the patch passed
+1 💚	javac	5m 25s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 27s		tez-api: The patch generated 0 new + 118 unchanged - 3 fixed = 118 total (was 121)
+1 💚	checkstyle	0m 23s		The patch passed checkstyle in tez-common
+1 💚	checkstyle	0m 27s		The patch passed checkstyle in tez-runtime-library
+1 💚	checkstyle	0m 24s		tez-examples: The patch generated 0 new + 10 unchanged - 1 fixed = 10 total (was 11)
-0 ⚠️	checkstyle	0m 34s	/results-checkstyle-tez-dag.txt	tez-dag: The patch generated 2 new + 610 unchanged - 2 fixed = 612 total (was 612)
-0 ⚠️	checkstyle	0m 46s	/results-checkstyle-root.txt	root: The patch generated 2 new + 740 unchanged - 6 fixed = 742 total (was 746)
+1 💚	hadolint	0m 1s		No new issues.
+1 💚	shellcheck	0m 0s		No new issues.
+1 💚	javadoc	3m 57s		the patch passed
+1 💚	spotbugs	13m 41s		the patch passed
			_ Other Tests _
+1 💚	unit	2m 35s		tez-api in the patch passed.
+1 💚	unit	0m 43s		tez-common in the patch passed.
+1 💚	unit	6m 12s		tez-runtime-library in the patch passed.
+1 💚	unit	0m 31s		tez-examples in the patch passed.
+1 💚	unit	5m 54s		tez-dag in the patch passed.
+1 💚	unit	72m 52s		root in the patch passed.
+1 💚	asflicense	2m 47s		The patch does not generate ASF License warnings.
		178m 42s

Subsystem	Report/Notes
Docker	ClientAPI=1.52 ServerAPI=1.52 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/40/artifact/out/Dockerfile
GITHUB PR	#427
Optional Tests	dupname asflicense codespell detsecrets hadolint shellcheck shelldocs javac javadoc unit xmllint compile spotbugs checkstyle
uname	Linux be3143d28af1 5.15.0-161-generic #171-Ubuntu SMP Sat Oct 11 08:17:01 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	/home/jenkins/jenkins-home/workspace/tez-multibranch_PR-427/src/.yetus/personality.sh
git revision	master / `0f9a678`
Default Java	Ubuntu-21.0.9+10-Ubuntu-124.04
Test Results	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/40/testReport/
Max. process+thread count	1306 (vs. ulimit of 5500)
modules	C: tez-api tez-common tez-runtime-library tez-examples tez-dag . U: .
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/40/console
versions	git=2.43.0 maven=3.8.7 hadolint=1.18.0-0-g76eee5c codespell=2.0.0 shellcheck=0.7.1
Powered by	Apache Yetus 0.15.1 https://yetus.apache.org

This message was automatically generated.

abstractdog · 2026-01-12T13:26:57Z

hm, something happened to spotbugs, even the latest version didn't help:
0f9a678

abstractdog · 2026-01-14T07:57:57Z

hm, something happened to spotbugs, even the latest version didn't help: 0f9a678

created TEZ-4669 about this, assuming spotbugs to be clean with this patch according to earlier runs (like this)

ayushtkn

LGTM

Thanx @abstractdog for the nice work & bearing with my slow reviews :-)

abstractdog · 2026-01-14T10:14:07Z

Thanks, @ayushtkn, for the approval!
I'm about to merge this soon, let me wait another few hours to maybe let @okumin, @zhangbutao chime in if I addressed all their comments regarding the code

zhangbutao · 2026-01-14T10:55:38Z

Thank you! @abstractdog I can't wait to experience the features of Tez on Cloud. Let's move on to the next step!

tez-yetus · 2026-01-14T12:37:25Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 45s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+0 🆗	xmllint	0m 1s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 14 new or modified test files.
			_ master Compile Tests _
+0 🆗	mvndep	2m 26s		Maven dependency ordering for branch
+1 💚	mvninstall	9m 37s		master passed
+1 💚	compile	5m 24s		master passed
+1 💚	checkstyle	3m 18s		master passed
+1 💚	javadoc	3m 58s		master passed
-1 ❌	spotbugs	1m 19s	/branch-spotbugs-tez-api.txt	tez-api in master failed.
-1 ❌	spotbugs	0m 30s	/branch-spotbugs-tez-common.txt	tez-common in master failed.
-1 ❌	spotbugs	0m 37s	/branch-spotbugs-tez-runtime-library.txt	tez-runtime-library in master failed.
-1 ❌	spotbugs	0m 30s	/branch-spotbugs-tez-examples.txt	tez-examples in master failed.
-1 ❌	spotbugs	0m 44s	/branch-spotbugs-tez-dag.txt	tez-dag in master failed.
-1 ❌	spotbugs	0m 24s	/branch-spotbugs-root.txt	root in master failed.
-0 ⚠️	patch	12m 41s		Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 16s		Maven dependency ordering for patch
+1 💚	mvninstall	6m 22s		the patch passed
+1 💚	codespell	0m 51s		No new issues.
+1 💚	compile	5m 28s		the patch passed
+1 💚	javac	5m 28s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 26s		tez-api: The patch generated 0 new + 118 unchanged - 3 fixed = 118 total (was 121)
+1 💚	checkstyle	0m 23s		The patch passed checkstyle in tez-common
+1 💚	checkstyle	0m 28s		The patch passed checkstyle in tez-runtime-library
+1 💚	checkstyle	0m 23s		tez-examples: The patch generated 0 new + 10 unchanged - 1 fixed = 10 total (was 11)
-0 ⚠️	checkstyle	0m 33s	/results-checkstyle-tez-dag.txt	tez-dag: The patch generated 2 new + 610 unchanged - 2 fixed = 612 total (was 612)
-0 ⚠️	checkstyle	0m 46s	/results-checkstyle-root.txt	root: The patch generated 2 new + 740 unchanged - 6 fixed = 742 total (was 746)
+1 💚	javadoc	3m 58s		the patch passed
-1 ❌	spotbugs	0m 38s	/patch-spotbugs-tez-api.txt	tez-api in the patch failed.
-1 ❌	spotbugs	0m 31s	/patch-spotbugs-tez-common.txt	tez-common in the patch failed.
-1 ❌	spotbugs	0m 38s	/patch-spotbugs-tez-runtime-library.txt	tez-runtime-library in the patch failed.
-1 ❌	spotbugs	0m 28s	/patch-spotbugs-tez-examples.txt	tez-examples in the patch failed.
-1 ❌	spotbugs	0m 42s	/patch-spotbugs-tez-dag.txt	tez-dag in the patch failed.
-1 ❌	spotbugs	0m 24s	/patch-spotbugs-root.txt	root in the patch failed.
			_ Other Tests _
+1 💚	unit	2m 35s		tez-api in the patch passed.
+1 💚	unit	0m 42s		tez-common in the patch passed.
+1 💚	unit	6m 9s		tez-runtime-library in the patch passed.
+1 💚	unit	0m 30s		tez-examples in the patch passed.
+1 💚	unit	6m 28s		tez-dag in the patch passed.
+1 💚	unit	72m 8s		root in the patch passed.
+1 💚	asflicense	2m 36s		The patch does not generate ASF License warnings.
		147m 21s

Subsystem	Report/Notes
Docker	ClientAPI=1.52 ServerAPI=1.52 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/41/artifact/out/Dockerfile
GITHUB PR	#427
Optional Tests	dupname asflicense javac javadoc unit codespell detsecrets xmllint compile spotbugs checkstyle
uname	Linux 08e0e7068641 5.15.0-161-generic #171-Ubuntu SMP Sat Oct 11 08:17:01 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	/home/jenkins/jenkins-home/workspace/tez-multibranch_PR-427/src/.yetus/personality.sh
git revision	master / `ff289be`
Default Java	Ubuntu-21.0.9+10-Ubuntu-124.04
Test Results	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/41/testReport/
Max. process+thread count	1389 (vs. ulimit of 5500)
modules	C: tez-api tez-common tez-runtime-library tez-examples tez-dag . U: .
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-427/41/console
versions	git=2.43.0 maven=3.8.7 codespell=2.0.0
Powered by	Apache Yetus 0.15.1 https://yetus.apache.org

This message was automatically generated.

This comment was marked as outdated.

Sign in to view

okumin reviewed Sep 11, 2025

View reviewed changes

abstractdog changed the title ~~TEZ-4007: Zookeeper based FrameworkServices and AmExtensions (3/3) - initial patch~~ [DRAFT] TEZ-4007: Zookeeper based FrameworkServices and AmExtensions (3/3) - initial patch Sep 11, 2025

abstractdog force-pushed the TEZ-4007 branch from 4702ed6 to c24caeb Compare September 11, 2025 19:56

This comment was marked as outdated.

Sign in to view

abstractdog force-pushed the TEZ-4007 branch from c24caeb to fbe40c8 Compare October 13, 2025 11:21

This comment was marked as outdated.

Sign in to view

abstractdog force-pushed the TEZ-4007 branch from fbe40c8 to a32a6f5 Compare October 13, 2025 14:53

This comment was marked as outdated.

Sign in to view

abstractdog force-pushed the TEZ-4007 branch 2 times, most recently from d3c969a to b3ae071 Compare October 14, 2025 08:21

This comment was marked as outdated.

Sign in to view

abstractdog changed the title ~~[DRAFT] TEZ-4007: Zookeeper based FrameworkServices and AmExtensions (3/3) - initial patch~~ [DRAFT] TEZ-4007: Zookeeper based FrameworkServices and AmExtensions (3/3) + refactor Oct 15, 2025

abstractdog force-pushed the TEZ-4007 branch from b3ae071 to 15bc7cf Compare October 15, 2025 12:03

This comment was marked as outdated.

Sign in to view

abstractdog force-pushed the TEZ-4007 branch 5 times, most recently from ec30327 to db11fb6 Compare October 17, 2025 12:49

This comment was marked as outdated.

Sign in to view

abstractdog force-pushed the TEZ-4007 branch from db11fb6 to 3564579 Compare October 20, 2025 08:10

This comment was marked as outdated.

Sign in to view

abstractdog force-pushed the TEZ-4007 branch 2 times, most recently from 2e44662 to eec88ab Compare October 22, 2025 13:51

This comment was marked as outdated.

Sign in to view

abstractdog force-pushed the TEZ-4007 branch from eec88ab to 6fb7001 Compare October 31, 2025 10:02

This comment was marked as outdated.

Sign in to view

abstractdog force-pushed the TEZ-4007 branch from f30ec00 to 3938781 Compare December 2, 2025 09:11

ayushtkn reviewed Dec 11, 2025

View reviewed changes

abstractdog added 6 commits January 8, 2026 15:03

TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkService…

2efbf4c

…s - initial patch

TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkService…

daebf5a

…s - checkstyle, spotbugs, javadoc improvements, refactor, test fixes

checksytle, spotbugs

1b57912

PR comments

ca8ac8c

reworked am registry service lifecycle

3893e8f

PR comments 2

bd1161c

abstractdog force-pushed the TEZ-4007 branch from 3938781 to bd1161c Compare January 8, 2026 19:15

This comment was marked as outdated.

Sign in to view

shutdown hook

ad531a8

This comment was marked as outdated.

Sign in to view

okumin approved these changes Jan 12, 2026

View reviewed changes

abstractdog force-pushed the TEZ-4007 branch from cee34c8 to 0f9a678 Compare January 12, 2026 09:47

PR comments 3

ff289be

abstractdog requested a review from ayushtkn January 13, 2026 11:14

ayushtkn approved these changes Jan 14, 2026

View reviewed changes

abstractdog force-pushed the TEZ-4007 branch from 0f9a678 to ff289be Compare January 14, 2026 10:09

		void add(AMRecord server) throws Exception;

		void remove(AMRecord server) throws Exception;


		public class ZkFrameworkClient extends FrameworkClient {

		private AMRecord amRecord;

	@Override public FrameworkClient newFrameworkClient() {
	@Override
	public FrameworkClient newFrameworkClient() {

TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices #427

Are you sure you want to change the base?

TEZ-4007: Introduce AmExtensions and Zookeeper-based FrameworkServices #427

Uh oh!

Conversation

abstractdog commented Sep 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as outdated.

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

tez-yetus commented Dec 2, 2025

Uh oh!

ayushtkn left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

abstractdog Jan 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This comment was marked as outdated.

abstractdog commented Jan 9, 2026

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

abstractdog commented Sep 6, 2025 •

edited

Loading

abstractdog Jan 7, 2026 •

edited

Loading

abstractdog Jan 12, 2026 •

edited

Loading

abstractdog commented Jan 14, 2026 •

edited

Loading