Add new extension yagp hooks collector by leborchuk · Pull Request #1629 · apache/cloudberry

leborchuk · 2026-03-18T15:26:01Z

A diagnostic and monitoring extension for Cloudberry clusters

Briefly, the interaction scheme is:

Cloudberry (Run Executor hooks) -> hooks collector extension (Create protobuf messages and send them using UDS) -> local yagpcc agent

This is the part that sends data from the PG instance to an external agent.

External agent receives telemetry, store, aggregate and expose to the consumers. The agent source code is https://github.com/open-gpdb/yagpcc We're going to propose add it to Apache Cloudberry infrastructure later, after fixing all the tests.

This is quite similar to the #1085 idea. However, it has a different implementation. It is completely separate from database application and written in Go. Could be separately maintained. No influence to the main database, and so different development requirements.

Extension has pg_regress tests, they were copied from pg_stat_statements (PG18) and adjusted to the extension behaviour.

Original extension authors are Smyatkin-Maxim and NJrslv I left their commits as-is to have blame history commited source.

Just create the overal project structure with a Makefile to generate protobufs, compile it into a shared library extension and install it

as a fundation to build on

- borrow GlowByte code to generate plan text and SessionInfo - borrow code from our in-house pg_stat_statements to generate query id and plan id - refactor code to follow common name conventions and identations

- do some minor refactoring to follow common naming convention - add additional message right after ExecutorStart hook

1) Query instrumentation 2) /proc/self/* stats

It allows finer granularity than executor hooks. Also removed some code duplication and data duplicaton

1. Initialize query instrumentation to NULL so that it can be properly checked later (temporary solution, need to find a proper fix) 2. Don't collect spillinfo on query end. Reason: a) it will always be zero and b) it could crash if we failed to enlarge a spillfile. Seems like we need some cummulative statistics for spillinfo. Need to check what explain analyze use.

1. Sync with protobuf changes to collect segment info 2. Remove noisy logging 3. Fix some missing node types in pg_stat_statements

Reason: when query info hook is called with status 'DONE' planstate is already deallocated by ExecutorEnd

1) Give higher gRPC timeouts to query dispatcher as losing messages there is more critical 2) If we've failed to send a message via gRPC we notify a background thread about it and refuse sending any new message until this thread re-establishes the lost connection

Don't collect system queries with empty query text and ccnt == 0

Rethrowing them might break other extensions and even query execution pipeline itself

Change makefile, test and add it to CI of yagp_hooks_collector. Add option --with-yagp-hooks-collector. Similarly to [1]. [1] open-gpdb/gpdb@7be8893

Correct CI for yagp_hooks_collector to use correct env script.

Correct defines for token ids copied from gram.y.

Full copy of [1] for yagp_hooks_collector. [1] open-gpdb/gpdb@845278f#diff-fa2654417413bbb37d47ecf1644dc5af90c76c77f2a90e05c27107967b5f6fd8

Similarly to [1] add missing executor query info hooks. [1] open-gpdb/gpdb@87fc05d

Copy of [1] with additinal changed needed for Clouberry are described below: The testing C functions have changed to set-returning ones if comparing with [1] because we need a control over the place where function is executed - either on master or segments, and in Cloudberry these functions must return set of values so they were changed to return SETOF. [1] open-gpdb/gpdb@989ca06

Copy of [1] - send() may return -1 in case of an error, do not add -1 to total_bytes sent. [1] open-gpdb/gpdb@e1f6c08

Similarly to [1]. [1] open-gpdb/gpdb@bdfabca#diff-99e92ab48c7310c149ea2bd3414a20a1edf31f26823253906aae1981e088775aR39-R79

The extension generates normalized query text and plan using jumbling functions. Those functions may fail when translating to wide character if the current locale cannot handle the character set. Fix changes functions that generate normalized query text/plan to noexcept versions so we can check if error occured and continute execution. The test checks that even when those functions fail, the plan is still executed. This test is partially taken from src/test/regress/gp_locale.sql.

Cloudberry builds treat compiler warnings as errors. For consistency, this behavior has been enabled in yagp_hooks_collector. This commit also fixes the warnings in yagp_hooks_collector.

We faced an issue - segments fail with backtrace ``` #7 0x00007f9b2adbf2e0 in set_qi_error_message (req=0x55f24a6011f0) at src/ProtoUtils.cpp:124 #8 0x00007f9b2adc30d9 in EventSender::collect_query_done (this=0x55f24a5489f0, query_desc=0x55f24a71ca68, status=METRICS_QUERY_ERROR) at src/EventSender.cpp:222 #9 0x00007f9b2adc23e1 in EventSender::query_metrics_collect (this=0x55f24a5489f0, status=METRICS_QUERY_ERROR, arg=0x55f24a71ca68) at src/EventSender.cpp:53 ``` the root cause here is we're trying to send info about error message in a hooks collector. For some queries ErrorData struckture could be NULL despite the fact that an error has occurred. it depends on error type and location of the error. So we should check if we had info about error details before using it.

leborchuk · 2026-03-24T08:27:01Z

Right now we are not fully support all the features from #1085 but I believe one day solution will be won't be any worse.

tuhaihe

Thanks for your great work! Just left some comments, FYI.

tuhaihe · 2026-03-24T08:58:29Z

pom.xml

+            <exclude>gpcontrib/yagp_hooks_collector/protos/yagpcc_metrics.proto</exclude>
+            <exclude>gpcontrib/yagp_hooks_collector/.clang-format</exclude>
+            <exclude>gpcontrib/yagp_hooks_collector/Makefile</exclude>



can move these new lines to the Cloudberry part, like:

cloudberry/pom.xml

Lines 1263 to 1272 in 6c37f8f



<exclude>src/include/task/bitstring.h</exclude>

<exclude>src/include/task/cron.h</exclude>

<exclude>src/include/task/pg_cron.h</exclude>

<exclude>src/include/task/task_states.h</exclude>

<exclude>src/include/task/job_metadata.h</exclude>

<!-- The following files are used by PAX as the

tuhaihe · 2026-03-24T08:59:50Z

gpcontrib/yagp_hooks_collector/src/stat_statements_parser/README.md

@@ -0,0 +1 @@
+This directory contains a slightly modified subset of pg_stat_statements for PG v9.4 to be used in query and plan ID generation.


Can add the standard ASF license header to the markdown file, like:

cloudberry/CODE_OF_CONDUCT.md

Lines 1 to 18 in 6c37f8f

tuhaihe · 2026-03-24T09:02:39Z

.github/workflows/build-cloudberry.yml

+              {"test":"gpcontrib-yagp-hooks-collector",
+               "make_configs":["gpcontrib/yagp_hooks_collector:installcheck"],
+               "extension":"yagp_hooks_collector"
+              },


Is there a need to add the new test to build-cloudberry-rocky8.yml or *.*-deb.yml file? FYI.

Done - I don't see a reason to not include it.

tuhaihe · 2026-03-24T09:09:29Z

gpcontrib/yagp_hooks_collector/protos/yagpcc_metrics.proto

+
+package yagpcc;
+option java_outer_classname = "SegmentYAGPCCM";
+option go_package = "a.yandex-team.ru/cloud/mdb/yagpcc/api/proto/common;greenplum";


Line 5: Is it private？ If so, do we need to make it public?

I don't think we need them, so I just got rid of those options.

tuhaihe · 2026-03-24T09:12:12Z

gpcontrib/yagp_hooks_collector/src/stat_statements_parser/README.md

@@ -0,0 +1 @@
+This directory contains a slightly modified subset of pg_stat_statements for PG v9.4 to be used in query and plan ID generation.


Can add an ASF license header. FYI.

tuhaihe · 2026-03-24T09:19:27Z

gpcontrib/yagp_hooks_collector/README.md

+-   **GUCs:** `yagpcc.enable`, `yagpcc.min_analyze_time`, `yagpcc.enable_cdbstats`(ANALYZE), `yagpcc.enable_analyze`(BUFFERS, TIMING, VERBOSE).
+
+#### 4. Other Metrics
+-   **What:** Captures Instrument, Greenplum, System, Network, Interconnect, Spill metrics.


Suggested change

- **What:** Captures Instrument, Greenplum, System, Network, Interconnect, Spill metrics.

- **What:** Captures Instrument, Apache Cloudberry (Greenplum), System, Network, Interconnect, Spill metrics.

Just got rid of it - done

tuhaihe · 2026-03-24T09:21:06Z

gpcontrib/yagp_hooks_collector/yagp_hooks_collector.control

@@ -0,0 +1,5 @@
+# yagp_hooks_collector extension
+comment = 'Intercept query and plan execution hooks and report them to Yandex GPCC agents'


Suggested change

comment = 'Intercept query and plan execution hooks and report them to Yandex GPCC agents'

comment = 'Intercept query and plan execution hooks and report them to Cloudberry monitor agents'

?

tuhaihe · 2026-03-24T09:23:54Z

configure.ac

+# yagp_hooks_collector
+#
+PGAC_ARG_BOOL(with, yagp_hooks_collector, no,
+              [build with YAGP hooks collector extension])


Suggested change

[build with YAGP hooks collector extension])

[build with hooks collector extension])

?

tuhaihe · 2026-03-24T09:24:29Z

Makefile

 # to build Postgres with a different make, we have this make file
 # that, as a service, will look for a GNU make and invoke it, or show
 # an error message if none could be found.
-


If no actual changes to Makefile, we can restore it.

tuhaihe · 2026-03-24T09:26:06Z

configure.ac

 AC_SUBST(with_zstd)

+#
+# yagp_hooks_collector


Does this new extension need other dependencies? Not sure about this. If so, we can add the pre-required dependencies here. FYI.

We need the protoc, included it.

NJrslv · 2026-03-24T09:43:43Z

src/backend/tcop/pquery.c

 		increment_command_count();

+	/* null this field until set by YAGP Hooks collector */
+	qd->yagp_query_key = NULL;


Rename yagp_query_key, in similar places too.

NJrslv · 2026-03-24T09:53:07Z

Hi all!

I want to extend the description of this project a bit.

Here we collect state of the query during the executor stages and send it via unix sockets. It was originally made for Greenplum, and I adapted it for Cloudberry here.

For testing purposes only, we support writing data to the catalog table so it can be selected during regression tests. The tuples are logged even during transaction aborts, using frozen tuples (only for the tests).

The state of the query is created before the executor_start stage and released after the executor_end stage. If the executor_end is missing for some unknown reason, the state is not freed. To fix this problem, new changes to the code outside of the project were introduced to correctly handle queries with a missing executor_end stage.

* Rename internal names * Add gp-stats-collector test for rocky8 and deb

Smyatkin-Maxim added 30 commits March 18, 2026 13:43

Initial commit: completely useless yet

cff1cf7

Just create the overal project structure with a Makefile to generate protobufs, compile it into a shared library extension and install it

Add .gitignore

00d0798

Simple Executor{Start,Finish} logging

a7c6387

as a fundation to build on

Add GRPC into Makefile and some prototype senders

8b302a2

Make ExecuteStart and ExecuteFinish work (partially)

2d9eaea

Implement QueryInfo generation

e5a2fd5

- borrow GlowByte code to generate plan text and SessionInfo - borrow code from our in-house pg_stat_statements to generate query id and plan id - refactor code to follow common name conventions and identations

Better protobuf filling code

69dcbfd

Fix segfault in plan text generator

11dabf1

Add support of spill info and

a7a6db8

- do some minor refactoring to follow common naming convention - add additional message right after ExecutorStart hook

Sync with proto changes + generate normalized query and plan texts

92bc2fb

Implement the rest of statistics

2157ed9

1) Query instrumentation 2) /proc/self/* stats

Use llvm code style

933fe68

Trace events using query_info_collect_hook

22a602f

It allows finer granularity than executor hooks. Also removed some code duplication and data duplicaton

Don't call hooks from UTILITY mode and increase grpc timeout

e49a598

More improvements to metrics collector

06b6e0b

1. Sync with protobuf changes to collect segment info 2. Remove noisy logging 3. Fix some missing node types in pg_stat_statements

Add debian package configuration

f4a5878

Try older grpcpp headers for bionic builds

b5446ab

Add missing greenplum nodes to pg_stat_statements

10c3c86

Move query stats reporting to ExecutorEnd hook

c866e5b

Reason: when query info hook is called with status 'DONE' planstate is already deallocated by ExecutorEnd

Track CDB metrics and track query nesting

95d4a39

Change package name

5609141

Skip nested and utility statements

02b53c5

Slightly rework query skipping

d640cfc

Don't collect system queries with empty query text and ccnt == 0

Get resource groups back into query info

2384337

Add some configuration parameters

8d2d5ee

Capture query execution system stats instead of whole process lifetime

f1acf98

Don't rethrow errors from metrics collector

c94dd1d

Rethrowing them might break other extensions and even query execution pipeline itself

Add forgotten hooks deinitialization

61b9273

NJrslv and others added 13 commits March 18, 2026 14:04

[yagp_hooks_collector] Add CI test and with option

577ad15

Change makefile, test and add it to CI of yagp_hooks_collector. Add option --with-yagp-hooks-collector. Similarly to [1]. [1] open-gpdb/gpdb@7be8893

[yagp_hooks_collector] Change greenplum_path.sh to cloudberry-env.sh

5b9fd6c

Correct CI for yagp_hooks_collector to use correct env script.

[yagp_hooks_collector] Add comments for func args

6c2f8bd

[yagp_hooks_collector] Correct tokens from gram.y

211b33e

Correct defines for token ids copied from gram.y.

[yagp_hooks_collector] Add consistent filtering

19a5811

Full copy of [1] for yagp_hooks_collector. [1] open-gpdb/gpdb@845278f#diff-fa2654417413bbb37d47ecf1644dc5af90c76c77f2a90e05c27107967b5f6fd8

[yagp_hooks_collector] Add submit & done hooks

7605825

Similarly to [1] add missing executor query info hooks. [1] open-gpdb/gpdb@87fc05d

[yagp_hooks_collector] Correct add of bytes sent

fb8688d

Copy of [1] - send() may return -1 in case of an error, do not add -1 to total_bytes sent. [1] open-gpdb/gpdb@e1f6c08

[yagp_hooks_collector] Refactor

1befa6d

Similarly to [1]. [1] open-gpdb/gpdb@bdfabca#diff-99e92ab48c7310c149ea2bd3414a20a1edf31f26823253906aae1981e088775aR39-R79

[yagp_hooks_collector] Fix warnings and error them

8fe91a1

Cloudberry builds treat compiler warnings as errors. For consistency, this behavior has been enabled in yagp_hooks_collector. This commit also fixes the warnings in yagp_hooks_collector.

[yagp_hooks_collector] Add licence

2184c9f

leborchuk force-pushed the mergeYagpHooks branch from 4e8ec12 to fb6d611 Compare March 20, 2026 12:43

Move hooks collector files to the gpcontrib

c446926

leborchuk force-pushed the mergeYagpHooks branch from fb6d611 to c446926 Compare March 20, 2026 13:31

leborchuk added 2 commits March 20, 2026 16:31

Merge branch 'main' into mergeYagpHooks

392cde8

Merge branch 'main' into mergeYagpHooks

4e3f37e

leborchuk marked this pull request as ready for review March 24, 2026 08:23

leborchuk requested review from Smyatkin-Maxim, fanfuxiaoran, my-ship-it and tuhaihe March 24, 2026 08:23

tuhaihe reviewed Mar 24, 2026

View reviewed changes

NJrslv reviewed Mar 24, 2026

View reviewed changes

NJrslv and others added 2 commits March 25, 2026 18:37

[gp_stats_collector] Rename yagp_hooks_collector to gp_stats_collector

6e5fe07

* Rename internal names * Add gp-stats-collector test for rocky8 and deb

[gp_stats_collector] Restore erroneously deleted line

f603afb

NJrslv force-pushed the mergeYagpHooks branch from 39c6fe8 to f603afb Compare March 25, 2026 16:12

	<!-- The following files are from pg_cron under PostgreSQL
	license, introduced by Cloudberry.
	-->
	<exclude>src/include/task/bitstring.h</exclude>
	<exclude>src/include/task/cron.h</exclude>
	<exclude>src/include/task/pg_cron.h</exclude>
	<exclude>src/include/task/task_states.h</exclude>
	<exclude>src/include/task/job_metadata.h</exclude>

	<!-- The following files are used by PAX as the

		@@ -0,0 +1 @@
		This directory contains a slightly modified subset of pg_stat_statements for PG v9.4 to be used in query and plan ID generation.

	<!--
	Licensed to the Apache Software Foundation (ASF) under one
	or more contributor license agreements. See the NOTICE file
	distributed with this work for additional information
	regarding copyright ownership. The ASF licenses this file
	to you under the Apache License, Version 2.0 (the
	"License"); you may not use this file except in compliance
	with the License. You may obtain a copy of the License at

	http://www.apache.org/licenses/LICENSE-2.0

	Unless required by applicable law or agreed to in writing,
	software distributed under the License is distributed on an
	"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
	KIND, either express or implied. See the License for the
	specific language governing permissions and limitations
	under the License.
	-->

	- What: Captures Instrument, Greenplum, System, Network, Interconnect, Spill metrics.
	- What: Captures Instrument, Apache Cloudberry (Greenplum), System, Network, Interconnect, Spill metrics.

		@@ -0,0 +1,5 @@
		# yagp_hooks_collector extension
		comment = 'Intercept query and plan execution hooks and report them to Yandex GPCC agents'

	[build with YAGP hooks collector extension])
	[build with hooks collector extension])

Conversation

leborchuk commented Mar 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

leborchuk commented Mar 24, 2026

Uh oh!

tuhaihe left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

NJrslv Mar 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

NJrslv Mar 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

NJrslv commented Mar 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

leborchuk commented Mar 18, 2026 •

edited

Loading

tuhaihe left a comment •

edited

Loading

NJrslv Mar 25, 2026 •

edited

Loading

NJrslv Mar 24, 2026 •

edited

Loading

NJrslv commented Mar 24, 2026 •

edited

Loading