Conversation
python-dameng
Adds initial scaffolding for Dameng database support in Archery. Includes: - DamengEngine class in `sql/engines/dameng.py` with core methods. - Updates to `sql/engines/__init__.py` for engine registration. - Addition of 'dameng' to `DB_TYPE_CHOICES` in `sql/models.py`. - Basic documentation in `docs/dameng_integration.md`. NOTE: This integration is currently NON-FUNCTIONAL due to: 1. Failure to install the `dmPython` driver (requires manual SDK setup and build). 2. Lack of any testing against a Dameng instance. Further work is required to resolve driver issues and test functionality before this integration can be considered complete.
Modified the `query` method in `sql/engines/dameng.py` to execute a `SET SCHEMA` command before running the user's query. This is an attempt to address issues where tables might not be found due to incorrect schema context when not using fully qualified table names. The exact `SET SCHEMA` syntax for Dameng should be verified. This change is still contingent on a working `dmPython` driver and proper Dameng client SDK setup.
Modified the `execute_workflow` method in `sql/engines/dameng.py` to execute a `SET SCHEMA` command at the beginning of the workflow if a `db_name` is provided. This aims to address issues where SQL statements might fail due to incorrect schema context. The exact `SET SCHEMA` syntax for Dameng should be verified. This change is still contingent on a working `dmPython` driver and proper Dameng client SDK setup. If setting the schema fails, the workflow will be marked as failed.
- Removed Git merge conflict markers from sql/engines/dameng.py. - Corrected the logic for setting schema in the `query` method by removing the problematic `_archery_schema_set` flag. The method will now attempt to `SET SCHEMA` if `db_name` is provided on each call, logging a warning on failure but allowing the query to proceed. This addresses the `SyntaxError` and the `AttributeError` related to `_archery_schema_set` previously reported.
…qw4285632q/Archery into feat/dameng-integration-partial
Ensured that the sql/engines/dameng.py file is free of Git merge conflict markers (<<<<<<<, =======, >>>>>>>) that were causing SyntaxErrors during application startup. This commit specifically addresses the markers found at the end of the kill_connection method.
Modified `execute_workflow` in `sql/engines/dameng.py` to explicitly call `conn.commit()` after all statements in a workflow are executed successfully, and `conn.rollback()` if any statement fails or if there's a setup error after connection. This is critical for database systems like Dameng (if similar to Oracle) where DML operations require an explicit commit when `autoCommit` is false on the connection.
…qw4285632q/Archery into feat/dameng-integration-partial
wangwenfeng@youzhicai.com
This reverts commit b281e3b.
This reverts commit be80c5a.
Implements two new features for the Dameng SQL submission workflow:
1. **SQL Validation:**
- The `execute_check` method for the Dameng engine now uses `EXPLAIN`
to validate DML statements (`INSERT`, `UPDATE`, `DELETE`).
- This checks for both syntax errors and the existence of
tables and columns before the workflow is submitted, preventing
runtime errors.
2. **Data Backup:**
- A new `is_backup` flag in the submission UI allows you to
request a backup of data before it's modified.
- When enabled, the `execute_workflow` method will, for `UPDATE` and
`DELETE` statements, select the affected rows and store them in a
new `SqlBackupHistory` table before execution.
- A new `SqlBackupHistory` model is added to store this data.
I've implemented several new features and fixes for the Dameng SQL workflow:
1. **SQL Validation:**
- The `execute_check` method for the Dameng engine now uses `EXPLAIN FOR`
to validate DML statements (`INSERT`, `UPDATE`, `DELETE`).
- This checks for both syntax errors and the existence of
tables and columns before the workflow is submitted.
2. **Data Backup:**
- I've added a new `SqlBackupHistory` model to store data backups.
- When the `is_backup` flag is enabled for a workflow, the `execute_workflow`
method will, for `UPDATE` and `DELETE` statements, select the
affected rows and store them in the `SqlBackupHistory` table
before execution.
- The backup option in the UI is now correctly displayed for Dameng
and MySQL instances.
3. **Execution Plan in SQL Query:**
- The "Execution Plan" button in the SQL Query view now correctly
generates and executes `EXPLAIN FOR <statement>` for Dameng databases.
This change implements the `get_rollback` method for the Dameng engine. This method generates rollback SQL for `UPDATE` and `DELETE` statements by reading the backup data from the `SqlBackupHistory` table. - For `DELETE` statements, it generates `INSERT` statements. - For `UPDATE` statements, it generates `UPDATE` statements to restore the previous values, using the primary key of the table. - A helper method `_get_primary_key` is added to retrieve the primary key of a table from the database metadata.
This change improves the error handling in the `execute_workflow` method for the Dameng engine. - If any error occurs during the data backup process (e.g., SQL parsing, SELECT query execution), the entire workflow will now fail. - The specific error message from the backup process will be propagated to the workflow's execution result, making it visible to you. This prevents silent failures where backup does not run but the workflow continues.
This commit addresses several issues related to enabling and using the Dameng (dm) database engine. 1. **Enable Dameng Engine**: The initial error was `ValueError: engine dm not enabled or not supported`. This was fixed by explicitly adding a handler for the 'dm' db_type in `sql/engines/__init__.py`, ensuring the `DamengEngine` is loaded. This also included cleaning up duplicate function definitions and fixing a typo in an error message. 2. **Fix Transaction Handling**: After enabling the engine, a new error `AttributeError: 'dmPython.Connection' object has no attribute 'autocommit'` occurred. The `dmPython` driver does not support setting the `autocommit` property on the connection object. The fix was to remove the lines that set `conn.autocommit` in the `execute_workflow` method in `sql/engines/dameng.py` and rely on the existing `commit()` and `rollback()` calls. 3. **Fix Schema Switching in `execute_check`**: The final error was `dmPython.DatabaseError: 无效的表或视图名` (Invalid table or view name). This was happening because the `execute_check` method, which validates SQL, was not switching to the correct database schema before running `EXPLAIN`. The fix was to add the same schema-switching logic that exists in the `query` and `execute_workflow` methods to `execute_check`.
|
达梦和mysql 和oracle 都可以兼容，你新做的模块新增了什么功能吗？ |
archery什么时候对达梦数据库进行了支持。我没看到官方文档有此功能介绍呀 。当初提issue的时候 得到的回复是得自己开发呀 |
|
不好意思, 我可能回复的太武断了, 我认知里, 达梦是支持使用 mysqlclient 进行连接的, 我也不太清楚这个认知是否正确, 我这个认知是否有一定的误区? 如果我说错了欢迎指正. 如果达梦支持使用 mysqlclient 进行连接, 那么直接使用 mysql engine 即可. 如果达梦仅支持达梦自己的sdk 进行连接, 那么非常欢迎你的 PR. 另外PR 当前有一个问题, 就是将 另一个功能 另外, 我非常不推荐关闭当前的 pr 并重开完全相同的 pr, 我推荐你继续在此分支上进行推送, 如果新开 pr, 由于讨论不连贯, 会导致审阅变得不方便, 希望你能配合. |
达梦数据库模式级别支持设置参数兼容oracle或者mysql级别。只能选择兼容一种，而且只是在语句语法层面。但是连接协议层面并非像tidb那样直接使用mysql协议连接。导致审核平台无法兼容达梦数据库 |
|
你当前的备份逻辑还是有一些值得探讨的地方的, 我还是比较建议你将其抽离出来单独提交, 比如你当前的设计, 备份的回滚语句是用 json 方式存储在数据库中的, 那么这个数据是否会太大? 是否会有效率问题? 这都需要再探讨一下, 如果我们两个功能一起来讨论, 势必会增大复杂性, 并且拖延你达梦数据库适配的进度. |
其实当前的备份逻辑也仅仅影响到达梦数据库，并非侵入到其他数据类型的代码。而且关于备份原理相当于一对一一行一行存储。并非将所有的记录备份到同一行的一个json中。当然也不得不承认 如果删除的数据过多。这块是否回台会hang。这块后期会增加一定限制。对于影响行数过多的数据。 |
There was a problem hiding this comment.
考虑到insert语句如果不使用达梦的logmgr功能。是无法捕获到cdc中的受影响记录的具体内容。如果仅仅根据查询很难获取到稳定的记录，因此暂时关闭insert操作的备份功能。同时对于update和delete的复杂性语句如join in exist等子查询联表删除更新的sql进行了兼容
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #3027 +/- ##
==========================================
- Coverage 78.51% 76.64% -1.88%
==========================================
Files 125 126 +1
Lines 17822 18324 +502
==========================================
+ Hits 13993 14044 +51
- Misses 3829 4280 +451 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
2 participants
通过Jules 自动编写代码，进行线下调试和生产环境测试。目前已实现达梦数据库的sql上线、查询、备份、sql检测、sql回滚和自动备份功能。sql脚本新增dm.sql 进行备份表的创建。后续将增加tidb 、sqlserver的备份功能。全部采用新的备份逻辑即更新和删除前，将原数据按行解析到sql_backup_history表中。避免因各数据的不同导致的备份逻辑区别