[Refactor](Multi Catalog) External meta cache framework with engine adapters#60937
Open
suxiaogang223 wants to merge 38 commits intoapache:masterfrom
Open
[Refactor](Multi Catalog) External meta cache framework with engine adapters#60937suxiaogang223 wants to merge 38 commits intoapache:masterfrom
suxiaogang223 wants to merge 38 commits intoapache:masterfrom
Conversation
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run external |
Contributor
Author
|
run buildall |
suxiaogang223
force-pushed
the
Refact-ExternalMetaCache
branch
from
d86d58c to
d3bbd5e
Compare
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 28796 ms |
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 27864 ms |
TPC-DS: Total hot run time: 154005 ms |
Contributor
FE Regression Coverage ReportIncrement line coverage |
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 27849 ms |
TPC-DS: Total hot run time: 153786 ms |
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
FE Regression Coverage ReportIncrement line coverage |
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 27550 ms |
TPC-DS: Total hot run time: 152802 ms |
Contributor
FE UT Coverage ReportIncrement line coverage |
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 27606 ms |
TPC-DS: Total hot run time: 152510 ms |
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Part of #60686
Summary
Introduce a unified external metadata cache framework and migrate Iceberg / Paimon / Hudi / MaxCompute / Doris to engine-level cache adapters while keeping Common/Hive on the legacy path for incremental rollout.
New Meta Cache Framework
The new framework standardizes external metadata caching into a 3-level model:
ExternalMetaCacheMgrroutes by engine (hive/iceberg/paimon/hudi/maxcompute/doris).MetaCacheEntryDef(name, key type, value type, loader, default spec).Structure (simplified):
Core components:
ExternalMetaCache: unified engine-level contract (initCatalog, scoped invalidation,stats).AbstractExternalMetaCache: shared implementation for entry registration, per-catalog entry group creation, type-safe entry lookup, lifecycle management.MetaCacheEntryDef: immutable declaration of an entry.MetaCacheEntry: generic cache runtime (load on miss, invalidate by key/predicate/all, per-entry stats).CacheSpec: unified cache policy (enable,ttl-second,capacity) and compatibility key mapping.CatalogEntryGroup: container for all entries in one catalog.Initialization and lifecycle:
ExternalCatalog.makeSureInitialized(),ExternalMetaCacheMgr.prepareCatalog(...)eagerly initializes engine entries for that catalog.engine.entry -> metric map, so each entry can be observed independently.Configuration model:
meta.cache.<engine>.<entry>.enable|ttl-second|capacityMetaCacheEntryDefdefaultCacheSpec.CacheSpec.applyCompatibilityMap(...)supports smooth migration from legacy keys.ExternalMetaCacheMgr
engineCachesOrganizationengineCachesis a concurrent map:Map<String, ExternalMetaCache>.initEngineCaches()pre-registers built-in engines.engine(engineName)normalizes to lowercase and usescomputeIfAbsent(...).routeEngine(engine, action):engine == nullMigration Status (Engine View)
iceberg(table,view,manifest)paimon(table)hudi(partition,fs_view,meta_client)maxcompute(metadata)doris(backends)commonhiveKey Changes
datasource.metacache:ExternalMetaCacheAbstractExternalMetaCacheMetaCacheEntryDefMetaCacheEntryCacheSpecCatalogEntryGroupExternalMetaCacheMgrto route cache lifecycle by engine (prepareCatalog,invalidateCatalog/db/table/partitions,stats).ExternalCatalog.makeSureInitialized().IcebergExternalMetaCacheentries (table,view,manifest) and move call sites to the engine cache path.PaimonExternalMetaCache(table) and route related call sites via engine cache.HudiExternalMetaCacheentries (partition,fs_view,meta_client) and route scan/utils through the new path.MaxComputeExternalMetaCacheand removeMaxComputeMetadataCacheMgr.DorisExternalMetaCacheand removeDorisExternalMetaCacheMgr.CacheSpecfor gradual key migration.ENGINE_COMMONandENGINE_HIVEonLegacyExternalMetaCacheto preserve existing behavior.IcebergExternalMetaCacheTestPaimonExternalMetaCacheTestMetaCacheDeadlockTestCompatibility / Behavior
meta.cache.<engine>.<entry>.enablemeta.cache.<engine>.<entry>.ttl-secondmeta.cache.<engine>.<entry>.capacityCheck List (For Reviewer who merge this PR)