[opt](memory) Replace boxed Long-keyed maps with fastutil primitive collections in FE by dataroaring · Pull Request #61292 · apache/doris

dataroaring · 2026-03-13T01:39:35Z

Summary

Replace HashMap<Long, V>, ConcurrentHashMap<Long, V>, HashSet<Long> and similar boxed collections with fastutil primitive-type-specialized collections (Long2ObjectOpenHashMap, Long2LongOpenHashMap, LongOpenHashSet, etc.) across FE hot paths to reduce memory footprint and eliminate autoboxing overhead.

Key changes:

New classes: ConcurrentLong2ObjectHashMap<V> and ConcurrentLong2LongHashMap — thread-safe wrappers over fastutil maps using segment-based locking, replacing ConcurrentHashMap<Long, V> where concurrent access is needed
Gson TypeAdapters: Added serialization/deserialization support for Long2ObjectOpenHashMap, Long2LongOpenHashMap, LongOpenHashSet, and the concurrent variants. Wire format is backward-compatible with HashMap<Long, V> (string-keyed JSON objects) for rolling upgrade safety
27 files converted across catalog, transaction, statistics, alter, clone, cloud, and load subsystems

Memory savings per collection type:

Pattern	Before (per entry)	After (per entry)	Savings
`HashMap<Long, Long>`	~64 bytes	~16 bytes	4x
`HashMap<Long, Object>`	~48 bytes	~16 bytes	3x
`HashSet<Long>`	~48 bytes	~8 bytes	6x
`ConcurrentHashMap<Long, V>`	~64 bytes	~16 bytes	4x

Scope of changes:

Priority 1 (tablet/replica scale, millions of entries):

MaterializedIndex.idToTablets
DeleteBitmapUpdateLockContext — 4× Long2Long maps + nested maps
TransactionState — deltaRows, loadedTblIndexes, errorReplicas
PublishVersionTask / PublishVersionDaemon fields
ReportHandler.ReportTask fields
DeleteJob tablet sets

Priority 2 (per-partition scale, thousands+):

PartitionInfo — 7 maps keyed by partition_id
OlapTable.idToPartition
TableCommitInfo.idToPartitionCommitInfo
DatabaseTransactionMgr — running/final transaction maps, subTxnIdToTxnId
TableStatsMeta / AnalysisInfo / ColStatsMeta — partition update rows, indexes row count

Priority 3-4 (alter, cloud, load):

SchemaChangeJobV2 / RollupJobV2 tablet maps
CloudGlobalTransactionMgr fields
RoutineLoadManager fields
TabletScheduler fields

Estimated total heap savings for a cluster with 1.3M tablets: 350-700MB.

Test plan

Unit tests for ConcurrentLong2ObjectHashMap (432 lines, covers CRUD, concurrency, iteration)
Unit tests for ConcurrentLong2LongHashMap (455 lines, covers CRUD, concurrency, default values)
Existing FE unit test suite passes
Verify Gson serialization/deserialization backward compatibility (rolling upgrade from old format)
Cluster-level validation with large tablet count

🤖 Generated with Claude Code

…ollections in FE Replace HashMap<Long, V>, ConcurrentHashMap<Long, V>, HashSet<Long> and similar boxed collections with fastutil primitive-type-specialized collections across FE hot paths. This eliminates Long autoboxing overhead and reduces per-entry memory footprint by 3-6x depending on the collection type. Key changes: - Add ConcurrentLong2ObjectHashMap and ConcurrentLong2LongHashMap as thread-safe wrappers over fastutil maps with segment-based locking - Add Gson TypeAdapters for Long2ObjectOpenHashMap, Long2LongOpenHashMap, LongOpenHashSet, and the concurrent variants for serialization compatibility - Replace collections in 27 files across catalog, transaction, statistics, alter, clone, cloud, and load subsystems For a cluster with 1.3M tablets, estimated heap savings: 350-700MB. Co-Authored-By: Claude Opus 4.6 <[email protected]>

Thearas · 2026-03-13T01:39:40Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

morrySnow · 2026-03-13T01:44:02Z

fe/fe-core/src/main/java/org/apache/doris/catalog/MaterializedIndex.java

@@ -72,7 +73,7 @@ public enum IndexExtState {

    public MaterializedIndex() {
        this.state = IndexState.NORMAL;
-        this.idToTablets = new HashMap<>();
+        this.idToTablets = new Long2ObjectOpenHashMap<>();


should update attr defination to Long2ObjectOpenHashMap too?

morrySnow · 2026-03-13T01:45:04Z

fe/fe-core/src/main/java/org/apache/doris/common/ConcurrentLong2LongHashMap.java

put this class into fe-foundation module

Copilot

Pull request overview

This PR introduces primitive-key/primitive-value concurrent maps and expands the use of fastutil primitive collections across FE code to reduce boxing overhead and heap usage, while keeping Gson persistence compatible during rolling upgrades.

Changes:

Added ConcurrentLong2LongHashMap and ConcurrentLong2ObjectHashMap (segmented ReentrantReadWriteLock design) plus comprehensive unit tests.
Replaced a variety of Map<Long, …> / Set<Long> usages with fastutil primitive collections (Long2*OpenHashMap, LongOpenHashSet) and the new concurrent primitive maps in transaction/statistics/catalog paths.
Extended GsonUtils with adapters to serialize/deserialize the new concurrent maps and fastutil primitive collections in a backward-compatible JSON shape.

Reviewed changes

Copilot reviewed 31 out of 31 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
fe/fe-core/src/test/java/org/apache/doris/common/ConcurrentLong2ObjectHashMapTest.java	Adds unit tests for the new concurrent long→object map and its Gson round-trip behavior.
fe/fe-core/src/test/java/org/apache/doris/common/ConcurrentLong2LongHashMapTest.java	Adds unit tests for the new concurrent long→long map, including concurrency and Gson checks.
fe/fe-core/src/main/java/org/apache/doris/transaction/TransactionState.java	Switches several txn structures to fastutil primitive maps/sets to reduce boxing.
fe/fe-core/src/main/java/org/apache/doris/transaction/TableCommitInfo.java	Replaces partition commit info map with `Long2ObjectOpenHashMap`.
fe/fe-core/src/main/java/org/apache/doris/transaction/PublishVersionDaemon.java	Uses fastutil primitive maps for visible-version bookkeeping.
fe/fe-core/src/main/java/org/apache/doris/transaction/DatabaseTransactionMgr.java	Uses `ConcurrentLong2LongHashMap` for subTxn mapping/counts and fastutil sets for replica IDs.
fe/fe-core/src/main/java/org/apache/doris/task/PublishVersionTask.java	Uses `Long2ObjectOpenHashMap` for delta-row tracking.
fe/fe-core/src/main/java/org/apache/doris/statistics/util/StatisticsUtil.java	Updates stats logic to use `ConcurrentLong2LongHashMap` fields.
fe/fe-core/src/main/java/org/apache/doris/statistics/TableStatsMeta.java	Migrates internal counters/maps to `ConcurrentLong2LongHashMap` and adjusts stale cleanup.
fe/fe-core/src/main/java/org/apache/doris/statistics/ColStatsMeta.java	Migrates per-partition update rows map to `ConcurrentLong2LongHashMap`.
fe/fe-core/src/main/java/org/apache/doris/statistics/BaseAnalysisTask.java	Updates local variable types to the new concurrent primitive map.
fe/fe-core/src/main/java/org/apache/doris/statistics/AnalysisManager.java	Initializes `partitionUpdateRows` with `ConcurrentLong2LongHashMap`.
fe/fe-core/src/main/java/org/apache/doris/statistics/AnalysisInfo.java	Migrates persisted `partitionUpdateRows`/`indexesRowCount` to `ConcurrentLong2LongHashMap`.
fe/fe-core/src/main/java/org/apache/doris/persist/gson/GsonUtils.java	Registers Gson adapters for the new concurrent maps and fastutil primitive collections.
fe/fe-core/src/main/java/org/apache/doris/master/ReportHandler.java	Uses `Long2ObjectOpenHashMap` when building tablet ID maps.
fe/fe-core/src/main/java/org/apache/doris/load/routineload/RoutineLoadManager.java	Replaces blacklist and txn→job mapping with `ConcurrentLong2LongHashMap`.
fe/fe-core/src/main/java/org/apache/doris/load/DeleteJob.java	Replaces several `Set<Long>` fields with `LongOpenHashSet`.
fe/fe-core/src/main/java/org/apache/doris/common/ConcurrentLong2ObjectHashMap.java	Adds segmented concurrent primitive-key long→object map implementation.
fe/fe-core/src/main/java/org/apache/doris/common/ConcurrentLong2LongHashMap.java	Adds segmented concurrent primitive-key/value long→long map implementation.
fe/fe-core/src/main/java/org/apache/doris/cloud/transaction/DeleteBitmapUpdateLockContext.java	Migrates multiple internal maps to fastutil primitive maps for lower overhead.
fe/fe-core/src/main/java/org/apache/doris/cloud/transaction/CloudGlobalTransactionMgr.java	Replaces multiple concurrent maps with `ConcurrentLong2LongHashMap` and adjusts cleanup logic.
fe/fe-core/src/main/java/org/apache/doris/cloud/catalog/CloudTabletInvertedIndex.java	Uses `Long2ObjectOpenHashMap` for tablet→replica metadata.
fe/fe-core/src/main/java/org/apache/doris/clone/TabletScheduler.java	Uses `Long2ObjectOpenHashMap` for scheduler tracking maps.
fe/fe-core/src/main/java/org/apache/doris/catalog/TempPartitions.java	Uses `Long2ObjectOpenHashMap` for temp partition ID lookup.
fe/fe-core/src/main/java/org/apache/doris/catalog/TabletInvertedIndex.java	Uses `Long2ObjectOpenHashMap` for tablet metadata storage.
fe/fe-core/src/main/java/org/apache/doris/catalog/PartitionInfo.java	Migrates multiple partition metadata maps to `Long2ObjectOpenHashMap`.
fe/fe-core/src/main/java/org/apache/doris/catalog/Partition.java	Uses `Long2ObjectOpenHashMap` for index maps inside partitions.
fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java	Migrates index meta map to `Long2ObjectOpenHashMap` and updates meta copying logic.
fe/fe-core/src/main/java/org/apache/doris/catalog/MaterializedIndex.java	Migrates tablet ID maps to `Long2ObjectOpenHashMap`.
fe/fe-core/src/main/java/org/apache/doris/alter/SchemaChangeJobV2.java	Migrates multiple job metadata maps to fastutil primitive maps.
fe/fe-core/src/main/java/org/apache/doris/alter/RollupJobV2.java	Migrates rollup tablet mapping structures to fastutil primitive maps.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

fe/fe-core/src/main/java/org/apache/doris/transaction/DatabaseTransactionMgr.java

+        subTxnIdToTxnId.forEach((subTxnId, txnId) -> {
+            if (txnId == transactionId) {
+                subTxnIdToTxnId.remove(subTxnId);
            }
-        }
+        });


fe/fe-core/src/main/java/org/apache/doris/cloud/transaction/CloudGlobalTransactionMgr.java

+        subTxnIdToTxnId.forEach((subTxnId, txnId) -> {
+            if (txnId == transactionId) {
+                subTxnIdToTxnId.remove(subTxnId);
            }
-        }
+        });


fe/fe-core/src/test/java/org/apache/doris/common/ConcurrentLong2LongHashMapTest.java

+        Long boxedResult = map.getOrDefault(999L, map.defaultReturnValue());
+        Assertions.assertEquals(0L, boxedResult);


fe/fe-core/src/main/java/org/apache/doris/common/ConcurrentLong2ObjectHashMap.java

+            V newValue = mappingFunction.apply(key);
+            seg.map.put(key, newValue);
+            return newValue;


fe/fe-core/src/main/java/org/apache/doris/common/ConcurrentLong2ObjectHashMap.java

+            }
+            V newValue = mappingFunction.get(key);
+            seg.map.put(key, newValue);
+            return newValue;


fe/fe-core/src/main/java/org/apache/doris/common/ConcurrentLong2LongHashMap.java

+     * Applies the given action to each entry under read-lock per segment.
+     */
+    public void forEach(LongLongConsumer action) {
+        for (Segment seg : segments) {
+            seg.lock.readLock().lock();
+            try {
+                for (Long2LongMap.Entry entry : seg.map.long2LongEntrySet()) {
+                    action.accept(entry.getLongKey(), entry.getLongValue());
+                }
+            } finally {
+                seg.lock.readLock().unlock();
+            }
+        }


fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java

        table.state = state;
-        table.indexIdToMeta = ImmutableMap.copyOf(indexIdToMeta);
+        table.indexIdToMeta = new Long2ObjectOpenHashMap<>(indexIdToMeta);
        table.indexNameToId = ImmutableMap.copyOf(indexNameToId);


fe/fe-core/src/main/java/org/apache/doris/transaction/TransactionState.java

+import it.unimi.dsi.fastutil.longs.Long2ObjectOpenHashMap;
+import it.unimi.dsi.fastutil.longs.LongOpenHashSet;
+


fe/fe-core/src/main/java/org/apache/doris/common/ConcurrentLong2LongHashMap.java

+import it.unimi.dsi.fastutil.longs.LongBinaryOperator;
+import it.unimi.dsi.fastutil.longs.LongOpenHashSet;
+import it.unimi.dsi.fastutil.longs.LongSet;
+import it.unimi.dsi.fastutil.objects.ObjectArrayList;
+import it.unimi.dsi.fastutil.objects.ObjectOpenHashSet;


dataroaring requested review from gavinchou and w41ter as code owners March 13, 2026 01:39

Copilot AI review requested due to automatic review settings March 13, 2026 01:39

Copilot started reviewing on behalf of dataroaring March 13, 2026 01:40 View session

morrySnow reviewed Mar 13, 2026

View reviewed changes

fe/fe-core/src/main/java/org/apache/doris/common/ConcurrentLong2LongHashMap.java

Copy link

Contributor

morrySnow Mar 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

put this class into fe-foundation module

Copilot AI reviewed Mar 13, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[opt](memory) Replace boxed Long-keyed maps with fastutil primitive collections in FE#61292

[opt](memory) Replace boxed Long-keyed maps with fastutil primitive collections in FE#61292
dataroaring wants to merge 1 commit intomasterfrom
feature/fastutil-memory-optimization

dataroaring commented Mar 13, 2026

Uh oh!

Thearas commented Mar 13, 2026

Uh oh!

morrySnow Mar 13, 2026

Uh oh!

morrySnow Mar 13, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

		Long boxedResult = map.getOrDefault(999L, map.defaultReturnValue());
		Assertions.assertEquals(0L, boxedResult);

		import it.unimi.dsi.fastutil.longs.Long2ObjectOpenHashMap;
		import it.unimi.dsi.fastutil.longs.LongOpenHashSet;

Conversation

dataroaring commented Mar 13, 2026

Summary

Key changes:

Memory savings per collection type:

Scope of changes:

Test plan

Uh oh!

Thearas commented Mar 13, 2026

Uh oh!

morrySnow Mar 13, 2026

Choose a reason for hiding this comment

Uh oh!

morrySnow Mar 13, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants