Skip to content

Commit a61a4a8

Browse files
fix: do not delete existing object on upload (#5778)
This increases performance for concurrent writes for azure and gcs backends as they had the check inside lock. For all backends this reduces the number of roundtrips to backend needed. For the blob backends the writes atomically overwrites any existing blob. So this should be faster and safer, there is otherwise a short window of time where no blob exists which could lead to unexpected results for reads. This PR have have been tested with #5779, see it for example and timings. --- TYPE: IMPROVEMENT DESC: Reduce latency of object storage uploads
1 parent a21dc5b commit a61a4a8

3 files changed

Lines changed: 0 additions & 14 deletions

File tree

tiledb/sm/filesystem/azure.cc

Lines changed: 0 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1115,11 +1115,6 @@ void Azure::write_blocks(
11151115

11161116
auto state_iter = block_list_upload_states_.find(uri.to_string());
11171117
if (state_iter == block_list_upload_states_.end()) {
1118-
// Delete file if it exists (overwrite).
1119-
if (is_file(uri)) {
1120-
remove_file(uri);
1121-
}
1122-
11231118
// Instantiate the new state.
11241119
BlockListUploadState state;
11251120

tiledb/sm/filesystem/gcs.cc

Lines changed: 0 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1034,11 +1034,6 @@ Status GCS::write_parts(
10341034

10351035
// Downgrade to read lock, expected below outside the create
10361036
unique_wl.unlock();
1037-
1038-
// Delete file if it exists (overwrite).
1039-
if (is_file(uri)) {
1040-
remove_file(uri);
1041-
}
10421037
} else {
10431038
// If another thread switched state, switch back to a read lock
10441039
state = &multi_part_upload_states_.at(uri.to_string());

tiledb/sm/filesystem/s3.cc

Lines changed: 0 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1227,10 +1227,6 @@ void S3::global_order_write(
12271227
state_ptr = &state_iter->second;
12281228

12291229
unique_wl.unlock();
1230-
1231-
if (is_file(uri)) {
1232-
remove_file(uri);
1233-
}
12341230
} else {
12351231
state_ptr = &state_iter->second;
12361232
// Unlock, as make_upload_part_req will reaquire as necessary.

0 commit comments

Comments
 (0)