Skip to content

HIVE-29377: Fix inconsistency for partition filter LIKE clause #6240

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wecharyu wants to merge 1 commit into
base: master
Choose a base branch
from
+97 −5
Open

HIVE-29377: Fix inconsistency for partition filter LIKE clause #6240

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 5 additions & 2 deletions ql/src/test/queries/clientpositive/ppd_like_filter.q
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,5 +28,8 @@ select * from test_tbl where b like '___';
select * from test_tbl where b like '%%%';
select * from test_tbl where b like '%\\\\%';



select * from test_tbl where b like 'a.*';
select * from test_tbl where b like 'd.\%.*';
select * from test_tbl where b like 'd.\\%.*';
select * from test_tbl where b like 'd.\\\\%.*';
select * from test_tbl where b like '.\\_.*';
48 changes: 48 additions & 0 deletions ql/src/test/results/clientpositive/llap/ppd_like_filter.q.out
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -256,3 +256,51 @@ POSTHOOK: type: QUERY
POSTHOOK: Input: default@test_tbl
POSTHOOK: Input: default@test_tbl@b=d_%5C%25ae
#### A masked pattern was here ####
PREHOOK: query: select * from test_tbl where b like 'a.*'
Copy link
Contributor

@zhangbutao zhangbutao Dec 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The conversion rule for this PR is: if the character .* is encountered in the partition predicate of Hive SQL, it is mapped to % in MySQL (matching zero or more characters), then the partition predicate a.* should be converted to a% in MySQL, and the direct SQL would query two partition entries, namely abc and af%.

However, why is the query result from q.out empty here?

Copy link
Contributor Author

@wecharyu wecharyu Dec 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The POSTHOOK Input shows that it got matching partitions from HMS, but hive optimizer PartitionConditionRemover will filter partitions again by original pattern like a.*, and does not convert to a% so it will filter out the matching partitions.

POSTHOOK: Input: default@test_tbl
POSTHOOK: Input: default@test_tbl@b=abc
POSTHOOK: Input: default@test_tbl@b=af%25

Copy link
Contributor

@zhangbutao zhangbutao Dec 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So there is an issue here. The HMS can return partition information based on like .*, but the PartitionConditionRemover considers these partitions invalid. This seems inconsistent—we need to maintain consistency. cc @dengzhhu653 @deniskuzZ

Additionally, I am not sure if .* is a commonly used expression in SQL LIKE filtering. Could you give examples of similar LIKE matching usages in other sql engines?

Copy link
Contributor Author

@wecharyu wecharyu Dec 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SQL engines usually not use .* as a filter, but HMS thrift APIs support it as a pattern, I think it's because HMS uses JDO matches() for like filter.

And Spark is using the .* pattern for partition filter like here:
https://github.com/apache/spark/blob/8cc2e4677d596bada67d05792f6fd896e5fc640f/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala#L827-L834

Copy link
Member

@dengzhhu653 dengzhhu653 Dec 30, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

per my understanding, we don't have a single pattern for both jdo and the direct sql, jdo accepts .* (Java-type pattern) for any sequences and the sql allows _% for the same game. When comes to HMS API, the user needs to take care of the pattern, whether it should be a Java-type pattern or sql pattern.

Not sure why PartitionConditionRemover would remove the partition condition, given the pruned partitions from Metastore is not empty.

Copy link
Member

@dengzhhu653 dengzhhu653 Feb 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The jdo like is much more complex than the sql type. In the direct sql we assume the like is a Jdo pattern, and the sql pattern in the jdo query for the same API, ObjectStore#getPartitionsByFilter for example, this could make things worse, if we disable the direct sql, it could give us a different answer for the same input.

do you think we should not support java pattern filter?

We use the Java filter somewhere, the tablePattern in getTableObjectsByName for example. If an ObjectStore API has implemented the direct sql or going to be, then for this API, I would suggest the sql type pattern for the filter.

Copy link
Contributor Author

@wecharyu wecharyu Feb 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if we disable the direct sql, it could give us a different answer for the same input.

The result could be different before this patch just like the updated ppd_like_filter.q test shows, we are fixing it here. Pls correct me if I miss something.

If an ObjectStore API has implemented the direct sql or going to be, then for this API, I would suggest the sql type pattern for the filter.

HMS users may not know the API details, it's a bit difficult for them to know which api should use sql filter and which one should use java pattern. In another word, such difference seems not friendly to users.

I think we have 2 choices to keep consistency:

  1. Declare in the user document that filter is SQL only, and escape the java pattern in JDO implement of each API.
  2. Both direct implement and JDO implement supports java pattern and sql filter, which is what we do here.

Copy link
Member

@dengzhhu653 dengzhhu653 Feb 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The result could be different before this patch just like the updated ppd_like_filter.q test shows, we are fixing it here. Pls correct me if I miss something.

I don't get it here, why would it cause different result without this PR? in case both the jdo and direct sql accept the SQL type pattern.

Both direct implement and JDO implement supports java pattern and sql filter, which is what we do here.

The reason is the java pattern is much more complex, imaging a java regex for IP: ^(\d{1,3})\.(\d{1,3})\.(\d{1,3})\.(\d{1,3})$, changing it to a sql filter looks quite hard and not easy to test.

Declare in the user document that filter is SQL only, and escape the java pattern in JDO implement of each API.

We can add some annotations in IMetastoreClient to tell which pattern type is accepted

Copy link
Member

@dengzhhu653 dengzhhu653 Feb 5, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The result could be different before this patch just like the updated ppd_like_filter.q test shows, we are fixing it here. Pls correct me if I miss something.

I somehow get the point, if the input is a java pattern which the user thinks this API should accept, then the direct sql might return a different result compared to the jdo, however this also might be a problem if another user insists that his input is a sql type pattern.

Copy link
Contributor Author

@wecharyu wecharyu Feb 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if the input is a java pattern which the user thinks this API should accept, then the direct sql might return a different result compared to the jdo

Yep, that's the motivation of this patch.

I think user should know it supports both java pattern and sql filter for now, otherwise it's also a problem if a user think he is input a java pattern filter.

PREHOOK: type: QUERY
PREHOOK: Input: default@test_tbl
PREHOOK: Input: default@test_tbl@b=abc
PREHOOK: Input: default@test_tbl@b=af%25
#### A masked pattern was here ####
POSTHOOK: query: select * from test_tbl where b like 'a.*'
POSTHOOK: type: QUERY
POSTHOOK: Input: default@test_tbl
POSTHOOK: Input: default@test_tbl@b=abc
POSTHOOK: Input: default@test_tbl@b=af%25
#### A masked pattern was here ####
PREHOOK: query: select * from test_tbl where b like 'd.\%.*'
PREHOOK: type: QUERY
PREHOOK: Input: default@test_tbl
#### A masked pattern was here ####
POSTHOOK: query: select * from test_tbl where b like 'd.\%.*'
POSTHOOK: type: QUERY
POSTHOOK: Input: default@test_tbl
#### A masked pattern was here ####
PREHOOK: query: select * from test_tbl where b like 'd.\\%.*'
PREHOOK: type: QUERY
PREHOOK: Input: default@test_tbl
#### A masked pattern was here ####
POSTHOOK: query: select * from test_tbl where b like 'd.\\%.*'
POSTHOOK: type: QUERY
POSTHOOK: Input: default@test_tbl
#### A masked pattern was here ####
PREHOOK: query: select * from test_tbl where b like 'd.\\\\%.*'
PREHOOK: type: QUERY
PREHOOK: Input: default@test_tbl
PREHOOK: Input: default@test_tbl@b=d_%5C%25ae
#### A masked pattern was here ####
POSTHOOK: query: select * from test_tbl where b like 'd.\\\\%.*'
POSTHOOK: type: QUERY
POSTHOOK: Input: default@test_tbl
POSTHOOK: Input: default@test_tbl@b=d_%5C%25ae
#### A masked pattern was here ####
PREHOOK: query: select * from test_tbl where b like '.\\_.*'
PREHOOK: type: QUERY
PREHOOK: Input: default@test_tbl
PREHOOK: Input: default@test_tbl@b=d_%5C%25ae
#### A masked pattern was here ####
POSTHOOK: query: select * from test_tbl where b like '.\\_.*'
POSTHOOK: type: QUERY
POSTHOOK: Input: default@test_tbl
POSTHOOK: Input: default@test_tbl@b=d_%5C%25ae
#### A masked pattern was here ####
9 changes: 9 additions & 0 deletions ...tore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/DatabaseProduct.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -856,6 +856,15 @@ public int getMaxRows(int batch, int paramSize) {
return batch;
}

public String escapeClause() {
if (isMYSQL()) {
// MySQL does not support ESCAPE '\'
return " ESCAPE '\\\\' ";
} else {
return " ESCAPE '\\' ";
}
}

// This class implements the Configurable interface for the benefit
// of "plugin" instances created via reflection (see invocation of
// ReflectionUtils.newInstance in method determineDatabaseProduct)
Expand Down
38 changes: 35 additions & 3 deletions ...e/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1528,7 +1528,11 @@ public void visit(LeafNode node) throws MetaException {
String tableValue = "\"FILTER" + partColIndex + "\".\"PART_KEY_VAL\"";

if (node.isReverseOrder && nodeValue != null) {
params.add(nodeValue);
if (node.operator == Operator.LIKE) {
params.add(makeDirectFilterForLike(nodeValue.toString()));
} else {
params.add(nodeValue);
}
}
String tableColumn = tableValue;
if ((colType != FilterType.String) && (!isDefaultPartition)) {
Expand Down Expand Up @@ -1560,12 +1564,16 @@ public void visit(LeafNode node) throws MetaException {
}

if (!node.isReverseOrder && nodeValue != null) {
params.add(nodeValue);
if (node.operator == Operator.LIKE) {
params.add(makeDirectFilterForLike(nodeValue.toString()));
} else {
params.add(nodeValue);
}
}

// The following syntax is required for using LIKE clause wildcards '_' and '%' as literals.
if (node.operator == Operator.LIKE) {
nodeValue0 = nodeValue0 + " ESCAPE '\\' ";
nodeValue0 = nodeValue0 + dbType.escapeClause();
}
String filter = node.isReverseOrder
? nodeValue0 + " " + node.operator.getSqlOp() + " " + tableValue
Expand Down Expand Up @@ -1607,6 +1615,30 @@ public void visit(LeafNode node) throws MetaException {

filterBuffer.append("(" + filter + ")");
}

private String makeDirectFilterForLike(String likePattern) {
StringBuilder sb = new StringBuilder();
for (int i = 0; i < likePattern.length(); i++) {
char n = likePattern.charAt(i);
// MySQL default ESCAPE is "\\\\", but prepared statement parameters
// are transmitted as raw values, so only "\\" is needed.
if (dbType.isMYSQL() && n == '\\' && i + 1 < likePattern.length()
&& likePattern.charAt(i + 1) == '\\') {
i++;
}
if (n == '.') {
if (i + 1 < likePattern.length() && likePattern.charAt(i + 1) == '*') {
sb.append("%");
i++;
} else {
sb.append("_");
}
continue;
}
sb.append(n);
}
return sb.toString();
}
}

/**
Expand Down