Commit 84bd2fe
Feat/sql redis query (#467)
# Spec for SQLQuery class
Make sql-like commands available to be translated into Redis queries via
redisvl to cut down on syntax overhead for engineers.
Ex:
```py
from redisvl.query import SQLQuery
from redisvl.index import SearchIndex
redis_index = SearchIndex.from_existing(
"my_book_index",
redis_url="my_redis_connection"
)
sql_query = SQLQuery("""
SELECT title, author, price
FROM my_book_index
WHERE category = "scify"
"""
)
response = redis_index.query(sql_query)
```
This code would then produce the equivalent redis query to be executed
against the database:
```py
FT.search my_book_index
"@category:{scify}"
LOAD 3 @title @author @price
DIALECT 2
```
Expose a method on the object: `.redis_query_string()` such that you can
easily inspect the resulting redis query constructed from SQLQuery class
invocation.
Ex:
```py
from redisvl.query import SQLQuery
sql_str = """
SELECT user, credit_score, job, age
FROM user_simple
WHERE age > 17
"""
sql_query = SQLQuery(sql_str)
sql_query.redis_query_string(redis_url="redis://localhost:6379")
# result:
# 'FT.SEARCH user_simple "@Age:[(17 +inf]" RETURN 4 user credit_score job age'
```
# Packaging and dependencies
In order to use the `SQLQuery` class, user will have to install the
optional dependency on
[sql-redis](https://github.com/redis-developer/sql-redis). This can be
accomplished with the command `pip install redisvl[sql-redis]`.
# Tested/supported operators
| Datatype | Operator | Tested | SQL Example | Redis Result |
|----------|----------|--------|-------------|--------------|
| tag | = | ✅ | `SELECT title, category FROM {index} WHERE category =
'electronics'` | `FT.SEARCH test_index "@category:{electronics}" RETURN
2 title category` |
| | != | ✅ | `SELECT title, category FROM {index} WHERE category !=
'electronics'` | `FT.SEARCH test_index "-@category:{electronics}" RETURN
2 title category` |
| | IN | ✅ | `SELECT title, category FROM {index} WHERE category IN
('books', 'accessories')` | `FT.SEARCH test_index
"@category:{books\|accessories}" RETURN 2 title category` |
| numeric | > | ✅ | `SELECT title, price FROM {index} WHERE price > 100`
| `FT.SEARCH test_index "@price:[(100 +inf]" RETURN 2 title price` |
| | >= | ✅ | `SELECT title, price FROM {index} WHERE price >= 25 AND
price <= 50` | `FT.SEARCH test_index "@price:[25 +inf] @price:[-inf 50]"
RETURN 2 title price` |
| | = | ✅ | `SELECT title, price FROM {index} WHERE price = 45` |
`FT.SEARCH test_index "@price:[45 45]" RETURN 2 title price` |
| | != | ✅ | `SELECT title, price FROM {index} WHERE price != 45` |
`FT.SEARCH test_index "-@price:[45 45]" RETURN 2 title price` |
| | < | ✅ | `SELECT title, price FROM {index} WHERE price < 50` |
`FT.SEARCH test_index "@price:[-inf (50]" RETURN 2 title price` |
| | <= | ✅ | `SELECT title, price FROM {index} WHERE price >= 25 AND
price <= 50` | `FT.SEARCH test_index "@price:[25 +inf] @price:[-inf 50]"
RETURN 2 title price` |
| | BETWEEN | ✅ | `SELECT title, price FROM {index} WHERE price BETWEEN
40 AND 60` | `FT.SEARCH test_index "@price:[40 60]" RETURN 2 title
price` |
| text | = | ✅ | `SELECT title, name FROM {index} WHERE title =
'laptop'` | `FT.SEARCH test_index "@title:laptop" RETURN 2 title name` |
| | != | ✅ | `SELECT title, name FROM {index} WHERE title != 'laptop'` |
`FT.SEARCH test_index "-@title:laptop" RETURN 2 title name` |
| | prefix | ✅ | `SELECT title, name FROM {index} WHERE title = 'lap*'`
| `FT.SEARCH test_index "@title:lap*" RETURN 2 title name` |
| | suffix | ✅ | `SELECT title, name FROM {index} WHERE name = '*book'`
| `FT.SEARCH test_index "@name:*book" RETURN 2 title name` |
| | fuzzy | ✅ | `SELECT title, name FROM {index} WHERE title =
'%laptap%'` | `FT.SEARCH test_index "@title:%laptap%" RETURN 2 title
name` |
| | phrase | ✅ | `SELECT title, name FROM {index} WHERE title = 'gaming
laptop'` | `FT.SEARCH test_index "@title:gaming laptop" RETURN 2 title
name` |
| | phrase with stopword | ✅ | `SELECT title, name FROM {index} WHERE
title = 'laptop and keyboard'` | `FT.SEARCH test_index "@title:laptop
keyboard" RETURN 2 title name` |
| | IN | ❌ | `SELECT title, name FROM {index} WHERE title IN ('Python',
'Redis')` | NOT SUPPORTED |
| vector | vector_distance | ✅ | `SELECT title,
vector_distance(embedding, :vec) AS score FROM {index} LIMIT 3` |
`FT.SEARCH test_index "*=>[KNN 3 @Embedding $vector AS score]" PARAMS 2
vector $vector DIALECT 2 RETURN 2 title score LIMIT 0 3` |
| | cosine_distance | ✅ | `SELECT title, cosine_distance(embedding,
:vec) AS vector_distance FROM {index} LIMIT 3` | `FT.SEARCH test_index
"*=>[KNN 3 @Embedding $vector AS vector_distance]" PARAMS 2 vector
$vector DIALECT 2 RETURN 2 title vector_distance LIMIT 0 3` |
| date | >, >=, =, !=, <, <=, IN, BETWEEN | | | TODO |
| geo | = | | | TODO |
# Tested/supported aggregation reducer functions
| Reducer | Tested | SQL Example | Redis Result |
|---------|--------|-------------|--------------|
| COUNT | ✅ | `SELECT COUNT(*) as total FROM {index}` | `FT.AGGREGATE
test_index "*" GROUPBY 0 REDUCE COUNT 0 AS total` |
| COUNT | ✅ | `SELECT category, COUNT(*) as count FROM {index} GROUP BY
category` | `FT.AGGREGATE test_index "*" LOAD 1 category GROUPBY 1
@category REDUCE COUNT 0 AS count` |
| COUNT_DISTINCT | ✅ | `SELECT category, COUNT_DISTINCT(title) as
unique_titles FROM {index} GROUP BY category` | `FT.AGGREGATE test_index
"*" LOAD 2 category title GROUPBY 1 @category REDUCE COUNT_DISTINCT 1
@title AS unique_titles` |
| SUM | ✅ | `SELECT SUM(price) as total FROM {index}` | `FT.AGGREGATE
test_index "*" LOAD 1 price GROUPBY 0 REDUCE SUM 1 @price AS total` |
| SUM | ✅ | `SELECT category, SUM(price) as total_price FROM {index}
GROUP BY category` | `FT.AGGREGATE test_index "*" LOAD 2 category price
GROUPBY 1 @category REDUCE SUM 1 @price AS total_price` |
| MIN | ✅ | `SELECT MIN(price) as min_price FROM {index}` |
`FT.AGGREGATE test_index "*" LOAD 1 price GROUPBY 0 REDUCE MIN 1 @price
AS min_price` |
| MIN | ✅ | `SELECT category, MIN(price) as min_price FROM {index} GROUP
BY category` | `FT.AGGREGATE test_index "*" LOAD 2 category price
GROUPBY 1 @category REDUCE MIN 1 @price AS min_price` |
| MAX | ✅ | `SELECT MAX(price) as max_price FROM {index}` |
`FT.AGGREGATE test_index "*" LOAD 1 price GROUPBY 0 REDUCE MAX 1 @price
AS max_price` |
| MAX | ✅ | `SELECT category, MAX(price) as max_price FROM {index} GROUP
BY category` | `FT.AGGREGATE test_index "*" LOAD 2 category price
GROUPBY 1 @category REDUCE MAX 1 @price AS max_price` |
| AVG | ✅ | `SELECT category, AVG(price) as avg_price FROM {index} GROUP
BY category` | `FT.AGGREGATE test_index "*" LOAD 2 category price
GROUPBY 1 @category REDUCE AVG 1 @price AS avg_price` |
| STDDEV | ✅ | `SELECT STDDEV(price) as price_stddev FROM {index}` |
`FT.AGGREGATE test_index "*" LOAD 1 price GROUPBY 0 REDUCE STDDEV 1
@price AS price_stddev` |
| QUANTILE | ✅ | `SELECT category, QUANTILE(price, 0.5) as median_price
FROM {index} GROUP BY category` | `FT.AGGREGATE test_index "*" LOAD 2
price category GROUPBY 1 @category REDUCE QUANTILE 2 @price 0.5 AS
median_price` |
| TOLIST | ✅ | `SELECT category, ARRAY_AGG(title) as titles FROM {index}
GROUP BY category` | `FT.AGGREGATE test_index "*" LOAD 2 category title
GROUPBY 1 @category REDUCE TOLIST 1 @title AS titles` |
| FIRST_VALUE | ✅ | `SELECT category, FIRST_VALUE(title) as first_title
FROM {index} GROUP BY category` | `FT.AGGREGATE test_index "*" LOAD 2
category title GROUPBY 1 @category REDUCE FIRST_VALUE 1 @title AS
first_title` |
# TODO: subsequent work that will follow in separate PRs
- DATE datatype support
- ISMISSING / EXISTS null check support
- Advanced text search BM25 etc.
- Advanced hybrid search
- GEO datatype support
---------
Co-authored-by: Nitin Kanukolanu <nitinkanukolanu@gmail.com>1 parent 00211d5 commit 84bd2fe
File tree
12 files changed
+3893
-101
lines changed- docs
- api
- user_guide
- redisvl
- index
- query
- tests
- integration
- unit
12 files changed
+3893
-101
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
231 | 231 | | |
232 | 232 | | |
233 | 233 | | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
Large diffs are not rendered by default.
0 commit comments