✨ feat(engine): qwant for the search engine#784
Merged
Conversation
- Provide the implementation for fetching search results from the `qwant` search engine using the api. - List the new implementation in the engine models. - Make the implementation module public. Co-authored-by: nrabulinski <[email protected]> Co-authored-by: Franz Kafka <[email protected]>
Co-authored-by: nrabulinski <[email protected]> Co-authored-by: Franz Kafka <[email protected]>
Co-authored-by: nrabulinski <[email protected]> Co-authored-by: Franz Kafka
Contributor
📝 WalkthroughWalkthroughAdds a new Qwant search engine: implements a Qwant engine module with JSON parsing and request handling, exports and registers it in the engine handler, adds a disabled-by-default config toggle, and bumps the crate version to 1.28.0. Changes
Sequence DiagramsequenceDiagram
participant Client as Client/User
participant Engine as Qwant Engine
participant Upstream as Qwant API
participant Parser as JSON Parser
Client->>Engine: results(query, page, user_agent)
activate Engine
Engine->>Engine: build URL (q=..., offset=page*10)
Engine->>Engine: set headers (User-Agent, Referer)
Engine->>Upstream: HTTP GET (Accept: application/json)
activate Upstream
Upstream-->>Engine: JSON bytes
deactivate Upstream
Engine->>Parser: parse_json_response(bytes)
activate Parser
Parser->>Parser: deserialize JSON -> check errorCode
Parser->>Parser: extract result.items.mainline where type == "web"
Parser-->>Engine: Vec<(url, SearchResult)> or EngineError
deactivate Parser
Engine-->>Client: Result<Vec<(String, SearchResult)>, EngineError>
deactivate Engine
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Possibly related PRs
Suggested labels
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@src/engines/qwant.rs`:
- Around line 146-148: The calculation let offset = page * count can overflow
for large page values; replace the direct multiplication with a checked or
saturating operation on the page/count symbols (e.g. use
page.checked_mul(count).ok_or(...) to return an error or use
page.saturating_mul(count) to cap the value) and handle the overflow case
appropriately (return an error or clamp) so offset is never produced by a
wrapped u32; update the code that uses offset to expect the error or clamped
value as needed.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 414a54e3-42ef-48b6-ac60-cbc119f5a8e5
⛔ Files ignored due to path filters (1)
Cargo.lockis excluded by!**/*.lock
📒 Files selected for processing (5)
Cargo.tomlsrc/engines/mod.rssrc/engines/qwant.rssrc/models/engine.rswebsurfx/config.lua
Contributor
There was a problem hiding this comment.
🧹 Nitpick comments (1)
src/engines/qwant.rs (1)
158-162: Refactor HeaderMap construction to avoid unnecessary HashMap allocation in per-request hot path.Building a temporary
HashMap<String, String>viaTryFromadds allocation and runtime parsing overhead that repeats for every request. ConstructHeaderMapdirectly using the typed constants andHeaderValue::from_str()for dynamic headers instead.♻️ Suggested refactor
-use reqwest::{Client, header::HeaderMap}; +use reqwest::{ + Client, + header::{HeaderMap, HeaderValue, REFERER, USER_AGENT}, +}; -use std::collections::HashMap; @@ - let header_map = HeaderMap::try_from(&HashMap::from([ - ("User-Agent".to_string(), user_agent.to_string()), - ("Referer".to_string(), "https://www.qwant.com/".to_string()), - ])) - .change_context(EngineError::UnexpectedError)?; + let mut header_map = HeaderMap::new(); + header_map.insert( + USER_AGENT, + HeaderValue::from_str(user_agent).change_context(EngineError::UnexpectedError)?, + ); + header_map.insert(REFERER, HeaderValue::from_static("https://www.qwant.com/"));🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@src/engines/qwant.rs` around lines 158 - 162, The HeaderMap is being built via HeaderMap::try_from(&HashMap::from(...)) which allocates a temporary HashMap per request; replace this by constructing a HeaderMap directly (the variable header_map) and insert headers using typed header names and HeaderValue::from_str(user_agent) for the dynamic user_agent and HeaderValue::from_static("https://www.qwant.com/") (or HeaderValue::from_str if preferred), propagating any HeaderValue::from_str errors into EngineError::UnexpectedError just like the current change_context; update the code around HeaderMap::try_from, HashMap::from, and user_agent usage to avoid the temporary allocation.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Nitpick comments:
In `@src/engines/qwant.rs`:
- Around line 158-162: The HeaderMap is being built via
HeaderMap::try_from(&HashMap::from(...)) which allocates a temporary HashMap per
request; replace this by constructing a HeaderMap directly (the variable
header_map) and insert headers using typed header names and
HeaderValue::from_str(user_agent) for the dynamic user_agent and
HeaderValue::from_static("https://www.qwant.com/") (or HeaderValue::from_str if
preferred), propagating any HeaderValue::from_str errors into
EngineError::UnexpectedError just like the current change_context; update the
code around HeaderMap::try_from, HashMap::from, and user_agent usage to avoid
the temporary allocation.
github-project-automation
Bot
moved this from Under Review
to Done
in Task list for v2.0.0 release cycle
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Adds Qwant as a new upstream search engine, querying the Qwant v3 JSON API (
api.qwant.com/v3/search/web).Changes:
src/engines/qwant.rsimplementing theSearchEnginetraitfetch_json_as_bytes_from_upstreamfor HTTP fetching (consistent with SepiaSearch)offsetparameter, 10 results per pagesafesearchquery parametermod.rs, the engine match statement inengine.rs, andconfig.luaWhy is this change important?
Qwant is a privacy-focused search engine that provides an alternative to Google/Bing-based results. Adding it gives websurfx users another upstream option that respects user privacy.
How to test this PR locally?
websurfx/config.lua:cargo test --lib engines::qwantAuthor's checklist
SearchEnginetrait impl + constructor)fetch_json_as_bytes_from_upstreamfor HTTP fetchingform_urlencodedform_urlencodedalready present from SepiaSearch)v1.28.0Related issues
Takes over #605 #783 (original work by @nrabulinski, and Franz Kafka). Closes #317.
Credit
Based on initial implementation by @nrabulinski in #605 and Franz Kafka in #783 — the serde response models and API structure were ported from their work.
Summary by CodeRabbit
New Features
Tests
Chores