Skip to content

refactor(miniapp): migrate to v2 data layer#14049

Open
chengcheng84 wants to merge 112 commits intoCherryHQ:v2from
chengcheng84:v2-miniapps
Open

refactor(miniapp): migrate to v2 data layer#14049
chengcheng84 wants to merge 112 commits intoCherryHQ:v2from
chengcheng84:v2-miniapps

Conversation

@chengcheng84
Copy link
Copy Markdown
Contributor

@chengcheng84 chengcheng84 commented Apr 5, 2026

What this PR does

After this PR:

  • Miniapp data unified in SQLite via DataApi
  • Types centralized in @shared/data/types/miniapp.ts
  • New useMinapps hook using useQuery/useMutation from DataApi
  • All components migrated to use appId instead of id
  • Routes migrated to /app/miniapp/$appId

Breaking changes

None

Special notes for your reviewer

Checklist

  • PR: The PR description is expressive enough and will help future contributors
  • Code: Write code that humans can understand and Keep it simple
  • Refactor: Left the code cleaner than I found it (removed file I/O, unified types)
  • Upgrade: Migration handled by existing MiniAppMigrator
  • Documentation: User-guide update not required (internal refactoring)
  • Self-review: Reviewed via gh pr diff

Release note

NONE - Internal v2 data refactoring, no user-facing changes

chengcheng84 and others added 30 commits March 14, 2026 21:56
Signed-off-by: suyao <[email protected]>

# Conflicts:
#	migrations/sqlite-drizzle/meta/0006_snapshot.json
#	migrations/sqlite-drizzle/meta/_journal.json
#	packages/shared/data/api/schemas/index.ts
#	src/renderer/src/config/minapps.ts
Critical fixes:
- C4: Derive BUILTIN_APP_LOGO_MAP from shared preset data instead of manual map
- C5: Return OffsetPaginationResponse from GET /miniapps (consistent with other endpoints)
- C6: Add Zod runtime validation schemas and parsing in all miniapp handlers

Important fixes:
- I1: update() returns merged preset for builtin apps (consistent with getByAppId/list)
- I2: Validate .returning() result in update/updateStatus, throw notFound if missing
- I3: Move ensureDefaultAppPref into transaction + batch queries in reorder()
- I4: Replace SELECT-then-INSERT with INSERT ON CONFLICT DO NOTHING (TOCTOU fix)
- I5: Whitelist preference-only fields for default app updates
- I6: loadCustomMiniApp only overwrites on ENOENT, not on transient errors
- I9: Add CHECK constraints on status/type enum columns
- I10: Add MiniAppSelect/MiniAppInsert type aliases (project convention)
- I11: Fix naming — path param :appId→:id, remove handler alias, StatusUpdateResponse
- I12: Remove redundant COUNT query in list()

Also fixes:
- Add missing reset() method to MiniAppMigrator (BaseMigrator contract)
- Add missing Minimax/Dangbei/Ima icon imports and switch cases in getMiniAppsLogo
- Update moonshot test expectation to match preset-derived logo key

Co-Authored-By: Claude Opus 4.6 <[email protected]>
Signed-off-by: suyao <[email protected]>
Only background and configuration support nullable (explicit clear to null),
based on business logic:
- logo: must always have a value (visual identity)
- supportedRegions: must be explicitly set (region visibility control)
- background: nullable — clearing resets to default/transparent
- configuration: nullable — clearing removes custom config

Co-Authored-By: Claude Opus 4.6 <[email protected]>
Signed-off-by: suyao <[email protected]>
Replace CompoundIcon reference with string key 'openclaw' to match
the v2 miniapp logo resolution pattern.

Co-Authored-By: Claude Opus 4.6 <[email protected]>
Signed-off-by: suyao <[email protected]>
Every custom miniapp must have:
- logo: visual identity is mandatory
- bordered: UI rendering needs explicit value
- supportedRegions: region filtering depends on this field

background and configuration remain optional.

Co-Authored-By: Claude Opus 4.6 <[email protected]>
Signed-off-by: suyao <[email protected]>
@chengcheng84
Copy link
Copy Markdown
Contributor Author

chengcheng84 commented Apr 11, 2026

Note

This comment was translated by Claude.

@DeJeune Are there any other tasks?


Original Content

@DeJeune 还有什么任务吗

DeJeune and others added 5 commits April 16, 2026 13:36
Signed-off-by: suyao <[email protected]>

# Conflicts:
#	packages/shared/data/api/schemas/index.ts
#	src/main/data/api/handlers/index.ts
#	src/main/data/services/MiniAppService.ts
#	src/main/data/services/__tests__/MiniAppService.test.ts
Signed-off-by: suyao <[email protected]>

# Conflicts:
#	packages/shared/data/api/schemas/index.ts
#	packages/shared/data/bootConfig/bootConfigSchemas.ts
#	packages/shared/data/preference/preferenceSchemas.ts
#	src/main/data/migration/v2/migrators/mappings/ComplexPreferenceMappings.ts
#	src/main/data/migration/v2/migrators/mappings/__tests__/ComplexPreferenceMappings.test.ts
#	src/main/data/services/MiniAppService.ts
#	src/main/data/services/__tests__/MiniAppService.test.ts
#	src/renderer/src/components/TopView/index.tsx
#	src/renderer/src/components/app/Sidebar.tsx
#	src/renderer/src/hooks/useAppInit.ts
Signed-off-by: suyao <[email protected]>

# Conflicts:
#	migrations/sqlite-drizzle/meta/_journal.json
#	src/main/data/services/MiniAppService.ts
oxlint --deny-warnings was failing CI on unused variables from
useTabs() destructuring in Sidebar.tsx.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Signed-off-by: suyao <[email protected]>
@chengcheng84
Copy link
Copy Markdown
Contributor Author

@DeJeune any updates

Signed-off-by: suyao <[email protected]>

# Conflicts:
#	packages/shared/data/api/schemas/index.ts
#	src/main/data/api/handlers/index.ts
#	src/main/data/migration/v2/migrators/mappings/__tests__/ComplexPreferenceMappings.test.ts
#	v2-refactor-temp/tools/data-classify/data/target-key-definitions.json
@DeJeune
Copy link
Copy Markdown
Collaborator

DeJeune commented Apr 17, 2026

@DeJeune any updates

No issues, currently having the maintainer review again.


Original Content

@DeJeune any updates

没有什么问题,正在让maintainer 再次review

- Fix stale doc comment in MiniAppService (miniappTable → miniAppTable)
- Rename private miniappsStatusKeyMap → miniAppsStatusKeyMap in i18n/label.ts
- Fix JSX comment casing in MiniAppSettings

Generated routeTree.gen.ts intentionally keeps lowercase 'miniapp' since
it derives from the URL path /app/miniapp/.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Signed-off-by: suyao <[email protected]>
Copy link
Copy Markdown
Member

@0xfullex 0xfullex left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the thorough v2 data-layer migration. The overall architecture (DataApi + Cache + Preference split, builtin-preset + DB-row merge, migrator pipeline) is solid and CI is green. However, several blocking issues — around correctness, silent failures, and a naming-convention decision the team needs to commit to — should be resolved before merge.


Critical (must fix)

C1. Dynamic path is not URL-encoded / input is not whitelisted

File: src/renderer/src/hooks/useMiniApps.ts:174-190

const result = await dataApiService.patch(`/miniapps/${appId}`, { body })

CreateMiniAppSchema.appId only enforces z.string().min(1), so /, ?, #, whitespace, or " can escape path matching. The same applies to MiniAppPage.tsx:108 where document.querySelector('webview[data-miniapp-id="${appId}"]') can be broken by a " in the id.

Fix: Tighten the Zod schema (z.string().regex(/^[A-Za-z0-9_-]+$/)), use encodeURIComponent(appId) in API paths and CSS.escape(appId) in DOM selectors.

C2. MiniAppService.update() — side effect before validation, and non-atomic ensure→update

File: src/main/data/services/MiniAppService.ts:244-277

  1. ensureDefaultAppPref(appId) runs before the "no applicable fields" validation. Sending { name: 'x' } for a default app returns 400, but a default-preference row has already been inserted as a side effect.
  2. ensureDefaultAppPrefupdate is two separate statements. A concurrent resetDefaults() between them makes update return 0 rows, surfacing as a misleading notFound for a builtin id.

Fix: Validate updates is non-empty before ensureDefaultAppPref(); wrap both steps in a single db.transaction, or use INSERT ... ON CONFLICT DO UPDATE for builtin apps.

C3. useMiniAppPopup module-level mutable state breaks multi-instance usage

File: src/renderer/src/hooks/useMiniAppPopup.ts:32-58, 99-148

miniAppsCache, cacheCallbacksRef, and cacheVersion are module-scoped globals mutated on every render. In multi-window or concurrent-component scenarios, only the most recently mounted consumer's setOpenedKeepAliveMiniApps is invoked from disposeAfter/onInsert, silently desyncing others. The resize effect reassigns the global cache, but parent components holding the old reference don't re-render.

Fix: Move the cache into useRef or a lifecycle-managed singleton service, and pass setters via proper React refs.

C4. Promise.allSettled swallows partial failures in updateMiniApps family

File: src/renderer/src/hooks/useMiniApps.ts:204-288

return results.filter(r => r.status === 'fulfilled').map(r => r.value)
window.toast?.error('Failed to update miniapps')
  • Callers can't distinguish "all succeeded" from "partially failed"; local UI drifts from DB with no refetch/rollback.
  • window.toast?.error silently no-ops if window.toast isn't registered (e.g. early boot or mini-window context).
  • Hardcoded English string — violates the i18n rule.

Fix: Aggregate rejections and throw (or return { fulfilled, rejected }). Route the message through i18next. Trigger invalidate('/miniapps') to resync UI.

C5. loadCustomMiniApp silently drops user data on any file error

File: src/renderer/src/config/miniapps.ts:91-117

Top-level await loadCustomMiniApp() returns [] on any JSON parse / read / permission error at module evaluation. A single corrupt character in custom-miniapps.json permanently hides all custom mini-apps with no user-facing notice.

Fix: On failure, quarantine the bad file as custom-miniapps.json.broken-{ts} before returning [], and surface a toast on next render.

C6. MiniAppMigrator.test.ts violates the DB-testing rule

File: src/main/data/migration/v2/migrators/__tests__/MiniAppMigrator.test.ts

CLAUDE.md: "For any service/handler/seeder that reads or writes SQLite, use setupTestDatabase() from @test-helpers/db". The test hand-rolls vi.fn().mockReturnThis() chains, so it can't catch batch-size regressions, UNIQUE/FK constraint issues, or validate-step ordering bugs.

Fix: Convert to setupTestDatabase() and assert against real DB reads.

C7. Naming convention: adopt Plan A (mini-app / mini_app)

See the Naming Decision section below. This PR carries the name in its first persistent form (SQLite table, API endpoints, preference keys). Fixing it now is a rename; fixing it later is a migration.


Important (strongly recommend fixing)

I1. API path design

  • 【这个先HOLD住,暂不用改。reorder的需求比较多,我会做一个统一的规范】PATCH /miniapps for batch reorder conflicts with "PATCH = partial update of one resource". Per the api-design-guidelines decision tree, change to PUT /mini-apps/order with body { items: [...] }.
  • DELETE /miniapps/defaults is semantically a reset action, not a resource deletion. Consider POST /mini-apps/defaults/reset.

I2. Fake pagination in list()

File: src/main/data/services/MiniAppService.ts:187-191

return { items: allItems, total: allItems.length, page: 1 }

query has no page/limit, so declaring OffsetPaginationResponse<MiniApp> misleads clients. Align with TagSchemas and return bare MiniApp[].

I3. Double merge of builtin + DB rows across processes

MiniAppService.builtinToMiniApp (main) and useMiniApps.mergeWithPreset (renderer) both fold logo, bordered, background, nameKey, supportedRegions from presets. Drift between them will surface as silent behavior differences.

Fix: Centralize merge in main (Registry Service pattern, per data-api-in-main.md). Renderer consumes the already-merged MiniApp.

I4. Missing withSqliteErrors wrappers

File: src/main/data/services/MiniAppService.ts

create() wraps inserts correctly. update, delete, reorder, resetDefaults, and ensureDefaultAppPref call Drizzle raw, leaking DrizzleQueryError instead of DataApiError on constraint violations.

I5. reorder silently skips missing appIds

File: src/main/data/services/MiniAppService.ts:357-370

result.length === 0 → skipped.push(...) only logs a warning and the handler returns 204 — renderer cannot detect skips.

Fix: Either throw notFound on the first missing id, or return { skipped: [...] } in the response.

I6. antd / styled-components in v2 files

Files: MiniAppPopupContainer.tsx:25 (Alert from antd), MiniAppTabsPool.tsx, MiniAppPage.tsx, WebviewContainer.tsx (all heavy styled.div usage).

CLAUDE.md v2 rule forbids antd / styled-components in new code. Either migrate to @cherrystudio/ui + Tailwind, or mark files as *.v2.tsx WIP per the naming convention.

I7. Hand-maintained MiniApp type drifts from DB schema

packages/shared/data/types/miniapp.ts is hand-written and already diverges from MiniAppSelect on nullability (logo, bordered).

Fix: Derive from $inferSelect, the same way tag DTOs are derived from their Zod schema.

I8. sortOrder collisions between migrator and builtin defaults

File: MiniAppMigrator.ts:96-111

Per-status reindexing to 0..N-1 collides with builtinMiniAppDefaultSortOrder and with create()'s Math.max(builtinCount, ...) logic. Custom apps end up with sortOrder colliding with builtins until the user reorders.

Fix: Offset custom-app sortOrder by builtinMiniAppDefaultSortOrder.size.

I9. Stale closures and ref churn

  • MiniAppPopupContainer.tsx:300-320, 345-360useEffect with react-hooks/exhaustive-deps disabled reads currentAppInfo/currentMiniAppId that aren't in deps.
  • WebviewContainer.tsx:32-43setRef(appid) returns a new function every render, causing React to call the old ref with null and the new one with the element on every render.

Fix: Memoize handlers with useCallback and include them in deps.

I10. any types in WebviewContainer

WebviewContainer.tsx:66, 220event: any, 'true' as any. Use Electron.DidNavigateInPageEvent and allowpopups={'true'}.

I11. MiniAppKind half-rename

DB layer renamed the discriminator type MiniAppTypeMiniAppKind, but the column (type), indexes (miniapp_type_idx, miniapp_status_type_idx), check constraint (miniapp_type_check), Zod schema (MiniAppTypeSchema), query field, and entity field on MiniApp all still say type. A partial rename is worse than either extreme.

Fix: Complete the rename — column → kind, indexes → mini_app_kind_idx / mini_app_status_kind_idx, check → mini_app_kind_check, Zod → MiniAppKindSchema, entity field → kind. Combine with Plan A below.


Naming Decision: adopt Plan A

The team has confirmed MiniApp is a compound term (the capital A in PascalCase is the evidence). By the mechanical mapping rule the repo already applies to every other compound noun (MCPServer → mcp-server, KnowledgeBase → knowledge-base, ActiveNode → active-node, RotatedKey → rotated-key, AuthConfig → auth-config), the forms must be:

Form Value
PascalCase MiniApp
camelCase miniApp
kebab-case mini-app
snake_case mini_app
SCREAMING_SNAKE MINI_APP

Rule: a case boundary in any one form must appear as a separator in all other forms.

Required renames in this PR

A. API paths (stateless, free to change)

  • /miniapps/mini-apps
  • /miniapps/:id/mini-apps/:id
  • /miniapps/defaults/mini-apps/defaults

B. File names (per-directory sub-convention)

  • api/schemas/miniapps.tsapi/schemas/miniApps.ts (schemas uses camelCase, cf. mcpServers.ts, fileProcessing.ts, temporaryChats.ts)
  • types/miniapp.tstypes/miniApp.ts
  • presets/miniapps.tspresets/mini-apps.ts (presets uses kebab-case, cf. file-processing.ts, translate-languages.ts)

C. Routes / URL segments

  • routes/app/miniapp/$appId.tsxroutes/app/mini-app/$appId.tsx
  • routes/app/miniapp/index.tsxroutes/app/mini-app/index.tsx
  • pages/miniapps/pages/mini-apps/
  • All literal path strings (TabsService.ts:67, useMiniAppPopup.ts navigation, etc.)

D. DB schema (persisted — but this PR introduces the table for the first time, so no migration chain yet; regenerate)

  • Table: 'miniapp''mini_app'
  • Indexes: miniapp_status_sort_idxmini_app_status_sort_idx, miniapp_type_idxmini_app_kind_idx, miniapp_status_type_idxmini_app_status_kind_idx
  • Checks: miniapp_status_checkmini_app_status_check, miniapp_type_checkmini_app_kind_check
  • Re-run pnpm agents:generate and replace the in-PR migration file.
  • Variable miniAppTable and column names (app_id, sort_order, supported_regions) are already correct.

E. Preference / Cache keys (persisted, but brand new in v2 and not yet shipped)

  • feature.miniapp.regionfeature.mini_app.region
  • feature.miniapp.max_keep_alivefeature.mini_app.max_keep_alive
  • miniapp.opened_keep_alivemini_app.opened_keep_alive
  • miniapp.current_idmini_app.current_id
  • miniapp.showmini_app.show
  • miniapp.opened_oneoffmini_app.opened_oneoff
  • miniapp.detected_regionmini_app.detected_region
  • Touchpoints: preferenceSchemas.ts, cacheSchemas.ts, cacheValueTypes.ts, PreferencesMappings.ts, v2-refactor-temp/tools/data-classify/data/classification.json; then re-run cd v2-refactor-temp/tools/data-classify && npm run generate.
  • Do not rename Redux source keys in migrators ('minapps', 'minAppRegion', 'minappsOpenLinkExternal') — they must match historical state.

F. Lingering legacy spellings (cleanup)

  • ORIGIN_DEFAULT_MIN_APPSORIGIN_DEFAULT_MINI_APPS (21 references across 13 files)
  • loadCustomMiniApploadCustomMiniApps (returns an array)
  • classification.json: "DEFAULT_MIN_APPS""DEFAULT_MINI_APPS"

G. Components (already correct)

  • components/MiniApp/* — PascalCase component directory stays as-is (React convention).

Protected (do not rename — v2 freeze)

Redux-persisted identifiers that must remain to preserve user state:

  • src/renderer/src/store/miniapps.ts: MinAppsState, slice name 'minApps', setMinApps, setDisabledMinApps, setPinnedMinApps
  • src/renderer/src/store/settings.ts: minappsOpenLinkExternal, setMinAppRegion
  • src/renderer/src/store/migrate.ts: all legacy key references
  • v2-refactor-temp/.../classification.json / inventory.json: source-key mirrors

Test coverage gaps

Must add

  • Concurrent create() → 409Promise.all([create(dto), create(dto)]) yields exactly one CONFLICT. Guards against reintroduction of a select-then-insert race.
  • MiniAppMappings.transformMiniApp unit tests — logo URL vs icon-key vs fallback, supportedRegions normalization, bordered/bodered typo compatibility.
  • reorder transaction rollback — simulate an in-loop throw and assert earlier updates rolled back.
  • Region auto-detection — reset regionDetectionPromise between tests; mock window.api.getIpCountry resolving 'CN'/'US' and rejecting; verify setDetectedRegion calls.
  • updateMiniApps partial-failure branch — mixed-result Promise.allSettled, assert toast invoked and fulfilled values returned.

Should add

  • reorder([]) no-op.
  • update() with {} and { name: undefined } for both default and custom apps.
  • list({ type: 'custom', status: 'pinned' }) combined filter.
  • LRU cache resize path when maxKeepAliveMiniApps changes between renders (the in-component copy branch).

Shallow tests to tighten

  • reorderMiniApps hook test only asserts typeof === 'function' — tautology. Assert the mutation trigger was called with the items.
  • Migrator "dedup-then-reindex" input contains no duplicates; make dedup visible.
  • Service "sort by status priority then sortOrder" only checks two positions; assert full ordering including the within-status tiebreak.
  • useMiniApps "no-op when visible list unchanged" should assert patchApp was never called, not just allApps.length === 1.

Minor / cleanup

  • Delete dead commented references in runtime.ts:59-65, 150-155, i18n/label.ts:180, types/index.ts:665, sidebar.ts:15.
  • Update // MinApps section headers in settings.ts:196, 383 to // MiniApps.
  • Wrap LRU disposeAfter/onInsert callbacks (useMiniAppPopup.ts:91-103) in try/catch + logger.error.
  • Replace setTimeout(..., 100) retry in TabsService.ts:67 with a proper readiness Signal.
  • Region-detection fallback to 'CN' should emit a one-time structured error for observability.
  • MiniAppMigrator.validate should track original-source count in stats.sourceCount (currently equal to preparedRows.length by construction, hiding silent drops).

Strengths

  • DataApi scope is fully compliant — every endpoint backs a SQLite row.
  • create() correctly relies on DB UNIQUE + withSqliteErrors instead of select-then-insert (race-free pattern from the docs).
  • Migrator pipeline is robust: priority-based dedup (pinned > enabled > disabled), bodered typo tolerance, batched insert in a transaction.
  • Cache/Preference keys follow the domain.snake_case convention consistently.
  • Region filtering is correctly modeled as a view concern; mutations operate on raw DataApi data.
  • Centralized logging via loggerService.withContext(...); no console.log introduced.
  • Clean deletion of legacy files: old MinApp/, useMinapps.ts, useMinappPopup.ts, MinAppIcon.tsx, routes/app/minapp/.

Suggested merge path

  1. Resolve C1–C7 (7 blockers).
  2. Complete I1–I11 (API shape, merge centralization, error translation, v2 UI rule, type derivation, half-rename).
  3. Apply Plan A rename — highest leverage on D (DB) and E (preferences) because they're first-write persistent state; leaving them now forces a migration later.
  4. Add Must-add tests (especially migrator → real DB).
  5. Clean up dead comments.

Thanks again for the thorough work — this is a large, well-structured migration. With these adjustments, the naming gets locked in once and for all before any of it hits production.

chengcheng84 and others added 11 commits April 17, 2026 20:42
- API paths: /miniapps → /mini-apps
- DB table: miniapp → mini_app
- Indexes/checks: miniapp_* → mini_app_*
- Preference keys: feature.miniapp.* → feature.mini_app.*
- Cache keys: miniapp.* → mini_app.*
- Constants: ORIGIN_DEFAULT_MIN_APPS → ORIGIN_DEFAULT_MINI_APPS
- Sidebar icon: 'miniapp' → 'mini_app'
- Update all tests and mock paths accordingly
- Add migration 0012_striped_hex
Critical:
- C1: appId regex validation + encodeURIComponent + CSS.escape
- C2: validate before side effects, wrap update in transaction
- C3: module-level state → sharedCache + useRef
- C4: Promise.allSettled → settleAndInvalidate + i18n error
- C5: quarantine broken custom-miniapps.json
- C6: MiniAppMigrator tests → setupTestDatabase()
- C7: Plan A naming (mini-app, mini_app, mini_app.*)

Important:
- I4: ensureDefaultAppPref atomic within transaction
- I11: mini_app_type_* → mini_app_kind_*

Remaining I1-I3, I5-I10 follow-up PRs.
- Rename `type` to `kind` in MiniApp schema, DB table, and all references
- Update `MiniAppService.list()` to return `MiniApp[]` directly instead
  of a paginated wrapper
- Add TODO annotations for API endpoints that deviate from design
  guidelines (batch reorder and reset-defaults)
- Fix `WebviewContainer` ref handling and type safety
- Remove unused styled component in `MiniAppsPage`
- Add missing `withSqliteErrors` wrappers in service methods
@chengcheng84
Copy link
Copy Markdown
Contributor Author

@0xfullex @DeJeune Thanks for your detailed review

Critical and Important issues have been fixed:

  • C1-C7 All fixed
  • I2-I5 I7-I11 All fixed

Remaining:

  • I1 — held for unified API spec
  • I6 — antd/styled-components → follow-up PR
  • Other suggestions → follow-up PR

Ready for final review

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants