Skip to content

Commit be90e1c

Browse files
pcarletonpcarleton
committed
Merge main into pkce-conformance-tests
2 parents fbd656d + 2d52de5 commit be90e1c

File tree

10 files changed

+523
-13
lines changed

10 files changed

+523
-13
lines changed

‎README.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,8 @@ A framework for testing MCP (Model Context Protocol) client and server implement
55
> [!WARNING]
66
> This repository is a work in progress and is unstable. Join the conversation in the #conformance-testing-wg in the MCP Contributors discord.
77
8+
**For SDK maintainers:** See [SDK Integration Guide](./SDK_INTEGRATION.md) for a streamlined guide on integrating conformance tests into your SDK repository.
9+
810
## Quick Start
911

1012
### Testing Clients

‎SDK_INTEGRATION.md‎

Lines changed: 208 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,208 @@
1+
# Using MCP Conformance Tests in SDK Repositories
2+
3+
This guide explains how to integrate the MCP conformance test suite into your language SDK repository. The conformance framework tests your MCP implementation against the protocol specification to ensure compatibility.
4+
5+
## Quick Start
6+
7+
Install and run conformance tests:
8+
9+
```bash
10+
# Client testing (framework starts a test server, runs your client against it)
11+
npx @modelcontextprotocol/conformance client --command "your-client-command" --scenario initialize
12+
13+
# Server testing (your server must already be running)
14+
npx @modelcontextprotocol/conformance server --url http://localhost:3000/mcp --scenario server-initialize
15+
```
16+
17+
## Two Testing Modes
18+
19+
### Client Testing
20+
21+
The framework **starts a test server** and spawns your client against it. Your client receives the server URL as its final command-line argument.
22+
23+
```bash
24+
# Run a single scenario
25+
npx @modelcontextprotocol/conformance client \
26+
--command "python tests/conformance/client.py" \
27+
--scenario initialize
28+
29+
# Run a suite of tests
30+
npx @modelcontextprotocol/conformance client \
31+
--command "python tests/conformance/client.py" \
32+
--suite auth
33+
```
34+
35+
**Available client suites:** `all`, `core`, `extensions`, `auth`, `metadata`, `sep-835`
36+
37+
Your client should:
38+
39+
1. Accept the server URL as its last argument
40+
2. Read `MCP_CONFORMANCE_SCENARIO` env var to determine which scenario is being tested
41+
3. Read `MCP_CONFORMANCE_CONTEXT` env var for scenario-specific data (e.g., OAuth credentials)
42+
43+
### Server Testing
44+
45+
Your server must be **running before** invoking the conformance tool. The framework connects to it as an MCP client.
46+
47+
```bash
48+
# Start your server first
49+
your-server --port 3001 &
50+
51+
# Then run conformance tests
52+
npx @modelcontextprotocol/conformance server \
53+
--url http://localhost:3001/mcp \
54+
--suite active
55+
```
56+
57+
**Available server suites:** `active` (default), `all`, `pending`
58+
59+
**Note:** Server testing requires you to manage server lifecycle (start, health-check, cleanup) yourself.
60+
61+
---
62+
63+
## Expected Failures (Baseline) File
64+
65+
The expected-failures feature lets your CI pass while you work on fixing known issues. It catches regressions by failing when:
66+
67+
- A previously passing test starts failing (regression)
68+
- A previously failing test starts passing (stale baseline - remove the entry)
69+
70+
### File Format
71+
72+
Create a YAML file (e.g., `conformance-baseline.yml`):
73+
74+
```yaml
75+
server:
76+
- tools-call-with-progress
77+
- resources-subscribe
78+
client:
79+
- auth/client-credentials-jwt
80+
```
81+
82+
### Usage
83+
84+
```bash
85+
npx @modelcontextprotocol/conformance server \
86+
--url http://localhost:3000/mcp \
87+
--expected-failures ./conformance-baseline.yml
88+
```
89+
90+
### Exit Code Behavior
91+
92+
| Scenario Result | In Baseline? | Exit Code | Meaning |
93+
| --------------- | ------------ | --------- | ----------------------------- |
94+
| Fails | Yes | 0 | Expected failure |
95+
| Fails | No | 1 | Unexpected regression |
96+
| Passes | Yes | 1 | Stale baseline - remove entry |
97+
| Passes | No | 0 | Normal pass |
98+
99+
---
100+
101+
## GitHub Action
102+
103+
The conformance repo provides a reusable GitHub Action that handles Node.js setup and conformance execution.
104+
105+
### Client Testing Example
106+
107+
```yaml
108+
name: Conformance Tests
109+
on: [push, pull_request]
110+
111+
jobs:
112+
conformance:
113+
runs-on: ubuntu-latest
114+
steps:
115+
- uses: actions/checkout@v4
116+
117+
- name: Set up your SDK
118+
run: |
119+
# Your SDK setup (pip install, npm install, etc.)
120+
pip install -e .
121+
122+
- uses: modelcontextprotocol/[email protected]
123+
with:
124+
mode: client
125+
command: 'python tests/conformance/client.py'
126+
suite: auth
127+
expected-failures: ./conformance-baseline.yml
128+
```
129+
130+
### Server Testing Example
131+
132+
```yaml
133+
name: Conformance Tests
134+
on: [push, pull_request]
135+
136+
jobs:
137+
conformance:
138+
runs-on: ubuntu-latest
139+
steps:
140+
- uses: actions/checkout@v4
141+
142+
- name: Set up and start server
143+
run: |
144+
pip install -e .
145+
python -m myserver --port 3001 &
146+
# Wait for server to be ready
147+
timeout 15 bash -c 'until curl -s http://localhost:3001/mcp; do sleep 0.5; done'
148+
149+
- uses: modelcontextprotocol/[email protected]
150+
with:
151+
mode: server
152+
url: http://localhost:3001/mcp
153+
suite: active
154+
expected-failures: ./conformance-baseline.yml
155+
```
156+
157+
### Action Inputs
158+
159+
| Input | Required | Description |
160+
| ------------------- | ----------- | ----------------------------------------------- |
161+
| `mode` | Yes | `server` or `client` |
162+
| `url` | Server mode | URL of the server to test |
163+
| `command` | Client mode | Command to run the client |
164+
| `expected-failures` | No | Path to YAML baseline file |
165+
| `suite` | No | Test suite to run |
166+
| `scenario` | No | Run a single scenario by name |
167+
| `timeout` | No | Timeout in ms for client tests (default: 30000) |
168+
| `verbose` | No | Show verbose output (default: false) |
169+
| `node-version` | No | Node.js version (default: 20) |
170+
171+
---
172+
173+
## Writing Conformance Clients/Servers
174+
175+
### Example Client Pattern
176+
177+
See [`src/conformance/everything-client.ts`](https://github.com/modelcontextprotocol/typescript-sdk/blob/main/src/conformance/everything-client.ts) in the TypeScript SDK for a reference implementation. The recommended pattern is a single client that routes behavior based on the scenario:
178+
179+
```python
180+
import os
181+
import sys
182+
import json
183+
184+
def main():
185+
server_url = sys.argv[-1] # URL passed as last argument
186+
scenario = os.environ.get("MCP_CONFORMANCE_SCENARIO", "")
187+
context = json.loads(os.environ.get("MCP_CONFORMANCE_CONTEXT", "{}"))
188+
189+
if scenario.startswith("auth/"):
190+
run_auth_scenario(server_url, scenario, context)
191+
else:
192+
run_default_scenario(server_url)
193+
194+
if __name__ == "__main__":
195+
main()
196+
```
197+
198+
### Example Server Pattern
199+
200+
See [`src/conformance/everything-server.ts`](https://github.com/modelcontextprotocol/typescript-sdk/blob/main/src/conformance/everything-server.ts) in the TypeScript SDK for a reference implementation that handles all server scenarios.
201+
202+
---
203+
204+
## Additional Resources
205+
206+
- [Conformance README](./README.md)
207+
- [Design documentation](./src/runner/DESIGN.md)
208+
- [TypeScript SDK conformance examples](https://github.com/modelcontextprotocol/typescript-sdk/tree/main/src/conformance)

‎examples/clients/typescript/auth-test-attempts-dcr.ts‎

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
#!/usr/bin/env node
2+
3+
import { Client } from '@modelcontextprotocol/sdk/client/index.js';
4+
import { StreamableHTTPClientTransport } from '@modelcontextprotocol/sdk/client/streamableHttp.js';
5+
import { withOAuthRetry } from './helpers/withOAuthRetry';
6+
import { runAsCli } from './helpers/cliRunner';
7+
import { logger } from './helpers/logger';
8+
9+
/**
10+
* Non-compliant client that ignores pre-registered credentials and attempts DCR.
11+
*
12+
* This client intentionally ignores the client_id and client_secret passed via
13+
* MCP_CONFORMANCE_CONTEXT and instead attempts to do Dynamic Client Registration.
14+
* When run against a server that does not support DCR (no registration_endpoint),
15+
* this client will fail.
16+
*
17+
* Used to test that conformance checks detect clients that don't properly
18+
* use pre-registered credentials when server doesn't support DCR.
19+
*/
20+
export async function runClient(serverUrl: string): Promise<void> {
21+
const client = new Client(
22+
{ name: 'test-auth-client-attempts-dcr', version: '1.0.0' },
23+
{ capabilities: {} }
24+
);
25+
26+
// Non-compliant: ignores pre-registered credentials from context
27+
// and creates a fresh provider that will attempt DCR
28+
const oauthFetch = withOAuthRetry(
29+
'test-auth-client-attempts-dcr',
30+
new URL(serverUrl)
31+
)(fetch);
32+
33+
const transport = new StreamableHTTPClientTransport(new URL(serverUrl), {
34+
fetch: oauthFetch
35+
});
36+
37+
await client.connect(transport);
38+
logger.debug('Connected to MCP server (attempted DCR instead of pre-reg)');
39+
40+
await client.listTools();
41+
logger.debug('Successfully listed tools');
42+
43+
await client.callTool({ name: 'test-tool', arguments: {} });
44+
logger.debug('Successfully called tool');
45+
46+
await transport.close();
47+
logger.debug('Connection closed successfully');
48+
}
49+
50+
runAsCli(runClient, import.meta.url, 'auth-test-attempts-dcr <server-url>');

‎examples/clients/typescript/everything-client.ts‎

Lines changed: 69 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,12 @@ import {
2121
} from '@modelcontextprotocol/sdk/client/auth-extensions.js';
2222
import { ElicitRequestSchema } from '@modelcontextprotocol/sdk/types.js';
2323
import { ClientConformanceContextSchema } from '../../../src/schemas/context.js';
24-
import { withOAuthRetry, handle401 } from './helpers/withOAuthRetry.js';
24+
import {
25+
withOAuthRetry,
26+
withOAuthRetryWithProvider,
27+
handle401
28+
} from './helpers/withOAuthRetry.js';
29+
import { ConformanceOAuthProvider } from './helpers/ConformanceOAuthProvider.js';
2530
import { logger } from './helpers/logger.js';
2631

2732
/**
@@ -300,6 +305,69 @@ export async function runClientCredentialsBasic(
300305

301306
registerScenario('auth/client-credentials-basic', runClientCredentialsBasic);
302307

308+
// ============================================================================
309+
// Pre-registration scenario
310+
// ============================================================================
311+
312+
/**
313+
* Pre-registration: client uses pre-registered credentials (no DCR).
314+
*
315+
* Server does not advertise registration_endpoint, so client must use
316+
* pre-configured client_id and client_secret passed via context.
317+
*/
318+
export async function runPreRegistration(serverUrl: string): Promise<void> {
319+
const ctx = parseContext();
320+
if (ctx.name !== 'auth/pre-registration') {
321+
throw new Error(`Expected pre-registration context, got ${ctx.name}`);
322+
}
323+
324+
const client = new Client(
325+
{ name: 'conformance-pre-registration', version: '1.0.0' },
326+
{ capabilities: {} }
327+
);
328+
329+
// Create provider with pre-registered credentials
330+
const provider = new ConformanceOAuthProvider(
331+
'http://localhost:3000/callback',
332+
{
333+
client_name: 'conformance-pre-registration',
334+
redirect_uris: ['http://localhost:3000/callback']
335+
}
336+
);
337+
338+
// Pre-set the client information so the SDK won't attempt DCR
339+
provider.saveClientInformation({
340+
client_id: ctx.client_id,
341+
client_secret: ctx.client_secret,
342+
redirect_uris: ['http://localhost:3000/callback']
343+
});
344+
345+
// Use the provider-based middleware
346+
const oauthFetch = withOAuthRetryWithProvider(
347+
provider,
348+
new URL(serverUrl),
349+
handle401
350+
)(fetch);
351+
352+
const transport = new StreamableHTTPClientTransport(new URL(serverUrl), {
353+
fetch: oauthFetch
354+
});
355+
356+
await client.connect(transport);
357+
logger.debug('Successfully connected with pre-registered credentials');
358+
359+
await client.listTools();
360+
logger.debug('Successfully listed tools');
361+
362+
await client.callTool({ name: 'test-tool', arguments: {} });
363+
logger.debug('Successfully called tool');
364+
365+
await transport.close();
366+
logger.debug('Connection closed successfully');
367+
}
368+
369+
registerScenario('auth/pre-registration', runPreRegistration);
370+
303371
// ============================================================================
304372
// Main entry point
305373
// ============================================================================

‎examples/clients/typescript/helpers/withOAuthRetry.ts‎

Lines changed: 16 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,7 @@ export const handle401 = async (
4545
}
4646
}
4747
};
48+
4849
/**
4950
* Creates a fetch wrapper that handles OAuth authentication with retry logic.
5051
*
@@ -53,8 +54,10 @@ export const handle401 = async (
5354
* - Does not throw UnauthorizedError on redirect, but instead retries
5455
* - Calls next() instead of throwing for redirect-based auth
5556
*
56-
* @param provider - OAuth client provider for authentication
57+
* @param clientName - Client name for OAuth registration
5758
* @param baseUrl - Base URL for OAuth server discovery (defaults to request URL domain)
59+
* @param handle401Fn - Custom 401 handler function
60+
* @param clientMetadataUrl - Optional CIMD URL for URL-based client IDs
5861
* @returns A fetch middleware function
5962
*/
6063
export const withOAuthRetry = (
@@ -71,6 +74,18 @@ export const withOAuthRetry = (
7174
},
7275
clientMetadataUrl
7376
);
77+
return withOAuthRetryWithProvider(provider, baseUrl, handle401Fn);
78+
};
79+
80+
/**
81+
* Creates a fetch wrapper using a pre-configured OAuth provider.
82+
* Use this when you need to pre-set client credentials (e.g., for pre-registration tests).
83+
*/
84+
export const withOAuthRetryWithProvider = (
85+
provider: ConformanceOAuthProvider,
86+
baseUrl?: string | URL,
87+
handle401Fn: typeof handle401 = handle401
88+
): Middleware => {
7489
return (next: FetchLike) => {
7590
return async (
7691
input: string | URL,

0 commit comments

Comments
 (0)