|
| 1 | +# Pinecone Universal Agent Guide |
| 2 | + |
| 3 | +> **Official docs**: [https://docs.pinecone.io/](https://docs.pinecone.io/) - For complete API reference, advanced features, and detailed guides. |
| 4 | +
|
| 5 | +This guide covers critical gotchas, best practices, and common patterns for Pinecone across multiple programming languages. For language-specific examples and patterns, see the appropriate language file below. |
| 6 | + |
| 7 | +## Choosing the Right Guide |
| 8 | + |
| 9 | +Based on what the user is asking, consult these guides: |
| 10 | + |
| 11 | +### Getting Started |
| 12 | + |
| 13 | +- **User wants to learn Pinecone** → [PINECONE-quickstart.md](./PINECONE-quickstart.md) |
| 14 | +- **User needs a specific use case** → [PINECONE-quickstart.md](./PINECONE-quickstart.md) (then link to language-specific examples) |
| 15 | + |
| 16 | +### Installation & Setup |
| 17 | + |
| 18 | +- **CLI installation/usage** → [PINECONE-cli.md](./PINECONE-cli.md) |
| 19 | +- **SDK installation by language**: |
| 20 | + - Python → [PINECONE-python.md](./PINECONE-python.md#installation--setup) |
| 21 | + - TypeScript/Node.js → [PINECONE-typescript.md](./PINECONE-typescript.md#installation--setup) |
| 22 | + - Go → [PINECONE-go.md](./PINECONE-go.md#installation--setup) |
| 23 | + - Java → [PINECONE-java.md](./PINECONE-java.md#installation--setup) |
| 24 | + |
| 25 | +### Implementation |
| 26 | + |
| 27 | +- **Building features in Python** → [PINECONE-python.md](./PINECONE-python.md) |
| 28 | +- **Building features in TypeScript/Node.js** → [PINECONE-typescript.md](./PINECONE-typescript.md) |
| 29 | +- **Building features in Go** → [PINECONE-go.md](./PINECONE-go.md) |
| 30 | +- **Building features in Java** → [PINECONE-java.md](./PINECONE-java.md) |
| 31 | + |
| 32 | +### Universal Concepts |
| 33 | + |
| 34 | +- **Use this file** for CLI vs SDK guidance, common mistakes, constraints, error handling |
| 35 | +- **Language-specific info** → See language-specific files above |
| 36 | + |
| 37 | +## Language Detection |
| 38 | + |
| 39 | +Determine the primary programming language by checking for these files: |
| 40 | + |
| 41 | +- **`package.json`** → TypeScript/Node.js (see [PINECONE-typescript.md](./PINECONE-typescript.md)) |
| 42 | +- **`requirements.txt` or `pyproject.toml`** → Python (see [PINECONE-python.md](./PINECONE-python.md)) |
| 43 | +- **`pom.xml` or `build.gradle`** → Java (see [PINECONE-java.md](./PINECONE-java.md)) |
| 44 | +- **`go.mod`** → Go (see [PINECONE-go.md](./PINECONE-go.md)) |
| 45 | +- **Default fallback** → Python (see [PINECONE-python.md](./PINECONE-python.md)) |
| 46 | + |
| 47 | +## Universal Concepts (All Languages) |
| 48 | + |
| 49 | +### ⚠️ Critical: Installation & SDK |
| 50 | + |
| 51 | +> **Before installing anything**: ALWAYS verify if CLI/SDK is already installed before asking users to install or update: |
| 52 | +> |
| 53 | +> - **CLI**: Run `pc version` - only install if command fails |
| 54 | +> - **SDK**: Check package files or use language-specific verification commands |
| 55 | +> - Only prompt for installation when verification shows it's missing |
| 56 | +
|
| 57 | +**ALWAYS use the current SDK:** |
| 58 | + |
| 59 | +- **Python**: `pip install pinecone` (not `pinecone-client`) |
| 60 | +- **TypeScript**: `npm install @pinecone-database/pinecone` |
| 61 | +- **Java**: Add to `pom.xml` or `build.gradle` |
| 62 | +- **Go**: `go get github.com/pinecone-io/go-pinecone/pinecone` |
| 63 | + |
| 64 | +### 🔧 CLI vs SDK: When to Use Which |
| 65 | + |
| 66 | +**Use the Pinecone CLI for one-time or automated administrative tasks:** |
| 67 | + |
| 68 | +- ✅ **Creating indexes** - `pc index create` |
| 69 | +- ✅ **Deleting indexes** - `pc index delete` |
| 70 | +- ✅ **Configuring indexes** - `pc index configure` (replicas, deletion protection) |
| 71 | +- ✅ **Listing indexes** - `pc index list` |
| 72 | +- ✅ **Describing indexes** - `pc index describe` |
| 73 | +- ✅ **Creating API keys** - `pc api-key create` |
| 74 | +- ✅ **One-off inspection** - Checking stats, configuration |
| 75 | +- ✅ **Automated deployment pipelines** - All initial infrastructure setup |
| 76 | + |
| 77 | +**Use the SDK for application code:** |
| 78 | + |
| 79 | +- ✅ **Ensuring index existence and correctness** - Creating/updating indexes as part of application startup |
| 80 | +- ✅ **Dynamic index management** - based on application's logic and requirements |
| 81 | +- ✅ **Vector operations** - upsert, query, search, delete vectors |
| 82 | +- ✅ **Records operations** - upsert, query, search, delete RECORDS (automatic embeddings generation) |
| 83 | +- ✅ **Other services** - explicit embeddings generation, reranking, etc. |
| 84 | +- ✅ **Unit and integration tests** |
| 85 | + |
| 86 | +## CLI Setup and Usage |
| 87 | + |
| 88 | +For detailed CLI installation, authentication, and command reference, see [PINECONE-cli.md](./PINECONE-cli.md). |
| 89 | + |
| 90 | +## Index Creation |
| 91 | + |
| 92 | +> **⚠️ Use CLI (`pc index create`), NOT SDK in application code. See [PINECONE-cli.md](./PINECONE-cli.md) for detailed commands.** |
| 93 | +
|
| 94 | +### Available embedding models (current) |
| 95 | + |
| 96 | +- `llama-text-embed-v2`: High-performance, configurable dimensions, recommended for most use cases |
| 97 | +- `multilingual-e5-large`: For multilingual content, 1024 dimensions |
| 98 | +- `pinecone-sparse-english-v0`: For keyword/hybrid search scenarios |
| 99 | + |
| 100 | +## Data Operations |
| 101 | + |
| 102 | +### Upserting records (text with integrated embeddings) |
| 103 | + |
| 104 | +**Always use namespaces for data isolation:** |
| 105 | + |
| 106 | +- Multi-user apps: `user_123` |
| 107 | +- Session-based: `session_456` |
| 108 | +- Content-based: `knowledge_base`, `chat_history` |
| 109 | + |
| 110 | +### Updating records |
| 111 | + |
| 112 | +Use the same upsert operation with existing IDs. Only changed fields need to be included for partial updates. |
| 113 | + |
| 114 | +### Fetching records |
| 115 | + |
| 116 | +Use the fetch method with namespace and record IDs. Always handle errors gracefully. |
| 117 | + |
| 118 | +### Listing record IDs |
| 119 | + |
| 120 | +Use paginated listing with optional prefix filters for efficient ID retrieval. |
| 121 | + |
| 122 | +## Search Operations |
| 123 | + |
| 124 | +### Semantic search with reranking (best practice) |
| 125 | + |
| 126 | +**Always rerank for production quality:** |
| 127 | + |
| 128 | +- Get 2x candidates initially |
| 129 | +- Rerank with `bge-reranker-v2-m3` model |
| 130 | +- Return final count |
| 131 | + |
| 132 | +### Lexical search (keyword-based) |
| 133 | + |
| 134 | +Use for exact keyword matching with optional required terms and reranking. |
| 135 | + |
| 136 | +### Metadata filtering |
| 137 | + |
| 138 | +**Supported filter operators:** |
| 139 | + |
| 140 | +- `$eq`: equals |
| 141 | +- `$ne`: not equals |
| 142 | +- `$gt`, `$gte`: greater than, greater than or equal |
| 143 | +- `$lt`, `$lte`: less than, less than or equal |
| 144 | +- `$in`: in list |
| 145 | +- `$nin`: not in list |
| 146 | +- `$exists`: field exists |
| 147 | +- `$and`, `$or`: logical operators |
| 148 | + |
| 149 | +## 🚨 Common Mistakes (Must Avoid) |
| 150 | + |
| 151 | +### 1. **Nested Metadata** (will cause API errors) |
| 152 | + |
| 153 | +- ❌ Nested objects not allowed |
| 154 | +- ✅ Flat structure only |
| 155 | +- ✅ String lists are OK |
| 156 | + |
| 157 | +### 2. **Batch Size Limits** (will cause API errors) |
| 158 | + |
| 159 | +- Text records: MAX 96 per batch, 2MB total |
| 160 | +- Vector records: MAX 1000 per batch, 2MB total |
| 161 | + |
| 162 | +### 3. **Missing Namespaces** (causes data isolation issues) |
| 163 | + |
| 164 | +- ❌ No namespace |
| 165 | +- ✅ Always use namespaces |
| 166 | + |
| 167 | +### 4. **Skipping Reranking** (reduces search quality) |
| 168 | + |
| 169 | +- ⚠️ OK but not optimal |
| 170 | +- ✅ Always rerank in production |
| 171 | + |
| 172 | +### 5. **Hardcoded API Keys** |
| 173 | + |
| 174 | +- ❌ Hardcoded keys |
| 175 | +- ✅ Use environment variables |
| 176 | + |
| 177 | +## Key Constraints |
| 178 | + |
| 179 | +| Constraint | Limit | Notes | |
| 180 | +| ------------------- | ------------------------------------------ | --------------------------------- | |
| 181 | +| Metadata per record | 40KB | Flat JSON only, no nested objects | |
| 182 | +| Text batch size | 96 records | Also 2MB total per batch | |
| 183 | +| Vector batch size | 1000 records | Also 2MB total per batch | |
| 184 | +| Query response size | 4MB | Per query response | |
| 185 | +| Metadata types | strings, ints, floats, bools, string lists | No nested structures | |
| 186 | +| Consistency | Eventually consistent | Wait ~1-5s after upsert | |
| 187 | + |
| 188 | +## Error Handling (Production) |
| 189 | + |
| 190 | +### Error Types |
| 191 | + |
| 192 | +- **4xx (client errors)**: Fix your request - DON'T retry (except 429) |
| 193 | +- **429 (rate limit)**: Retry with exponential backoff |
| 194 | +- **5xx (server errors)**: Retry with exponential backoff |
| 195 | + |
| 196 | +### Retry Pattern |
| 197 | + |
| 198 | +Implement exponential backoff with max retries for transient errors only. |
| 199 | + |
| 200 | +## Use Cases |
| 201 | + |
| 202 | +### Search |
| 203 | + |
| 204 | +Build a semantic search system that returns ranked results from your knowledge base. Ideal for search interfaces where users need relevant documents with confidence scores. |
| 205 | + |
| 206 | +### RAG |
| 207 | + |
| 208 | +Build a multi-tenant RAG (Retrieval-Augmented Generation) system that retrieves relevant context per tenant and feeds it to an LLM. Each tenant has isolated data stored in separate Pinecone namespaces. |
| 209 | + |
| 210 | +### Recommendations |
| 211 | + |
| 212 | +Build a recommendation engine that suggests similar items based on semantic similarity. Ideal for e-commerce, content platforms, and user personalization systems. |
| 213 | + |
| 214 | +## Quick Troubleshooting |
| 215 | + |
| 216 | +| Issue | Solution | |
| 217 | +| ------------------------------------ | ------------------------------------------------------------ | |
| 218 | +| `ModuleNotFoundError: pinecone.grpc` | Wrong SDK - reinstall with correct package | |
| 219 | +| `Metadata too large` error | Check 40KB limit, flatten nested objects | |
| 220 | +| `Batch too large` error | Reduce to 96 records (text) or 1000 (vectors) | |
| 221 | +| Search returns no results | Check namespace, wait for indexing (~5s), verify data exists | |
| 222 | +| Rate limit (429) errors | Implement exponential backoff, reduce request rate | |
| 223 | +| Nested metadata error | Flatten all metadata - no nested objects allowed | |
| 224 | + |
| 225 | +## Official Documentation Resources |
| 226 | + |
| 227 | +For advanced features not covered in this quick reference: |
| 228 | + |
| 229 | +- **API reference**: [https://docs.pinecone.io/reference/api/introduction](https://docs.pinecone.io/reference/api/introduction) |
| 230 | +- **Bulk imports** (S3/GCS): [https://docs.pinecone.io/guides/index-data/import-data](https://docs.pinecone.io/guides/index-data/import-data) |
| 231 | +- **Hybrid search**: [https://docs.pinecone.io/guides/search/hybrid-search](https://docs.pinecone.io/guides/search/hybrid-search) |
| 232 | +- **Error handling**: [https://docs.pinecone.io/guides/production/error-handling](https://docs.pinecone.io/guides/production/error-handling) |
| 233 | +- **Database limits**: [https://docs.pinecone.io/reference/api/database-limits](https://docs.pinecone.io/reference/api/database-limits) |
| 234 | + |
| 235 | +**Remember**: Always use namespaces, always rerank, always handle errors with retry logic. |
0 commit comments