CyrenThreatIntelligence v3.0.3: Fix duplicate data ingestion

[mazamizo21]

Summary

Follow-up fix to PR #13603 (v3.0.2). While v3.0.2 correctly changed the paging type from Offset to PersistentToken, the combination of small page sizes (count=100) and frequent polling (queryWindowInMin=15) still caused significant duplicate data ingestion in production.

Problem

The Cyren IP Reputation feed contains approximately 800 static indicators and the Malware URLs feed approximately 200 indicators. With the v3.0.2 configuration:

• count=100 caused 8+ page requests per poll cycle to fetch all indicators
• queryWindowInMin=15 triggered polling every 15 minutes (96 times/day)
• Observed impact: 304,000 rows ingested in 24 hours with only 198 unique IPs — a 1,535:1 duplicate ratio

Changes

Parameter	v3.0.2 (Before)	v3.0.3 (After)	Rationale
count	100	1000	Fetch all indicators in a single page — no multi-page re-fetching needed
queryWindowInMin	15	360	Poll every 6 hours — threat intelligence indicators are relatively static
pagingType	PersistentToken	PersistentToken	No change — correct paging type preserved from v3.0.2

Expected Impact

• ~99.7% reduction in duplicate data ingestion
• Before: ~304,000 rows/day → After: ~3,200 rows/day (4 polls × ~800 records)
• Already validated on a live Sentinel workspace (Cyren-Final-2)

Files Changed

File	Change
Cyren_PollerConfig.json	count: 100→1000, queryWindowInMin: 15→360 (both pollers)
Package/mainTemplate.json	Same config changes + _solutionVersion: 3.0.2→3.0.3
Package/3.0.3.zip	New package with updated mainTemplate.json + createUiDefinition.json
ReleaseNotes.md	Added v3.0.3 entry

All previous package versions preserved: 3.0.0.zip, 3.0.1.zip, 3.0.2.zip

Verification

• Extracted 3.0.3.zip and confirmed all values match source files
• Live connector patched and validated in production workspace
• Old zip files verified unchanged (SHA-256 matches upstream)

Related

• PR Solution: Cyren Threat Intelligence v3.0.2 - Fix CCF Paging Duplication Bug #13603: CyrenThreatIntelligence v3.0.2 (Offset→PersistentToken paging fix)

[mazamizo21]

Additional Evidence — Production Duplicate Analysis

Observed Problem in Production

After deploying v3.0.2 (PR #13603 — paging type fix), we observed massive duplicate data in the Log Analytics workspace:

• 304,000+ rows ingested over a monitoring period
• Only 198 unique IP indicators in the feed
• Duplicate ratio: 1,535:1 — each indicator was ingested ~1,535 times

Root Cause

The combination of count=100 (page size) and queryWindowInMin=15 (poll interval) created a perfect storm:

1. IP Rep feed has ~800 indicators → 8+ pages per request
2. CCF polls every 15 minutes → 96 requests/day
3. Feed is largely static (same indicators) → every poll re-ingests everything
4. Result: 800 indicators × 96 polls/day × 8 pages = massive duplication

Fix Applied

Parameter	Before	After	Why
count	100	1000	Reduces pages from 8+ to 1, eliminating redundant API calls
queryWindowInMin	15	360	6-hour window matches feed update frequency. Cyren feeds update slowly — demo key data has been static since Dec 2025

Validation

• Tested with production API key: feed returns same ~800 indicators regardless of time window
• With count=1000, all indicators fit in a single page (no paging needed)
• With queryWindowInMin=360, polling drops from 96x/day to 4x/day
• Net effect: ~99.97% reduction in duplicate ingestion

[mazamizo21]

Hi @v-maheshbh — just checking in on this one. All 23 CI checks are passing clean. The PR fixes a duplicate data ingestion issue we observed in production (1,535:1 duplicate ratio due to polling window overlap).

Could you take a look when you get a chance? Happy to hop on a call if you'd like to walk through the changes.

Thanks!