Skip to content

README_Inventory_components.md

Latest commit

 

History

History
492 lines (367 loc) · 12.4 KB

README_Inventory_components.md

File metadata and controls

492 lines (367 loc) · 12.4 KB

Network Inventory Component Tracking

Automated hardware component extraction and database population from device inventory captures

Overview

The inventory loader parses vendor-specific inventory command outputs using TextFSM templates and populates the components table in assets.db. This provides automated tracking of:

  • Chassis and switch stack members
  • Line cards and modules
  • Power supplies and fans
  • Network transceivers (SFPs/QSFPs)
  • Serial numbers and part identifiers

Current Deployment Stats:

  • 147 devices with inventory data
  • 2,165 hardware components tracked
  • 100% parsing success rate across Cisco IOS/NX-OS and Arista EOS
  • Multi-vendor template support with strict validation

Architecture

Components

inventory_loader.py - Primary parsing engine

  • Reads inventory captures from device_captures_current table
  • Uses TextFSM templates for vendor-specific parsing
  • Populates components table with normalized data
  • Template-only approach with explicit failure reporting

component_cleanup.py - Database cleanup utility

  • Removes component records without full database reset
  • Supports selective cleanup by device, vendor, or source
  • Statistics reporting for loaded components

Database Schema - Components table structure:

CREATE TABLE components (
    id INTEGER PRIMARY KEY,
    device_id INTEGER NOT NULL,
    name TEXT NOT NULL,              -- Component identifier
    description TEXT,                -- Detailed description
    serial TEXT,                     -- Serial number
    position TEXT,                   -- Slot/port position
    have_sn BOOLEAN DEFAULT 0,       -- Serial number present flag
    type TEXT,                       -- chassis/module/psu/fan/transceiver
    subtype TEXT,                    -- Additional classification
    extraction_source TEXT,          -- 'inventory_capture'
    extraction_confidence REAL,      -- Template match score
    FOREIGN KEY (device_id) REFERENCES devices(id)
);

Supported Vendors

Cisco IOS/IOS-XE

Template: cisco_ios_show_inventory

Command: show inventory

Fields Extracted:

  • NAME - Component name/identifier
  • DESCR - Description
  • PID - Product ID (model number)
  • VID - Version ID
  • SN - Serial number

Sample Output:

NAME: "Switch 1", DESCR: "WS-C3850-48P"
PID: WS-C3850-48P, VID: V07, SN: FCW1234A5BC

NAME: "Power Supply 1", DESCR: "FRU Power Supply"
PID: C3KX-PWR-715WAC, VID: V02, SN: LIT1234A5BC

Components Tracked:

  • Stack members (Switch 1, Switch 2, etc.)
  • Modules and line cards
  • Power supplies
  • Fans
  • Stack cables
  • Transceivers

Cisco NX-OS

Template: cisco_nxos_show_inventory

Command: show inventory

Fields Extracted: Same as IOS (NAME, DESCR, PID, VID, SN)

Differences from IOS:

  • Different naming conventions for modules
  • Fabric extenders (FEX) tracked separately
  • Supervisor modules explicitly identified

Arista EOS

Template: arista_eos_show_inventory

Command: show inventory

Fields Extracted:

  • PORT - Port/slot identifier
  • NAME - Component model name
  • SN - Serial number
  • DESCR - Description
  • VID - Hardware version

Sample Output:

System information
  Model: DCS-7050SX3-48YC8
  Serial number: JMX2333A0LW
  Hardware revision: 12.21

System (power supply 1)
  SN: ABC1234567

System (power supply 2)
  SN: ABC1234568

Special Handling:

  • Chassis info in first record (NAME field populated)
  • Power supplies have empty NAME, use DESCR
  • Transceivers have empty NAME, use DESCR
  • Fans typically don't have serial numbers

Workflow

1. Inventory Capture

Captures must exist in device_captures_current table with capture_type = 'inventory':

# Captured via batch_spn_concurrent.py
python batch_spn_concurrent.py sessions.yaml \
    --fingerprinted-only \
    --capture-types "inventory" \
    --max-processes 8

2. Component Extraction

# Process all inventory captures
python inventory_loader.py

# Process with debug output
python inventory_loader.py --debug

# Process specific devices
python inventory_loader.py --device-filter "core-switch"

# Limit processing for testing
python inventory_loader.py --max-files 10

Processing Flow:

  1. Query v_capture_details view for inventory captures
  2. Read capture file content
  3. Identify vendor from device metadata
  4. Select appropriate TextFSM templates
  5. Parse content with template scoring
  6. Map fields to component schema
  7. Determine component types
  8. Store in components table

3. Cleanup and Reprocessing

# View statistics
python component_cleanup.py --stats

# Clean all components
python component_cleanup.py --all --confirm

# Clean specific device
python component_cleanup.py --device-id 42

# Clean by device name pattern
python component_cleanup.py --device-name "leaf-switch"

# Clean by extraction source
python component_cleanup.py --source "inventory_capture" --confirm

4. Verification

-- Component counts by device
SELECT 
    d.name,
    v.name as vendor,
    COUNT(c.id) as component_count,
    COUNT(CASE WHEN c.have_sn = 1 THEN 1 END) as with_serial
FROM devices d
LEFT JOIN components c ON d.id = c.device_id
LEFT JOIN vendors v ON d.vendor_id = v.id
GROUP BY d.id
ORDER BY component_count DESC;

-- Components by type
SELECT 
    type,
    COUNT(*) as count,
    COUNT(CASE WHEN have_sn = 1 THEN 1 END) as with_serial
FROM components
GROUP BY type
ORDER BY count DESC;

-- Devices missing components
SELECT d.name, v.name as vendor
FROM devices d
LEFT JOIN vendors v ON d.vendor_id = v.id
WHERE NOT EXISTS (
    SELECT 1 FROM components WHERE device_id = d.id
);

Template Validation

Strict Matching Requirements

The loader uses template-only parsing with no regex fallback. This ensures:

  • Explicit failure when templates are inadequate
  • Clear identification of which vendors need work
  • No silent data corruption from pattern guessing

Minimum Template Score: 20 (configurable in InventoryLoader.MINIMUM_SCORE)

Failure Modes:

✗ REJECTED: Best score 15 < minimum 20. Template needs improvement for vendor 'HP'

When this occurs:

  1. Check if TextFSM template exists for vendor
  2. Review template field names and patterns
  3. Test template against sample output
  4. Adjust template or create new one

Field Mapping

The loader handles vendor variations in field naming:

field_mappings = {
    'name': ['NAME', 'name'],
    'description': ['DESCR', 'description', 'DESCRIPTION'],
    'serial': ['SN', 'serial', 'SERIAL_NUMBER'],
    'model': ['PID', 'model', 'MODEL'],
    'version': ['VID', 'version', 'VERSION'],
    'position': ['PORT', 'SLOT', 'position', 'POSITION']
}

Fallback Logic:

  • If name field empty, use description as name
  • Required: At least one of name or description
  • Optional: serial, model, version, position

Component Type Detection

Automatic classification based on keywords:

Type Keywords
chassis chassis, stack, "switch "
module module, linecard, "line card"
psu "power supply", psu, "power-supply"
fan fan, cooling
transceiver transceiver, sfp, qsfp, gbic
supervisor supervisor, sup, management
unknown (default if no match)

Performance Metrics

Processing Speed:

  • 147 devices in ~45 seconds
  • Average 3 seconds per device
  • Bulk processing recommended for initial load

Success Rates:

  • Cisco IOS: 100% (60+ devices)
  • Cisco NX-OS: 100% (10+ devices)
  • Arista EOS: 100% (35+ devices)
  • HP ProCurve: Requires template development
  • Other vendors: Requires template development

Component Extraction:

  • Average 15 components per device
  • Range: 4-40 components depending on device complexity
  • Serial number coverage: ~80% of components

Data Quality

Serial Number Tracking

The have_sn flag indicates serial number presence:

-- Devices with full serial coverage
SELECT d.name, COUNT(c.id) as total, 
       COUNT(CASE WHEN c.have_sn = 1 THEN 1 END) as with_sn
FROM devices d
JOIN components c ON d.id = c.device_id
GROUP BY d.id
HAVING COUNT(c.id) = COUNT(CASE WHEN c.have_sn = 1 THEN 1 END);

Common Missing Serials:

  • Fans (typically not serialized)
  • Stack cables (not always tracked)
  • Some transceivers (vendor dependent)

Extraction Confidence

The extraction_confidence field stores the TextFSM template match score:

-- Low confidence extractions
SELECT d.name, c.name, c.extraction_confidence
FROM components c
JOIN devices d ON c.device_id = d.id
WHERE c.extraction_confidence < 0.5
ORDER BY c.extraction_confidence;

Scores below 0.3 (30%) indicate potential template issues.

Integration Points

Device Management

Components automatically link to devices via device_id foreign key. Deleting a device cascades to components (if cascade is configured).

Web Dashboard

The Flask web interface displays components in device detail pages:

  • Tab showing component list
  • Serial numbers
  • Component types
  • Position/slot information

Change Detection

Future enhancement: Track component additions/removals over time by comparing snapshots.

Inventory Reconciliation

Compare extracted components against asset management systems for:

  • Missing devices
  • Serial number discrepancies
  • Hardware lifecycle tracking

Troubleshooting

No Components Extracted

Symptom: ✓ Loaded 0 components from file

Causes:

  1. Template score below minimum threshold
  2. Template field mapping mismatch
  3. Empty or malformed inventory output

Debug:

python inventory_loader.py --device-filter "problem-device" --debug

Look for:

  • Template match scores
  • Field extraction results
  • Mapping failures

Database Constraint Errors

Symptom: NOT NULL constraint failed: components.name

Cause: Both name and description fields are empty

Fix: Update template to capture at least one identifier field

Duplicate Components

Symptom: Multiple identical components for same device

Cause: Running loader multiple times without cleanup

Fix:

python component_cleanup.py --device-name "device" --confirm
python inventory_loader.py --device-filter "device"

Future Enhancements

Phase 2: Advanced Features

Stack Member Detail Extraction

  • Populate stack_members table from parsed inventory
  • Link stack member serials to component records
  • Track stack master/member roles

Module Position Tracking

  • Parse slot/port hierarchies
  • Track module insertion positions
  • Blade server chassis support

Component Lifecycle

  • Historical tracking of component changes
  • Addition/removal detection
  • Warranty and EOL correlation

Multi-Vendor Expansion

  • HP Comware templates
  • Juniper JunOS support
  • Fortinet inventory parsing
  • Palo Alto component tracking

Phase 3: Advanced Analytics

Component Search

  • Web interface for component lookup
  • Serial number search across devices
  • Model/PID filtering

Inventory Reports

  • Hardware asset summaries
  • Serial number exports
  • Compliance reporting

Predictive Maintenance

  • Component age tracking
  • Failure prediction based on model/age
  • Proactive replacement planning

Related Documentation

  • README_Pipeline.md - Data collection and capture workflow
  • README_Fingerprinting.md - TextFSM template development
  • README_Network_Mgmt_Flask.md - Web interface integration
  • README_DB_Loaders.md - Database loading patterns

Command Reference

# Full inventory load
python inventory_loader.py

# Debug mode
python inventory_loader.py --debug

# Limited processing
python inventory_loader.py --max-files 10

# Device filter
python inventory_loader.py --device-filter "cisco"

# Statistics
python component_cleanup.py --stats

# Full cleanup
python component_cleanup.py --all --confirm

# Selective cleanup
python component_cleanup.py --device-name "switch-1"
python component_cleanup.py --source "inventory_capture"

Status: Production ready for Cisco IOS/NX-OS and Arista EOS. Additional vendor support requires template development.