Confirm that kepler fixtures are required for initial database population

[waltronix]

Before deleting CVE data from the repository, check if it is downloaded during deployment.

[dommyrock]

Changes were introduced here: (switch to new 2.0 api + /data .gz sources updated)
#40

[dommyrock]

Explanation

When building the kepler image in the scope of the docker-compose.yaml there is this section in which we bind local filesystem /data directory into a container.

 volumes:
      - ./data:/data:Z

Until now this /data directory was pre-populated with .gz files, so that we save time for dev-setup.

When we remove the .gz files and we only bind the empty /data directory, NIST source download step will be required.

This works and its tested bellow.
Even without binding .gz files as part of /data volume inside the kepler container.

This means that we can freely remove the contents of this /data folder to reduce initial image size when being deployed.

Deployed version will likely have to update those sources quite often, since they are being updated quite often (see picture in the end),

Steps taken when testing

Test 1 - Normal import

# Destroy previous volumes 
podman-compose down -v

# Re-build a new image
podman compose build 

# Spin up a new kepler + kepler_db cluster   
podman-compose up

# Run the import task
for year in $(seq 2002 2025); do
  podman exec -it kepler kepler import_nist $year -d /data
done

[Note]

• make sure you removed the old /data and only have v2.0
• when you start kepler it doesn't auto populate the db from .gz untill you call the import import_nist command

2025 log output example

[2025-09-15T09:17:46Z INFO  domain_db::cve_sources::nist] reading /data/nvdcve-2.0-2025.json ...
[2025-09-15T09:17:46Z INFO  domain_db::cve_sources::nist] loaded 11536 CVEs in 351.54686ms
[2025-09-15T09:17:46Z INFO  kepler] connected to database, importing records ...
[2025-09-15T09:17:46Z INFO  kepler] configured 'KEPLER__BATCH_SIZE' 5000
[2025-09-15T09:17:46Z INFO  kepler] 11536 CVEs pending import
[2025-09-15T09:17:47Z INFO  domain_db::db] batch imported 5000 object records ...
[2025-09-15T09:17:47Z INFO  domain_db::db] batch imported 5000 object records ...
[2025-09-15T09:17:47Z INFO  domain_db::db] batch imported 1536 object records ...
[2025-09-15T09:17:48Z INFO  kepler] batch imported 5000 cves ...
[2025-09-15T09:17:48Z INFO  kepler] batch imported 10000 cves ...
[2025-09-15T09:17:48Z INFO  kepler] batch imported 15000 cves ...
[2025-09-15T09:17:48Z INFO  kepler] batch imported 20000 cves ...
[2025-09-15T09:17:48Z INFO  kepler] batch imported 25000 cves ...
[2025-09-15T09:17:48Z INFO  kepler] batch imported 30000 cves ...
[2025-09-15T09:17:49Z INFO  kepler] batch imported 35000 cves ...
[2025-09-15T09:17:49Z INFO  kepler] imported 37592 records Total
[2025-09-15T09:17:49Z INFO  kepler] 37592 new records created

Test 2 - Delete /data and try to re-import

Steps:

# Delete all `.gz` files from `/data`

# Destroy the existing volume where we bound populated `/data`. 
podman-compose down -v

# Build a new image with an empty `/data` mount.
poodman compose build

# Re trigger import (this time we will have to download all the years `.gz` fists and than kepler can proceed with `.json` extraction and db import step.  

for year in $(seq 2002 2025); do
  podman exec -it kepler kepler import_nist $year -d /data
done

Example output:

Notice "downloading" extra step here compared to normal import when we had /data populated.

 for year in $(seq 2002 2025); do   podman exec -it kepler kepler import_nist $year -d /data; done

[2025-09-15T09:20:59Z INFO  domain_db::cve_sources] downloading https://nvd.nist.gov/feeds/json/cve/2.0/nvdcve-2.0-2002.json.gz to /data/nvdcve-2.0-2002.json.gz ...
[2025-09-15T09:21:00Z INFO  domain_db::cve_sources::nist] extracting /data/nvdcve-2.0-2002.json.gz to /data/nvdcve-2.0-2002.json ...
[2025-09-15T09:21:00Z INFO  domain_db::cve_sources::nist] reading /data/nvdcve-2.0-2002.json ...
[2025-09-15T09:21:00Z INFO  domain_db::cve_sources::nist] loaded 6546 CVEs in 92.942702ms
[2025-09-15T09:21:00Z INFO  kepler] connected to database, importing records ...
[2025-09-15T09:21:00Z INFO  kepler] configured 'KEPLER__BATCH_SIZE' 5000
[2025-09-15T09:21:00Z INFO  kepler] 6546 CVEs pending import
[2025-09-15T09:21:01Z INFO  domain_db::db] batch imported 5000 object records ...
[2025-09-15T09:21:01Z INFO  domain_db::db] batch imported 1546 object records ...
[2025-09-15T09:21:01Z INFO  kepler] batch imported 5000 cves ...
[2025-09-15T09:21:01Z INFO  kepler] imported 9159 records Total
[2025-09-15T09:21:01Z INFO  kepler] 9159 new records created
[2025-09-15T09:21:01Z INFO  domain_db::cve_sources] downloading https://nvd.nist.gov/feeds/json/cve/2.0/nvdcve-2.0-2003.json.gz to /data/nvdcve-2.0-2003.json.gz ...
[2025-09-15T09:21:02Z INFO  domain_db::cve_sources::nist] extracting /data/nvdcve-2.0-2003.json.gz to /data/nvdcve-2.0-2003.json ...

Conclusion:

As you can see from the logs , even when you don't have the .qz cve files in the /data locally (or part of the built image) kepler still tries to download them before importing to the kepler-db .

So both cases should work:

• when there is no /data(takes alot longer since latest years are quite big to download)
• when there is /data populated we just extract .json and populate kepler-db

Proof that NIST does frequent CVE source updates

Since we last merged these .gz files in the repo (a week ago), there is already quite a few checksum diffs for years 2010 - 2025.

This is just a reminder that these sources change quite often , therefore we should be aware of them.
We also have AWS fargate service (ecs) running on a chron schedule to keep those sources updated.

Kepler doesn't re-insert the same objects or cve entries multiple times

This is because we have constraints on cve field for object table.

Source

kepler/domain-db/src/db/mod.rs

Line 105 in 72cfcbd

pub fn batch_insert_objects(

This is because we have constrants on cve, vendor, product fields on 'cves' table.

Source
https://github.com/exein-io/kepler/blob/72cfcbdee1f02899fc7e482b7f77cd6b4972bf6d/domain-db/src/db/mod.rs#L141irep

[dommyrock]

Note:

• Might need to update Kepler docs to make them more clear for the other teams in the future.