Skip to content

Improve NIST data source importing#35

Merged
dommyrock merged 6 commits intomainfrom
improve_nist_source_import
May 19, 2025
Merged

Improve NIST data source importing#35
dommyrock merged 6 commits intomainfrom
improve_nist_source_import

Conversation

@dommyrock
Copy link
Member

@dommyrock dommyrock commented May 15, 2025

Improve NIST data source importing

This primarily takes care of batch importing YEARLY CVE dataset imports from NIST .
Current BATCH_SIZE = 5000.

You can see a demo from importing and updating existing records in the ticket comments.

I have also touched on some minor details :

  • Added UNIQUE Constraint's to cves and objects tables. Stopping us from constantly having to ping the DB to check if CVE-ID and (cve-vendor-product) combination already exists.
  • Fixing compiler warnings for cargo.toml workspace missing workspace =resolver = "2" verssion.
  • moved some common crates to [workspace.dependencies]
  • updated few shared crate versions where it made sense to do so
  • update score,severity and vector values in db (they were just defaulting to type values before)
  • updated workspace to rust 2024 edition (which re-formatted some unrelated files)

I'm not sure where the final place for the BATCH_SIZE will be (maybe it makes sense to move to ENV Variable)
But for now i kept it in kepler/domain-db/src/db/mod.rs since this is where PostgresRepository is defined and where most of DB transactions are taking place.

I have

  • run cargo fmt;
  • run cargo clippy;
  • run cargo testand all tests pass;
  • linked to the originating issue (if applicable).

@dommyrock dommyrock requested review from aleksanderkrauze, banditopazzo, davidristov and dergecko and removed request for aleksanderkrauze May 15, 2025 14:20
@dommyrock dommyrock force-pushed the improve_nist_source_import branch from a1cefbe to 9266415 Compare May 15, 2025 14:26
@dommyrock dommyrock force-pushed the improve_nist_source_import branch from 9266415 to 837e49b Compare May 15, 2025 14:29
@dommyrock dommyrock linked an issue May 15, 2025 that may be closed by this pull request
@dommyrock dommyrock requested a review from krsh May 15, 2025 14:43
@dommyrock dommyrock force-pushed the improve_nist_source_import branch from e03c901 to 9373615 Compare May 15, 2025 19:12
Copy link
Contributor

@dergecko dergecko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

praise

Really like the comments and structure of the code.
Also there a lot iof tests which makes it more maintainable in the future.

@dommyrock
Copy link
Member Author

dommyrock commented May 17, 2025

I think I covered all of your suggestions .

  • I have also updated rust edition to 2024 (I remembered we mentioned it for other project )
    Thanks for the detailed review btw : )

@dommyrock dommyrock force-pushed the improve_nist_source_import branch 2 times, most recently from 18f036d to 65bfeb6 Compare May 17, 2025 20:40
@dommyrock dommyrock force-pushed the improve_nist_source_import branch 2 times, most recently from bee05f8 to 648d1c1 Compare May 17, 2025 21:01
@dommyrock dommyrock force-pushed the improve_nist_source_import branch from 648d1c1 to 1c9f9a0 Compare May 17, 2025 21:03
Copy link
Contributor

@dergecko dergecko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice really like the comments.
Found some minor typos and one memroy related issue.

@dommyrock dommyrock force-pushed the improve_nist_source_import branch from 08f50ea to 79bf690 Compare May 19, 2025 12:38
Copy link
Contributor

@dergecko dergecko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me

@dommyrock dommyrock merged commit 28d7b8b into main May 19, 2025
6 checks passed
@dommyrock dommyrock deleted the improve_nist_source_import branch May 19, 2025 12:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Improve data source importing

2 participants