Rewrite BBOT's Web Engine in Rust?

[TheTechromancer]

HTTP performance is a big bottleneck in BBOT right now, as httpx is written in pure python and isn't very efficient. We need a low-level web library, a lot like ProjectDiscovery's httpx, that we can call from python.

Right now, Rust's PyO3 + Tokio is the best candidate language-wise.

Requirements:

• Response format compatible with httpx/requests
• Extremely fast
• Ability to execute batch requests (for dirbusting etc.)
• Async
• Should probably release the GIL (Global Interpreter Lock)

@stryker2k2
@TMDeal
@liquidsec

[TheTechromancer]

Waiting on #519

[SpamFaux]

This will be tested after #519 is completed. It isn't in stone if this is the way forward but will be considered.

[TheTechromancer]

2023-09-20: A small update to this, httpx now takes more than a second longer to start:

# time echo example.com | ./httpx 

    __    __  __       _  __
   / /_  / /_/ /_____ | |/ /
  / __ \/ __/ __/ __ \|   /
 / / / / /_/ /_/ /_/ /   |
/_/ /_/\__/\__/ .___/_/|_|
             /_/              v1.2.5

		projectdiscovery.io

https://example.com

real	0m0.364s

# time echo example.com | httpx

    __    __  __       _  __
   / /_  / /_/ /_____ | |/ /
  / __ \/ __/ __/ __ \|   /
 / / / / /_/ /_/ /_/ /   |
/_/ /_/\__/\__/ .___/_/|_|
             /_/

		projectdiscovery.io

[INF] Current httpx version v1.3.5 (latest)
https://example.com

real	0m1.602s

Lately it has been bogged down with features we'll never use such as web screenshot capability. Get it outta here!

[TheTechromancer]

Work has started on this:

#2447