Rustls failure on network startup

[akru]

Is there an existing issue?

• I have searched the existing issues

Experiencing problems? Have you tried our Stack Exchange first?

• This is not a support question.

Description of bug

Description

The problem rises after uplift to latest stable polkadot-sdk, node fails for live networks, --dev still works normal.

Configuration

• polkadot-sdk: stable2512-2
• polkadot-omni-node-lib: 0.13.0
• robonomics: https://github.com/airalab/robonomics/tree/release/4.1

Error log

Version: robonomics-ca86f71a059 :: polkadot omni node v1.21.0

   0: sp_panic_handler::set::{{closure}}
   1: std::panicking::rust_panic_with_hook
   2: std::panicking::begin_panic_handler::{{closure}}
   3: std::sys::backtrace::__rust_end_short_backtrace
   4: __rustc::rust_begin_unwind
   5: core::panicking::panic_fmt
   6: core::option::expect_failed
   7: rustls::client::client_conn::ClientConfig::builder_with_protocol_versions
   8: rustls::client::client_conn::ClientConfig::builder
   9: tokio_tungstenite::tls::encryption::rustls::wrap_stream::{{closure}}
  10: <tokio::time::timeout::Timeout<T> as core::future::future::Future>::poll
  11: <litep2p::transport::websocket::WebSocketTransport as litep2p::transport::Transport>::open::{{closure}}::{{closure}}
  12: futures_util::stream::stream::StreamExt::poll_next_unpin
  13: <futures_util::abortable::Abortable<Fut> as core::future::future::Future>::poll
  14: <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll
  15: <futures_util::future::try_future::UnwrapOrElse<Fut,F> as core::future::future::Future>::poll
  16: <futures_util::stream::futures_unordered::FuturesUnordered<Fut> as futures_core::stream::Stream>::poll_next
  17: <litep2p::transport::websocket::WebSocketTransport as futures_core::stream::Stream>::poll_next
  18: <litep2p::transport::manager::TransportContext as futures_core::stream::Stream>::poll_next
  19: litep2p::transport::manager::TransportManager::next::{{closure}}
  20: <core::future::poll_fn::PollFn<F> as core::future::future::Future>::poll
  21: <sc_network::litep2p::Litep2pNetworkBackend as sc_network::service::traits::NetworkBackend<B,H>>::run::{{closure}}
  22: sc_service::build_network_future::{{closure}}::{{closure}}::{{closure}}
  23: <futures_util::future::poll_fn::PollFn<F> as core::future::future::Future>::poll
  24: <sc_service::task_manager::prometheus_future::PrometheusFuture<T> as core::future::future::Future>::poll
  25: <futures_util::future::select::Select<A,B> as core::future::future::Future>::poll
  26: <tracing_futures::Instrumented<T> as core::future::future::Future>::poll
  27: tokio::runtime::park::CachedParkThread::block_on
  28: tokio::runtime::handle::Handle::block_on
  29: <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll
  30: tokio::runtime::task::core::Core<T,S>::poll
  31: tokio::runtime::task::harness::Harness<T,S>::poll
  32: tokio::runtime::blocking::pool::Inner::run
  33: std::sys::backtrace::__rust_begin_short_backtrace
  34: core::ops::function::FnOnce::call_once{{vtable.shim}}
  35: std::sys::pal::unix::thread::Thread::new::thread_start
  36: start_thread
  37: __GI___clone3


Thread 'tokio-runtime-worker' panicked at '
Could not automatically determine the process-level CryptoProvider from Rustls crate features.
Call CryptoProvider::install_default() before this point to select a provider manually, or make sure exactly one of the 'aws-lc-rs' and 'ring' features is enabled.
See the documentation of the CryptoProvider type for more information.
            ', /home/akru/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/rustls-0.23.36/src/crypto/mod.rs:249

Steps to reproduce

No response

[akru]

Btw, set network-backend to libp2p on relay and collator fixes it, but it looks like default backend not works properly.

[bkchr]

@akru have you done this?

Call CryptoProvider::install_default() before this point to select a provider manually, or make sure exactly one of the 'aws-lc-rs' and 'ring' features is enabled.

CC @lexnv

[akru]

@bkchr I belive it's somewhere in omni-node-lib internals, unfortunately, I can't change anything without polkadot fork. All I do is use polkadot_omni_node_lib here: https://github.com/airalab/robonomics/blob/release/4.1/bin/src/main.rs

[bkchr]

@akru it tells you that you should enable these features. So, you add the rustls crate to your downstream node and enable the features. Please try this.

[akru]

Unfortunately, nothing changed.

[lexnv]

@akru This is the fix to unblock your chain:

diff --git a/Cargo.lock b/Cargo.lock
index e92dc6e..e8c8b38 100644
--- a/Cargo.lock
+++ b/Cargo.lock
@@ -11036,6 +11036,7 @@ dependencies = [
  "color-eyre",
  "polkadot-omni-node-lib",
  "robonomics-runtime",
+ "rustls 0.23.37",
  "sp-genesis-builder 0.21.0",
  "substrate-build-script-utils",
 ]
diff --git a/Cargo.toml b/Cargo.toml
index 00c2b32..543264c 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -33,6 +33,9 @@ members = [
 resolver = "2"

 [workspace.dependencies]
+
+rustls = { version = "0.23", default-features = false, features = ["ring"] }
+
 # General deps
 # (wasm)
 bounded-collections = { version = "0.1.8", default-features = false }
diff --git a/bin/Cargo.toml b/bin/Cargo.toml
index 64b3cf5..52699c9 100644
--- a/bin/Cargo.toml
+++ b/bin/Cargo.toml
@@ -17,6 +17,7 @@ color-eyre.workspace = true
 polkadot-omni-node-lib.workspace = true
 robonomics-runtime.workspace = true
 robonomics-runtime.features = ["dev-runtime"]
+rustls.workspace = true
 sp-genesis-builder.workspace = true

 [dev-dependencies]
diff --git a/bin/src/main.rs b/bin/src/main.rs
index 9e2dc1c..b8c55e3 100644
--- a/bin/src/main.rs
+++ b/bin/src/main.rs
@@ -94,6 +94,10 @@ impl LoadSpec for RobonomicsChainSpecLoader {
 fn main() -> color_eyre::eyre::Result<()> {
     color_eyre::install()?;

+    // Install the ring CryptoProvider for rustls before any TLS connections are made.
+    // Ignore the error if a provider was already installed.
+    let _ = rustls::crypto::ring::default_provider().install_default();
+
     let config = RunConfig::new(
         Box::new(DefaultRuntimeResolver),
         Box::new(RobonomicsChainSpecLoader),

In the meanwhile, we ll prepare a proper fix for litep2p to use the default provider in the networking backend. Please do not panic on install_default (ie, the let _ is intentional since we expect the network backends to handle this)

[akru]

@lexnv Yes, works well, thank you!

[lexnv]

Glad to hear! Please feel free to reach out again if you have any other questions 🙏

In the meanwhile Im going to close this issue, the fix you added should be entirely compatible with the improvements we are going to merge soon on the networking backends. In other words, whenever you update your polkadot version you won't need to revert or handle anything else (although I would recommand using the next release version without this intermediate fix) 🙏