Rustls failed tls handshake on private server with self signed cert

server:
127.0.0.1:60588 connected!
received fatal alert: CertificateUnknown

client:
Error: invalid peer certificate: Other(OtherError(CaUsedAsEndEntity))

bash:
mkdir .keys/
cd .keys/

openssl req -newkey rsa:2048 -nodes -keyout private_key.pem -x509 -out cert.pem
openssl rsa -in private_key.pem -outform DER -out private_key.der
openssl x509 -in cert.pem -outform DER -out cert.der

rust server:
pub async fn bind(
        addr: &'static str,
        cert_path: &'static str,
        key_path: &'static str,
    ) -> Result<Self> {
        let cert: Vec<CertificateDer<'static>> = vec![read(cert_path)?.into()];

        let private_key_der: PrivatePkcs8KeyDer<'static> = read(key_path)?.into();

        let config = rustls::ServerConfig::builder()
            .with_no_client_auth()
            .with_single_cert(cert, PrivateKeyDer::Pkcs8(private_key_der))?;

        let acceptor = TlsAcceptor::from(Arc::new(config));

        let listener = TcpListener::bind(&addr).await?;

        Ok(TlsListener { acceptor, listener })
    }

rust client:
pub fn connect(addr: &'static str, cert: &'static [u8]) -> Result<Self> {
        let socket_addr: SocketAddr = addr.parse()?;

        let cert: CertificateDer<'static> = cert.into();

        let mut root_store = RootCertStore::empty();
        root_store.add(cert)?;

        let config = rustls::ClientConfig::builder()
            .with_root_certificates(root_store)
            .with_no_client_auth();

        let server_name = ServerName::IpAddress(socket_addr.ip().into());

        let conn = rustls::ClientConnection::new(Arc::new(config), server_name)?;
        let sock = TcpStream::connect(addr)?;
        let tls = rustls::StreamOwned::new(conn, sock);
        Ok(TlsStream { stream: tls })
    }

rustls doesn't support use of self-signed CA certificates as leafs. You can use self-signed certificates that aren't CA's though.

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.