A simple way to hide an SSH server behind TLS
If you host an SSH server on a public IPv4 address, then you’ll be greeted within minutes by an onslaught of password guessing attempts from bots. On a quiet system the resulting failed logins can easily dominate the logs, and with some configurations they can even interfere with authorized logins. One solution is to hide the SSH server alongside an HTTPS server on the same port, and in this post I’ll describe how I’m doing that.
A note about security
The technique below does not increase security. You should still apply the usual best practices like using keys, disabling password authentication, and staying up to date on security patches. What I’m about to describe is akin to moving from a busy city street to a quiet country road, but it’s orthogonal to what kind of lock you put on your door.
Concept
My goals include:
- Stop all the unauthorized login attempts. Not just 50%, or 90%, or some other number that will vary as bots tune their behavior.
- Allow authorized users to come from any IP address. I don’t want to be locked out while traveling.
- Do not require specialized client software. I sometimes connect from my work computer, and don’t want to install programs on it.
- (optionally) Support multiplexing to various backends.
Introducing ALPN
Application-Layer Protocol Negotiation is a TLS extension that enables a client to communicate the list of protocols that it supports, and enables the server to pick one. It was originally developed for HTTP/2 so that clients and servers could negotiate which HTTP version speak without adding any more roundtrips than were already inherent in the TLS handshake.
A “next” protocol negotiated by ALPN is identified by a variable length byte sequence, usually consisting of the protocol name as a UTF-8 string. Common values are registered with the IANA, but crucially, TLS libraries just treat them as opaque strings. An application can thus define its own values for private use.
We will be overloading the protocol name to identify both that the next protocol is SSH, and which backend host to connect to. Including the backend host enables multiplexing and makes it harder to probe for an SSH server given only the IP of the proxy.
Server
The server is a small Go binary:
package main
import (
"crypto/tls"
"fmt"
"net/http"
"strings"
"github.com/inetaf/tcpproxy"
)
func proxy(_ *http.Server, conn *tls.Conn, _ http.Handler) {
alpn := conn.ConnectionState().NegotiatedProtocol
host := strings.TrimPrefix(alpn, "ssh/")
p := &tcpproxy.DialProxy{
Addr: host + ":22",
}
p.HandleConn(conn)
}
func main() {
s := &http.Server{
Handler: http.RedirectHandler("https://blog.sigi.net", http.StatusFound),
TLSConfig: &tls.Config{
NextProtos: []string{
"ssh/avocado.example.com",
"ssh/banana.example.com",
"ssh/coconut.example.com",
},
},
TLSNextProto: map[string]func(*http.Server, *tls.Conn, http.Handler){
"ssh/avocado.example.com": proxy,
"ssh/banana.example.com": proxy,
"ssh/coconut.example.com": proxy,
},
}
err := s.ListenAndServeTLS("fullchain.pem", "privkey.pem")
if err != nil {
fmt.Println(err)
}
}
Going through this piece-by-piece:
- The
NextProtosslice defines the allowed set of protocols (and in our case, destination hosts) that the server will accept. - The
TLSNextProtomap specifies the function(s) that will proxy a connection to its appropriate backend. - The
proxyfunction does the heavy lifting of connecting to a backend and forwarding data in both directions. TCP proxies have enough subtleties to make a good homework assignment for graduate students, so we use the excellent tcpproxy as a library. - The
Handlerserves a simple redirect to any HTTP clients that didn’t negotiate an SSH destination. You could put a full blown web site here, or omit the line to serve a default 404. ListenAndServeTLSstarts the HTTPS server with a certificate obtained outside of this program (e.g. using certbot).
Client
The client consists of a few lines for each destination host added to your ~/.ssh/config:
Host avocado-proxied
Hostname avocado.example.com
ProxyCommand openssl s_client -connect ssh.example.com:443 -quiet -tls1_3 -alpn ssh/%h
When you ssh to avocado-proxied, the connection will be tunneled through the TLS server at ssh.example.com and ultimately to the SSH server avocado.example.com. The client uses OpenSSL, which is widely installed on Linux and Mac systems, so meets the requirement of not depending on custom software.
And that’s it! Simple!
Caveats
You will have noticed that we enumerate the allowed backends in the server config. The TLS implementation in the Go standard library imposes a constraint that these are provided in advance, but that’s not inherent to ALPN. The client sends its supported protocol(s) first, so in principle a server could choose a backend dynamically.
That said, I consider the explicit list of backends to be a positive feature because it prevents bad actors from abusing the proxy to probe arbitrary hosts. In my production deployment I added an additional indirection of using opaque tokens instead hostnames for the ALPN protocol IDs. These tokens are harder to guess than a backend hostname and can be rotated if desired, but they do need to be shared between the server and clients in advance.
The server in this example disables HTTP/2, because it does not provide a next protocol function for “h2”. This doesn’t matter in my environment because I’m running it behind an SNI proxy (which is another instance of tcpproxy). That first proxy forwards traffic to a web server (which supports HTTP/2) or to the SSH proxy (which doesn’t host any meaningful HTTP content) according to SNI hostname.
Alternatives considered
To conclude, let’s look at some other approaches to reducing the failed logins and why I didn’t choose those.
Use IPv6
I’m already doing this, and it’s a good solution today. The address space is sparsely populated by design, so it’s much harder to find responsive servers just by scanning IPs; I have not yet seen any password guessing attempts over IPv6. Various researchers have found some of my SSH servers however, and the techniques for doing so are continually improving. I expect we’ll start to see failed logins in due time.
I still routinely encounter IPv4-only networks when traveling, so it’s important to have a fallback for those cases. One dual-stack instance of the TLS proxy can give IPv4 clients access to many IPv6 servers.
Use alternative ports
Many online guides suggest moving SSH to a port other than the default 22. I’ve tried this before and it worked briefly, long enough to declare victory and call it a day. But come back later and the bots will have found the new port. It’s evidently not very hard to scan 65K ports and find which ones are presenting an SSH banner.
Use some form of IP-based filtering
Fail2Ban monitors system logs for failed login attempts and adjusts firewall rules to block the offending IPs. This will certainly slow bots down, but many of the login attempts are coming from cloud providers that make it easy to rotate through IPs, so it turns into an endless game of whac-a-mole.
It’s possible to block by country or reputation by consulting an IP database. The datasets required to do this are difficult to build, frought with erroneous entries, and often sold on a tiered pricing model where the free option is the least curated. In my opinion this can never be robust enough.
Share a port with plain HTTP
In HTTP the client speaks first, and in SSH the client and server both send their banner without waiting for the other side. It’s possible to share a port by having the server refrain from sending anything until it’s seen the first few bytes from the client and decided whether they look like an HTTP request or an SSH banner.
This requires no client software, not even OpenSSL, which is great! It doesn’t support multiplexing though, and it doesn’t gate acceptance of a connection on receiving anything unique (backend hostname or opaque token) from the client. I haven’t tried this so don’t know how effective it would be against the failed logins.