SOS - Serial Over SSH

There are times when a server becomes unreachable over SSH due to broken networking, a bad install, misconfiguration, a kernel upgrade, bad firewall rules, etc. Equinix Metal™ offers an out-of-band, Serial Over SSH (SOS) console.

Equinix Metal runs the SOS console application in each of our facilities. It listens for incoming network connections and allows authenticated connections administrative access to our servers, effectively acting as a "jump box" to your server.

You can disable the ability of SOS to access your server in the server's operating system, but in the event that you can't reach the OS, SOS access can not be restored.

SOS Console Authentication

Access to the SOS console requires that you have your server's root password and valid a SSH key.

If you do not have the root password, you will not be able to use the SOS console.

If you provisioned your server with an either your user account's SSH key or a Project SSH key, the SOS console will use those keys for authentication.

If you provisioned your server without SSH keys, you can add either a new user SSH key or a new Project SSH key, associate the SSH key with the server while uploading the SSH key, and the SOS console will use the new SSH key for authentication. You must associate the new SSH key with the server.

The SOS console has been updated to support most modern key types. If you do not have the root password or a valid SSH key, you should leverage our Rescue Mode.

The SOS Hostname

Console

In the Equinix Metal console, you can find the SSH command for SOS on the server's Overview page, by clicking on >_.

Copy the SSH command containing the SOS hostname.

API

To retrieve the SOS hostname from the API, send GET request to the /devices/{id} endpoint. The response contains an "sos" object.

curl -X GET -H 'X-Auth-Token: <API_TOKEN>' \
https://api.equinix.com/metal/v1/devices/{id}

Example Response (truncated):

{
"id": "ca4f6a80-d797-4a78-8c0f-a2b1edb3f7ae",
"hostname": "sv-m3-small-x86-02",
...
"sos": "ca4f6a80-d797-4a78-8c0f-a2b1edb3f7ae@sos.sv15.platformequinix.com",
}

Connecting to SOS

To access SOS, paste the provided command from the console into a terminal, or SSH to the hostname provided by the API from your local machine.

ssh <SOS_hostname>

If you have multiple keys locally, use the -i flag to specifically choose the correct key.

ssh <SOS_hostname>
-i </path/to/correct/key>

Once you have connected to the SOS console, you should be able to log in to the server.

[SOS Session Ready. Use ~? for help.]
[Note: You may need to press RETURN or Ctrl+L to get a prompt.]

sos-testing login: root
Password:

We don’t keep a persistent connection to the serial console on every server, so when you connect there won’t be any previous output. You will likely need to press enter/return before it asks for username and password.

Limitations

• This method allows only one connection at a time because servers only have a single serial console.
• If you try to connect to the console while the server is still provisioning, it will place you in a read-only mode for up to 30 seconds after the server has finished provisioning.
• If you don't see any output after connecting to the console, (common when using a custom image, operating system, or iPXE) please ensure that you have your console settings setup correctly. For x86 servers, use ttyS1,115200n8. For aarch64 servers, use ttyAMA0,115200.
• If you're connecting with PuTTY, use the SOS host as the hostname/IP in your PuTTY Session. For example, if your server is in provisioned in SV - SJC1, your SOS host is sos.sjc1.console.equinix.com. When prompted for the username, enter the server-uuid.

Using SOS with Windows

You can use the SOS console to access a server running Windows, but you have to enable Emergency Management Services (EMS) in Windows first. Enabling EMS is what allows the SOS console to access Windows when something goes wrong. More information and a brief how-to is in our Setting Up a Windows Server guide.

Disconnecting from SOS

First logout of the server.

root@sos-testing:~# exit
> logout

Then, in order to disconnect from the SOS console you need to enter ~. (tilde followed by period). SSH only accepts the escape character after a new line, so you may need to press enter/return first, followed by ~..

sos-testing login: ~.
> Connection to <SOS_hostname>closed.

This method of disconnection is built into SSH, not part of our console service. If you disconnected previously without logging out, it may automatically log you in on subsequent reconnects.