Skip to main content

Event Streams

!!! Info "Notice"

Equinix Metal Observability features are currently in Beta and not available to all customers. This is intended for evaluation and testing only. This feature is currently non-billed and not covered by an SLA.

You can monitor specific event and activity data from your Metal organizations in near real-time using event streams. Event streams can provide critical event data to help you understand and analyze your business needs. These events include:

  • Account and Organization Events
  • Services and Billing events
  • Device Management Events
  • Networking Management Events

Event Streams are only visible to Owners and Admins for an Organization.

Limitations​

Currently, event streams supports only Splunk as the destination. Observability Stream data is sent to Splunk through the Splunk HTTP Event Collector ("HEC") interface. In order to receive Observability Stream data, you must first configure an HTTP Event Collector in your Splunk instance. We recommend creating a dedicated collector specifically for receiving Observability Stream data.

Sending Observability Stream data to Splunk is subject to the following limitations:

  • To ensure encryption of data in transit, only HTTPS connections are allowed. Sending data to a Splunk HEC using plain, unsecured HTTP is NOT supported.
  • All Observability Stream data is sent to Splunk as JSON events; raw data is NOT supported.
  • JSON events will be sent to the standard /services/collector/event endpoint; alternative endpoint URLs are NOT supported.
  • JSON events will be sent to the default index that was configured when creating the Splunk HTTP Event Collector. Overriding the index of events sent to Splunk is NOT supported.
  • Indexer acknowledgement is NOT supported.

The Observability API​

If you are using the Observability API, you should note some differences between it and the current Equinix Metal API.

  • new API endpoints
  • new Authentication mechanism

First, Observability has its own API endpoints that can be reached at https://observability.equinixmetal.net.

Second, the Observability API has a authentication mechanism where you exchange your Equinix Metal API key for a short-lived JSON web token (JWT) that you use for authentication for all observability requests. The token expires after 5 minutes. To retrieve a JWT, send a POST request to the iam.metalctrl.io/api-keys/exchange endpoint.

curl -X POST \
-H "Authorization: bearer <API_TOKEN>" \
https://iam.metalctrl.io/api-keys/exchange

The response will be a JSON "access_token" which you use to authenticate your requests to the Observability API.

{
"access_token": "eyJ....98"
}

The Observability API does not accept Equinix Metal API keys for authentication.

Creating an Event Stream​

Create an event stream to export server data to Splunk for storage and analysis. To create a stream, do the following:

  1. Go to Equinix Metal Portal and enter your credentials to sign in to the portal.

  2. From the Organization's drop-down menu, select the organization to create an event stream.

  3. Click the Observability tab.

    Event Streams page from the Observability tab

  4. Click, Create Event Stream.

  5. In the Create Event Stream modal, enter a user-friendly name in the Name field.

  6. Optional. Enter a description for the event stream in the description field.

  7. Enter your Splunk hostname and API key in the fields provided to send data to Splunk.

    The connection hostname should be taken from the HTTP Event Collector URI. The hostname should be specified as a string with an optional port-number suffix; valid values include http-inputs-my-org.splunkcloud.com or splunk.example.com:8443.

    The API Key should be the HTTP Event Collector token that was generated when creating the Splunk HEC endpoint. This will be stored in an encrypted format that can only be decrypted by the worker sending Observability Stream data.

Testing the Stream Connection​

Before you stream a data connection to Splunk, you must test it to ensure its validity. To test the stream connection, do the following:

  1. In the Create Event Stream modal, enter your Splunk hostname and API key in the fields provided, if you have not already done so.

  2. Click, Test Stream Connection.

    The connection is valid if a Test Passed message appears on the screen. An error message displays if the test connection does NOT pass. You must receive a Test Passed message to ensure that the test connection is a valid to stream data to Splunk.

    Testing an event stream in the console

  3. Click Save to save the event stream. The Event stream is saved in the Event Streams table on the Observability page.

Managing an Event Stream​

Once the event stream is created in the Metal console, you can manage the event stream from your Organization's Observability tab.

Event Streams Table

Deleting an Event Stream​

To stop the data stream to your Splunk instance, you can delete the event stream. To delete an event stream, do the following:

  1. In the Event Streams table, click the trash can icon from the Delete column. The Delete event stream window appears.

  2. Enter Delete in the field and click, Delete.

Once a stream is deleted, it will immediately stop accepting new data from sources, but will continue delivering in-flight data. Once all in-flight data has been delivered, the stream is deleted and all resources are freed.

Splunk Event Metadata​

The Observability Service sets the following Splunk Event metadata keys for every audit log event that it sends to your stream:

  • time - The timestamp at which the event occurred, specified in seconds since the Unix epoch.
  • source - All events will specify a source of equinix.
  • sourcetype- Audit log events will specify a source type of log.
  • fields - Additional fields for indexing as described below.

Index Fields​

In addition to the standard Splunk metadata keys described above, the Observability Service sets the following custom index fields for each log event that it sends to your stream. These fields are optional and will be omitted if absent.

  • level - A human-readable indication of the logging level, such as INFO or DEBUG.
  • severity - A numeric severity level; higher numbers indicate more severe events (such as errors or violations).

Splunk Event Data​

The content of the event key sent to the Splunk HTTP Event Collector as "Event data" is a structured JSON document with application-specific information about the events being logged.

For a complete reference, refer to the Splunk Event Schema Documentation.

{
"stream" : {
    "streamId" : "dce13c1d-0589-406b-af40-854156a0621e",
    "streamName" : "Example Stream"
},
"source" : {
    "category" : "audit",
    "type" : "api_request",
    "service" : "metal",
    "organizationId" : "99be473c-ee3c-4aeb-a678-eba3fdae7ca6",
    "projectId" : "99be473c-ee3c-4aeb-a678-eba3fdaebeef"

},
"schema" : "v1",
"timestamp" : "2024-04-16T14:58:21.442334Z",
"level" : "INFO",
"eventId" : "6910f03f-ec60-42fc-9e9d-c5f6af2f732d",
"event" : {
    "eventName" : "instance_provision_requested",
    "status" : "failed",
    "auth" : {
        "authType" : "user",
        "user" : {
            "userId" : "582865f9-904b-4061-b536-2420eb01ecdc",
            "userName" : "jdoe@equinix.com"
        },
        "role" : {
            "roleName" : "collaborator"
        }
    },
    "httpRequest" : {
        "host" : "api.equinix.com",
        "method" : "PUT",
        "path" : "/metal/v1/projects/99f8e7f1-fe4a-441a-ade9-687743f080f6",
        "scheme" : "http",
        "statusCode" : 200,
        "userAgent" : "metal-cli/metal equinix-sdk-go/0.30.0",
        "sourceIpAddress" : "111.111.111.11"
    }
}
}