Session Recordings

How the Gateway records sessions, and how to route recordings to storage.

The Gateway records interactive sessions, such as kubectl exec and SSH shells, in asciicast v2 format. Recordings stay in your environment and are never uploaded to Twingate. Replay them with the Twingate session player or asciinema play.

Recording format

By default, the Gateway writes each recording as a structured JSON log entry to stderr. Because recordings are ordinary log lines, any log pipeline (Vector, Fluentd, Fluent Bit) can route them to storage. To keep recordings separate from other logs, set log.sessionRecording.output to stdout or to a file, with optional rotation.

A recording log entry has these fields:

FieldDescription
loggerAlways session_recording. Use it to separate recordings from other logs.
asciicastThe recording content in asciicast v2 format.
asciicast_sequence_numThe segment number. It increases each time the Gateway flushes a segment.
conn_idThe connection ID. All segments of a session share it.
userThe Twingate user who ran the session.
tsThe log timestamp.

Long sessions produce multiple log entries, one per segment. To rebuild a session, collect every entry with the same conn_id in asciicast_sequence_num order.

Segment size and duration

The Gateway flushes a segment when it reaches a maximum duration or size. Set both under log.sessionRecording.segment in the Gateway configuration file or Helm values:

log:
sessionRecording:
segment:
maxDuration: 5m
maxSize: "64KiB"

maxSize defaults to 64KiB (65,536 bytes) and can be at most 256KiB (262,144 bytes). It accepts bytes with no unit or B, decimal units (KB or K, multiples of 1,000), and binary units (KiB or Ki, multiples of 1,024).

Ship recordings to Google Cloud Storage with Vector

This example uses the Vector Helm chart to collect recordings from Gateway pods and write each segment as a .cast file to a Google Cloud Storage bucket. It reads container logs, so it assumes recordings go to stderr or stdout.

Prerequisites

  • A Google Cloud Storage bucket.
  • A Google Cloud service account with write access to the bucket. Vector authenticates as this service account through Workload Identity.
  • The label vector.dev/twingate-gateway=true on the Gateway pods. Set it with the Gateway’s podLabels Helm value.

Install Vector

  • Create a values.yaml file for Vector. Replace <SERVICE_ACCOUNT_EMAIL> and <YOUR_GCS_BUCKET_NAME>:

    role: Agent
    logLevel: "info"
    env:
    - name: VECTOR_SELF_NODE_NAME
    valueFrom:
    fieldRef:
    fieldPath: spec.nodeName
    rbac:
    create: true
    serviceAccount:
    create: true
    annotations:
    iam.gke.io/gcp-service-account: <SERVICE_ACCOUNT_EMAIL>
    name: vector
    customConfig:
    data_dir: /vector-data-dir
    api:
    enabled: true
    address: 0.0.0.0:8686
    playground: false
    sources:
    gateway_logs:
    type: kubernetes_logs
    extra_label_selector: vector.dev/twingate-gateway=true
    internal_metrics:
    type: internal_metrics
    transforms:
    gateway_json_logs:
    type: remap
    inputs:
    - gateway_logs
    drop_on_abort: true
    metric_tag_values: single
    source: |-
    parsed_json = parse_json!(.message)
    if parsed_json == null {
    abort
    }
    if parsed_json.logger != "session_recording" {
    abort
    }
    . = parsed_json
    timezone: local
    session_logging:
    type: remap
    inputs:
    - gateway_json_logs
    drop_on_abort: true
    metric_tag_values: single
    source: |-
    if .asciicast == null {
    abort
    }
    parsed_ts = parse_timestamp!(.ts, "%Y-%m-%dT%H:%M:%S%.3fZ")
    . = {
    "filename_ts_part": format_timestamp!(parsed_ts, "%Y%m%d%H%M%S"),
    "user_id": replace!(.user.username, r'@|\.', "_"),
    "conn_id": .conn_id,
    "message": .asciicast,
    "asciicast_sequence_num": .asciicast_sequence_num
    }
    timezone: local
    sinks:
    gcs:
    type: gcp_cloud_storage
    inputs:
    - session_logging
    bucket: <YOUR_GCS_BUCKET_NAME>
    key_prefix: sessionrecordings/{{"{{"}} .user_id {{"}}"}}_{{"{{"}} .filename_ts_part {{"}}"}}_{{"{{"}} .asciicast_sequence_num {{"}}"}}
    filename_extension: cast
    filename_append_uuid: false
    compression: none
    encoding:
    codec: text
    batch:
    max_events: 1
    timeout_secs: 1

    The pipeline reads logs from pods labeled vector.dev/twingate-gateway=true, keeps entries whose logger is session_recording, and writes each segment to the bucket as sessionrecordings/<user>_<timestamp>_<sequence>.cast.

  • Add the Vector Helm repository:

    helm repo add vector https://helm.vector.dev
    helm repo update
  • Install Vector in its own namespace:

    helm install vector vector/vector -n vector --create-namespace -f values.yaml

Limitations

  • Kubernetes session recording requires the WebSocket streaming protocol. Sessions that fall back to SPDY are not recorded. See Kubernetes known issues.
  • SSH: only interactive shell sessions are recorded. Remote commands (ssh host <command>), file transfer, and port forwarding are not recorded.

Last updated