Skip to content

Networking and connection modes ​

Enterprise module

Connection modes are part of the GPU Fleet module (gpu-fleet) of your Cognipeer package.

The Connection tab of a machine's page decides how the console reaches the models on that machine. How the agent connects to the console (heartbeat, commands) is the same in every mode: the agent always connects outbound to the console over HTTPS. What changes is the traffic from the console to the model.

ModeThe console connects toWhat must be open on the machineWhen to use it
Directhttp://<service address>:<model port>The port of every modelThe console and the machine are on the same network (VNet, VPN).
Agent gatewayThe Endpoint URL, or http://<service address>:<gateway port>Only the gateway port (default 7480), or a tunnelMachines behind a tunnel, an HTTPS proxy or a VPN.
RelayThe connection the agent opened to the consoleNothingMachines behind NAT that accept no inbound connections (for example a laptop Mac).

Direct ​

The default. The console connects to each model's port on the service address. The service address is an IP or a DNS name (10.0.0.4, host.tailnet.ts.net), not a URL.

On a Mac, MLX models listen on 127.0.0.1 only; in direct mode they are reached through the gateway port on the same service address.

Agent gateway ​

The agent puts one authenticated port (default 7480) in front of all models. The console adds a token valid only for that deployment to each request; the gateway checks it and forwards the request to the model on the machine. In this mode Docker models also listen on 127.0.0.1 only.

You can put a tunnel or an HTTPS name in front of the gateway and enter it in Endpoint URL:

bash
# ngrok
ngrok http 7480
# Endpoint URL: https://<your-name>.ngrok.app

# Cloudflare Tunnel
cloudflared tunnel --url http://localhost:7480

# Tailscale Funnel
tailscale funnel 7480

The Endpoint URL must start with http:// or https:// and cannot contain a user name, password, query or fragment. A path prefix (https://proxy.example.com/gpu-01) is accepted. The console connects to the Endpoint URL over public addresses only.

If a tunnel on the same machine is the only way in, you do not need to expose the gateway: set COGNIPEER_GATEWAY_BIND=127.0.0.1 on the agent. The gateway port is set with COGNIPEER_GATEWAY_PORT (0 or off turns the gateway off; direct mode then cannot reach MLX models).

Relay ​

The agent opens a persistent WebSocket to the console outward, and model requests travel over it. No port, tunnel or public IP is needed. When the console runs on several nodes, requests are routed between the nodes through Redis.

Relay also carries streaming responses. A single request body can be at most 32 MB.

Testing the connection ​

Test connection on the card checks:

  • in relay mode, whether the agent is connected;
  • in gateway mode, whether the gateway answers;
  • a GET /v1/models call for up to five deployments on the machine.

Each step is listed with its reason when it fails (connection refused, name not resolved, timeout).

Changing the mode ​

  • After a change, the inference monitoring records point to the new address. In relay mode these records are removed (monitoring does not go through the relay); Model Hub and pool traffic keep working through the relay.
  • Docker models started while the machine was in gateway or relay mode listen on 127.0.0.1. When you go back to Direct, restart these models so their ports are reachable from outside.

See also Mac (Apple Silicon) and MLX.

Studio · Pulse · Console · Agent SDK and more — the Cognipeer documentation hub