Operations

How vws runs against its database and how a running process reports its state. Configuration variables are listed in Configuration.

Listeners

Metrics are never served on the public listener. Inside containers set VWS_OPS_ADDR=:9091 so Prometheus can scrape it; leave that port unpublished on the load balancer. VWS_OPS_ADDR=off disables the listener.

Health

Readiness runs each check concurrently with a 2-second timeout per check and returns JSON:

{"status":"ok","checks":{"postgres":"ok","redis":"ok"}}
{"status":"unavailable","checks":{"postgres":"fail","redis":"ok"}}
{"status":"draining"}

Check names and results are the only details exposed; the error itself is logged once when a check starts failing (readiness check failed, warning) and once when it recovers (readiness check recovered). A failing dependency never stops the process: readiness recovers on its own when the dependency returns.

Behind a load balancer, point its health check at /readyz. In a container, vws health requests /healthz from inside, as the image has no shell or curl.

Realtime connections

Clients hold one WebSocket each at GET /api/v1/realtime on the public listener, in modes all and world. The server pings every client every 25 seconds and drops a connection that sends nothing for two intervals, so the load balancer's idle timeout must stay above 25 seconds.

A session outlives a dropped connection for 30 seconds, so that a client that reconnects resumes it without missing anything. Meanwhile it stays in its room and the server keeps up to 256 KiB of the frames it sent; a client that misses more than that starts over with a new session. Sessions live in the memory of their task, so the target group needs sticky sessions: a client that reconnects to another task starts over too. The rooms themselves run on one task each, which the others reach (Cluster).

Every process in modes all and world publishes who stands in its rooms to Redis every 5 seconds, for 15 (vws:presence:world-users:<tenant id>:<world id>), which the world's user list reads; while Redis is unreachable it logs cannot publish who is in the rooms, and the lists empty out within 15 seconds.

A world's capacity counts the accounts in all its rooms, wherever they run (Cluster).

Each room runs in its own goroutine. A room that panics logs room stopped after a panic at error level, which reaches Sentry when it is configured, closes the sessions in it with code 4000 so that their clients start over, and starts afresh on the next join; other rooms carry on.

A connection may send 30 messages a second on average, in bursts of up to 60; the server refuses the excess with ERROR_CODE_RATE_LIMITED. A frame larger than VWS_MAX_FRAME_BYTES closes the connection with code 1009.

The server sends a batch of events 100 ms after its first event, or as soon as it holds 128. Up to 64 KiB may wait for a client's socket; beyond that the session holds batches back, and once 512 events are pending it drops position updates that a later update of the same player replaces. A client that falls behind with events that cannot be dropped loses its session with code 4000. vws_realtime_queued_bytes and vws_realtime_dropped_events_total show slow clients.

Cluster

Several processes in modes all and world share the rooms and spaces of a platform when each sets VWS_CLUSTER_ADDR, such as :9090, and the same VWS_CLUSTER_SECRET (Configuration). Without it, each process runs every room its visitors join, so two visitors of a room on two processes would not meet: such a platform runs one process in modes all and world.

• Placement. A room or a space runs on one process at a time, the holder of its lease in Redis (vws:lease:room:<tenant id>:<room id>, vws:lease:space:<tenant id>:<world id>:<name>), which the process renews every 2 seconds and which lapses 6 seconds after the last renewal. A room nobody runs starts on the process that consistent hashing names among the living ones, so that a process that joins or leaves moves only its share of the new rooms; the running ones stay where they are.
• Registry. Every process announces itself every second in vws:presence:cluster-nodes, with the address of VWS_CLUSTER_ADVERTISE (by default the machine's private IPv4 address and the port of VWS_CLUSTER_ADDR), and the others forget it 5 seconds after its last announcement.
• The seam. The visitor's session stays on the process its WebSocket reached. When another process runs the room, the session's process links the member to it over a stream of vws.cluster.v1.ClusterService (proto/vws/cluster/v1), the gRPC protocol over unencrypted HTTP/2 on VWS_CLUSTER_ADDR, which carries the member's calls one way and the room's events and answers the other, in order. Every call carries the shared secret, compared in constant time, so the listener must stay on the private network. Spaces, the partner API's room calls and the media server's webhooks follow their rooms the same way: a webhook about a space another process runs goes there over Redis (vws:topic:media-events).
• Capacity. A world's capacity counts accounts across the processes: each marks the accounts it holds in vws:presence:census:<tenant id>:<world id>, renewed every 10 seconds, for 30.
• A process that dies. Its streams end at once, or within 6 seconds of unanswered pings, and its members start over, with code 4000; their next join waits for the dead process's lease to lapse and starts the room on a living process, within about 10 seconds. A process that stops deliberately leaves the registry at once, closes its cluster listener without waiting, and releases its leases as its rooms close, so its members move over within a second or two.
• Logs. room moved to another process and space moved to another process mean a process lost a lease it held, another process took over a lease that its renewal found it gone, and a process of the cluster did not answer that a join set a silent process aside until it renews its place. Redis being unreachable logs cannot announce this process to the cluster and cannot renew the leases of this process's rooms and spaces; rooms keep running meanwhile, and a lease another process took in the meantime moves its room once Redis answers again.

Media

Media runs on the LiveKit server of VWS_LIVEKIT_URL, self-hosted or LiveKit Cloud; without it, media features are off on every site. The API secret must have at least 32 characters, as LiveKit's generate-keys makes them. Browsers connect to the URL directly, over wss:// in production, and the server calls LiveKit's room and egress APIs over HTTPS at the same address, or at VWS_LIVEKIT_API_URL when it reaches LiveKit by another address, as inside a VPC, so tasks need outbound access to it.

Each space with media, such as a bubble, meets in a LiveKit room named at random for the life of the space; LiveKit creates the room when the first member connects, which is its default, and the server deletes it when the space ends. A member's token admits it under its space member id for 10 minutes, long enough to connect, after which LiveKit renews it for as long as the member stays connected. A member that leaves the space is removed from the room at once. Calls to LiveKit go out in order from one queue per process, each with a 5-second timeout; a failed call is logged as a call to the media server failed and not repeated, and LiveKit closes rooms left empty by itself.

Point LiveKit's webhooks at the site, with the platform's API key:

webhook:
  api_key: <VWS_LIVEKIT_API_KEY>
  urls:
    - https://<a host of the platform>/api/v1/livekit/webhook

The route answers 401 to a webhook whose signature does not match, which in LiveKit's logs means the keys differ, and 404 when the platform has no LiveKit. A participant that joins a space's room without being a member of the space, such as one whose token outlived its membership, is removed, unlike the participants of LiveKit's own services. Webhooks reach the process that runs the space only while one process runs the rooms.

Recordings

Recordings need the recording platform feature in VWS_FEATURES and LiveKit's egress service beside the LiveKit server, on the same Redis; LiveKit Cloud runs its own. The server asks egress to record a room into one MP4 file under tenants/<tenant id>/recordings/<recording id>.mp4:

• s3 backend. Egress uploads into VWS_S3_BUCKET, in VWS_S3_REGION and at VWS_S3_ENDPOINT, with the credentials of its own environment, which need s3:PutObject on tenants/*/recordings/*. Downloads are URLs that S3 signs for at least an hour.
• disk backend. Egress writes into VWS_STORAGE_DISK_PATH at the same absolute path, so it must share that directory with the server, as the development stack of deploy/dev does; downloads go through the server with a ticket that works once.

Egress reports recordings to the webhook route above, which keeps them. A start that egress refuses, because none runs or it lacks capacity, is logged as cannot start a recording and told to its recorder. Every hour, one process in modes all and control, holding the lease vws:lease:sweep-recordings, deletes the completed recordings that ended longer ago than their tenant's recordings.retentionDays with their files (Tenancy), and logs deleted expired items with sweep=recordings per tenant.

Accounts

• Signed-out sessions. Every process in modes all and world listens on vws:topic:session-revoked and closes the realtime connections of a session that ends anywhere, with code 4000. While Redis is unreachable it logs cannot listen for signed-out sessions; retrying and tries again every 5 seconds; access tokens of revoked sessions stop working as soon as Redis is back, and at the latest when they expire.
• Kicks and bans. The process that takes a kick or a ban ends the removed account's sessions at once and announces it on vws:topic:removals, on which every process in modes all and world ends that account's sessions in the room or the world. While Redis is unreachable it logs cannot listen for removed players; retrying; a banned account cannot join again, wherever it reconnects.
• Expired accounts. Every 10 minutes, one process in modes all and control, holding the lease vws:lease:sweep-accounts, deletes up to 500 anonymous accounts per tenant whose last session expired 7 days ago and up to 500 guest accounts whose period ended, after revoking the guests' sessions, and logs deleted expired accounts with the count.
• Email. VWS_EMAIL_BACKEND=ses sends sign-in links through Amazon SES with the AWS SDK's default credentials, and mailgun through Mailgun's HTTP API with VWS_MAILGUN_API_KEY on VWS_MAILGUN_DOMAIN, at api.eu.mailgun.net when VWS_MAILGUN_REGION is eu; a failure logs cannot send a sign-in link with the provider's answer and answers the client 500. In development, log writes each message to the log as email not sent in development, where the link can be copied. Production refuses log.
• Meeting invitations. Every process in modes all and world listens on vws:topic:invitations and hands each invitation and answer to its own sessions of the invited account; while Redis is unreachable it logs cannot listen for meeting invitations; retrying, and an invitation reaches only the sessions on the process that sent it.
• Player variables. A kept change of a player variable of the world scope is announced on vws:topic:player-variables, and every process in modes all and world passes it to the account's sessions in the other rooms of the world; while Redis is unreachable it logs cannot listen for player variables; retrying, and those sessions read the change when they next join. Every 5 minutes one process in modes all and world, holding the lease vws:lease:sweep-variables, deletes the player variables whose time to live ran out, in every tenant, and logs deleted expired items with sweep=player-variables per tenant.
• Blocks. A block made or lifted is announced on vws:topic:blocks, and every process in modes all and world hides the two accounts from each other in its running rooms, or shows them again; while Redis is unreachable it logs cannot listen for blocks; retrying, and the change applies when one of the two next enters a room.
• Encryption key. VWS_ENCRYPTION_KEY seals identity providers' client secrets and the tokens of connected calendars. Without it, production refuses to start; with a different key, those providers fail to sign anyone in until their secrets are entered again, and every calendar has to be connected again, its reading failing meanwhile with cannot read a calendar.

Calendars

People connect their Google and Microsoft calendars through the platform's OAuth clients, one per provider, whose redirect URI is <VWS_PUBLIC_URL>/api/v1/calendars/callback for every tenant (Extensions).

• Google. In the Google Cloud console, enable the Google Calendar API and the Google Tasks API, configure the OAuth consent screen with the scopes openid, email, https://www.googleapis.com/auth/calendar.events.readonly and https://www.googleapis.com/auth/tasks.readonly, and create an OAuth client of the type web application with the redirect URI. The calendar scope is sensitive: until Google verifies the app, only test users listed on the consent screen can connect. Set VWS_GOOGLE_CLIENT_ID and VWS_GOOGLE_CLIENT_SECRET.
• Microsoft. In Microsoft Entra, register an application for accounts in any organizational directory and personal Microsoft accounts, with a web redirect URI, the delegated Microsoft Graph permissions User.Read, Calendars.Read and Tasks.Read, and a client secret. Set VWS_MICROSOFT_CLIENT_ID and VWS_MICROSOFT_CLIENT_SECRET, and renew the secret before it expires.
• Turning them off. Without either client, the platform offers no calendars and every tenant's flag calendars reads off; connections already made stay until their people remove them, and are not read.
• Failures. A grant the provider cannot complete logs cannot complete a calendar grant with the provider and sends the person back with the error failed. A calendar that cannot be read logs cannot read a calendar with the provider, the tenant and the account, and the person's day lists it as unavailable; a refresh token the provider refuses expires the connection without a log. Requests to the providers keep to public addresses outside development, as identity providers' do.

License

• Daily check. Where the build checks licenses (Licensing), one process in modes all and control, holding the lease vws:lease:check-license, renews the activation once a day at the license API, and logs cannot check the platform's license when it cannot reach it, or the license API refused the platform's license with the refusal. The install runs on the last answer for 7 days; rooms close once it runs out, or at once after a refusal, and open again with the next good check. A license the console or the administration activates or checks applies on every process within 30 seconds.
• Start. A multi-tenant platform whose kept license is missing or standard refuses to start; run vws license activate <key> with the server's environment.
• Moving. vws license release gives the key back before an install is retired, so that another can activate it; otherwise the website's admin resets it.

Tenants

• Running for one organisation. Set VWS_TENANCY=single and VWS_PUBLIC_URL to the address people use. The first start creates the tenant main (or the slug of VWS_TENANT) and logs serving one tenant on every host; every later start finds it again. Any host that reaches the server serves it, so one DNS record and one certificate for the public host do. Grant the first admin with vws tenant admin --email <address> once that person has signed in; the rest of this section is for multi-tenant platforms (Tenancy).
• The central admin. A multi-tenant platform's operators run its tenants at https://<VWS_OPERATOR_HOST>/operator (Central admin), which needs VWS_OIDC_* and a redirect URI https://<VWS_OPERATOR_HOST>/api/v1/operator/callback at that provider. Add the first operator with vws operator add --email <address>, as a one-off task like the commands below.
• Suspended and deleted tenants. Suspending a tenant ends its live sessions on every process in modes all and world as soon as the tenancy change reaches it, and its hosts refuse everything until it resumes. Every hour one process in modes all and control, holding the lease vws:lease:purge-tenants, deletes the tenants whose deletion is older than 30 days, objects first, logging deleted a tenant whose grace period ended and writing tenant.purged to the platform's log; a tenant whose objects cannot all be deleted stays for the next hour.
• Creating a tenant. vws tenant create --slug acme --name "Acme" creates a tenant whose primary domain is acme.<VWS_TENANT_DOMAIN>, verified at once; each --host adds a host the operator controls, also verified at once. Without VWS_TENANT_DOMAIN, the first --host becomes the primary domain.
• The first admin. Once the tenant's admin has signed in, vws tenant admin --host acme.<VWS_TENANT_DOMAIN> --email ada@acme.com gives their registered account the admin role, which opens /api/v1/admin to them.
• Custom domains. Every 5 minutes one process in modes all and control, holding the lease vws:lease:verify-domains, looks up the TXT record _vws-challenge.<host> of every pending domain and verifies those that publish their challenge, logging verified custom domains with the count. A verified host resolves at once on the process that verified it and within 10 seconds elsewhere. The certificate is the edge's business.
• Usage. Every 15 minutes one process in modes all and control, holding the lease vws:lease:review-usage, reviews each tenant's month: it records the peaks of its rooms and storage, emails its admins the alerts of usage at 80 % and 100 % of a limit or of the cap, and queues the reports of the billing webhook (Central admin), logging sent usage alerts and reports with sweep=usage and the count where something went out. A mail that fails leaves the tenant's alerts and reports for the next round.
• Changes. Every process listens on vws:topic:tenancy and drops what it cached of a changed tenant, and processes in modes all and world reload their running rooms of it; while Redis is unreachable they log cannot listen for tenancy changes; retrying or cannot listen for tenancy changes in rooms; retrying, cached hosts, settings and themes expire within a minute, and rooms keep their flags and bubbles until they next start.
• Worlds and rooms. A changed or deleted world or room is announced on vws:topic:directory-changed, and every process in modes all and world reloads its running rooms of it, or tells their players of the deletion; while Redis is unreachable it logs cannot listen for changed worlds and rooms; retrying, and rooms keep the old configuration until they next start.

Partner API and webhooks

Partner API describes the API and the deliveries; this is how they run.

• Where calls go. Processes in modes all and world serve the partner API at /vws.partner.v1.PartnerService/ on every host of the platform. Its calls on rooms reach the rooms of the answering process, so they see every room while one process runs them. The public listener speaks HTTP/2 without TLS, by prior knowledge, beside HTTP/1.1, so a load balancer that ends TLS can pass gRPC on to it over HTTP/2.

• Rate limits. Each key's calls are counted in Redis under vws:rate:partner:<key id>; while Redis is unreachable, calls go through uncounted and the process logs cannot count the calls of an API key.

• Deliveries. Every 2 seconds each process in modes all and control asks the database for the tenants with deliveries due, those waiting longest first, and posts up to 10 deliveries of each, for 4 tenants at a time. A delivery it claims is its own for a minute. Failures to reach the database log cannot find the webhook deliveries due, cannot claim webhook deliveries or cannot record how a webhook delivery went. Deliveries go straight to the webhooks' addresses, never through a proxy, so the tasks need a route to the internet, such as a NAT gateway for private subnets.

• Billing deliveries. Every 2 seconds each process in modes all and control claims up to 10 deliveries of the platform's billing webhook that are due, each its own for a minute, and posts them; failures to reach the database log cannot claim the billing webhook's deliveries or cannot record how a billing delivery went. Once an hour each such process deletes the billing deliveries settled more than a week ago, logging deleted settled billing deliveries. The outbox is the table billing_delivery, which only transactions outside any tenant read.

• Joins and leaves. Each process in modes all and world queues the joins and leaves of its rooms in memory, up to 4096, and writes them to the outbox; a full queue logs dropped a webhook event. A process that shuts down writes its queue before it exits.

• Pruning. Every hour one process in modes all and control, holding the lease vws:lease:prune-deliveries, deletes the deliveries settled more than a week ago, in every tenant, and logs deleted expired items with sweep=webhook-deliveries per tenant.

• A webhook's deliveries. The outbox is the table outbox_event: status is pending, delivered or failed, attempts counts the tries, last_error says why the last one failed and available_at when the next is due. As the owner role, which row-level security does not restrict:

  SELECT id, topic, status, attempts, last_error, available_at
  FROM outbox_event
  WHERE tenant_id = '<tenant id>' AND webhook_id = '<webhook id>'
  ORDER BY id DESC
  LIMIT 50;

Chat

The chat of a space lives in Postgres while the space lives on its process. When the space ends, or the process shuts down, its channel is archived: chat_channel.archived_at is set and nobody reads it through the product any more. Archived channels and deleted messages (chat_message.deleted_at) stay for the platform's administrators to review, as the owner role, which row-level security does not restrict:

SELECT chat_message.created_at, app_user.display_name, chat_message.kind, chat_message.body, chat_message.deleted_at
FROM chat_message
JOIN chat_channel ON chat_channel.tenant_id = chat_message.tenant_id AND chat_channel.id = chat_message.channel_id
LEFT JOIN app_user ON app_user.tenant_id = chat_message.tenant_id AND app_user.id = chat_message.sender_user_id
WHERE chat_channel.tenant_id = '<tenant id>' AND chat_channel.space_name = '<space name>'
ORDER BY chat_message.created_at;

A space renews its channel, chat_channel.renewed_at, every 30 seconds while it lives. A process that stops without shutting down, such as one the kernel killed, leaves the channels of its spaces open: every minute, one process in modes all and world, holding the lease vws:lease:sweep-space-chats, archives the space channels nobody renewed for 2 minutes, takes them away from their last members, and logs archived the chats of spaces that stopped with sweep=space-chats per tenant. Their last members read them over HTTP until then, about three minutes at most. The renewals live in Postgres rather than in Redis, so that a restart of Redis, which forgets every key, archives no live chat. A room likewise renews, every 30 seconds, chat_member.renewed_at of its channel for the accounts standing in it, and admits any of them the channel lacks. Every minute, one process in modes all and world, holding the lease vws:lease:sweep-room-chats, takes the rooms' channels away from the accounts nobody renewed for 2 minutes, those a crashed process's rooms left behind, and logs took the chats of rooms that stopped away from their players with sweep=room-chats per tenant.

Chat attachments are files of the object store under tenants/<tenant id>/chat-attachments/, with a row of file_object and of chat_attachment each. Every hour, one process in modes all and control, holding the lease vws:lease:sweep-chat-attachments, deletes the uploads that no message took within a day, with their files, and logs deleted expired items with sweep=chat-attachments per tenant; a message's files go when it is deleted. With the s3 backend, browsers fetch attachments from the bucket after a redirect, which needs a CORS rule on the bucket that allows the site's origin to GET.

The changes of rooms' and worlds' channels travel on vws:topic:chat, on which every process in modes all and world hands them to its sessions that watch the channels. While Redis is unreachable a process logs cannot listen for chat; retrying, and its sessions miss the messages of other processes until they load the history again.

Admin messages

The process that accepts an admin's message announces it on vws:topic:announcements, and every process in modes all and world hands it to the members of its rooms it is for. While Redis is unreachable a process logs cannot listen for announcements; retrying, and messages sent meanwhile do not reach its rooms; nothing sends them again. Audio messages are files of the object store under tenants/<tenant id>/audio-messages/, with a row of file_object each. Every hour, one process in modes all and control, holding the lease vws:lease:sweep-audio-messages, deletes those older than a week with their files, and logs deleted expired items with sweep=audio-messages per tenant.

Maps

Map uploads are part of the control API, so processes in modes all and control take them; processes in modes all and content serve map files. An upload's checks hold the package in memory, up to 100 MiB plus the 256 MiB it may unpack to, so size the control tasks' memory for the uploads that may run at once. A replaced package is announced on vws:topic:map-changed, and every process in modes all and world reloads its rooms that use the map; while Redis is unreachable it logs cannot listen for changed maps; retrying, and rooms keep the old map until they next start. Layout commands go out the same way on vws:topic:layout-applied; a client that misses some catches up from the command log when it notices the gap. A changed prefab goes out on vws:topic:prefabs-changed, and the tenant's running rooms read again the cells their entities block.

Capacity

vws load opens realtime sessions against a running server the way a crowd of visitors would. Each session signs up as an anonymous account, joins the rooms named with --room in turn and answers pings; with --move it also walks back and forth at the position interval the server announces. It prints how long joining took and, after --duration, how many frames a second the crowd received and how many sessions it lost. It exits 1 when a session failed to join or dropped.

The server allows 20 anonymous accounts per address at once and 10 a minute after that (Identity), so each session signs up from its own address in 198.18.0.0/15, the range RFC 2544 sets aside for benchmarks, sent as X-Forwarded-For. The server believes that header only from a trusted proxy: start it with the load generator's address in VWS_TRUSTED_PROXIES, such as 127.0.0.1/32 on one machine.

vws load --url https://office.example.com --room /office/lobby --connections 200 --move --duration 2m

On a laptop with one vws all process, 1,000 idle sessions over 20 rooms of 50 kept the server at 118 MB resident, and 200 walking sessions in 4 rooms of 50, which received 1,000 frames a second between them, at 120 MB. During a run, watch the process's resident memory, vws_realtime_sessions, vws_realtime_queued_bytes and vws_realtime_dropped_events_total.

Shutdown

On SIGTERM or SIGINT the process:

1. reports draining on /readyz, so the load balancer stops routing new requests;
2. stops accepting connections and gives in-flight requests up to VWS_SHUTDOWN_TIMEOUT (20 seconds by default);
3. closes every realtime connection with code 1001 and ends every realtime session, so clients reconnect and start a new session, stops the room actors, and archives the chats of its spaces, giving up after 5 seconds;
4. flushes pending traces and error reports, with a 5-second limit;
5. exits with status 0, or 1 if the shutdown did not complete cleanly.

A second signal during this sequence terminates the process immediately. Give the process manager a stop timeout above VWS_SHUTDOWN_TIMEOUT plus 5 seconds; the self-hosted systemd units do.

Database

The server connects through VWS_DATABASE_URL as a login role that is a member of vws_app, so row-level security applies to every query. Migrations connect through VWS_DATABASE_MIGRATION_URL as vws_owner, the role that owns the schema. db/bootstrap.sql creates them and the database, run by a role that may create roles and databases:

psql "$ADMIN_URL" -v owner_password=... -v app_password=... -v database=vws -f db/bootstrap.sql

Apply migrations in one of two ways:

• Run vws migrate up with the new version before starting it.
• Set VWS_MIGRATE_ON_START=true. Every starting task then tries to migrate, and a Postgres advisory lock makes them take turns.

Without VWS_MIGRATE_ON_START, a process exits at startup with the database schema is behind this build when the database lacks a migration the build expects. A database that is ahead of the build is accepted, which allows rolling the service back. Every migration must therefore keep working with the previous release.

vws migrate down --yes rolls back the latest migration. The initial migration's Down drops every table.

Each process opens at most VWS_DATABASE_MAX_CONNS connections, 10 by default. Keep the number of processes times that limit below the database's max_connections.

Object storage

Map files are stored through VWS_STORAGE_BACKEND, under keys of the form tenants/<tenant id>/maps/<map id>/<file>.

• disk writes under VWS_STORAGE_DISK_PATH, which every task of a deployment would have to share, so it suits a single process. GET /api/v1/maps/{map}/files/{path} streams the file itself.
• s3 uses VWS_S3_BUCKET with the AWS SDK's default credentials. A process in mode all, world or content checks at startup that it can reach the bucket, and refuses to start otherwise. The file route answers 302 with a signed URL, so browsers download from S3 directly. A signed URL stays the same for a whole hour and works for at least one more, which lets browsers cache what they fetch through it; the redirect itself may be cached for 5 minutes.

Files whose name carries a content hash or a UUID are served as immutable for a year, and every other file is revalidated on each use. Uploading a map package again deletes the files the new package no longer contains.

Redis

Every mode except content connects to VWS_REDIS_URL at startup and adds a redis readiness check. Use rediss:// for ElastiCache with in-transit encryption, with the auth token as the URL password. Redis holds only ephemeral state, so it needs no backups or persistence.

Logs

Logs go to standard error, one JSON object per line (VWS_LOG_FORMAT=text for terminals), at the level set by VWS_LOG_LEVEL. A self-hosted install keeps them in journald: journalctl -u vws.

Every HTTP request produces one http request record. Health probes log at debug level, everything else at info.

Records written while serving a request carry the same request_id and trace fields. Secrets in the configuration are redacted wherever they are logged.

Metrics

GET /metrics on the operations listener, in Prometheus text format.

route is the matched pattern, so label cardinality stays bounded; unknown methods are reported as OTHER.

Tracing

Tracing is off unless OTEL_EXPORTER_OTLP_ENDPOINT or OTEL_EXPORTER_OTLP_TRACES_ENDPOINT is set; OTEL_SDK_DISABLED=true turns it off again. Spans are exported over OTLP/HTTP, so point the endpoint at port 4318 of a collector such as the AWS Distro for OpenTelemetry sidecar. Every other standard variable applies: OTEL_EXPORTER_OTLP_HEADERS, OTEL_TRACES_SAMPLER, OTEL_TRACES_SAMPLER_ARG, OTEL_RESOURCE_ATTRIBUTES, OTEL_SERVICE_NAME.

Resources carry service.name=vws, service.version, deployment.environment.name from VWS_ENV, vws.mode, and the host name. Incoming requests continue W3C traceparent contexts, and each request span is named after its route.

Error reporting

With VWS_SENTRY_DSN set, every error-level log record and every recovered panic is sent to Sentry, tagged with the release vws@<version> and the environment VWS_SENTRY_ENVIRONMENT (default: VWS_ENV). An error attribute on the record becomes the reported exception; the other attributes travel as event context. Warnings, such as a dependency check failing, are not reported.