Voice

Two directions, one shape. Speech in (a voice message becomes text) and speech out (text becomes a voice message) are each a declared fallback chain: the first step that answers wins, a step whose credential is missing is skipped with a warning rather than failing the boot, and declaring a chain means declaring all of it.

Neither is required. Omit both keys and you get sensible chains.

Voice in (transcription)

Every voice message your brain receives is transcribed before anything else sees it — the agent gets words, the chat gets a 🎤 quote, the transcript is cached by the file’s identity and the hearing that made it, so re-reading a nine-minute recording costs nothing. Agents can also transcribe on demand with the audio_transcribe tool (below). Any role can hear in a way of its own (below), and every person in a way of theirs (below).

Long recordings (past ~8 000 chars of transcript, about ten minutes of speech) arrive differently: the agent’s turn carries the first ~1 500 chars plus the path of a file holding the full transcript (and, when the provider returned timing, a timeline: file), and the 🎤 quote posts the excerpt only. A CLI agent reads the files natively; nothing is lost. Files over Telegram’s 20 MB bot limit — an hour-long meeting recording — are fetched over MTProto automatically when an account that can see the message is authorized via /tg_auth, so they transcribe like any other voice message. That covers channels and supergroups (the default account must be able to read the chat) and Secretary-mode business chats (the connected owner’s own account). It does not cover a private chat with the bot or a basic group: there message numbers belong to each account, so no user account can fetch the bot’s message. A file past 20 MB sent there arrives as a note that it could not be downloaded — send it in a supergroup or channel instead.

The brain-wide block lives in constitution/MANIFEST.yaml — hot: an agent changes it with manifest_update (patch: { transcription: { vocabulary: [...] } }, a list replaces the key whole; unset: [['transcription']] goes back to the engine’s), and the next voice message is heard by it:

# constitution/MANIFEST.yaml
transcription:
  languages: [ru, nl, en]
  vocabulary: [Largo, Tenpu, KNM, inburgeringsexamen]

Omit the block and you get elevenlabs → openai → groq → deepgram → local, quality first, auto-detect, no vocabulary. That is not greedy: each step drops out when its key or its binary is absent (the trail says so — elevenlabs (ELEVENLABS_API_KEY is not set)), so the default collapses to exactly what your deployment can do. Declaring a chain means declaring all of it:

transcription:
  chain:
    - provider: elevenlabs
    - provider: openai
      model: gpt-transcribe
    - provider: local
      model: large-v3
      timeoutMs: 1800000

The engine knows those limits and skips a step it cannot satisfy instead of paying for the upload — you see it in the failure reason as openai (62 min over its 25 min limit), not as an opaque provider error. It also keeps most recordings under those limits in the first place: see below.

OpenAI retires gpt-4o-transcribe, gpt-4o-mini-transcribe, whisper-1 and gpt-4o-transcribe-diarize on 2027-02-26. Pin gpt-transcribe (the default) or no model at all; after that date no openai step returns timing.

Privacy and cost. With ELEVENLABS_API_KEY set, the default chain sends the audio to ElevenLabs first (about $0.2–0.4 per hour of audio); ElevenLabs keeps request logs unless the account has enterprise zero retention. A brain that does not want that declares a chain without it.

When a voice is answered late

From the person’s side, every slow stage looks the same: the bot ignores the voice, and then answers all at once. The bot’s logs name the stage. Search them (log_list with search, service bot) around the time the voice was sent:

The first two lines appear only past 10 s, so a healthy chat writes none of them. The last three appear on every voice, which gives a baseline to compare against. After the transcript, the agent’s own thinking is the turn, and the agent’s history shows it (agent_history).

Long recordings just work

Before anything is uploaded, a file bigger than the smallest limit in your chain is converted to a 16 kHz mono opus speech track — video dropped, an hour of recording down from ~65 MB to ~5 MB. Providers that would have answered 413 Request Entity Too Large take it without chunking, and 25 MB then buys about 3.5 hours of audio.

Nothing to configure. Two things worth knowing:

• It needs ffmpeg, which is baked into the engine image. On an engine-dev workstation without it, an oversized recording still goes to the steps whose provider has no size limit (elevenlabs, deepgram, local) — it is only refused when every step in your chain would have rejected it anyway, and then the reason says so (omnislash doctor → Audio transcoding). Everything under the limit is unaffected either way.
• A file carrying no audio track at all fails immediately, in ffmpeg’s own words, instead of walking the whole chain to collect four rejections.

Languages

transcription:
  languages: [ru]          # one: a commitment
# or
  languages: [ru, nl, en]  # several: the set the speaker switches between

ISO-639-1 codes (two letters), lowercase. Omitted = auto-detect.

• One language is a commitment, not a hint: every provider is told it, and whisper honours it literally — a block that says ru and receives an English recording gets Russian letters spelling out English words. Set it only where the input really is one language.
• Several languages are the set a speaker mixes. A provider that takes a set gets it — gpt-transcribe as languages, Deepgram nova-3 as multi — and one that takes a single language is told none and auto-detects: ElevenLabs Scribe detects per segment, which is why it leads the chain for mixed speech.
• The engine translates for every provider and every path — what a block says, what a person says (below), and what an agent passes to audio_transcribe. Deepgram’s nova-3 gets one language as it is and multi for several or none (its English default decodes anything else as silence).

A speaker of two languages on whisper — prompt

Whisper picks ONE language per recording and spells everything in it: a Russian speaker practising Dutch who says «сегодня я был bij de huisarts» gets the Dutch back in Cyrillic. What helps a whisper step (groq, openai, local) is a prompt: text it reads as if it were the transcript so far, so it keeps writing in the same style. Write a short sample the way the speaker actually talks:

transcription:
  languages: [ru, nl]
  chain:
    - provider: elevenlabs
    - provider: groq
      prompt: "Hoi! Сегодня я был bij de huisarts, и он сказал dat alles goed is."

• It is a hint the model imitates, not an instruction it obeys.
• It lives on the step: each whisper step reads its own. elevenlabs and deepgram take no prompt — the key fails the load there.
• Keep it short — whisper reads only the last ~224 tokens of a prompt; a longer one is cut from the front, silently. Names go in vocabulary, not here.

Vocabulary

transcription:
  vocabulary: [Largo, Tenpu, KNM, inburgeringsexamen, Arie]

Names, places and jargon the recordings contain. Each provider gets them in its own form: ElevenLabs keyterms, gpt-transcribe keywords, a Glossary: … line after a whisper step’s prompt (at most 400 characters of it — whisper keeps the end of its prompt, so the terms survive); Deepgram gets none (its term list is one value in the SDK the engine uses). A term a provider cannot take — over 50 characters or 5 words, or with < > { } [ ] \ for ElevenLabs; with <, > or a line break for OpenAI — is dropped for that provider only.

ElevenLabs bills key terms: about +20 % on a request that sends any, and a request with more than 100 terms is billed for at least 20 seconds. Keep the list to the words it actually mishears.

The vocabulary of a hearing is the union of the call’s, the speaker’s and the block’s, most specific first — a provider’s cap keeps the specific ones.

Per-person hearing

How a person speaks is theirs, not the role’s: list it under their identity in constitution/access.yaml (chat-access.md):

identities:
  "tg:user:123456789": { tier: owner, hearing: { languages: [ru, nl, en], vocabulary: [Сергей, Arie] } }
  "tg:chat:-1001134511467": { tier: owner, note: diary channel, hearing: { languages: [ru, nl, en] } }

• Their languages replace the block’s for what they say; their vocabulary joins it. The chain stays the role’s.
• Applied when a recording arrives and the speaker is known: their own message in a chat (a post on behalf of a channel is the channel’s). A forwarded message and audio_transcribe are heard by the role alone. A voice message someone replies to, and a thread root, were heard when they arrived: their transcript is reused whatever hearing made it, and only one nobody heard yet is heard then — the media a reply points at by the role alone, a thread root with its author’s hearing.
• An agent writes it with access_update — { op: 'identity', identity: 'tg:user:<id>', hearing: { languages: [...], vocabulary: [...] } }, hearing: null to remove it. A hearing is not access: the edit widens nothing. Someone not listed yet needs a tier in the same edit — tier: owner for an owner (an env owner included: listing them changes nothing), else their real tier (tier: stranger for someone with no access).

Deadlines

Each step runs under its own deadline; past it the step is abandoned and the next one runs (timed out after N s in the trail). The engine’s: for a remote step a minute plus 6 s per audio minute, at most 15 minutes (a recording of unknown length — an agent’s audio_transcribe of a file — is sized by its file size, generously); for local, 15 minutes. A step’s timeoutMs overrides it, in MANIFEST’s block or a role’s.

Per-role transcription

A role hears in a way of its own: give it a transcription: block in its file — the same shape as MANIFEST’s:

# features/dutch-coach/dutch-coach.agent.md
---
name: dutch-coach
transcription:
  languages: [ru, nl]
  vocabulary: [inburgeringsexamen, KNM, Arie]
  chain:
    - provider: elevenlabs
    - provider: openai
      prompt: "Hoi! Сегодня я был bij de huisarts, и он сказал dat alles goed is."
    - provider: groq
      prompt: "Hoi! Сегодня я был bij de huisarts, и он сказал dat alles goed is."
---

The role that hears is the role that answers — the same role whose voice speaks back: the role serving the chat the voice message arrives in (its bot’s DM, the group its rule names, a Secretary counterpart’s route — their people."tg:user:<id>".role in access.yaml, else the bot’s own), and the calling role for audio_transcribe. A chat no rule serves, and your own messages in a Secretary chat, are heard by the brain’s block.

• The role’s block replaces the brain’s whole — chain, languages and vocabulary. What it leaves out is the engine’s: a role that declares a chain and no languages is auto-detected even when the brain says [ru], and it restates the brain’s vocabulary it still wants. List every step it should fall back through.
• It is hot. An agent changes it with role_update (patch: { transcription: { chain: [...] } }, null to go back to the brain’s) and the next voice message is heard by it.
• A mistyped key fails the role load, as for the brain.
• Each hearing keeps its own transcripts. The cache is keyed by the file and the hearing (its steps, models, prompts, languages and vocabulary), so the same voice message forwarded into another role’s chat is heard again by that role, and an edited block is a fresh transcription, never the old text. The chat-history lines a turn is shown (a group’s or thread’s context) and the media a reply points at are not a new hearing: they show the role’s own transcript of the file when it has one, else whichever hearing made one, and hear only a file no hearing transcribed.
• Earlier transcripts stay readable. A chat-history line still shows any transcript a file has; a hearing whose key changed (every default-chain key did in 1.28.0) transcribes the file again the next time it is heard.
• model_list names every role that hears in a way of its own (roleTranscription: its chain, languages, vocabulary), and each step’s capabilities.

audio_transcribe

The tool hears through the calling role’s block — a url, a local path, or widget: { widgetId, name } (a recording a widget’s page uploaded, widgets.md § 3):

• languages, vocabulary — for this call; languages override every tier, terms join the role’s and the brain’s.
• timestamps: 'segment' | 'word' — the answer carries segmentsPath, a JSON timeline file ({ granularity, items: [{ text, start, end, speaker? }] }); steps that cannot time are skipped.
• diarize: true / numSpeakers — tell speakers apart (ElevenLabs only); speakers counts the labels. Use both for a meeting.
• steps: [...] (+ compare: true) — force the provider/model: run exactly these steps, in order, instead of the recommended chain (the role’s, else the brain’s, else the engine’s). Omitted, the recommended chain hears; a step’s model omitted, that provider’s recommended model runs (the table above). [{ provider: elevenlabs }] pins Scribe for one call; with compare every step runs and results holds each one’s text, language, timeline and time. Languages and vocabulary still apply. Nothing is cached.

Every answer carries attempts — the chain’s trail, each step with its outcome and reason — and the detected language, as the provider spells it (ElevenLabs answers ISO-639-3: rus, nld).

The local step

local runs whisper-ctranslate2 (or whisper) as a subprocess. There is no flag to turn it on: the binary on PATH is the declaration, and it gets there by enabling the whisper-ctranslate2 skill and rebuilding your image. Enable the skill and the step lights up; don’t, and the step is skipped — the first voice message heard through a chain naming it logs why, and every trail says local (no whisper CLI on PATH …).

It is last for a reason — a CPU transcription of a long recording takes minutes, against seconds for an API — and it is worth having for exactly that reason too: it needs no network, no key and no credit balance, so it is what answers when every provider is down, rate-limited or unpaid. One run is capped at 15 minutes.

If you want to run the CLI yourself — subtitles, --task translate — that is what the skill’s own instructions are for. Plain speech-to-text should go through audio_transcribe, which already ends in the same binary and caches what it gets.

Voice out (TTS)

Which voice your brain speaks in is one config key — and any role can speak in a voice of its own (below). It applies to both speech tools — voice_send (synthesizes and delivers a voice message — see notifications) and voice_render (renders and delivers nothing: with widget: { widgetId, name } the audio goes straight onto that open widget’s page as widget.media.<name>, the widget’s own file — widgets.md § 3; without it an agent gets a file in its artifacts directory and its path, for file_send / artifact_publish, and only a caller with no agent — a pipeline step — gets base64) — and to voice replies in chat (below). elevenlabs_sound_effect answers the same way.

defineProject({
  speech: {
    chain: [
      { provider: 'openai', voice: 'cedar', instructions: 'Unhurried and low. Warm, but not soft.' },
      { provider: 'elevenlabs', voice: 'nPczCjzI2devNBz1zQrb' },
    ],
  },
});

A voice name names a voice — it is not a secret, so it belongs in config, not env (provider keys stay in OPENAI_API_KEY / ELEVENLABS_API_KEY). Omit speech entirely and you get ElevenLabs → OpenAI with stock voices. Config is baked into your image, so swapping the brain’s voice ships with a rebuild + redeploy, not a brain-sync pull. A role’s own voice (below) does not.

Per-role voice

Two personas rarely share a voice: a warm Russian coach and a teacher who reads listening drills in standard Dutch need different timbres, languages and directions. Give a role its own speech: block in its file — the same shape as defineProject({ speech }):

# features/dutch-coach/dutch-coach.agent.md
---
name: dutch-coach
speech:
  chain:
    - provider: openai
      voice: marin
      instructions: Spreek helder, standaard Nederlands (ABN), rustig tempo, duidelijke articulatie.
---

Whenever that role’s agent is the one speaking, its chain is used instead of the brain’s: voice_send and voice_render called by the role’s agent (a heartbeat or hook running as the role included), and voice replies in a chat the role answers. Pipeline code speaks as a role by naming it on the call — callTool(ctx, 'voice_send', { text, caption: null }, { role: 'dutch-coach' }) (pipelines.md § callTool) — so a text that must be read exactly as written (a listening drill) is spoken in the role’s voice without an agent in between. Everything else — a role with no speech: block, a pipeline call that names no role, a device calling voice_render with no role — speaks in the brain’s voice.

• The role’s chain replaces the brain’s whole, exactly as the brain’s replaces the engine’s. Nothing is merged step by step: a Dutch role does not inherit a Russian fallback. List every step you want, fallback included.
• It is hot. Role files sync with brain-sync, so an agent changes a voice with role_update (patch: { speech: { chain: [...] } }, null to go back to the brain’s) and the next synthesis uses it — no rebuild.
• A mistyped key fails the role load (instruction, voiceID), the same strictness as the brain tier — a silently ignored voice is invisible until someone hears the wrong one.
• model_list names every role that speaks in a voice of its own (roleSpeech).

The voice is the role’s, never the call’s: neither tool takes a voice or an instructions argument. A voice id means something only to its own provider, and a persona that has to restate its voice on every call is a prompt rule the model must remember each turn.

Voice replies

A person who would rather listen than read switches it on in the chat itself: /settings → Voice replies. From then on every answer in that chat arrives twice — the text, as always, and right after it a voice note of the same answer. Nothing to configure and nothing for your agents to do: the engine speaks what the agent wrote, and the agent never learns that it did.

• In the answering role’s voice. A role with its own speech: block speaks its answers in it; every other role in the brain’s.
• Off by default, per chat. Each chat (and each bot in a shared group) has its own switch; an owner flips it. A chat’s topics and threads share it.
• The voice reads the answer, not the markup. Formatting is dropped, a table is read as lines, a link as its label; code blocks and bare URLs are left out — they stay in the text above.
• Text first, voice right after. The text is never held back for the synthesis; while the voice is being made the chat shows “recording a voice message”. The voice note arrives silently — the text already notified.
• Only answers. A failed turn’s ⚠️ line, cards, questions and command replies stay text. So does an answer delivered after a bot restart.
• No chain, no switch. A brain whose speech chain has no usable step refuses the switch and says why. A synthesis that fails mid-run (a provider down, credits gone) drops that one voice note; the text already arrived.
• Long answers are long voice notes, billed like any synthesis. A provider that refuses the length hands off to the next step in your chain.

Business (Secretary) chats have no settings: there the bot speaks as you, in text.

The speech chain

Synthesis falls over across providers exactly like transcription does: the first step that answers wins, and a step that fails hands off to the next one. You declare the order, because who speaks first and what voice they speak in are one decision — which is why voice lives on the step, not on the chain.

Two rules worth knowing before you write one:

• A step whose key is missing is skipped, not fatal. Declare an ElevenLabs step in a brain with no ELEVENLABS_API_KEY and you get a warning at boot and the rest of the chain. The same config works before and after you buy a key.
• Declaring a chain means declaring all of it. The engine appends no hidden last resort. A one-step chain that fails is silence — which is a legitimate choice, as long as it is the one you made.

openai

instructions is the steerable half and the reason this provider is worth the first slot: the preset picks the timbre, instructions picks the reading. Write it as direction to a voice actor — pace, weight, warmth, what to avoid — not as a description of a person:

instructions: 'Unhurried, low, plenty of air between sentences. Calm authority, ' +
  'never brisk. No broadcaster polish and no motivational lift.'

Voices are auditionable at openai.fm with your own text and your own instructions, which beats guessing from the names.

elevenlabs

ElevenLabs splits voices into premade (usable on every plan, including free) and Voice Library (paid plans only — a free account gets 402 paid_plan_required, and adding the voice to “My Voices” does not lift that).

A voice your plan can’t use fails silently. The chain treats a 402 like any other provider failure: it logs a warning and moves to the next step. You hear a voice, delivery succeeds, no tool call errors — it is just never the voice you configured. Same for a typo’d or deleted id. Nothing about a wrong voice id is visible at boot.

So verify the id before you commit it. The elevenlabs_voices tool lists what your key can actually reach, including each voice’s category — premade is the free-safe bucket. Or check it directly:

curl -s -o /dev/null -w '%{http_code}\n' \
  -X POST "https://api.elevenlabs.io/v1/text-to-speech/<VOICE_ID>" \
  -H "xi-api-key: $ELEVENLABS_API_KEY" -H 'Content-Type: application/json' \
  -d '{"text":"test","model_id":"eleven_flash_v2_5"}'
# 200 = usable · 402 = library voice on a free plan · 400/404 = no such voice

Note that ElevenLabs moves voices between those buckets over time: a premade voice can be retired into the Voice Library, at which point it starts 402’ing for free-plan brains that were happily using it.

Running out of credits looks exactly the same. The allowance is a hard stop, not an overdraft: past it every request 401s with quota_exceeded and the chain moves on. If your brain quietly stops sounding like itself mid-month, check the credit balance before checking anything else — and consider whether ElevenLabs belongs first in your chain at all, since a pay-as-you-go step above it has no cliff to fall off.