Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions docs/prompting.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,10 +20,31 @@ Probe `transcribe_model_supports()` for the matching feature bit.
| Canary 180m-flash / 1b-flash / 1b-v2 | | | | yes |
| Fun-ASR-Nano | yes | | | |
| MOSS-Transcribe-Diarize | yes | | | |
| Parakeet (TDT and RNN-T heads) | yes | | | |

Per-model formats and restrictions are in each model doc's **Prompting:**
note under [`models/`](models/).

## Parakeet vocabulary

Parakeet has no prompt slot, so its transducer decoders take vocabulary terms
as phrase boosting: each term is tokenized the way the model writes it, starting
at a word boundary, and the terms form an Aho-Corasick automaton over token
ids. The model alone decides between blank and a token on every greedy step.
When it emits a token, the vocabulary re-ranks the non-blank tokens. Once the
model has begun a term on its own, each further piece that extends the match
earns a logit bonus of 3.0, a token that breaks the match gives back what the
unfinished match earned, and a completed term keeps its bonus. 3.0 is the
lowest bonus that kept the term-recall gains while leaving transcripts of
unrelated speech unchanged. Each term also matches with the case of its first
letter flipped, since sentence position decides that letter's case in the
transcript; other letters match as given. Biasing therefore changes which word is
written, never whether one is, so it cannot invent speech on silence. The
reported token confidences stay the model's own, and with no terms decoding is
bit-exact. Terms with characters the tokenizer cannot write are skipped with a
WARN. Offline runs and batches take terms; streaming ignores them with a WARN,
and CTC heads do not take them yet.

## Limits

- Over the prompt budget, `vocabulary` drops terms from the end of the list and
Expand Down
1 change: 1 addition & 0 deletions src/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ add_library(transcribe
transcribe-model.cpp
transcribe-tokenizer.cpp
transcribe-prompting.cpp
transcribe-phrase-boost.cpp
transcribe-unicode.cpp
transcribe-unicode-data.cpp
transcribe-debug.cpp
Expand Down
70 changes: 56 additions & 14 deletions src/arch/parakeet/decoder.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,8 @@
#include "transcribe-log.h"
#include "weights.h"

#include <limits>

// ggml-backend.h, not ggml-cpu.h: under GGML_BACKEND_DL the CPU backend
// is a loadable module, so backend-specific entry points must be reached
// through the registry (ggml_backend_init_by_type / get_proc_address),
Expand Down Expand Up @@ -966,6 +968,31 @@ int argmax_range(const float * data, int n) {
return best_i;
}

// Token argmax, biased toward the vocabulary when there is one.
//
// The model alone decides between blank and a token: only when its own
// argmax emits something does the vocabulary re-rank the non-blank tokens.
// Biasing can then change which word is written but never whether one is,
// so it cannot invent speech on silence or noise. The model's own logits are
// left untouched for the confidence that follows.
static int biased_argmax(const float * token_logits,
int n_token_cls,
int blank_id,
const transcribe::PhraseBoost * boost,
int boost_state,
std::vector<float> & scratch) {
const int unbiased = argmax_range(token_logits, n_token_cls);
if (boost == nullptr || boost->empty() || unbiased == blank_id) {
return unbiased;
}
scratch.assign(token_logits, token_logits + n_token_cls);
boost->apply(boost_state, scratch.data(), n_token_cls, blank_id);
if (blank_id >= 0 && blank_id < n_token_cls) {
scratch[static_cast<size_t>(blank_id)] = -std::numeric_limits<float>::infinity();
}
return argmax_range(scratch.data(), n_token_cls);
}

// Entropy-based confidence over a token-logit slice. Mirrors the
// reference ParakeetTDT.decode_greedy path:
//
Expand Down Expand Up @@ -1015,12 +1042,13 @@ float token_confidence(const float * token_logits, int n_token_classes, std::vec
// Greedy decode driver
// ---------------------------------------------------------------------------

transcribe_status decode_tdt_greedy(const HostDecoderWeights & w,
const float * enc_out,
int T_enc,
int d_enc,
int n_threads,
std::vector<TdtToken> & out_tokens) {
transcribe_status decode_tdt_greedy(const HostDecoderWeights & w,
const float * enc_out,
int T_enc,
int d_enc,
int n_threads,
std::vector<TdtToken> & out_tokens,
const transcribe::PhraseBoost * boost) {
if (enc_out == nullptr || T_enc <= 0 || d_enc <= 0) {
return TRANSCRIBE_ERR_INVALID_ARG;
}
Expand Down Expand Up @@ -1097,6 +1125,10 @@ transcribe_status decode_tdt_greedy(const HostDecoderWeights & w,
// the unconditional path.
bool predictor_dirty = true;

// Vocabulary biasing state; untouched when no terms were given.
int boost_state = transcribe::PhraseBoost::root;
std::vector<float> boosted;

// See ParakeetHParams::tdt_global_symbol_budget.
const bool global_budget = w.tdt_global_symbol_budget;
int64_t steps_remaining = static_cast<int64_t>(w.tdt_max_symbols) * T_enc;
Expand Down Expand Up @@ -1131,7 +1163,7 @@ transcribe_status decode_tdt_greedy(const HostDecoderWeights & w,
const float * token_logits = logits.data();
const float * duration_logits = logits.data() + n_token_cls;

const int pred_token = argmax_range(token_logits, n_token_cls);
const int pred_token = biased_argmax(token_logits, n_token_cls, blank_id, boost, boost_state, boosted);
const int decision = argmax_range(duration_logits, n_dur);
const int duration = w.tdt_durations[static_cast<size_t>(decision)];

Expand Down Expand Up @@ -1172,6 +1204,9 @@ transcribe_status decode_tdt_greedy(const HostDecoderWeights & w,
last_token = pred_token;
std::swap(state, next_state); // commit
predictor_dirty = true;
if (boost != nullptr) {
boost_state = boost->next(boost_state, pred_token);
}
}

if (global_budget) {
Expand Down Expand Up @@ -1249,12 +1284,13 @@ transcribe_status decode_tdt_greedy(const HostDecoderWeights & w,
// duration). Matches the reference dump points (`dec.embed.0`,
// `dec.lstm.{layer}.{h,c}.0`, `dec.joint.0`) emitted on iter 1.

transcribe_status decode_rnnt_greedy(const HostDecoderWeights & w,
const float * enc_out,
int T_enc,
int d_enc,
int n_threads,
std::vector<TdtToken> & out_tokens) {
transcribe_status decode_rnnt_greedy(const HostDecoderWeights & w,
const float * enc_out,
int T_enc,
int d_enc,
int n_threads,
std::vector<TdtToken> & out_tokens,
const transcribe::PhraseBoost * boost) {
if (enc_out == nullptr || T_enc <= 0 || d_enc <= 0) {
return TRANSCRIBE_ERR_INVALID_ARG;
}
Expand Down Expand Up @@ -1316,6 +1352,9 @@ transcribe_status decode_rnnt_greedy(const HostDecoderWeights & w,
// RNN-T iters/token ratio.
bool predictor_dirty = true;

int boost_state = transcribe::PhraseBoost::root;
std::vector<float> boosted;

while (step < T_enc && iter < max_iters) {
++iter;

Expand All @@ -1342,7 +1381,7 @@ transcribe_status decode_rnnt_greedy(const HostDecoderWeights & w,

// RNNT joint output is just `n_token_cls` floats (no duration extras).
const float * token_logits = logits.data();
const int pred_token = argmax_range(token_logits, n_token_cls);
const int pred_token = biased_argmax(token_logits, n_token_cls, blank_id, boost, boost_state, boosted);

if (iter == 1 && transcribe::debug::enabled()) {
const long long s_h = H;
Expand Down Expand Up @@ -1377,6 +1416,9 @@ transcribe_status decode_rnnt_greedy(const HostDecoderWeights & w,
last_token = pred_token;
std::swap(state, next_state);
predictor_dirty = true;
if (boost != nullptr) {
boost_state = boost->next(boost_state, pred_token);
}

new_symbols += 1;
if (w.tdt_max_symbols > 0 && new_symbols >= w.tdt_max_symbols) {
Expand Down
31 changes: 19 additions & 12 deletions src/arch/parakeet/decoder.h
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@

#include "ggml-backend.h"
#include "ggml.h"
#include "transcribe-phrase-boost.h"
#include "transcribe.h"

#include <cstdint>
Expand Down Expand Up @@ -218,24 +219,30 @@ struct TdtToken {
//
// Returns TRANSCRIBE_OK on success; cannot fail except via invalid args
// (validated by the family driver before this is called).
transcribe_status decode_tdt_greedy(const HostDecoderWeights & w,
const float * enc_out,
int T_enc,
int d_enc,
int n_threads,
std::vector<TdtToken> & out_tokens);
//
// `boost`, when non-null and non-empty, biases the token argmax toward the
// caller's vocabulary terms (transcribe_run_params::vocabulary); see
// transcribe-phrase-boost.h. Token confidences stay the model's own.
transcribe_status decode_tdt_greedy(const HostDecoderWeights & w,
const float * enc_out,
int T_enc,
int d_enc,
int n_threads,
std::vector<TdtToken> & out_tokens,
const transcribe::PhraseBoost * boost = nullptr);

// Run RNNT greedy decode end-to-end. Same predictor + joint code as TDT,
// but the joint emits `vocab+1` logits (no duration extras) and the step
// rule is "blank → advance one frame, non-blank → emit + stay, capped by
// tdt_max_symbols". Per-emit duration_frames is fixed at 1. Same I/O
// contract and TdtToken result type as decode_tdt_greedy.
transcribe_status decode_rnnt_greedy(const HostDecoderWeights & w,
const float * enc_out,
int T_enc,
int d_enc,
int n_threads,
std::vector<TdtToken> & out_tokens);
transcribe_status decode_rnnt_greedy(const HostDecoderWeights & w,
const float * enc_out,
int T_enc,
int d_enc,
int n_threads,
std::vector<TdtToken> & out_tokens,
const transcribe::PhraseBoost * boost = nullptr);

// Streaming variant of RNN-T greedy decode. Consumes T_enc_new encoder
// frames (the chunk just produced) and APPENDS emitted tokens to
Expand Down
96 changes: 94 additions & 2 deletions src/arch/parakeet/model.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@
#include "transcribe-log.h"
#include "transcribe-mel.h"
#include "transcribe-meta.h"
#include "transcribe-prompting.h"
#include "transcribe/parakeet.h"
#include "weights.h"

Expand All @@ -40,6 +41,7 @@
#include <cmath>
#include <cstdint>
#include <cstdio>
#include <cstdlib>
#include <cstring>
#include <fstream>
#include <ios>
Expand Down Expand Up @@ -587,6 +589,10 @@ transcribe_status load(Loader & loader, const transcribe_model_load_params * par
return st;
}

// Vocabulary terms bias the greedy transducer decoders (TDT and RNN-T);
// CTC heads do not take them yet.
transcribe::set_feature(m.get(), TRANSCRIBE_FEATURE_VOCABULARY, m->host_decoder.head_kind != HostHeadKind::CTC);

// Multitalker bundle post-load: fuse the diarizer conformer's BN (needs
// uploaded data), build its ceil-framed mel frontend, and publish the
// DIARIZATION capability — for a bundle it is a property of the file.
Expand Down Expand Up @@ -789,6 +795,81 @@ static double parakeet_ms_per_enc_frame(const ParakeetHParams & hp) {
static_cast<double>(hp.fe_sample_rate);
}

// Logit bonus per matched vocabulary token after a term's first, in
// natural-log units. The lowest value that kept the term-recall gains while
// leaving transcripts of unrelated speech unchanged; see docs/prompting.md.
constexpr float kVocabularyBoost = 3.0f;

// `term` with the case of its first letter flipped ("Postgres" ->
// "postgres"), or "" when it does not start with an ASCII letter. Sentence
// position decides that letter's case in the transcript, not the term list.
static std::string flip_first_letter(const std::string & term) {
std::string out = transcribe::prompting::strip(term);
if (out.empty()) {
return "";
}
const char c = out[0];
if (c >= 'a' && c <= 'z') {
out[0] = static_cast<char>(c - 'a' + 'A');
} else if (c >= 'A' && c <= 'Z') {
out[0] = static_cast<char>(c - 'A' + 'a');
} else {
return "";
}
return out;
}

// The phrase-boost automaton for this run's vocabulary, or nullptr when the
// run has none. Terms are tokenized as the model writes them (with the word
// boundary marker), so a term matches only from the start of a word. Each
// term also matches with its first letter's case flipped.
static const transcribe::PhraseBoost * vocabulary_boost(ParakeetSession * pc,
const ParakeetModel * pm,
const transcribe_run_params * params) {
std::vector<std::string> terms = transcribe::prompting::terms(params);
if (terms.empty() || pm->host_decoder.head_kind == HostHeadKind::CTC) {
return nullptr;
}
if (terms != pc->boost_terms || pc->boost.empty()) {
std::vector<std::vector<int32_t>> phrases;
for (const std::string & term : terms) {
std::vector<int32_t> ids;
if (pm->tok.encode_sentencepiece_bpe(term, ids, 0, -1, /*remove_extra_whitespaces=*/true) !=
TRANSCRIBE_OK) {
log_msg(TRANSCRIBE_LOG_LEVEL_WARN,
"parakeet: this tokenizer cannot encode vocabulary terms; ignoring them");
return nullptr;
}
bool unknown = false;
for (const int32_t id : ids) {
unknown = unknown || pm->tok.is_unknown(id);
}
if (unknown) {
log_msg(TRANSCRIBE_LOG_LEVEL_WARN,
"parakeet: vocabulary term '%s' has characters the model cannot write; skipped", term.c_str());
continue;
}
phrases.push_back(std::move(ids));

const std::string flipped = flip_first_letter(term);
if (!flipped.empty() &&
pm->tok.encode_sentencepiece_bpe(flipped, ids, 0, -1,
/*remove_extra_whitespaces=*/true) == TRANSCRIBE_OK) {
bool flipped_unknown = false;
for (const int32_t id : ids) {
flipped_unknown = flipped_unknown || pm->tok.is_unknown(id);
}
if (!flipped_unknown) {
phrases.push_back(std::move(ids));
}
}
}
pc->boost.build(phrases, kVocabularyBoost);
pc->boost_terms = std::move(terms);
}
return pc->boost.empty() ? nullptr : &pc->boost;
}

// Host-side TDT/RNNT/CTC decode + public result-hierarchy build for one
// utterance's encoder output. `enc` is the row-major [T_enc, d_enc]
// activation for ONE utterance; utt_index >= 0 tags the per-utterance
Expand Down Expand Up @@ -827,10 +908,12 @@ static transcribe_status decode_and_populate(ParakeetSession * pc,
transcribe_status st = TRANSCRIBE_OK;
switch (pm->host_decoder.head_kind) {
case HostHeadKind::TDT:
st = decode_tdt_greedy(pm->host_decoder, enc, T_enc, d_enc, pc->n_threads, pc->raw_tokens);
st = decode_tdt_greedy(pm->host_decoder, enc, T_enc, d_enc, pc->n_threads, pc->raw_tokens,
vocabulary_boost(pc, pm, params));
break;
case HostHeadKind::RNNT:
st = decode_rnnt_greedy(pm->host_decoder, enc, T_enc, d_enc, pc->n_threads, pc->raw_tokens);
st = decode_rnnt_greedy(pm->host_decoder, enc, T_enc, d_enc, pc->n_threads, pc->raw_tokens,
vocabulary_boost(pc, pm, params));
break;
case HostHeadKind::CTC:
st = decode_ctc_greedy(pm->host_decoder, enc, T_enc, d_enc, pc->n_threads, pc->raw_tokens);
Expand Down Expand Up @@ -3406,6 +3489,15 @@ transcribe_status stream_begin(transcribe_session * session,

pc->stream_audio_input_samples = 0;

// Vocabulary biasing is wired into the offline decoders only; say so
// rather than drop the terms silently.
if (run_params->n_vocabulary > 0) {
log_msg(TRANSCRIBE_LOG_LEVEL_WARN,
"parakeet: vocabulary terms are not applied on the streaming API yet; ignoring %d term(s) "
"(use transcribe_run)",
run_params->n_vocabulary);
}

// -------- Buffered streaming path (parakeet-unified-en-0.6b) --------
//
// chunked_limited_with_rc with a 3-tuple training menu. Re-runs the
Expand Down
13 changes: 8 additions & 5 deletions src/arch/parakeet/parakeet.h
Original file line number Diff line number Diff line change
Expand Up @@ -313,11 +313,14 @@ struct ParakeetSession final : public transcribe_session {
ggml_tensor * encoder_out = nullptr;

// Per-context scratch reused across runs.
std::vector<float> mel_buf;
std::vector<float> pos_buf;
std::vector<float> pos_div_term;
std::vector<float> enc_host;
std::vector<TdtToken> raw_tokens;
std::vector<float> mel_buf;
std::vector<float> pos_buf;
std::vector<float> pos_div_term;
std::vector<float> enc_host;
std::vector<TdtToken> raw_tokens;
// Vocabulary biasing, rebuilt only when the run's terms change.
std::vector<std::string> boost_terms;
transcribe::PhraseBoost boost;

// Multitalker bundle scratch: streaming-diarizer AOSC/FIFO state for
// the embedded sortformer forward (diarize=ON runs only).
Expand Down
Loading
Loading