Skip to content
azerothl
Rbitnet
Public

About

Port bitnet to Rust

azerothl.github.io/Rbitnet/

Resources

Readme
MIT license
Activity

Stars

2 stars

Watchers

0 watching

Forks

0 forks
Report repository
BranchesTags

Repository files navigation

Rbitnet

Rbitnet pixel rabbit mark

Native Rust GGUF inference for Llama-family, BitNet, Qwen, Mixtral, GPT-OSS, GLM-4.7-Flash and Spark-X2.5 models, plus an OpenAI-compatible HTTP server for Akasha (BitNetProvider). Support depends on the exact architecture, tensor layout and backend; see the architecture matrix and known limits.

Where Rbitnet fits

Train with Unsloth, TRL, or Hugging Face, export a supported GGUF, then serve it with Rbitnet. Native paths cover Llama-family, BitNet, dense Qwen3 and Qwen3.5, Mixtral, GPT-OSS, GLM-4.7-Flash and Spark-X2.5 exports. Real-model validation and backend coverage vary by family; consult the architecture matrix and documented limits before choosing a checkpoint. Serving needs no Python, PyTorch or llama.cpp inference engine.

Stage Owner
Fine-tuning and GGUF export Unsloth / TRL / Hugging Face and upstream converters
Local GGUF inference and OpenAI HTTP API Rbitnet (akasha-infer), used by Akasha's BitNetProvider
Assistant UI and tools; optional OS daemon Akasha; akasha-os

Unsloth Studio has its own training and serving workflow. Rbitnet provides the GGUF runtime for Akasha; it does not offer a Studio, RAG or MCP platform. Follow the Unsloth / TRL to Rbitnet walkthrough, compatibility rules and native-first policy.

Project name: akasha-infer · Code / crates: Rbitnet — see docs/AKASHA_INFER.md for the frozen /v1 + /metrics contract. Optional in-process embed: docs/ENGINE_API.md (bitnet-core semver surface; HTTP remains the Akasha default).

akasha-infer = Rbitnet: the local GGUF engine for Akasha (BitNetProvider). Akasha is the assistant; akasha-os is the optional caps/daemon layer; this repo is the Rust inference satellite — not a second product. Contract checklist + smoke: docs/AKASHA_INFER.md · scripts/smoke_openai.sh.

Brand & site: docs/BRAND.md · presentation site in website/ (GitHub Pages; English default, docs on-site under website/docs/).

Serve profiles (rbitnet tune)

Profile Intent Key flags
interactive Low-latency chat (≥2 sessions) PREFIX_KV=1, CB off, CPU
throughput (batch) Concurrent packing CB on, KV_POOL=1, KV_QUANT=q8
bitnet-cpu Akasha BitNetProvider CPU ARCHITECTURE=bitnet, prefix KV, no CUDA graph
battery Minimal power concurrency 1, heavy paths off
latency Low latency + CUDA prefix KV, CUDA_GRAPH=1 
rbitnet tune interactive          # set env in this process
rbitnet tune throughput --export  # print export lines for the shell

Details: docs/USAGE.md. BitNet recipe notes point at bitnet-cpu.

Do I need Python?

Not to run Rbitnet. The server and bitnet-core are self-sufficient in Rust: mmap the GGUF, dequantize weights, run the transformer, sample tokens. No external inference engine is required.

You only need Python (or other tools) if you are converting a Hugging Face / Safetensors checkpoint into GGUF upstream (for example Microsoft BitNet or llama.cpp converters). That is export-time, not a runtime dependency.

Optional helper: [scripts/setup_env.py](scripts/setup_env.py) — download HF weights (huggingface_hub), print RBITNET_* lines; calling Microsoft BitNet is optional — see docs/MODEL_TESTING.md.

Start here: docs/USAGE.md (models, tokenizer, env vars, hot reload, TUI chat, curl examples). Quick start (FR): docs/DEMARRAGE_5MIN.md.

Installation

The installer downloads the latest tagged release. It does not clone the repository and it does not require Rust.

Windows

irm https://raw.githubusercontent.com/azerothl/Rbitnet/main/scripts/install.ps1 | iex

Binaries and compatible_models.json go in %LOCALAPPDATA%\Rbitnet, which is added to the user PATH. Open a new terminal, then rbitnet --version.

.\scripts\install.ps1 -Version 0.2.0
.\scripts\install.ps1 -Uninstall

macOS / Linux

curl -fsSL https://raw.githubusercontent.com/azerothl/Rbitnet/main/scripts/install.sh | sh

Binaries and compatible_models.json go in ~/.local/share/rbitnet. The script appends that directory to PATH in ~/.profile, ~/.bashrc and ~/.zprofile. Open a new terminal, then rbitnet --version.

./scripts/install.sh --version 0.2.0
./scripts/install.sh --uninstall

Release archives are rbitnet-server-vX.Y.Z-windows-x86_64.zip, rbitnet-server-vX.Y.Z-linux-x86_64.tar.gz and rbitnet-server-vX.Y.Z-macos-arm64.tar.gz on GitHub Releases. Each archive contains rbitnet, rbitnet-server, rbitnet-runner and rbitnet-proxy.

Build from source

Developers with a checkout can compile the CLI and server:

cargo build -p bitnet-server -p rbitnet-cli -p rbitnet-runner -p rbitnet-proxy --release --locked

Docker

The included [Dockerfile](Dockerfile) builds rbitnet-server and is usable for server-only deployments:

docker build -t rbitnet:local .
docker run --rm -e RBITNET_MODEL=/model/model.gguf -e RBITNET_TOKENIZER=/model/tokenizer.json \
  -v /path/on/host:/model:ro -p 8080:8080 rbitnet:local

Status

Works Today

Rbitnet supports multiple native GGUF architecture paths; it is not limited to Llama-shaped models. The first curated BitNet target is Microsoft's microsoft-bitnet-b1.58-2b-4t bundle. Exact tensor layouts, backend availability and real-model validation differ by family, so check docs/ARCHITECTURE_GGUF_MATRIX.md and docs/LIMITATIONS.md. For BitNet installation and setup, see docs/BITNET_NATIVE.md.

Recent inference-engine features include:

  • RBITNET_QUANT_KERNEL=auto for the shared CPU-parallel quantized matvec path.
  • RBITNET_BACKEND=hybrid plus RBITNET_HYBRID_POLICY={layers,hotcold,auto} for CPU/GPU offload planning.
  • RBITNET_LLAMA_PAGED_KV=1 and RBITNET_KV_QUANT={off,q8,q4} for paged Llama KV (Q8/Q4 = compact CPU pages, no F32 twin; see scripts/bench_kv_q8.sh).
  • RBITNET_PREFILL_CHUNK_TOKENS, RBITNET_CONTINUOUS_BATCHING + RBITNET_ITERATION_TOKEN_BUDGET (Sarathi stall-free), speculative draft hooks (RBITNET_DRAFT_PATH), and experimental JSON/tool sampler hooks (RBITNET_STRUCTURED_OUTPUT); structured generation returns HTTP 501 until a tokenizer-aware grammar is validated.
  • Runtime admin reload through POST /v1/admin/reload, useful for switching models/configs while the server stays up.

Examples of public Llama-family GGUF repos checked through the Hugging Face model API (gguf.architecture=llama):

  • TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF with tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf (small CPU smoke model).
  • unsloth/Llama-3.2-1B-Instruct-GGUF with Llama-3.2-1B-Instruct-Q4_K_M.gguf.
  • NousResearch/Hermes-2-Pro-Llama-3-8B-GGUF with Hermes-2-Pro-Llama-3-8B-Q4_K_M.gguf (RBITNET_CHAT_FORMAT=chatml recommended).

Native BitNet curated install (stable tag bitnet:2b):

rbitnet models install bitnet:2b --dir ./models
# paths also written to models/rbitnet.manifest.json
rbitnet recipe recipes/bitnet-b158.recipe.json
# or: rbitnet tune bitnet-cpu && rbitnet serve

Install the release, pull a tagged starter model, then serve:

curl -fsSL https://raw.githubusercontent.com/azerothl/Rbitnet/main/scripts/install.sh | sh
rbitnet models install tinyllama:q4 --dir ./models
rbitnet up tinyllama:q4 --dir ./models
rbitnet serve

Stable tags are listed in data/compatible_models.json (bitnet:2b, tinyllama:q4, …). Hub repo ids still work with quickstart / up.

For Windows quick install:

irm https://raw.githubusercontent.com/azerothl/Rbitnet/main/scripts/install.ps1 | iex
rbitnet quickstart TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF --file tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf

Run the server (real GGUF)

Place **tokenizer.json** (or tokenizer.model) beside the .gguf, or set RBITNET_TOKENIZER. rbitnet serve and rbitnet-server also read rbitnet.toml from the current directory, RBITNET_CONFIG, or the user config directory; explicit env vars still win.

export RBITNET_MODEL=/absolute/path/to/model.gguf
export RBITNET_BIND=127.0.0.1:8080
cargo run -p bitnet-server --bin rbitnet-server --release

Stub (no weights, integration text):

export RBITNET_STUB=1
export RBITNET_BIND=127.0.0.1:8080
cargo run -p bitnet-server --bin rbitnet-server

Toy LM (no GGUF):

export RBITNET_TOY=1
cargo run -p bitnet-server --bin rbitnet-server

Open the local static UI from the same server:

RBITNET_STUB=1 rbitnet serve --open-ui
# or visit http://127.0.0.1:8080/ui

Quick terminal testing

Use the terminal chatbot TUI to test a running server:

rbitnet chat --base-url http://127.0.0.1:8080/v1 --model rbitnet-llama

Or let the TUI launch and manage a local server:

rbitnet chat --serve \
  --model-path /absolute/path/to/model.gguf \
  --tokenizer /absolute/path/to/tokenizer.json \
  --chat-format raw

Useful keys:

Key Action
Enter Send the prompt.
F2 / F3 Decrease / increase max_tokens.
- / + Decrease / increase temperature.
F4 Fetch /v1/models.
Ctrl+R Call POST /v1/admin/reload when RBITNET_ADMIN_TOKEN is set.
Ctrl+U Call POST /v1/admin/unload when RBITNET_ADMIN_TOKEN is set.
Esc or Ctrl+C Quit.

Add --transcript chat.jsonl to append prompt/reply rows during quick regression tests.

Hot reload

Set an admin token before starting the server:

export RBITNET_ADMIN_TOKEN=dev-secret

Reload the current env/config model without restarting the HTTP process:

curl -sS -X POST http://127.0.0.1:8080/v1/admin/reload \
  -H 'X-Rbitnet-Admin-Token: dev-secret' \
  -H 'Content-Type: application/json' \
  -d '{}'

Reload a registry model:

curl -sS -X POST http://127.0.0.1:8080/v1/admin/reload \
  -H 'X-Rbitnet-Admin-Token: dev-secret' \
  -H 'Content-Type: application/json' \
  -d '{"active_model_id":"tiny"}'

Reload a specific GGUF:

curl -sS -X POST http://127.0.0.1:8080/v1/admin/reload \
  -H 'X-Rbitnet-Admin-Token: dev-secret' \
  -H 'Content-Type: application/json' \
  -d '{"model":"/models/tiny.gguf","tokenizer":"/models/tokenizer.json","architecture":"llama"}'

Reload metrics are exposed as rbitnet_model_reloads_total, rbitnet_model_reload_failures_total, and rbitnet_model_reload_ms_sum.

Benchmarks

Use docs/BENCHMARKS.md for the procedure and docs/BENCHMARKS_RESULTS.md for dated results. Rows may cover real GGUF runs, unit checks or stub-only HTTP measurements; read each row's method before comparing. The helper scripts start a stub server by default for API-overhead smoke checks. For real throughput numbers, start a model and run scripts/bench_matrix.* with NO_START_SERVER=1 / -NoStartServer, or use scripts/compare_llamacpp_rbitnet.sh when llama-bench is available. See docs/MODEL_MATRIX.md for reference models.

Inspect a GGUF

cargo run -p bitnet-core --example inspect_gguf -- /path/to/model.gguf

Test BitNet / Ternary GGUF

Use the curated microsoft-bitnet-b1.58-2b-4t bundle for native GGUF inference, or convert Safetensors with Microsoft BitNet tooling, then add the tokenizer and point RBITNET_MODEL at the .gguf. Walkthrough: docs/BITNET_NATIVE.md and docs/MODEL_TESTING.md.

Optional automated parse check (local only):

export RBITNET_TEST_GGUF=/path/to/model.gguf
cargo test -p bitnet-core optional_gguf_from_env_smoke -- --nocapture

Akasha

  1. Run rbitnet-server (see docs/USAGE.md).
  2. In your data directory, edit llm_router.yaml:
providers:
  bitnet:
    base_url: "http://127.0.0.1:8080"

task_types:
  conversation:
    primary:
      provider: bitnet
      model: rbitnet-llama
    fallback:
      - provider: akasha_core
        model: core

The model field must match an id from GET /v1/models (rbitnet-stub, rbitnet-toy, or rbitnet-<architecture> when a GGUF is loaded).

See the Akasha repo: spec/llm_router.example.yaml.

Hermes / Akasha ecosystem (ops)

Rbitnet is the local OpenAI-compatible backend in the Akasha multi-reference parity story. For self-hosted SLO alignment with the daemon router, scrape **GET /metrics** on Rbitnet and compare with Akasha’s **GET /api/router/metrics** (see docs/USAGE.md § correlation). Product parity tracking: [Akasha spec/dev/roadmap/reference-products-parity-matrix.md](https://github.com/azerothl/Akasha/blob/main/spec/dev/roadmap/reference-products-parity-matrix.md) and [Akasha spec/dev/roadmap/hermes-integration-remainder.md](https://github.com/azerothl/Akasha/blob/main/spec/dev/roadmap/hermes-integration-remainder.md).

Environment

Variable Meaning
RBITNET_BIND Host:port (default 127.0.0.1:8080)
RBITNET_MODEL Path to .gguf for real inference
RBITNET_TOKENIZER Path to tokenizer.json or tokenizer.model if not beside the GGUF
RBITNET_STUB 1 = stub text (no inference)
RBITNET_TOY 1 = tiny in-process F32 toy LM (no GGUF)
RBITNET_BACKEND auto selects an available backend for the architecture; explicit cpu / cuda / hybrid / rocm paths have model-specific limits. Vulkan/Metal/Intel GPU inference is not implemented. See LIMITATIONS.md.
RBITNET_QUANT_KERNEL auto, scalar, or cuda quantized matvec backend
RBITNET_HYBRID_POLICY layers, hotcold, or auto layer offload policy
RBITNET_LLAMA_PAGED_KV / RBITNET_KV_QUANT Enable paged KV and choose off, q8, or q4 KV format
RBITNET_ADMIN_TOKEN Enables POST /v1/admin/unload and POST /v1/admin/reload
RBITNET_CHAT_BASE_URL Default base URL for rbitnet chat
RBITNET_TOY_SEED Seed for toy weights (default 42)
RBITNET_TEST_GGUF Optional path for optional_gguf_from_env_smoke test only

Server tuning (rbitnet-server): RBITNET_MAX_BODY_BYTES, RBITNET_MAX_PROMPT_CHARS, RBITNET_MAX_TOKENS_CAP, RBITNET_MAX_CONCURRENT, RBITNET_INFERENCE_TIMEOUT_SECS, RBITNET_API_KEY — see docs/USAGE.md.

Benchmarks

cargo bench -p bitnet-core

License

MIT — see LICENSE.

About

Port bitnet to Rust

azerothl.github.io/Rbitnet/

Resources

Readme
MIT license
Activity

Stars

2 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages