Skip to content

About

个人转化的讯飞星火 Spark-13B (2024版) 的 Hugging Face 实现。供社区研究早期大模型架构演进与学习用途。A Hugging Face compatible implementation of the early 2024 iFlytek Spark 13B Chat model. Converted for educational and research purposes to study LLM architecture evolution.

Resources

Stars

5 stars

Watchers

0 watching

Forks

Repository files navigation

OpenSpark-13B-Chat

中文 | English

⚠️ Note: This is a relatively early version of the iFlytek Spark model (released in 2024). We converted it to Hugging Face format primarily for research purposes — to help the community study early LLM architectures, compare with modern models, and understand how the field has evolved.

This is a community-converted Hugging Face compatible version of the iFlytek Spark 13B model. The original weights were converted from the official Megatron-DeepSpeed format to work seamlessly with the transformers ecosystem.

Source

This repository contains the code and configuration for the 2024 version of the iFlytek Spark model, converted to Hugging Face format. It is intended for educational and research purposes, allowing the community to study the architecture of early large language models and compare them with modern iterations.

Requirements

pip install torch transformers sentencepiece

Usage

You can load this model using the transformers library. Ensure you have trust_remote_code=True set to load the model and tokenizer logic.

Basic Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_path = "freedomking/OpenSpark-13B-Chat"

tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_path, 
    torch_dtype=torch.bfloat16, 
    device_map="auto", 
    trust_remote_code=True
)

prompt = "<User> 你好,请自我介绍一下。<end><Bot>"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(**inputs, max_new_tokens=512)
print(tokenizer.decode(outputs[0], skip_special_tokens=False))

# 输出:<User> 你好, 请自我介绍一下。 <end><Bot> 您好,我是科大讯飞研发的认知智能大模型,我的名字叫讯飞星火。我有以下几个特点: <ret><ret>1. 专业知识:我具备丰富的专业知识,包括语言、历史、科学、技术、文化等各个领域的知识储备。<ret>2. 自然语言处理:我能够理解和生成自然、流畅的中文文本,进行多轮对话。<ret>3. 个性化定制:我可以根据用户的需求和喜好,提供个性化的服务和建议。<ret>4. 持续学习:我可以通过海量数据和大规模预训练,不断提升自己的知识和技能。<ret>5. 多模态交互:我不仅能够处理文本输入,还可以处理图片、语音、视频等多种形式的信息。<ret>6. 跨领域应用:我可以应用于教育、医疗、法律、金融、智能家居等多个领域,为用户提供智能化的解决方案。<end>

Using apply_chat_template (Recommended)

For multi-turn conversations, use the built-in chat template:

messages = [
    {"role": "user", "content": "你好,请自我介绍一下。"}
]

inputs = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    return_tensors="pt",
    add_generation_prompt=True
).to(model.device)

outputs = model.generate(
    inputs,
    max_new_tokens=8192,
    temperature=0.7,
    top_k=1,
    do_sample=True,
    repetition_penalty=1.02,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=False))

Multi-turn Conversation

messages = [
    {"role": "user", "content": "什么是人工智能?"},
    {"role": "assistant", "content": "人工智能是一种模拟人类智能的技术..."},
    {"role": "user", "content": "它有哪些应用场景?"}
]

inputs = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    return_tensors="pt",
    add_generation_prompt=True
).to(model.device)

outputs = model.generate(inputs, max_new_tokens=512)
print(tokenizer.decode(outputs[0], skip_special_tokens=False))

# <User> 什么是人工智能? <end><Bot> 人工智能是一种模拟人类智能的技术...<end><User> 它有哪些应用场景? <end><Bot> 人工智能的应用场景非常广泛,包括但不限于以下几个方面:<ret><ret>1. 自然语言处理:包括语音识别、机器翻译、文本摘要、情感分析等。<ret>2. 计算机视觉:包括图像识别、人脸识别、物体检测、视频分析等。<ret>3. 推荐系统:根据用户行为和偏好,为用户提供个性化的推荐服务,如新闻推荐、音乐推荐、商品推荐等。<ret>4. 自动驾驶:通过计算机视觉、传感器等技术,实现车辆的自主驾驶。<ret>5. 医疗诊断:通过分析医学影像、病历等数据,辅助医生进行疾病诊断和治疗。<ret>6. 金融风控:通过大数据分析,识别潜在的金融风险,提高金融机构的风险管理能力。<ret>7. 智能制造:通过物联网、传感器等技术,实现生产过程的自动化和智能化。<ret>8. 智能家居:通过语音识别、机器学习等技术,实现家居设备的智能化控制。<ret>9. 教育领域:通过智能教育平台,为学生提供个性化的学习资源和辅导。<ret>10. 农业领域:通过无人机、遥感等技术,实现农业生产的智能化管理。<ret><ret>这些只是人工智能应用的冰山一角,随着技术的发展,人工智能将在更多领域发挥重要作用。<end>

Model Details

Parameter Value
Architecture Transformer Decoder (Spark)
Parameters ~13B
Hidden Size 5120
Layers 40
Attention Heads 40
Vocab Size 60,000
Context Length 32K
RoPE Base (Theta) 1,000,000
Activation Fast GeLU

Generation Parameters

Parameter Recommended Value
max_new_tokens 8192
temperature 0.7
top_k 1
do_sample True
repetition_penalty 1.02

Why This Conversion?

This project serves several purposes for the research community:

  1. Historical Reference: Study the architecture of early Chinese LLMs
  2. Benchmark Comparison: Compare performance against modern models (Qwen, DeepSeek, etc.)
  3. Educational Value: Understand the evolution of LLM design choices
  4. Ecosystem Compatibility: Run the model using standard Hugging Face APIs

Knowledge & Analysis: Spark 13B Architecture

This model (early 2024) represents an interesting point in the evolution of Chinese LLMs. Compared to the currently dominant Llama-3/Qwen-2 architectures, Spark 13B shows several distinct design choices:

1. Position Encoding (RoPE)

  • Spark 13B: Uses a rope_theta of 1,000,000.
  • Comparison: Standard Llama-2 used 10,000. A larger theta allows the model to generalize better to longer sequence lengths (extrapolate) without explicit context window extension techniques, crucial for its 32K context support.

2. Attention Mechanism (MHA vs GQA)

  • Spark 13B: Uses standard Multi-Head Attention (MHA) (num_attention_heads=40, num_key_value_heads=40).
  • Comparison: Modern efficient models (like Llama-3, Qwen-2) typically employ Grouped Query Attention (GQA). GQA significantly reduces the KV cache size during inference. Spark 13B's use of MHA reflects the architectural standards before GQA became the default for large context models.

3. Layer Normalization

  • Spark 13B: Uses standard LayerNorm (with bias).
  • Comparison: Llama and Qwen use RMSNorm (Root Mean Square Layer Normalization), which is computationally simpler and more numerically stable for FP16/BF16 training.

4. Bias Terms

  • Spark 13B: use_bias=True in Linear layers and LayerNorm.
  • Comparison: Modern architectures (like PaLM, Llama) often remove bias terms (bias=False) from dense layers to improve training stability and reduce memory bandwidth usage.

5. Vocabulary

  • Spark 13B: 60,000 tokens.
  • Comparison:
    • Llama-3: 128k
    • Qwen-2: 152k
    • A smaller vocabulary size (60k) suggests a more focused coverage of Chinese/English, whereas newer models aim for massive multilingual support.

Why Comparison Matters?

Studying these differences helps researchers understand what worked and what changed. For instance, the shift from GeLU -> SiLU and LayerNorm -> RMSNorm in later models reflects a community-wide move towards training efficiency and stability.

  • Chat Template: Supports apply_chat_template for multi-turn dialogues (<User>...<end><Bot>... format).
  • Standardized Naming: Consistent with mainstream models like Qwen and Llama.
  • Custom Tokenizer: Handles Chinese punctuation, tab formatting, and special tokens (<ret>, <end>).
  • BFloat16 Support: Optimized for modern GPUs with BF16 precision.

License

This project is licensed under the Apache 2.0 License.

About

个人转化的讯飞星火 Spark-13B (2024版) 的 Hugging Face 实现。供社区研究早期大模型架构演进与学习用途。A Hugging Face compatible implementation of the early 2024 iFlytek Spark 13B Chat model. Converted for educational and research purposes to study LLM architecture evolution.

Resources

Stars

5 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages