中文 | English
⚠️ Note: This is a relatively early version of the iFlytek Spark model (released in 2024). We converted it to Hugging Face format primarily for research purposes — to help the community study early LLM architectures, compare with modern models, and understand how the field has evolved.
This is a community-converted Hugging Face compatible version of the iFlytek Spark 13B model. The original weights were converted from the official Megatron-DeepSpeed format to work seamlessly with the transformers ecosystem.
- Hugging Face Hub: freedomking/OpenSpark-13B-Chat
- Original Weights: iFlytek Spark-13B on Gitee
- Training Framework: Megatron-DeepSpeed
- Release Date: 2024 (Early Version)
This repository contains the code and configuration for the 2024 version of the iFlytek Spark model, converted to Hugging Face format. It is intended for educational and research purposes, allowing the community to study the architecture of early large language models and compare them with modern iterations.
pip install torch transformers sentencepieceYou can load this model using the transformers library. Ensure you have trust_remote_code=True set to load the model and tokenizer logic.
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
model_path = "freedomking/OpenSpark-13B-Chat"
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
model_path,
torch_dtype=torch.bfloat16,
device_map="auto",
trust_remote_code=True
)
prompt = "<User> 你好,请自我介绍一下。<end><Bot>"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=512)
print(tokenizer.decode(outputs[0], skip_special_tokens=False))
# 输出:<User> 你好, 请自我介绍一下。 <end><Bot> 您好,我是科大讯飞研发的认知智能大模型,我的名字叫讯飞星火。我有以下几个特点: <ret><ret>1. 专业知识:我具备丰富的专业知识,包括语言、历史、科学、技术、文化等各个领域的知识储备。<ret>2. 自然语言处理:我能够理解和生成自然、流畅的中文文本,进行多轮对话。<ret>3. 个性化定制:我可以根据用户的需求和喜好,提供个性化的服务和建议。<ret>4. 持续学习:我可以通过海量数据和大规模预训练,不断提升自己的知识和技能。<ret>5. 多模态交互:我不仅能够处理文本输入,还可以处理图片、语音、视频等多种形式的信息。<ret>6. 跨领域应用:我可以应用于教育、医疗、法律、金融、智能家居等多个领域,为用户提供智能化的解决方案。<end>For multi-turn conversations, use the built-in chat template:
messages = [
{"role": "user", "content": "你好,请自我介绍一下。"}
]
inputs = tokenizer.apply_chat_template(
messages,
tokenize=True,
return_tensors="pt",
add_generation_prompt=True
).to(model.device)
outputs = model.generate(
inputs,
max_new_tokens=8192,
temperature=0.7,
top_k=1,
do_sample=True,
repetition_penalty=1.02,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=False))messages = [
{"role": "user", "content": "什么是人工智能?"},
{"role": "assistant", "content": "人工智能是一种模拟人类智能的技术..."},
{"role": "user", "content": "它有哪些应用场景?"}
]
inputs = tokenizer.apply_chat_template(
messages,
tokenize=True,
return_tensors="pt",
add_generation_prompt=True
).to(model.device)
outputs = model.generate(inputs, max_new_tokens=512)
print(tokenizer.decode(outputs[0], skip_special_tokens=False))
# <User> 什么是人工智能? <end><Bot> 人工智能是一种模拟人类智能的技术...<end><User> 它有哪些应用场景? <end><Bot> 人工智能的应用场景非常广泛,包括但不限于以下几个方面:<ret><ret>1. 自然语言处理:包括语音识别、机器翻译、文本摘要、情感分析等。<ret>2. 计算机视觉:包括图像识别、人脸识别、物体检测、视频分析等。<ret>3. 推荐系统:根据用户行为和偏好,为用户提供个性化的推荐服务,如新闻推荐、音乐推荐、商品推荐等。<ret>4. 自动驾驶:通过计算机视觉、传感器等技术,实现车辆的自主驾驶。<ret>5. 医疗诊断:通过分析医学影像、病历等数据,辅助医生进行疾病诊断和治疗。<ret>6. 金融风控:通过大数据分析,识别潜在的金融风险,提高金融机构的风险管理能力。<ret>7. 智能制造:通过物联网、传感器等技术,实现生产过程的自动化和智能化。<ret>8. 智能家居:通过语音识别、机器学习等技术,实现家居设备的智能化控制。<ret>9. 教育领域:通过智能教育平台,为学生提供个性化的学习资源和辅导。<ret>10. 农业领域:通过无人机、遥感等技术,实现农业生产的智能化管理。<ret><ret>这些只是人工智能应用的冰山一角,随着技术的发展,人工智能将在更多领域发挥重要作用。<end>| Parameter | Value |
|---|---|
| Architecture | Transformer Decoder (Spark) |
| Parameters | ~13B |
| Hidden Size | 5120 |
| Layers | 40 |
| Attention Heads | 40 |
| Vocab Size | 60,000 |
| Context Length | 32K |
| RoPE Base (Theta) | 1,000,000 |
| Activation | Fast GeLU |
| Parameter | Recommended Value |
|---|---|
max_new_tokens |
8192 |
temperature |
0.7 |
top_k |
1 |
do_sample |
True |
repetition_penalty |
1.02 |
This project serves several purposes for the research community:
- Historical Reference: Study the architecture of early Chinese LLMs
- Benchmark Comparison: Compare performance against modern models (Qwen, DeepSeek, etc.)
- Educational Value: Understand the evolution of LLM design choices
- Ecosystem Compatibility: Run the model using standard Hugging Face APIs
This model (early 2024) represents an interesting point in the evolution of Chinese LLMs. Compared to the currently dominant Llama-3/Qwen-2 architectures, Spark 13B shows several distinct design choices:
- Spark 13B: Uses a
rope_thetaof 1,000,000. - Comparison: Standard Llama-2 used 10,000. A larger theta allows the model to generalize better to longer sequence lengths (extrapolate) without explicit context window extension techniques, crucial for its 32K context support.
- Spark 13B: Uses standard Multi-Head Attention (MHA) (
num_attention_heads=40,num_key_value_heads=40). - Comparison: Modern efficient models (like Llama-3, Qwen-2) typically employ Grouped Query Attention (GQA). GQA significantly reduces the KV cache size during inference. Spark 13B's use of MHA reflects the architectural standards before GQA became the default for large context models.
- Spark 13B: Uses standard
LayerNorm(with bias). - Comparison: Llama and Qwen use RMSNorm (Root Mean Square Layer Normalization), which is computationally simpler and more numerically stable for FP16/BF16 training.
- Spark 13B:
use_bias=Truein Linear layers and LayerNorm. - Comparison: Modern architectures (like PaLM, Llama) often remove bias terms (
bias=False) from dense layers to improve training stability and reduce memory bandwidth usage.
- Spark 13B: 60,000 tokens.
- Comparison:
- Llama-3: 128k
- Qwen-2: 152k
- A smaller vocabulary size (60k) suggests a more focused coverage of Chinese/English, whereas newer models aim for massive multilingual support.
Studying these differences helps researchers understand what worked and what changed. For instance, the shift from GeLU -> SiLU and LayerNorm -> RMSNorm in later models reflects a community-wide move towards training efficiency and stability.
- Chat Template: Supports
apply_chat_templatefor multi-turn dialogues (<User>...<end><Bot>...format). - Standardized Naming: Consistent with mainstream models like Qwen and Llama.
- Custom Tokenizer: Handles Chinese punctuation, tab formatting, and special tokens (
<ret>,<end>). - BFloat16 Support: Optimized for modern GPUs with BF16 precision.
This project is licensed under the Apache 2.0 License.