Issues

• SMP2020-EWECT的数据集开源使用问题

  Tencent/TencentPretrain#133 · Filter by author pxaklbe opened on Oct 31, 2024
  3

• 请问这里的中文模型支持的最大输入序列长度是512tokens吗？超过512tokens就会被截断嘛？可不可以在微调的时候扩大模型的位置编码数量？

  Tencent/TencentPretrain#130 · Filter by author chengzi-big opened on Jul 21, 2024

• 单机2卡预训练LLAMA-7B报错TypeError: an integer is required (got type NoneType)

  Tencent/TencentPretrain#112 · Filter by author smallYellowCat opened on Nov 29, 2023
  1

• 【问题】deepspeed如何对不同显存大小分配数据，我有32G和16G两种大小的GPU

  Tencent/TencentPretrain#98 · Filter by author 18liumin opened on Sep 26, 2023
  1

• LLaMA2-70B格式转换

  Tencent/TencentPretrain#96 · Filter by author Double-bear opened on Sep 5, 2023

• size mismatch for classifier.weight: copying a param with shape torch.Size([7, 768]) from checkpoint, the shape in current model is torch.Size([2, 768]).

  Tencent/TencentPretrain#93 · Filter by author byte4tide opened on Aug 19, 2023

• KeyError: 'd'

  Tencent/TencentPretrain#92 · Filter by author byte4tide opened on Aug 19, 2023
  1

• Resume from checkpoint

  Tencent/TencentPretrain#81 · Filter by author mohammadaminabbasi opened on Aug 15, 2023

• 发现几个bug，dynamic_masking 意思是不是写反了，多了not

  Tencent/TencentPretrain#80 · Filter by author boluoyu opened on Aug 10, 2023

• lora推理

  Tencent/TencentPretrain#71 · Filter by author aSmallsheep opened on Jun 6, 2023
  1

• DeepSpeedZeRoOffload initialization failed (can't allocate memory)

  Tencent/TencentPretrain#70 · Filter by author treya-lin opened on May 25, 2023

• 关于扩充词表再增量预训练的疑问

  Tencent/TencentPretrain#68 · Filter by author qiancheng99 opened on May 19, 2023

• 关于LLAMA预训练的疑问

  Tencent/TencentPretrain#66 · Filter by author zhanghaok opened on May 17, 2023
  1

• 有计划支持torch2.0的flashattention这些吗

  Tencent/TencentPretrain#61 · Filter by author zhuojianc opened on May 11, 2023

• 训练完怎么执行推理，发现训练保存的模型只有17M

  Tencent/TencentPretrain#59 · Filter by author jiangjingyao opened on May 4, 2023
  1

• 似乎没有保存rotary_emb.inv_freq的权重

  Tencent/TencentPretrain#58 · Filter by author cristianoc20 opened on May 2, 2023
  10

• pretrain.py of llama-7b model, Exception: : Current loss scale already at minimum

  Tencent/TencentPretrain#57 · Filter by author liukaiyueyuo opened on Apr 26, 2023
  2

• 单机8卡A100-80G deepspeed ZERO3 或者 非ZERO3 pretrain LLaMA-7B时，不能充分利用显卡

  Tencent/TencentPretrain#56 · Filter by author ShadowTeamCN opened on Apr 25, 2023
  1

• 支持bloom模型的预训练吗

  Tencent/TencentPretrain#53 · Filter by author nishiwen1214 opened on Apr 18, 2023

• llama训练时，best状态存储导致训练卡顿，建议删除存储best文件部分代码，望记得更新

  Tencent/TencentPretrain#52 · Filter by author baketbek opened on Apr 16, 2023
  4

• lora训练llama 貌似不支持？

  Tencent/TencentPretrain#51 · Filter by author wind91725 opened on Apr 15, 2023
  3

• Any plan of supporting Bloom model?

  Tencent/TencentPretrain#50 · Filter by author wind91725 opened on Apr 15, 2023

• 多卡跑llama预训练出错：RuntimeError: CUDA error: invalid resource handle，Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

  Tencent/TencentPretrain#43 · Filter by author baketbek opened on Apr 11, 2023
  1

• lora貌似不支持ddp？

  Tencent/TencentPretrain#41 · Filter by author wutaiqiang opened on Apr 11, 2023

• 用自己的中文数据的话，preprocess中需要把数据格式调整成什么形式即可？这部分相关说明有吗？目标是想做llama的增量预训练

  Tencent/TencentPretrain#40 · Filter by author baketbek opened on Apr 10, 2023
  1