Lila Sciences and Anyscale Achieve 1M-Token Training with GLM-5.3-Flash

Lila Sciences and Anyscale have successfully trained the GLM-5.3-Flash model on sequences of up to 1 million tokens, a significant milestone in long-context AI training. The collaboration utilized SkyRL, an open-source training stack, running on 64 NVIDIA B200 GPUs. This achievement represents a 64-fold increase in sequence length compared to the initial training capacity and addresses the growing need for AI models capable of handling extensive scientific datasets. The GLM-5.3-Flash model incorporates a hybrid architecture with 34 layers of Kimi Delta Attention (linear attention) and 11 layers of DeepSeek Sparse Attention (DSA).

Source: Blockchain.News