Hi Everyone, Unfortunately, we have a bunch of people traveling for conferences this week as well! So, let's skip the SAIL meeting this week. In the meantime, we are starting to plan for the Summer. In particular, there seems to be strong interest in running more focused reading groups and having tutorials about different open-source components (e.g., serving systems) and skills (e.g., GPU programming, dataset curation). Can you each please fill out this form https://docs.google.com/forms/d/e/1FAIpQLSekbsgCaRaMLa7uofOxUVm_tQMu8k6xQKjK... so we can get a sense of what participation would look like? Cheers, Ravi
I like the idea of building stuff to understand how it works. Stanford has a course like this to build LLMs from scratch: https://stanford-cs336.github.io/spring2025/.
In that spirit, here are a couple of suggestions for things we can build.
- An LLM serving system from scratch (~3k lines of code to cover important cases we'd care about for research)
- An LLM training system from scratch (~3k lines of code, just the basics to train a 1B to 8B dense model with reasonable MFU)
- An MoE training system.
If these sound fun, we can have volunteers to lead sections of those projects.
Tri
On Apr 29, 2025, at 4:17 PM, Ravi Netravali
participants (2)
-
Ravi Netravali -
Tri Dao