optimization

Streaming ConvolutionsAugust 24, 2023
Working out the math for streaming convolutions.
machine-learning speech algorithms optimization
Fast Attention ImplementationsJune 29, 2023
A reference collection of fast attention implementations.
machine-learning nlp optimization reference
RWKV Language Model MathJune 16, 2023
In-depth explanation of the math behind the RWKV model, with PyTorch implementations, plus a discussion of numerical stability.
machine-learning nlp deep-learning math optimization
Optimized Log-Sum-Exp PyTorch FunctionMay 20, 2020
A walkthrough of how to optimize the log-sum-exp function in PyTorch.
machine-learning math optimization software-engineering
Torch CUDA Extension TricksMay 6, 2020
Some tricks I found useful for writing CUDA extensions for PyTorch.
software-engineering gpu ml-frameworks optimization