Feb 26, 2025

ML Times

FFT Strikes Back: An Efficient Alternative to Self-Attention

• FFTNet offers an innovative solution to the quadratic complexity of self-attention by utilizing the Fast Fourier Transform (FFT), achieving global token mixing in $\mathcal{O}(n\log n)$ time.

Accelerating Generative AI with PyTorch: Segment Anything 2 - Fast and furious inference with low latency and fast cold starts

• Optimized SAM2: The implementation of AOTInductor’s ahead-of-time compilation via torch.export, along with reduced precision and batched prompts, achieves up to 13x improvement in p90 execution latency and queue times compared to standard eager mode PyTorch.

DeepSeek Open Sources DeepGEMM: Clean and efficient FP8 GEMM kernels

• DeepGEMM is a CUDA library for FP8 General Matrix Multiplications (GEMMs), optimized for NVIDIA Hopper tensor cores, featuring a lightweight Just-In-Time (JIT) compilation that eliminates the need for pre-installation compilation.

Hard problems that reduce to document ranking

• LLMs can effectively solve complex problems by transforming them into document ranking tasks, as demonstrated in locating N-day vulnerabilities through patch diffing, utilizing established ranking techniques from information retrieval.

The FFT Strikes Back: An Efficient Alternative to Self-Attention

• The Fast Fourier Transform (FFT) offers a scalable alternative to traditional self-attention by reducing computational complexity from O(n²) to O(n log n), enabling efficient modeling of long-range dependencies in sequences.

DeepSearcher: A local open-source Deep Research

• DeepSearcher is an advanced open-source research agent that enhances local deep research capabilities by integrating features like query routing and conditional execution flow, allowing for more efficient information retrieval and report generation.

Alexa+

• Alexa+ leverages generative AI to enhance user interaction, making conversations feel more natural and intuitive, akin to speaking with a knowledgeable friend.

A possible future of Python in the browser

• SPy is an emerging language designed for Python users, enabling native compilation to binaries or WebAssembly, thus enhancing performance significantly over existing solutions like Pyodide and MicroPython.

Train your own Reasoning model - GRPO works on just 5GB VRAM

• GRPO now operates on just 5GB of VRAM, enabling 10x longer context lengths without sacrificing accuracy, thanks to newly implemented Efficient GRPO algorithms.

Muon is Scalable for LLM Training

• Muon is an optimizing algorithm that achieves ∼2× computational efficiency over AdamW, saving about 50% FLOPs for training a 1.5B LLM on 39B tokens, demonstrating its scalability for larger models.

SpargeAttn: Accurate Sparse Attention Accelerating Any Model Inference

• SpargeAttn introduces a universal sparse and quantized attention mechanism that accelerates inference across various models by effectively predicting and filtering the attention map, thus reducing computational overhead.

Forecasting Rare Language Model Behaviors

• Anthropic's method leverages power-law scaling to predict rare AI risks, enabling early detection of issues like harmful responses or misaligned behavior before deployment.

From Records to Agents: The Overlooked Revolution in Enterprise Software

• Enterprise software is evolving from passive records to autonomous agents, enabling business objects like invoices to actively pursue their own approval and payment processes, fundamentally changing how organizations operate.

RAGSys: Real-Time Self-Improvement for LLMs Without Retraining

• RAGSys introduces a dynamic framework for LLMs that optimizes prompt composition by retrieving relevant few-shot examples and instructions tailored to each query, enhancing response quality in real-time.

Regarding quantization, what are the future directions in this topic for LLM/SLMs?

• Future directions in quantization for LLMs/SLMs may focus on enhancing model efficiency while maintaining performance, particularly through advanced techniques like post-training quantization and mixed-precision training.