Procedural Knowledge in Pretraining Drives Reasoning in Large Language Models

Procedural knowledge in pretraining significantly enhances reasoning capabilities in Large Language Models (LLMs), revealing that models utilize distinct data sets for factual versus reasoning tasks, with procedural documents being crucial for the latter.

DynaSaur: Large Language Agents Beyond Predefined Actions

DynaSaur introduces a novel framework for LLM agents that enables dynamic action creation and composition, overcoming limitations of fixed action sets in complex environments.

Large Language Models as Markov Chains

Large language models (LLMs) can be equivalently represented as Markov chains, revealing insights into their inference power and convergence speed based on vocabulary size and context window.

Controlling AI's Growing Energy Needs

AI training consumes vast energy, with models like ChatGPT-3 using nearly 1,300 megawatt hours, equivalent to the annual energy use of 130 homes; this exponential growth in energy demand poses sustainability challenges.

How We Optimize LLM Inference for AI Coding Assistant

Full codebase context is essential for developer AI, as it significantly enhances the quality of code predictions, with Augment achieving a time to first token (TTFT) of less than 300ms for 10k input tokens, outperforming existing solutions by threefold.

We need data engineering benchmarks for LLMs

Data engineering benchmarks are essential for evaluating LLMs, as current frameworks like SWE-bench fail to address the unique challenges of data workflows, which prioritize data quality, reliability, and scalability over mere code correctness.

[P] TIME-MOE: Billion-Scale Time Series Forecasting with Mixture-of-Experts

Time-MOE is a 2.4B parameter open-source model that leverages Mixture-of-Experts (MOE) for effective zero-shot forecasting in time series analysis.