Titans introduce a neural long-term memory module that enhances attention mechanisms by allowing models to utilize both current and historical context, improving dependency modeling without the quadratic cost of traditional attention.
MatterGen: A new paradigm of materials design with generative AI
MatterGen revolutionizes materials design by utilizing generative AI to directly create novel materials based on specified properties, rather than relying on traditional screening methods that exhaust known candidates.
Diffusion training from scratch on a micro-budget
SonyResearch's micro_diffusion demonstrates training a 1.16 billion parameter sparse transformer on a mere $1,890, utilizing 37M images to achieve an FID of 12.7 in zero-shot generation on the COCO dataset, showcasing cost-effective model training.
Framework for Artificial Intelligence Diffusion
The Bureau of Industry and Security (BIS) has introduced a framework to regulate the export and diffusion of advanced AI technologies, particularly focusing on AI model weights and advanced computing integrated circuits (ICs) to safeguard U.S. national security and foreign policy interests.
Let's talk about AI and end-to-end encryption
AI's integration with end-to-end encryption raises significant privacy concerns, as the need for powerful processing may lead to increased reliance on remote servers, potentially exposing sensitive data to unauthorized access.
Grokking at the Edge of Numerical Stability [Research]
Grokking reveals a phenomenon where models suddenly generalize after prolonged overfitting, yet the role of regularization in this process remains poorly understood, leading to the introduction of Softmax Collapse (SC) as a critical factor hindering grokking.
MuJoco Playground
MuJoCo Playground is an open-source framework designed for robot learning, enabling rapid training on diverse robotic platforms with a simple installation process via pip install playground.
Dr. TVAM – Inverse Rendering for Tomographic Volumetric Additive Manufacturing
Dr.TVAM is an inverse rendering framework designed for tomographic volumetric additive manufacturing, leveraging the Mitsuba renderer to optimize printing patterns through physically-based differentiable rendering.
Skyvern Browser Agent 2.0: How We Reached State of the Art in Evals
Skyvern 2.0 achieves a remarkable score of 85.85% on the WebVoyager Eval, showcasing its advanced capabilities in executing complex web navigation tasks autonomously, outperforming many closed-source competitors like Google Mariner.
[R] Multimodal Visualization-of-Thought: Enhancing MLLM Reasoning Through Visual Thinking
The Multimodal Visualization-of-Thought (MVoT) system enhances reasoning in large language models by integrating image generation, allowing AI to "visually think" and improve problem-solving capabilities.
NVIDIA Releases NIM Microservices to Safeguard Applications for Agentic AI
NVIDIA's new NIM microservices enhance the safety, precision, and scalability of generative AI applications, addressing critical enterprise concerns like trust and compliance through the NVIDIA NeMo Guardrails framework.
Ideas: AI for materials discovery with Tian Xie and Ziheng Lu
AI tools MatterGen and MatterSim revolutionize materials discovery by enabling the generation and simulation of materials tailored to specific properties, significantly enhancing efficiency in identifying novel materials for applications like batteries and energy storage.
Research Focus: Week of January 13, 2025
Microsoft Research introduces privacy enhancements for multiparty deep learning, significantly improving communication efficiency and speed in model training through Differentially Private, secure Multiparty Computation (DP-MPC) protocols, achieving up to 794 times more efficiency than previous methods.
Rational Tuning of LLM Cascades via Probabilistic Modeling
This paper introduces a probabilistic model for the joint performance distribution of LLM cascades, enabling rational tuning of confidence thresholds through continuous optimization, which enhances performance predictability.
To Retrieve or Not to Retrieve? Uncertainty Detection for Dynamic Retrieval Augmented Generation
Dynamic retrieval in Retrieval-Augmented Generation can significantly enhance efficiency by only invoking external knowledge when the LLM lacks necessary information, addressing limitations in long-form question answering.