# Mar 12, 2024 OpenAI – transformer debugger release - **Transformer Debugger (TDB)**, developed by OpenAI's Superalignment team, facilitates the investigation of small language models by combining [automated interpretability techniques](https://openai.com/research/language-models-can-explain-neurons-in-language-models) with [sparse autoencoders](https://transformer-circuits.pub/2023/monosemantic-features), enabling users to understand and manipulate model behaviors without coding. - Building Meta's GenAI infrastructure - Meta announces the creation of **two 24k GPU clusters** designed for high throughput and reliability across various AI workloads, including the training of **Llama 3**, a next-generation AI model. - Diffusion models from scratch, from a new theoretical perspective - **Diffusion models, known for their effectiveness in generative modeling across various domains like text-to-image, audio, video, and protein design, are explored from an optimization perspective**, offering insights into their training and implementation for both simple and complex datasets as detailed in a [tutorial](https://www.chenyang.co/diffusion.html) based on a [new paper](https://arxiv.org/abs/2306.04848). - Devin: AI Software Engineer - **Devin**, the world's first fully autonomous AI software engineer, has been introduced by Cognition Labs, showcasing abilities to plan, execute complex engineering tasks, and learn over time. - Simpson's paradox - **Simpson's paradox** occurs when a trend observed within multiple groups reverses when the groups are combined, highlighting the complexity of statistical data interpretation. - Stealing Part of a Production Language Model - Researchers have developed a **model-stealing attack** capable of extracting the **embedding projection layer** from black-box production language models like OpenAI's ChatGPT or Google's PaLM-2, revealing their hidden dimensions. - Behind the Compute: Benchmarking Compute Solutions - Stability AI's **benchmarking** reveals **Intel Gaudi 2 accelerators** outperform Nvidia's A100 and H100 in training **Stable Diffusion 3**, showcasing **1.5 times faster** image processing and **superior scalability**. - Show HN: Prompts as WASM Programs - **The Artificial Intelligence Controller Interface (AICI) by Microsoft Research enables real-time control and direction of Large Language Model outputs through flexible Controllers**, which can implement constrained decoding, dynamic editing, and coordinate execution across multiple generations. - Is Cosine-Similarity of Embeddings Really About Similarity? - **Cosine-similarity**, often used to quantify semantic similarity between high-dimensional objects, can produce **arbitrary and potentially meaningless results** when applied to embeddings from regularized linear models. - (How to Write a (Lisp) Interpreter (In Python)) (2010) - Peter Norvig demonstrates how to implement a Lisp interpreter in Python, focusing on the Scheme dialect, to illustrate the foundational concepts of software, as highlighted by Alan Kay's "Maxwell's Equations of Software." - [D] How does Gemini 1.5 Pro recall information in 10M context? - **Google's Gemini 1.5 Pro** utilizes modern approaches like **recurrent memory** and **ring attention** to enhance its **long-context capabilities**, as detailed in a [technical report](https://arxiv.org/ftp/arxiv/papers/2403/2403.05530.pdf). - [R] ShortGPT: Layers in Large Language Models are More Redundant Than You Expect - **ShortGPT** demonstrates that **many layers in Large Language Models (LLMs) are redundant**, with some contributing minimally to the model's overall functionality, as revealed through a novel metric called **Block Influence (BI)**. [Read the paper](https://arxiv.org/abs/2403.03853) - Among the A.I. doomsayers - **A.I. doomsayers** and **techno-optimists** are deeply divided over whether artificial intelligence will lead to humanity's utopia or its destruction, with debates often centered in the Bay Area's "Cerebral Valley." - [D] Can someone please clarify if web search LLMs like Perplexity, You.com, or Coral Search are crawling the entire web themselves? Otherwise, how do they differ from simply combining a search API with any LLM model? - **Web search LLMs like Perplexity, You.com, or Coral Search** may or may not be crawling the web themselves, raising questions about their operational uniqueness and the source of their search results. - Steve Wozniak and Stuart Brand Discuss Control of IP (1984) - At the first Hackers Conference in 1984, **Stewart Brand** famously articulated the dual nature of information, stating it wants to be both **expensive and free**, reflecting on the tension between intellectual property control and the inherent desire for information dissemination.