# Aug 13, 2024

## A C/C++ library for audio and music analysis
- **audioFlux** is a **deep learning library** designed for **audio and music analysis**, offering extensive **feature extraction** capabilities across various time-frequency analysis methods.

## [R] 1.5-Pints Technical Report: Pretraining in Days, Not Months -- Your Language Model Thrives on Quality Data (2408.03506)
- The **1.5-Pints Language Model** pre-trains in **just 9 days**, leveraging a **57 billion token dataset** to surpass benchmarks set by tech giants like Apple and Microsoft, focusing on **quality data** for enhanced instruction-following capabilities.

## The AI Scientist: Towards Automated Open-Ended Scientific Discovery
- **The AI Scientist automates the entire scientific research process**, from idea generation to experiment execution and manuscript writing, aiming to democratize and accelerate scientific discovery. [Read the full report on arXiv](https://arxiv.org/abs/2408.06292/)

## Someone's been messing with Python's floating point subnormals
- After encountering a persistent warning while importing Python packages, an investigation revealed that over **2,500 Python packages** could cause incorrect numerical results due to a few being compiled with a risky compiler option, `-ffast-math`.

## [R] The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery
- **The AI Scientist** framework enables **frontier large language models** to independently conduct scientific research, **generating novel ideas**, writing code, executing experiments, and **communicating findings** through full scientific papers, including a simulated review process. [Paper](https://arxiv.org/abs/2408.06292)

## "Mutual Reasoning" improves GSM8K accuracy from 13% to 64% [R]
- **rStar**, a **self-play mutual reasoning approach**, significantly **enhances the reasoning capabilities** of small language models (SLMs) by employing a novel generation-discrimination process without the need for fine-tuning or larger models.

## [D] New Llama scaling laws?
- **Llama 3's development** revealed that **model performance improves log-linearly** with training on up to **15T tokens**, surpassing the previously optimal 200B tokens for an 8B parameter model.

## Med42-v2: A Suite of Clinical LLMs
- **Med42-v2** introduces **clinical large language models (LLMs)** built on **Llama3 architecture**, fine-tuned with specialized clinical data to address healthcare-specific challenges.

## [R] Grokfast: Accelerated Grokking by Amplifying Slow Gradients
- **Grokfast** significantly **accelerates the grokking phenomenon** in machine learning by focusing on **amplifying slow-varying gradient components**, which are crucial for generalization. [Read the paper](https://arxiv.org/abs/2405.20233v2)

## [R] Transformers are Universal In-context Learners
- **Transformers** can **approximate continuous in-context mappings** with **arbitrary precision**, handling an **infinite number of context tokens** without increasing the embedding dimension or the number of heads.

## LLMs as Optimizers - Theory Paper Recommendation [R]
- **Transformers** are theorized to perform **first-order optimization**, such as gradient descent, expanding their utility beyond traditional applications.

## Wet-lab innovations will lead the AI revolution in biology
- **Abhishaike Mahajan** specializes in **biology posting** on Substack, offering insights into biology and machine learning through essays.

## ARPA-H announces awards to develop novel technologies for precise tumor removal
- **ARPA-H's PSI program** awards **$150 million** to develop technologies for **precise tumor removal**, aiming to **reduce repeat surgeries** and **unintentional injuries** during cancer operations. [PSI program page](https://arpa-h.gov/research-and-funding/programs/psi)

## Body Transformer: Leveraging Robot Embodiment for Policy Learning
- **Body Transformer (BoT)** introduces an **architecture that leverages robot embodiment**, enhancing the learning process by structuring it around the robot's physical form.

## 🤗Introduction to ggml
- **ggml is a minimalist, lightweight machine learning library** focusing on Transformer inference, designed for ease of compilation and memory efficiency, distinguishing itself from larger libraries like PyTorch and TensorFlow.