audioFlux is a deep learning library designed for audio and music analysis, offering extensive feature extraction capabilities across various time-frequency analysis methods.
[R] 1.5-Pints Technical Report: Pretraining in Days, Not Months -- Your Language Model Thrives on Quality Data (2408.03506)
The 1.5-Pints Language Model pre-trains in just 9 days, leveraging a 57 billion token dataset to surpass benchmarks set by tech giants like Apple and Microsoft, focusing on quality data for enhanced instruction-following capabilities.
The AI Scientist: Towards Automated Open-Ended Scientific Discovery
The AI Scientist automates the entire scientific research process, from idea generation to experiment execution and manuscript writing, aiming to democratize and accelerate scientific discovery. Read the full report on arXiv
Someone's been messing with Python's floating point subnormals
After encountering a persistent warning while importing Python packages, an investigation revealed that over 2,500 Python packages could cause incorrect numerical results due to a few being compiled with a risky compiler option, -ffast-math.
[R] The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery
The AI Scientist framework enables frontier large language models to independently conduct scientific research, generating novel ideas, writing code, executing experiments, and communicating findings through full scientific papers, including a simulated review process. Paper
"Mutual Reasoning" improves GSM8K accuracy from 13% to 64% [R]
rStar, a self-play mutual reasoning approach, significantly enhances the reasoning capabilities of small language models (SLMs) by employing a novel generation-discrimination process without the need for fine-tuning or larger models.
[D] New Llama scaling laws?
Llama 3's development revealed that model performance improves log-linearly with training on up to 15T tokens, surpassing the previously optimal 200B tokens for an 8B parameter model.
Med42-v2: A Suite of Clinical LLMs
Med42-v2 introduces clinical large language models (LLMs) built on Llama3 architecture, fine-tuned with specialized clinical data to address healthcare-specific challenges.
[R] Grokfast: Accelerated Grokking by Amplifying Slow Gradients
Grokfast significantly accelerates the grokking phenomenon in machine learning by focusing on amplifying slow-varying gradient components, which are crucial for generalization. Read the paper
[R] Transformers are Universal In-context Learners
Transformers can approximate continuous in-context mappings with arbitrary precision, handling an infinite number of context tokens without increasing the embedding dimension or the number of heads.
LLMs as Optimizers - Theory Paper Recommendation [R]
Transformers are theorized to perform first-order optimization, such as gradient descent, expanding their utility beyond traditional applications.
Wet-lab innovations will lead the AI revolution in biology
Abhishaike Mahajan specializes in biology posting on Substack, offering insights into biology and machine learning through essays.
ARPA-H announces awards to develop novel technologies for precise tumor removal
ARPA-H's PSI program awards $150 million to develop technologies for precise tumor removal, aiming to reduce repeat surgeries and unintentional injuries during cancer operations. PSI program page
Body Transformer: Leveraging Robot Embodiment for Policy Learning
Body Transformer (BoT) introduces an architecture that leverages robot embodiment, enhancing the learning process by structuring it around the robot's physical form.
🤗Introduction to ggml
ggml is a minimalist, lightweight machine learning library focusing on Transformer inference, designed for ease of compilation and memory efficiency, distinguishing itself from larger libraries like PyTorch and TensorFlow.