LLMs are evolving beyond their initial training on internet data, incorporating custom and synthetic data to address their traditional limitations.
PyCon US 2024 Recap
PyCon US 2024 showcased significant advancements in Python, including the implementation of a JIT Compiler in Python 3.13 and the option to disable the Global Interpreter Lock (GIL), marking a pivotal shift towards enhancing Python's performance and concurrency capabilities.
Here comes the Muybridge camera moment but for text
Embeddings transform text into mathematical coordinates, enabling a new way to visualize and analyze the semantic structure of text, as demonstrated by the visual plot of Douglas Engelbart's essay and the Braggoscope search engine.
X is justifiably slow (2022)
X's perceived slowness often stems from inefficient algorithms, implementations, or a failure to leverage modern hardware, rather than the necessity of its workload.
FineWeb: Decanting the web for the finest text data at scale
FineWeb is a new Hugging Face Space designed to extract high-quality text data from the web, enabling more refined and efficient data gathering for machine learning applications.
[R] CoPE: Contextual Position Encoding: Learning to Count What's Important
Contextual Position Encoding (CoPE) enhances Large Language Models by allowing positioning to be context-dependent, enabling models to attend to more abstract concepts like the i-th sentence.
Milvus Lite: The Lightweight Version of Milvus
Milvus Lite is a lightweight vector database designed for local use in Python applications, streamlining the development of GenAI applications by leveraging core components from the open-source Milvus vector database for efficient vector indexing and query parsing.
[Research] Tangles: a new mathematical ML tool in book announced by Diestel
Tangles offer a new structural approach to artificial intelligence, enabling the identification of complex structures in data by grouping qualities that frequently occur together, thus revealing clusters and types of various phenomena.
Implementing "Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet" paper for open source models.[D]
The paper "Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet" introduces a method using sparse autoencoders for extracting interpretable features from large language models, aiming to enhance understanding of model internals.
[P] Automated LoRA Discovery
The Automated LoRA Discovery project introduces a suite of methods to optimize the training of LoRAs by constraining the search space and using generative models, aiming to reduce the number of trainable parameters significantly. Full writeup and resources
[R] Tech report on FineWeb: decanting the web for the finest text data at scale
The FineWeb team has published a detailed blog post on the creation of high-quality web-scale datasets, sharing insights from their experience with the 15 trillion token FineWeb dataset.
TwoMinutePapers - OpenAI’s ChatGPT: This is Science Fiction!
OpenAI's ChatGPT demonstrates unprecedented versatility by adopting roles within virtual environments, such as organizing events and running a game development company, showcasing its ability to simulate complex human interactions and collaborative tasks.
[D] Teacher student training strategy
Utilizing a Large Language Model (LLM) like llama3 to generate training data through prompts aims to enhance a smaller model's accuracy by custom training, potentially reducing the need for extensive datasets.
YannicKilcher - xLSTM: Extended Long Short-Term Memory
xLSTM, an extended version of LSTM, incorporates learnings from the world of Transformers, aiming to explore the potential of LSTM architectures in competing with Transformer-based models in language modeling.
KServe Providers Dish Up NIMble Inference in Clouds and Data Centers
NVIDIA NIM integrates with KServe on Kubernetes platforms, enabling scalable deployment of large language models (LLMs) with a simple API call, facilitated by providers like Canonical, Nutanix, and Red Hat.