# Jun 2, 2024

## LLMs aren't "trained on the internet" anymore

- **LLMs are evolving beyond their initial training on internet data**, incorporating custom and synthetic data to address their traditional limitations.

## PyCon US 2024 Recap

- **PyCon US 2024** showcased significant advancements in Python, including **the implementation of a JIT Compiler in Python 3.13** and **the option to disable the Global Interpreter Lock (GIL)**, marking a pivotal shift towards enhancing Python's performance and concurrency capabilities.

## Here comes the Muybridge camera moment but for text

- **Embeddings** transform text into mathematical coordinates, enabling a new way to visualize and analyze the **semantic structure** of text, as demonstrated by the visual plot of Douglas Engelbart's essay and the **Braggoscope** search engine.

## X is justifiably slow (2022)

- **X's perceived slowness** often stems from **inefficient algorithms, implementations, or a failure to leverage modern hardware**, rather than the necessity of its workload.

## FineWeb: Decanting the web for the finest text data at scale

- **FineWeb** is a new **Hugging Face Space** designed to **extract high-quality text data** from the web, enabling more refined and efficient data gathering for machine learning applications.

## [R] CoPE: Contextual Position Encoding: Learning to Count What's Important

- **Contextual Position Encoding (CoPE)** enhances Large Language Models by allowing **positioning to be context-dependent**, enabling models to attend to more abstract concepts like the _i-th sentence_.

## Milvus Lite: The Lightweight Version of Milvus

- **Milvus Lite** is a **lightweight vector database** designed for local use in Python applications, streamlining the development of **GenAI applications** by leveraging core components from the open-source Milvus vector database for efficient vector indexing and query parsing.

## [Research] Tangles: a new mathematical ML tool in book announced by Diestel

- **Tangles** offer a **new structural approach** to artificial intelligence, enabling the identification of complex structures in data by grouping qualities that frequently occur together, thus revealing clusters and types of various phenomena.

## Implementing "Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet" paper for open source models.[D]

- The paper **"Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet"** introduces a method using **sparse autoencoders** for extracting interpretable features from large language models, aiming to enhance understanding of model internals.

## [P] Automated LoRA Discovery

- The **Automated LoRA Discovery** project introduces a suite of methods to **optimize the training of LoRAs** by constraining the search space and using generative models, aiming to **reduce the number of trainable parameters** significantly. [Full writeup and resources](https://sweet-hall-e72.notion.site/Automated-LoRA-Discovery-and-Teaching-Neural-Networks-to-make-Neural-Networks-22aa3b5ad66e4bc985ff2c93896538d2)

## [R] Tech report on FineWeb: decanting the web for the finest text data at scale

- The **FineWeb team** has **published a detailed blog post** on the **creation of high-quality web-scale datasets**, sharing insights from their experience with the **15 trillion token FineWeb dataset**.

## TwoMinutePapers - OpenAI’s ChatGPT: This is Science Fiction!

- OpenAI's ChatGPT demonstrates **unprecedented versatility** by adopting roles within virtual environments, such as organizing events and running a game development company, showcasing its ability to **simulate complex human interactions** and **collaborative tasks**.

## [D] Teacher student training strategy

- Utilizing a **Large Language Model (LLM) like llama3** to generate training data through prompts aims to **enhance a smaller model's accuracy** by custom training, potentially reducing the need for extensive datasets.

## YannicKilcher - xLSTM: Extended Long Short-Term Memory

- **xLSTM, an extended version of LSTM**, incorporates learnings from the world of Transformers, aiming to explore the potential of LSTM architectures in competing with Transformer-based models in language modeling.

## KServe Providers Dish Up NIMble Inference in Clouds and Data Centers

- **NVIDIA NIM** integrates with **KServe** on Kubernetes platforms, enabling scalable deployment of large language models (LLMs) with a simple API call, facilitated by providers like Canonical, Nutanix, and Red Hat.