# Jan 3, 2025

- **TinyStories**: How Small Can Language Models Be and Still Speak Coherent English? (2023)
  - **TinyStories** demonstrates that language models with **fewer than 10 million parameters** can generate coherent and grammatically correct text, challenging the notion that larger models are necessary for fluency.

- **Can LLMs write better code if you keep asking them to "write better code"?**
  - **Iterative prompting** of LLMs, such as asking to "write better code," can lead to significant improvements in code quality and performance, with Claude 3.5 Sonnet achieving up to **100x speedup** through optimizations like JIT compilation and parallel processing.

- **System76 built the fastest Windows Arm PC**
  - **System76's Thelio Astra** is the fastest Windows Arm PC, featuring an **Ampere Altra Max CPU** with 128 cores, designed primarily for **autonomous vehicle development** and capable of running Windows 11 seamlessly.

- **How is LLM changing your job as a ML engineer?**
  - **LLMs drastically reduce project timelines**, enabling traditional ML tasks that once took **6 months** to be completed in just a **weekend**, as highlighted by Andrew Ng's insights on zero-shot learning.

- **Numerical features with factorization machines**
  - The paper, "[Function Basis Encoding of Numerical Features in Factorization Machines](https://openreview.net/forum?id=M4222IBHsh)," introduces a novel approach to enhance **Factorization Machines (FMs)** by employing **parametric curves** for numerical feature encoding, improving their effectiveness in recommender systems.

- **Yi: A Family of Foundation Models Optimized Through Cascaded Data Processing and Targeted Finetuning**  
  - The **Yi family of foundation models** utilizes a **novel data processing pipeline** that combines rule-based filtering with learned models, enhancing data quality while ensuring safety during training.

- **Test-time compute for image generation?**
  - **Test-time reasoning** can enhance **image generation** by allowing models to take additional time for more accurate outputs, similar to O(1) approaches in other domains.

- **Scaling of Search and Learning: A Roadmap to Reproduce o1 from a Reinforcement Learning Perspective**
  - **OpenAI o1** achieves expert-level performance through **reinforcement learning**, emphasizing the importance of **policy initialization**, **reward design**, **search**, and **learning** as critical components for replicating its capabilities.

- **What are your thoughts on LLMs 'understanding' their domain and enhancing domain understanding?**
  - **Enhancing LLMs' domain understanding** through structured methods like ontologies and knowledge graphs could significantly improve their performance, as evidenced by a fine-tuning experiment yielding an F1 score of **76%** with only **400 examples**.