Jan 3, 2025

• TinyStories: How Small Can Language Models Be and Still Speak Coherent English? (2023)

  • TinyStories demonstrates that language models with fewer than 10 million parameters can generate coherent and grammatically correct text, challenging the notion that larger models are necessary for fluency.

• Can LLMs write better code if you keep asking them to "write better code"?

  • Iterative prompting of LLMs, such as asking to "write better code," can lead to significant improvements in code quality and performance, with Claude 3.5 Sonnet achieving up to 100x speedup through optimizations like JIT compilation and parallel processing.

• System76 built the fastest Windows Arm PC

  • System76's Thelio Astra is the fastest Windows Arm PC, featuring an Ampere Altra Max CPU with 128 cores, designed primarily for autonomous vehicle development and capable of running Windows 11 seamlessly.

• How is LLM changing your job as a ML engineer?

  • LLMs drastically reduce project timelines, enabling traditional ML tasks that once took 6 months to be completed in just a weekend, as highlighted by Andrew Ng's insights on zero-shot learning.

• Numerical features with factorization machines

  • The paper, "Function Basis Encoding of Numerical Features in Factorization Machines," introduces a novel approach to enhance Factorization Machines (FMs) by employing parametric curves for numerical feature encoding, improving their effectiveness in recommender systems.

• Yi: A Family of Foundation Models Optimized Through Cascaded Data Processing and Targeted Finetuning

  • The Yi family of foundation models utilizes a novel data processing pipeline that combines rule-based filtering with learned models, enhancing data quality while ensuring safety during training.

• Test-time compute for image generation?

  • Test-time reasoning can enhance image generation by allowing models to take additional time for more accurate outputs, similar to O(1) approaches in other domains.

• Scaling of Search and Learning: A Roadmap to Reproduce o1 from a Reinforcement Learning Perspective

  • OpenAI o1 achieves expert-level performance through reinforcement learning, emphasizing the importance of policy initialization, reward design, search, and learning as critical components for replicating its capabilities.

• What are your thoughts on LLMs 'understanding' their domain and enhancing domain understanding?

  • Enhancing LLMs' domain understanding through structured methods like ontologies and knowledge graphs could significantly improve their performance, as evidenced by a fine-tuning experiment yielding an F1 score of 76% with only 400 examples.