Training Ultra Long Context Language Model with Fully Pipelined Distributed Transformer
The Fully Pipelined Distributed Transformer (FPDT) significantly increases the sequence length capability for training Large Language Models (LLMs) on limited hardware, addressing the high resource demands of processing long contexts.
Inductive or Deductive? Rethinking the Fundamental Reasoning Abilities of LLMs
LLMs demonstrate remarkable inductive reasoning capabilities when isolated from deductive reasoning tasks, as shown by the novel SolverLearner framework which focuses on pure inductive reasoning by learning mappings from input to output.
AI-Implanted False Memories
Conversational AI, particularly generative chatbots using large language models (LLMs), significantly amplifies the formation of false memories in individuals during simulated crime witness interviews, compared to control, survey-based, and pre-scripted chatbot interactions.
Population Minimizer of The Categorical Cross Entropy Loss (a blog post)
The population minimizer of conditional risk associated with Categorical Cross Entropy is proven to be the true probability label distribution, a significant insight for neural network outputs.
Fine-tuning coding LLMs on Git histories rather than just final code?
Fine-tuning coding LLMs on Git histories could enhance software development by leveraging the evolutionary context of code, beyond its final state.
MemLong: Memory-Augmented Retrieval for Long Text Modeling
MemLong, a Memory-Augmented Retrieval method for Long Text Generation, addresses the challenge of handling long contexts in Large Language Models (LLMs) by utilizing an external retriever for historical information retrieval.
Investigating Neuron Ablation in Attention Heads: The Case for Peak Activation Centering
Neuron ablation in transformer-based models reveals varying impacts on performance, highlighting the complexity of attention mechanisms.
A Unified Benchmark for Federated Unsupervised Anomaly Detection in Tabular Data
The FedAD-Bench benchmark, introduced in a recent study, aims to standardize the evaluation of unsupervised anomaly detection in federated learning (FL) environments, specifically for tabular data. Read the paper
Look, Compare, Decide: Alleviating Hallucination in Large Vision-Language Models via Multi-View Multi-Path Reasoning
The MVP framework reduces hallucinations in Large Vision-Language Models ( LVLMs) without the need for retraining, leveraging Multi-View Multi-Path Reasoning to enhance comprehension of image content.
EMPOWER: Embodied Multi-role Open-vocabulary Planning with Online Grounding and Execution
EMPOWER introduces an open-vocabulary online grounding and planning framework for robots, enhancing their ability to perform tasks in real-life settings by addressing common challenges in task planning.
Dynamic Self-Consistency: Leveraging Reasoning Paths for Efficient LLM Sampling
Reasoning-Aware Self-Consistency (RASC) introduces an early-stopping framework that evaluates both the output answer and the reasoning paths to optimize sample usage and enhance answer reliability in Large Language Models.
Safety Layers of Aligned Large Language Models: The Key to LLM Security
Aligned Large Language Models (LLMs) are designed to be secure by refusing to answer malicious questions, but their security mechanisms, particularly at the parameter level, are not fully understood, especially under conditions of fine-tuning with non-malicious data.
UserSumBench: A Benchmark Framework for Evaluating User Summarization Approaches
UserSumBench is a benchmark framework designed to overcome challenges in developing LLM-based user summarization techniques, such as the lack of ground-truth labels and the costly nature of human evaluation.