Next Token tokenizer idx - DeepTalk - Deep Learning Community
Next Token tokenizer idx
post by boazlavon on May 5, 2025
Hi
Can I access the idx (from the tokenizer) of the next token some how?
Thanks!
52 views
Related topics
| Topic | Replies | Views | Activity |
|---|---|---|---|
| CoT content for deepseek-r1-671b | 0 | 42 | Apr 2025 |
| Tensor Book factory reset | 0 | 1.9k | Apr 2019 |
| Tensorbook recovery iso download access - please | 1 | 156 | Nov 2024 |
| Qwen 3 Models Support | 0 | 69 | Apr 2025 |
| Did inference api change recently? | 0 | 117 | Mar 2025 |