MetaNova 1
README.md currently being written... ☕
| Benchmark |
MetaNova-Test |
MetaNova-0.1 |
MetaNova-1 |
Supra-50M-Base |
OpenAI/GPT-2 |
| Params |
70.55M |
70.55M |
62.70M |
51.79M |
124M |
| HellaSwag |
25.37% |
27.40% |
27.08% |
31.65% |
31.26% |
| ARC-Easy |
26.89% |
31.73% |
30.56% |
45.58% |
39.35% |
| ARC-Challenge |
25.09% |
25.00% |
24.15% |
24.66% |
22.35% |
| PIQA |
52.29% |
58.54% |
56.04% |
61.53% |
62.08% |
| ArithMark-2 |
24.12% |
27.12% |
34.12% |
26.08% |
26.48% |
| ARC Avg |
25.99% |
28.37% |
27.35% |
35.12% |
30.85% |
| Final Avg |
31.94% |
35.36% |
36.15% |
38.60% |
37.67% |
DeepSeek and Qwen Format
| Thinking Mode |
Non-Thinking Mode |
<|im_start|>user
{query} /think<|im_end|>
<|im_start|>assistant
<think>
{thinking_content}
</think>
{response}<|im_end|>
|
<|im_start|>user
{query} /no_think<|im_end|>
<|im_start|>assistant
<think>
</think>
{response}<|im_end|>
|
