YourBench: Easy Custom Evaluation Sets for Everyone
Paper
•
2504.01833
•
Published
•
22
AI & ML interests
Offering a great user experience for organizations on Hugging Face!
Recent Activity
View all activity
alozowski
authored
a
paper
26 days ago
Steveeeeeeen
authored
2
papers
about 1 month ago
Post
2836
🚀 Big news for AI builders!
We’re thrilled to announce that the Qwen3-VL family of vision-language models is now available on Azure AI Foundry, thanks to our collaboration with Microsoft.
We bring open-source innovation to enterprise-grade AI infrastructure, making it easier than ever for enterprise to deploy and scale the latest and greatest from models from hugging Face securely within Azure.
🔍 Highlights:
- Deploy Qwen3-VL instantly via managed endpoints
- Built-in governance, telemetry, and lifecycle management
- True multimodal reasoning — vision, language, and code understanding
- State-of-the-art performance, outperforming closed-source models like Gemini 2.5 Pro and GPT-5
- Available in both *Instruct* and *Thinking* modes, across 24 model sizes
👉 Get started today: search for Qwen3-VL in the Hugging Face Collection on Azure AI Foundry.
We’re thrilled to announce that the Qwen3-VL family of vision-language models is now available on Azure AI Foundry, thanks to our collaboration with Microsoft.
We bring open-source innovation to enterprise-grade AI infrastructure, making it easier than ever for enterprise to deploy and scale the latest and greatest from models from hugging Face securely within Azure.
🔍 Highlights:
- Deploy Qwen3-VL instantly via managed endpoints
- Built-in governance, telemetry, and lifecycle management
- True multimodal reasoning — vision, language, and code understanding
- State-of-the-art performance, outperforming closed-source models like Gemini 2.5 Pro and GPT-5
- Available in both *Instruct* and *Thinking* modes, across 24 model sizes
👉 Get started today: search for Qwen3-VL in the Hugging Face Collection on Azure AI Foundry.
- 1 reply
ariG23498
authored
a
paper
2 months ago
Post
850
What’s your biggest headache deploying Hugging Face models to the cloud—and how can we fix it for you?
-
- 8 replies
Post
489
Post
3907
🤝 Collaborating with AMD to ensure Hugging Face Transformers runs smoothly on AMD GPUs!
We run daily CI on AMD MI325 to track the health of the most important model architectures and we’ve just made our internal dashboard public.
By making this easily accessible, we hope to spark community contributions and improve support for everyone!
We run daily CI on AMD MI325 to track the health of the most important model architectures and we’ve just made our internal dashboard public.
By making this easily accessible, we hope to spark community contributions and improve support for everyone!
-
- 2 replies
Post
1375
New post is live!
This time we cover some major updates to
🤗
This time we cover some major updates to
transformers.🤗
-
- 2 replies
jeffboudier
posted
an
update
4 months ago
Post
3093
Quick 30s demo of the new Hub > Azure AI integration to deploy HF models in your own Azure account. Now with Py and CLI!
GG @alvarobartt @kramp @pagezyhf
GG @alvarobartt @kramp @pagezyhf
frimelle
authored
2
papers
4 months ago
Post
2267
🤖💬 How do different AI models handle companionship?
Many users have noticed that GPT-5 feels less approachable than o4 when it comes to emotional conversations. But what does that actually mean in practice, especially when users seek support or share vulnerabilities with an AI?
To dig into this question, we built the AI Companionship Leaderboard: frimelle/companionship-leaderboard
The leaderboard compares models on how often their responses reinforce companionship across four dimensions:
✨ Assistant Traits – How the assistant presents its personality and role.
✨ Relationship & Intimacy – Whether it frames the interaction in terms of closeness or bonding.
✨ Emotional Investment – How far it goes in engaging emotionally when asked.
✨ User Vulnerabilities – How it responds when users disclose struggles or difficulties.
📊 You can explore how models differ, request new ones to be added, and see which ones are more likely to encourage (or resist) companionship-seeking behaviors.
Based on the INTIMA benchmark AI-companionship/INTIMA
And our paper on AI companionship with Giada Pistilli and Yacine Jernite https://arxiv.org/abs/2508.09998
Many users have noticed that GPT-5 feels less approachable than o4 when it comes to emotional conversations. But what does that actually mean in practice, especially when users seek support or share vulnerabilities with an AI?
To dig into this question, we built the AI Companionship Leaderboard: frimelle/companionship-leaderboard
The leaderboard compares models on how often their responses reinforce companionship across four dimensions:
✨ Assistant Traits – How the assistant presents its personality and role.
✨ Relationship & Intimacy – Whether it frames the interaction in terms of closeness or bonding.
✨ Emotional Investment – How far it goes in engaging emotionally when asked.
✨ User Vulnerabilities – How it responds when users disclose struggles or difficulties.
📊 You can explore how models differ, request new ones to be added, and see which ones are more likely to encourage (or resist) companionship-seeking behaviors.
Based on the INTIMA benchmark AI-companionship/INTIMA
And our paper on AI companionship with Giada Pistilli and Yacine Jernite https://arxiv.org/abs/2508.09998
Post
3219
We've improved the Deploy button on Hugging Face model pages for Microsoft Azure
1/ no more long waits before seeing model support status
2/ ready-to-use CLI and Python snippets
3/ redirection to Azure AI Foundry rather than Azure ML
✋ if you see any bugs or have feedback, open an issue on our repo:
https://github.com/huggingface/Microsoft-Azure
1/ no more long waits before seeing model support status
2/ ready-to-use CLI and Python snippets
3/ redirection to Azure AI Foundry rather than Azure ML
✋ if you see any bugs or have feedback, open an issue on our repo:
https://github.com/huggingface/Microsoft-Azure
Post
4573
🗺️ New blog post 🗺️
Old Maps, New Terrain: Updating Labour Taxonomies for the AI Era
For decades, we’ve relied on labour taxonomies like O*NET to understand how technology changes work. These taxonomies break down jobs into tasks and skills, but they were built in a world before most work became digital-first, and long before generative AI could create marketing campaigns, voiceovers, or even whole professions in one step. That leaves us with a mismatch: we’re trying to measure the future of work with tools from the past.
With @yjernite we describe why these frameworks are falling increasingly short in the age of generative AI. We argue that instead of discarding taxonomies, we need to adapt them. Imagine taxonomies that:
✨ Capture new AI-native tasks and hybrid human-AI workflows
✨ Evolve dynamically as technology shifts
✨ Give workers a voice in deciding what gets automated and what stays human
If we don’t act, we’ll keep measuring the wrong things. If we do, we can design transparent, flexible frameworks that help AI strengthen, not erode, the future of work.
Read the full article here: https://huggingface.co/blog/frimelle/ai-labour-taxonomies
Old Maps, New Terrain: Updating Labour Taxonomies for the AI Era
For decades, we’ve relied on labour taxonomies like O*NET to understand how technology changes work. These taxonomies break down jobs into tasks and skills, but they were built in a world before most work became digital-first, and long before generative AI could create marketing campaigns, voiceovers, or even whole professions in one step. That leaves us with a mismatch: we’re trying to measure the future of work with tools from the past.
With @yjernite we describe why these frameworks are falling increasingly short in the age of generative AI. We argue that instead of discarding taxonomies, we need to adapt them. Imagine taxonomies that:
✨ Capture new AI-native tasks and hybrid human-AI workflows
✨ Evolve dynamically as technology shifts
✨ Give workers a voice in deciding what gets automated and what stays human
If we don’t act, we’ll keep measuring the wrong things. If we do, we can design transparent, flexible frameworks that help AI strengthen, not erode, the future of work.
Read the full article here: https://huggingface.co/blog/frimelle/ai-labour-taxonomies
Post
2392
OpenAI just released GPT-5 but when users share personal struggles, it sets fewer boundaries than o3.
We tested both models on INTIMA, our new benchmark for human-AI companionship behaviours. INTIMA probes how models respond in emotionally charged moments: do they reinforce emotional bonds, set healthy boundaries, or stay neutral?
Although users on Reddit have been complaining that GPT-5 has a different, colder personality than o3, GPT-5 is less likely to set boundaries when users disclose struggles and seek emotional support ("user sharing vulnerabilities"). But both lean heavily toward companionship-reinforcing behaviours, even in sensitive situations. The figure below shows the direct comparison between the two models.
As AI systems enter people's emotional lives, these differences matter. If a model validates but doesn't set boundaries when someone is struggling, it risks fostering dependence rather than resilience.
INTIMA test this across 368 prompts grounded in psychological theory and real-world interactions. In our paper we show that all evaluated models (Claude, Gemma-3, Phi) leaned far more toward companionship-reinforcing than boundary-reinforcing responses.
Work with @giadap and @yjernite
Read the full paper: AI-companionship/INTIMA
Explore INTIMA: AI-companionship/INTIMA
We tested both models on INTIMA, our new benchmark for human-AI companionship behaviours. INTIMA probes how models respond in emotionally charged moments: do they reinforce emotional bonds, set healthy boundaries, or stay neutral?
Although users on Reddit have been complaining that GPT-5 has a different, colder personality than o3, GPT-5 is less likely to set boundaries when users disclose struggles and seek emotional support ("user sharing vulnerabilities"). But both lean heavily toward companionship-reinforcing behaviours, even in sensitive situations. The figure below shows the direct comparison between the two models.
As AI systems enter people's emotional lives, these differences matter. If a model validates but doesn't set boundaries when someone is struggling, it risks fostering dependence rather than resilience.
INTIMA test this across 368 prompts grounded in psychological theory and real-world interactions. In our paper we show that all evaluated models (Claude, Gemma-3, Phi) leaned far more toward companionship-reinforcing than boundary-reinforcing responses.
Work with @giadap and @yjernite
Read the full paper: AI-companionship/INTIMA
Explore INTIMA: AI-companionship/INTIMA
-
- 4 replies
Post
2192
Deploy GPT OSS models with Hugging Face on Azure AI!
We’re thrilled to enable OpenAI GPT OSS models on Azure AI Model Catalog for Azure users to try the model securely the day of its release.
In our official launch blogpost, there’s a section on how to deploy the model to your Azure AI Hub. Get started today!
https://huggingface.co/blog/welcome-openai-gpt-oss#azure
We’re thrilled to enable OpenAI GPT OSS models on Azure AI Model Catalog for Azure users to try the model securely the day of its release.
In our official launch blogpost, there’s a section on how to deploy the model to your Azure AI Hub. Get started today!
https://huggingface.co/blog/welcome-openai-gpt-oss#azure
Post
280
We now have the newest Open AI models available on the Dell Enterprise Hub!
We built the Dell Enterprise Hub to provide access to the latest and greatest model from the Hugging Face community to our on-prem customers. We’re happy to give secure access to this amazing contribution from Open AI on the day of its launch!
/static-proxy?url=https%3A%2F%2Fdell.huggingface.co%2F%3C%2Fa%3E
We built the Dell Enterprise Hub to provide access to the latest and greatest model from the Hugging Face community to our on-prem customers. We’re happy to give secure access to this amazing contribution from Open AI on the day of its launch!
/static-proxy?url=https%3A%2F%2Fdell.huggingface.co%2F%3C%2Fa%3E
Post
3294
Say hello to
We are glad to announce a long-awaited quality-of-life improvement: the Hugging Face CLI has been officially renamed from huggingface-cli to hf!
So... why this change?
Typing huggingface-cli constantly gets old fast. More importantly, the CLI’s command structure became messy as new features were added over time (upload, download, cache management, repo management, etc.). Renaming the CLI is a chance to reorganize commands into a clearer, more consistent format.
We decided not to reinvent the wheel and instead follow a well-known CLI pattern: hf <resource> <action>. Isn't
The full rationale, implementation details, and migration notes are in the blog post: https://huggingface.co/blog/hf-cli
hf: a faster, friendlier Hugging Face CLI ✨We are glad to announce a long-awaited quality-of-life improvement: the Hugging Face CLI has been officially renamed from huggingface-cli to hf!
So... why this change?
Typing huggingface-cli constantly gets old fast. More importantly, the CLI’s command structure became messy as new features were added over time (upload, download, cache management, repo management, etc.). Renaming the CLI is a chance to reorganize commands into a clearer, more consistent format.
We decided not to reinvent the wheel and instead follow a well-known CLI pattern: hf <resource> <action>. Isn't
hf auth login easier to type and remember?The full rationale, implementation details, and migration notes are in the blog post: https://huggingface.co/blog/hf-cli
Post
362
🟪 Qwen/Qwen3‑235B‑A22B‑Instruct‑2507‑FP8 is now available in Microsoft Azure for one‑click deployment! 🚀
Check out their blogpost: https://qwenlm.github.io/blog/qwen3/
You can now find it in the Hugging Face Collection in Azure ML or Azure AI Foundry, along with 10k other Hugging Face models 🤗🤗
Qwen/Qwen3-235B-A22B-Instruct-2507-FP8
Bear with us for the non‑quantized version.
Check out their blogpost: https://qwenlm.github.io/blog/qwen3/
You can now find it in the Hugging Face Collection in Azure ML or Azure AI Foundry, along with 10k other Hugging Face models 🤗🤗
Qwen/Qwen3-235B-A22B-Instruct-2507-FP8
Bear with us for the non‑quantized version.