AI & Agents
Stack Exchange · 2025-01-10
Flagship phones ship local LLMs, but nobody knows how they really compare across devices
旗舰机里塞着本地大模型，但没人知道各机型真实能力差多少
At a glance · 30-second read
- WhoStack Exchange · 2025-01-10 · 1 upvotes
- The askFlagship phones ship local LLMs, but nobody knows how they really compare across devices
- The openingA crowdsourced on-device LLM benchmark: users run a test suite and report anonymously; sell rankings to vendors and developers.
What was actually said
“High-end phones have embedded LLMs accessible via AICore, usable offline. I'm building an app on this SDK and want to crowdsource real responses across devices.”
— Stack Exchange · 2025-01-10 · softwarerecs.stackexchange.com/questions/91759/how-t…
Background and pain
The pain
On-device models are fragmented: same prompt, wildly different results per phone, no public benchmark — devs buy hardware to find out.
A buildable angle
The opening
A crowdsourced on-device LLM benchmark: users run a test suite and report anonymously; sell rankings to vendors and developers.
How to validate it
- Read the whole thread and note the recurring scenarios — repetition is the density of the need.
- Search existing tools around "on-device llm / benchmarks" and check which single step they actually cover.
- Put up a one-line landing page, share it back to Stack Exchange, and see whether anyone asks about pricing.
Existing options
实验室基准论坛零散反馈
More threads on this pain
- How are teams sharing context across Claude Code, Codex and Cursor?Cursor Forum · 2026-05-06
- Hand-written Postman regression suites can't keep up with API changesStack Exchange · 2026-10-13
- Paid plans still hit limits: want a VS Code AI assistant without strict usage capsStack Exchange · 2026-07-13
- Switching coding agents means re-explaining your project from scratch every timeHacker News · 2026-10-21
- APIs have Swagger; databases need the same — schema as a tracked file with nice docsStack Exchange · 2025-08-18