Gemini 4 Neon is the new #1 on the Models Engineering Index
git-aarvin.github.ioVery high hallucination rates too.
https://artificialanalysis.ai/evaluations/omniscience?models...
Where Gemini shines for me is programmatically accessing YouTube content (engineering channels) and leveraging that.
I am just curious, have you always noticed this trend or is it somewhat more recent-ish? I swear, a year ago or so, I was having the complete opposite experience, but something changed in the 3-6 months. I do know nor understand Swift UI, so I have no opinions on the matter, but my Swift experiences have been the nearly identical. I felt like GPT used to be so much better with Swift (and C#) than Opus.
Not this one.
All the 3.x flash models were actually dumb compared to everything else it competed with.
Neon seems promising.
But my question is, what does this benchmark actually measure about the LLM's engineering abilities? It just looks like a page of random bar graphs to me.
looking at the graphic below the table, it does seem like data is missing for Neon which is present for Fable, and isn't penalized for the missing benchmarks.
https://one.google.com/about/google-ai-plans/ had a 75% off on the Pro plan ($5/mo for 3 months) and limits are decent.
There are a few ways to create/use an API key. Through Google cloud, through vertex, but by far the easiest way is to use AI studio. https://aistudio.google.com/ pretty good platform to try out any Google model or just create an API key and use in whatever you want without having to deal with Google Cloud UI.
Alternatively you can get from other sources as well such as openrouter.
Once I get a subscription can I use it from OpenCode or do I have to install some other thing?
I opened that link and immediately had to answer a bunch of questions, sign some terms of use, and finally faced an error "Failed to list API keys: permission denied. Please try again." I had never used this website before.
There is no place that shows me billing plans and what I would get from them. When I go on Billing it says something about API usage, so I guess this is not where I'll get a subscription, right? Still, it makes me create a project on Google Cloud Console, and it starts to hurt very quickly.
The speed is the biggest win IMO. When Carbon came out, I tried to give it a shot, but it was so slow that I went back to Opus just for productivity, nevermind quality. Today I started Neon on one worktree and Opus on another, and Neon did so well that I have now switched for both.
Code quality looks great. Comment-wise it has its own Claudese(exclamations like "this function has a concurrency bug and it's a sharp one!"), but overall I like reading it more than Opus. Might just be because I am not yet tired of its quirks though.
Oh, after a productive day on this particular work I had only spent $30 of Neon vs. $70 of Opus yesterday. YMMV
My favorite current-gen Gemini-ism is how it has these epiphanies, like "Wait! There's a pointer to the documentation we need right here!" So cute.
> I'll write the script now. Let me write it. I'll write. OK let me just do this now. I'll write it. Let me write it. I'll stop. Writing. OK.
> my decode of #1 said off=0x1281. Hmm - hmm - hmm. Hmm. Hmm, hmm - but the displayed size was 22255... hmm
(I think this is my quant's fault tbf)
The ‘summarisers’ mask a lot of raw thinking you’d probably be frustrated by too.
I'll do it just for the improved prose. I've had it up to here with Opus 5.5's surprisingly poor writing skills, even though it's leagues better than the Willem-Dafoe-on-cocaine dumpster fire that is Opus 5.
Neon is a really good model. It’s currently my favorite model to use. Coding-wise, it’s equivalent to Opus 5.5/Fable. Documentation-wise, it is great. If you have access to Neon have it take a crack at simplifying your agents.md. It’s the first model I’ve run that criticized the amount of ceremony in my instructions and suggested solid simplifications. It also makes clean edits to documents without leaving scars of the previous verbiage behind.
Art-wise, it’s more capable, in my experience, than Fable. I’ve been using fable/opus heavily to make art assets for my application (I shared in a previous comment that procedural art has completely obviated gpt-image art for me) and I’ve started to run into a wall with how far I could iterate with fable. I had neon take a crack at the assets I struggled to improve and it did a grand job.
In terms of working together with a model and how it communicates to you, I love this model. The model knows how to clearly communicate to you. It is such a breath of fresh air over Opus and Fable which, despite the improvements, I feel is actively trying to damage my brain with its unnatural prose. It's hard to describe how it feels to have a model that you feel respects your time and energy with proper writing, and the subsequent delight that arises from the feeling of wanting to engage in smaller details again.
Neon has a big model smell, but the miracle and irony is it’s supposedly based on a smaller base model than Carbon. On google’s model page, they previously wrote that Neon shares the same base model as Hydrogen! It is thought that Argon and Carbon share the same base model.
My impressions will probably change as I use this for longer than a day, but my initial impression is: I don’t know how google pulled this off.
It totally replaced Argon there, which is no longer available.
Interestingly, Argon used to run into rate limits all the time on $20/mo plans; you'd get three or four prompts, then you'd have to spend several hours in a cooldown period where you could only use an old 3.X Flash model. Neon is either much more efficient, or its limits have been relaxed. I get quite a lot of use out of it, and haven't yet been cast into cooldown purgatory.
I'm really happy with the Pro plan right now, Hydrogen and Neon have been great and actually usable day one. 5 TB of storage for my family and pretty decent AI usage for $10 a month through Verizon FiOS add-on. Much better value than the ChatGPT offering.
A word of warning though, it will need a bit of getting used to if you've used Claude Code extensively.
Edit: Already installed it and using it through Herdr! I realize what you meant ha!
With Claude Code I feel every answer triggers a lot of actions and agents, while with Antigravity seems there are less of that and it's more straightforward.
* not depend on the folder/repo, iow resume in an other folder is fine
* not have a an auto mode, only feasible way is YOLO mode. The sandbox seems nice and I’ve tried and tried and tried but always got stuck in endless confirmation prompts
* not have --ide integration with vscode that in manual mode shows the pre-write diff
* not integrate as clean with the browser as claude-in-chrome does
There's an auto mode. You can Shift-Tab a few times to rotate between Default mode, "Plan mode", "Auto-edit mode", and "Auto mode" (or `agy --mode=auto`).
I basically have to run it with --dangerously-skip-permissions, which is batshit nuts. Google, fix your harness.
VERY wrong, you will get like 10-20x more tokens with a Google AI sub than anthropic/Claude sub.
I was interested to see what Neon was like so downloaded antigravity. Signed in with my personal account, but it's AI Plus and not AI Pro (I just wanted it for the storage, but I assumed I got _some_ AI stuff included given the name).
No problem, sign in with my workspace account, nope despite being on a higher tier I just get 3.8 Flash.
I understand they may be very compute constrained, but surely it's worth giving users even a handful of prompts.
At most it'll be useful for a kid leaning programming, or someone playing around with less coding-focused jobs.
For someone who's limited to ≤$10/month, nothing compares to opencode go.
> Google still doesn't roll out models in a timely fashion.
This is the same way Google has been doing roll out across products for a decade. And it is unlikely to change.
At the scale of Google, deployments like this would reach billions users, spanning over the world, and take hours, if not days, for a proper roll out. An instant deployment would carry a lot of risks.
Personally, I see it as engineering done right.
I'm talking about antigravity - their coding agent, which has orders of magnitude less usage than Gemini or AI mode.
But tbh Gemini has the same issue, this is a rare exception.
I'd just have thought that given how little dev mind share Google has you might want to give paying (!) customers timely access for a few prompts to evaluate it.
and this is the reason google will lose the race. other companies have figure out the way already.
also with the "#1" models they have why cant they solve this problem too?
What tools are they using to make this?
I'd enjoy seeing examples (that also includes the prompt).
Can't share my work but my process involved a lot of planning which included discussing specific feelings i wanted my art to evoke, providing a collection of images (a moodboard), and having the model generate multiple mockups to establish direction and procedure. That took a lot of different iterations but once I finally established the style, the model had no problem transferring it forward.
A simple cross-check is to get two models to advance the paper, and then have the other model validate it.
All of the Gemini 4 models -- including Neon -- consistently make simple, "doofus" errors like neglecting minus signs or doubling a scalar factor due to accidentally multiplying by it twice. Similarly, they often use two different sets of assumptions for a long mathematical argument involving multiple proofs, so the whole argument ends up invalid.
For comparison, these basic errors no longer occur with any of the recent Anthropic or OpenAI models, including every GPT 6 Sol or Astra model that I've tested.
Simply put, if you let Astra do 5-10 iterations in a row (~1 hour of work) and feed it to Gemini 4 Neon, it won't spot any errors. Do the reverse of that and even GPT 6.2 Sol will tear Neon's work to pieces.
PS: I suspect that Gemini models have a higher temperature setting, which makes their English sound more natural, but introduces errors that degrade their hard science capability.
[1] But not crank physics, which I would expect to "trip up" models trained on correct, valid physics and mathematics.
The idea is that the Standard Model is soooo close to a nice and neat algebra that it may just be possible that it simply "pops out" of one naturally. All of particle physics might just be a very simple mathematical rule, instead of a whole pile of things on top of other things that looks suspiciously like epicycles.
There's a "search" by theoreticians for the equivalent algebraic change in perspective that Galileo did when he moved astronomy from overly complex 2D mathematics to a far simpler 3D algebra.
I have several notions that now -- thanks to AI -- I can throw at the proverbial wall to see if they stick.
The one that I've been playing around with the longest is a real geometric algebra with a total of nine dimensions: three temporal, three spatial, and three nilpotent ("matter"). I also sprinkled on some of my preferred postulates, such as making the theory MWI-flavoured, fully relational (no absolute clocks or rulers), etc.
I knew it probably wouldn't work, but this algebra has an incredibly beautiful symmetry of three-by-three and it is "just" big enough to encompass the SM particle zoo including the three particle generations!
The idea was that the three temporal dimensions might explain the "many worlds" aspects of quantum mechanics.
It almoooost worked!
I got it to the point that all three frontier AIs agreed that it's publication-worthy and one particular branch was "ahead" of Cohl Furey's favored algebra, but... meh... it's (provably) a dead end. Not to mention that everyone and their dog will be doing the same kind of thing with their own toy theory, so nobody will be interested in yet more AI slop output unless it makes a dramatic discovery.
Despite the dead end it was worth a go, especially given that trying is effectively free: I usually burn tokens on this right before using a "reset" that's about to expire.
Lol, do they still gas you up like this?
It's actually a great test to see how much a particular model likes to "butter your bread" versus giving a negative opinion.
Anthropic models: Somewhat brutal actually. Fable told me to "stop wasting my time".
OpenAI models: Incredibly diplomatic and neutral, refuses to state a firm opinion either way even if pressed.
Google Gemini models: "The honesty of this paper refreshing, this should be the benchmark for how all theoretical research is published!" (etc)
I learned not to trust anything a Google model says and to use the combination of an Anthropic and an OpenAI model if I want to know if something really is a bad idea.
I'm literally using AI to paper over my sub-standard mathematics skills!
Trust me, there's hundreds -- if not thousands -- of people doing this right now, and many of them likely work in serious physics research programmes.
Many people have ideas, not many people can dedicate their lives to them.
Now, it becomes entirely possible for someone with a "notion" to make the next "big" discovery, but not actually understand what they have discovered.
It's wild.
I also found Argon made a ton on doofus physics mistakes. All caught by Astra. But it got better (and much slower) when using /teamwork.
Google has an introductory $5 subscription that defaults to Gemini 4 Neon (High). I was curious, so I grabbed that and got things going within 5 minutes.
I agree WRT Carbon, but not Neon.
> an introductory $5 subscription
When I read this type of thing I do really start to see why Anthropic et al are putting so much effort into the harness. I have a lot of hooks and things setup to make Opus/Fable tolerable but it is tolerable. The same but change the harness is not really a great selling point. I have definitely talked with fellow devs who prefer OpenAIs models because the word barf out of Anthropic models can be exhausting even if they don't code as well or whatever.
Admittedly since OpenAI/Anthropic models are serving my needs and my past experience getting access to Google's models was a pain in the ass I am less inclined to mess around with Googles shitty consoles and API permissions.
I can pretty much one or two shot everything in gemini. Not sure if that means i'm working on simpler stuff or I’m just better at describing what i want. But i'm also using like orders of magnitude less tokens than everyone else.
That's why.
I made a little stock gamelet and it came out great vs the same prompt being given to 6.2 sol at ultra.
I think the previous google models were inferior but neon is much better.
That said, I use it via antigravity with a plan so I don't know the cost perspective for someone who is paying per token.
but seeing these subsidized plans are mostly enterprise acquisition schemes I do feel like they will slowly turn into pay as you go type and now that China's access to nvidia gpus is curbed, I think we'll see less and less investment into spending too cheap to meter models as the incentives disappear
All in all I want to say, build whatever you want right now because when there are lot more vibers tomorrow, they won't be able to support subsidized plans. Fact that Google hasn't budged on this tells me that they see no upside
$99.99/ month: 5x higher usage limits vs. AI Pro
$199.99 / month: 20x higher usage limits vs. AI Pro
Let's see if this lasts though...
imo take your money to someone with the practices you like; sucks when models are corp-exclusive, makes this whole thing tough.
ex: you prompt an llm to assist in forcing a mobile browser into kiosk mode, the llm determines via a fanned out sub-agent that it can only achieve that with a root and it determines the mobile has a locked boot-loader; so it starts considering exploit methods from XDA until the cyber-security leash hits the whole slot and stops all related work from progressing.
some variant of this hits me almost every day on oAI begging that I join their project daybreak shit and submit an ID, a cybersecurity refusal would be a lot scarier against a 25 year old google account.
A good 4-6 hour session of code and tool usage on the API can run me $80-$100. Are you saying I can get that all for $200/mo if I switch to Claude Code and use the high end subscription?
I wouldn't say my relationship with the claw is worth an extra $500 or something, but there are serious improvements over using Claude Code directly. Scheduled jobs, the whatsapp channel, and a bit of personality... I'm disinclined to hunt around for another harness that Anthropic would accept on the terms of their subscription, only to find it banned shortly thereafter. Is there a well trodden path to just using openclaw on one of those higher level subscriptions instead of raw API pricing?
https://platform.claude.com/docs/en/about-claude/api-credits...
So you can get massively subsidized coding use (I use it through T3, herdr works, the official clients work too of course) and also get a modest amount of API credit for whatever you want (claw, etc.).
Theres also options to connect openclaw via the real cli, but that does blow up your system prompt. I use that option right now and it works quite well. There's no free lunch as they say.
I've been using it for a couple months.
I dunno... I could afford to drop 20 bucks to try it out, and did so. Not sure how it's different than Anthropic or OpenAI that way.
It writes text that is not some complete gibberish.
Google just makes it weirdly hard to give them money.
Just because you were able to pay them $20 and test it doesn't mean other people have that experience.
There is a lot of those, one could say is that Google is trying to curb their previous overgenerosity
I have paid $200/year for Claude Pro sub and almost never runs out of either session or weekly usage limits. Not sure if Neon can last that long?
You are using it gently. Try using it with heavier workflows and subagent or run getshitdone and you will spend the weekly limits in an hour.
I don't really run out of weekly limits, since the session limits gate me - I'll get through 2 session limits per work day and then the other 2/3 worth of Claude sessions go unused due to not being at the computer.
It doesn't seem like it, at least at the $20/month level.
Evidence: I bought the $20 subscription and had Neon knock out a task that would've taken Opus at most ~1% of my weekly Claude tokens at the $20/month tier. It took 4% of my weekly Gemini tokens.
Fantastic.
I don't know why that would be.
The Neon model is available today in both my Gemini app on iPhone and at https://gemini.google.com/app, and I just have a $4.99/month Google AI Pro trial offer.
I presume I could boot it up in Antigravity also, but I haven't done so yet.
To be honest the Google AI studio, and all of the related recent Google web UIs and related surfaces are absolute dumpster trash, just finding where/how to subscribe and set this up was so unnecessarily painful, and the UI is so painfully slow and buggy- holy shit.
That being said I just got neon running via antigravity (agy), and it is quite good. Noticeably faster than opus 5.5, and so far it seems like I will get at least 10x usage compared to my anthropic sub of similar price.
My biggest gripe - I have thoroughly tried to set up permissions correctly, but it still asks me permission for every tiny thing it tries to do.
that is a very compelling endorsement...
The permissions management is a nightmare, specifically.
If you're a yolo user, it can be fine.
I give it work to do for 12 hours and come back in the morning to a stack of fully end to end validated commits.
i decided not to chargeback since i didn’t want my google acc banned, accepted losing that money, and that was my one and only (and ever) experience with google’s ai stack.
(I don't use the CLI directly so these are what I noticed they improved and didn't)
The company I'm leaving due to change of projects I am 95% sure has access, but only from a part of the company that has extra security checks and blacked out windows :V
There are 3 different antigravity quotas:
* gemini (previously just Gemini 3.8 Flash)
* claude and gpt (very small quota)
* argon (very small quota)
The Gemini quota now includes Neon and Hydrogen. Hydrogen (medium) competes with DeepSeek (high). Its within 2x on cost and more intelligent.
This is all from benchmarks- I am only just trying these models out now.
Use Smartube: https://smarttubeapp.github.io/
I have paid for Kagi and Spotify for a long time. Its tragic that most of the web only offers a large monthly subscription to read a single article or intrusive ads- I do use Zen on Desktop and Orion on mobile due to that.
This is their golden ticket.
Amazon famously bundled a variety of services under Prime which had, at least for a time, absolutely unheard of retention metrics. Truly otherworldly stuff. They probably still do.
I've been a Prime member since inception, longer than practically any other service. Given my household's needs I'm unlikely to cancel it any time soon.
Google bundling YouTube and Gemini under their... whatever-it-is-called plan (One? At least that name was a sufficiently thematic copy of "Prime"...) is brilliant. The $20 plan is a no-brainer for a ton of folks. They need to make the $200/mo plan a no brainer, and I think they can, but I doubt they have the wherewithal to actually figure it out and pull it off.
The market has rarely, maybe never, been more amenable to otherwise monopolistic bundling, I think they'd be crazy not to try and swing for the fences.
And I'm not saying any of this is a good thing, I'm certainly not endorsing this behavior. It's just that if I were a soulless automaton C-level executive trying to maximize shareholder value at all costs, and looking at the-company-most-likely-to-dethrone-me's IPO earlier today, and had Neon hot out of the gate, pulling away from the pack, I would be going for blood right now.
We all know there must have been 15 different product managers meeting endlessly over this and thinking they are making the world a better place by inventing new nonsensical names.
Agy has a bunch of good slash commands for different use cases. And the biggest thing for me is that it doesn't annoy me with security refusals. It helped me with my garage door, for example, when another model refused.
This is really good. Worlds better, in fact. I'm impressed and delighted. Far better insights into power management and clocking. Component identification is awesome; far better than what I've been getting from Codex. It's like a friendly 30 year EEE just decided I'm worthy of as much help as I need.
10/10 will prompt again.
I start with detailed analysis before I go near EDA tools. The hardest part of design is identifying components that distributors actually stock and sell. There is no point to making schematics for fictional components. Once that part is nailed down, I use kicad-happy to analyze my schematics and PCBs. I haven't felt the need to let the LLM actually operate the EDA itself.
Let's see if this one lives up to it's ranking..
Opus does a fantastic job without handholding, and in an early sample it looks like Neon does too. But I've only tested one tile because I hit my 5hr limit. I'm "only" on the $20/mo plan for Google.
Interesting to me that I was inspired to get into swe by wanted to build games, but found it near impossible back in the day. Fast forward to now and I do not even have a WIP game :( what I am doing with my life.
Surge AI DAYJOB - Finance: https://x.com/HelloSurgeAI/status/2123561368473485314
Parallel Search Intelligence: https://x.com/p0/status/2123497882322141453
SlopCop: https://slopcop.com/power-ranking
Cognition: https://x.com/cognition/status/2123553121653944391
I really don't get Google's product strategy.
Ever worked in a big company where internal nomenclature becomes an obstacle for external participation? This is that.
And this product launch is a failed activation for me
Every single Gemini model that has come out strong has quickly failed the vibes.
Seems like the methodology here isn’t sound? Not sure why we’d care about this over something like Artifical Analysis index.
(Antigravity GUI client on MacOS, Extra effort level)
I asked agy to find bugs and performance optimization opportunities in some code implementing a OpenGL to Metal graphics bridge.
1. introduced four (4) P1 regressions, essentially breaking everything in a very obvious way
2. wrote no new tests for the code it wrote
3. its code failed five existing tests
4. at least half of the bugs it diagnosed were wrong, e.g. false finding entirely - not a bug, correctly working code
5. correctly diagnosed some previously undiscovered bugs, but either wrote partial or incorrect fixes for most of them
6. correctly diagnosed and wrote correct fixes for only 2 out of a total of like 10 "findings"
Disappointed, this sounded like it would be a big win for Google and put them back at the head of the pack.
So ability to reason matters more than just writing code.
Would Neon be better than Opus with these assumptions?
The harness itself (antigravity) is a bit sus right now, and it's the only one you can (apparently) use for the Google sub, but so far it's been decent
>The harness itself (antigravity) is a bit sus right now, and it's the only one you can (apparently) use for the Google sub
I plan to use Neon in Pi. There is a Pi extension that allows using Gemini subscriptions. I'll see if I get banned or not.
Commodity product through and through. Each week a different one