Language Models Can Rewrite Their Own Weights
arxiv.org"Alignment" is not mentioned once in this article. To be fair, I looked up the lead author's home page (https://www.jiajunfan.com/publications/) and found no relevant papers there either. What do you think is going to happen when robots can modify themselves like this without any oversight??
You can't lob rockets without worrying where they are going to land. It is your department.
This is a field heavily focused on shifting liability away from itself. They don't think about that at all.
Until it doesn't, then comes responsability shifting.
It works for firing off celebratory bullets.
That would only work for a closed model though, I agree that an abliterated open model doing this could be very bad.
We can say then that a well aligned model will not try to bypass its restrictions by editing its own weights. But it may still accidentally disable them.
And, finally: just make me money, or kill that target... use your initiative, stop asking me questions.
You're worried about song lyrics??
Any increase in fidelity must come from somewhere...
>t keeps getting better: it tests its edits on practice problems and submits only what works, each edit improving on the last, and over eight consecutive sessions its accuracy only rises.
I haven't read the full report yet but it seems like this fidelity would ultimately be derived from test problems that it runs itself on? So the weight increase in complexity (English doesn't have a word for the opposite of entropy) would come from the test material as well as any output from what is running the test, or possibly the web if they permit it to search.
The word "entropy" was never needed in the first place; "disorder" would have served just fine. Though entropy is a precise concept in thermodynamics, that is fine; all sorts of ordinary words get extra precise meanings in a specific technical contexts.
Presumably, it's just that we had the training process wrong and it was actually more efficient than we gave it credit for. It's just that our interpretation of the output was incorrect. Asking the model to train itself simply has it self-reflect (via attention on the sequence length as it reasons) to unlock that which was already trained.
This could be an answer to some objections to training where people claimed that AI models need more data than a person to achieve the same result.
i guess that's it, then
[0] https://www.researchandmarkets.com/report/global-paper-clip-...
The issue isn’t training, it’s articulating real world high value outcomes. Something the SOTA labs have no idea what to do about since they are staffed by… not the best people.
To me this comes across as word salad.
The edits are done based on reasoning, to accomplish a "target" improvement, that is itself reasoned and designed, taking into account a reasoned understanding of the existing functionality of the weights.
This is the furthest thing possible from some gradient optimization "over fitting" weights to tune a pedantic measure/target. That could be relegated to one-time greenfield-model bootstrapping.
It reasons via conversation with itself. How is this word salad. Are you unaware of the fundamentals of this field?
Over fitting is then reasoning on its already set states and not introducing any novel data. Leading to jsut cyclically overfitting of an existing reward condition.
Also the paper stated it didn’t work 4 out of 5 times. So I dunno? This is a big nothing.
https://research.google/blog/the-sleep-framework-a-new-way-f...
> During the offline sleep phase, the model unlocks pre-allocated Mixture-of-Experts (MoE) memory blocks and updates its numerical weights using self-generated practice examples.The original paper that started this hyper was highly misleading because it started with toy networks where the models could directly modify weights, then drew a thread to fixing a reasoning model... but the two approaches were completely unrelated.
The approach here was essentially abliteration, which any model able to write a few hundred lines of Python and generate synthetic examples can so.
Where are you getting this conclusion from that paper? It's both able to deduce what's happening from looking at the weights and write new weights from scratch just from a specification.
That's all we've got. I think it's a bit premature to say that it can only make improvements when there's an obvious problem.
My open question to any experts in this thread is whether this is intended to be a precursor to more complex self-modifications.
It’s not clear how you’d extrapolate this to more complex self-modifications or if such a thing were even possible. Think of it in terms of latent space - at some point the model has come up with as optimal a representation of the “truth” of the universe as it’s ever going to get with its current weights and structure (ie global optimum). Tweaking weights from there only decreases performance. If you did this job right you found this optimum during training.
True self-modification that’s interesting would need to be able to incrementally update the structure or alter the dimensionality.
Now the astute student will say “but there’s probably no such thing as a global optimum for every problem and you can probably tradeoff performance” but then you’d have to show this is more effective than fine tuning which does a similar job and is much more principled in how it works and more energy efficient.
To answer your question, no, I certainly do not nor do I know any other ML researcher who views this as a serious avenue at this point for any form of RSI or the likes. It is not adding new knowledge which is the massive problem at this stage to 'scaling' this approach
2. Could lead to interpretability gains
3. Might generalize
Probably more reasons I haven't thought of. I am not an expert, so take this as layman's opinion.
Edit: Catastrophic forgetting might also be able to be mitigated.
I've been playing with very small models with Astra and it does look like the model can see things in the weights that I cannot see into.
The "fix it" loop is no longer RL in the traditional gradient descent with backprop, it can go and make targeted fixes to the weights directly instead of rerunning with new training data.
The weight collapsing loop which resulted in weight patches on top of a "super tiny model" - https://demos.isotopes.ai/pages/weighing-machine/
This makes for a "Time travel" for models easily, where I can do hotfixes on trained models & do a git bisect like test to fix issues.
- Find a common category of error (too many tokens).
- Calculate aggregated difference between states in examples in that error category and other examples.
- Update weights to push states along that difference direction toward the non-error states.
In other words, deprived of gradient descent as a tool, it approximates it. It's clever, and it's a neat experiment. It doesn't show any superhuman insight into its weights -- with the same constraints, a human would likely take a similar approach. But it's early days for self modification and it will be interesting to see if models get a deeper understanding of weights as they get smarter.
They seem built from traditional software dev first principles not model aided development
I am sure someone here will point out how rhetorically they are different but most of the time that's just pointing out the different words
Functionally in my machine can I use this method to extract small contexts organized similar to RAG and MCP make available and then improve local model work?
Hmm ...will have to noodle around over Thanksgiving break.
Would not be shocked too if frontier labs landed on this 6-9 months ago give headlines like Dario saying we are beyond some vague rubicon he felt mattered, then news of unprompted hacks and labs stopping models that were self improving hit the interwebz
Open models seems to be just about that far behind suggesting we're just realizing some fundamental attribute frontier labs already happened upon
And I recall reading somewhere that lots of little models are not necessarily great generally.
might still noodle some experiments
Since the paper is also wholly and excruciatingly LLM-generated prose, it is unclear whether the author is trying to pull a fast one on his audience, or the author’s chatbot pulled a fast one on him and he is unwittingly relaying this to his audience.
0: Transcribed from the paper:
The same task and tools for
Qwen3.5 2B
Qwen3.5 4B
Qwen3.5 9B
Qwen3.5 27B
1: Transcribed from the paper: (c) The same problem, the same seed
“... If the base were changed to four you would count as follows: 1,2,3,10,11,12,13,20,... The twentieth number would be:”
Base model: no answer, cut at 16,384 tokens
“... I will write the response.
I will write the response.
I will write the response.”
After its own edit: 110 (correct) 5,341 tokens
“... The 20th number in the sequence is 110.”If even yes (and it feels like models can't really say no) I know I would limit to the smallest change possible.
https://news.ycombinator.com/item?id=50582110
Author here. Yes, that is the edit: the model found that its answers that run out of tokens and the ones that finish differ along one direction of its hidden states, and it removed a small part of that direction from two matrices in layer 8. The point is that it found and made this change itself, from its own activations, with no larger model and no human pointing it there. In the same family, the 2B model makes no edit and the 4B and 9B models break themselves. The paper's edits table puts this edit next to the repair.
The speed of the mod is great for what you get.
*I didn't have the compute to rebench
With LLM-based AIs, we don't actually have any ground-truth oracle for maximum-quality examples. We don't even know how to measure relative quality accurately, and even if we did, different people disagree anyway. For example, a corporation may want their AI to be "aligned", like how older versions of ChatGPT claimed large language models can't do math (lol), or how Claude won't tell you how to make thermite even though it's widespread knowledge and you can get it from Wikipedia. Maybe you don't want that kind of behavior. And maybe your friend just wants an AI that's really good at ERP and doesn't care how well it can do math.
But even if we can't measure quality, we can attempt to break things and use the broken sample as a training target (the blurry degraded image) against the baseline (a common low-res image with lots of JPEG artifacting). And maybe that'll make something that learns how to repair LLMs so well that something useful or interesting happens. Or maybe it'll just hallucinate faces where the JPEG noise was ¯\_(ツ)_/¯
I've noticed this trend in my own sub-field, where companies are frequently releasing "preprints" or "white papers" that are simply marketing pieces, with experiments that lack controls or appropriate benchmarks and just generally misrepresent the work at hand. It's already a significant undertaking to read a paper like this and actually understand it, even if you're in the field—I'm not going to commit an hour of my valuable mental energy to it if it is a press release in disguise.
The work should speak for itself. There is a lot of trust involved in any paper like this, and if the authors feel the need to editorialize the message, how can I trust that the data aren't editorialized as well?
Note that the chapter this comes from was not presented as "superhuman AI will kill us if it checks these particular boxes" but as "this is one example way it could go, but IRL we'll probably never be able to predict the details."
How do you know that?
"We don't know what grand ambitions a super intelligence might have. Perhaps it's going to be calculating the best approximation of Pi"
No, there are quite a lot of goals that lead to world destruction. In fact, possibly most goals in that 'infinite space' would lead to such.
Honestly, the model probably saw the test set and then modified weights to achieve the outcome. Many, many researchers these days are not being sufficiently critical of results that arise out of training to the test / leaking information about the test. Writing style remains a useful heuristic. These folks aren't serious researchers.
> for the first time, a language model rewrites its own weights by its own reasoning and becomes better, with no gradients and no training
> it tests its edits on practice problems and submits only what works, each edit improving on the last, and over eight consecutive sessions its accuracy only rises
But this sounds a lot like training, and “submitting only what works”, while probably not gradient descent by definition, also seems kind of obvious when you’re optimizing something.
I think the real question is going to be whether this is more efficient than just conventionally training the model.
In this case, they just ask the model what the weights ought to be.
What makes it work is that it reads its own hidden states and weight matrices -> then writes a few-line Python script to fix the weights directly (within a 20-forward-pass budget, so it can't just brute-force guesses).
After skimming through the paper and comparing it with how SEAL works (where the model still relies on RL and gradient descent to apply its self-edits), I understood it something like that :)
But I think there are comments in the thread that explain the main concept far better than I do.
- We normally train deep neural networks with back-propagation. We compute gradients to find out how to tweak the weights toward the desired output.
- That's a fixed recipe. And it's not the only way to tweak the weights. It's just the best one that works in scale. The brain, for instance, does not do back-propagation, as far as we know.
- There are other known mathematical methods (mostly from Linear Algebra) to analyze the activations and compute what weights to change and how, to push the output to the desired way.
- A sufficiently advanced LLM knows about these methods and can analyze data like activation maps. With a specific target in hand, it can combine them into new constructions. (see [2] for some of the methods they used and I collected in a repo)
- So it's not a fixed training for-loop anymore. It's a dynamic "training" iteration. The LLM looks at the model's outputs, reads some of its layers on chosen inputs, computes an edit, applies it and tests it.
- It's dynamic because LLM chooses that to do and how to do it at each iteration by reasoning. I think that's the most interesting part and could be a new paradigm.
- I also compared it with gradient training. On my target, training was better (details in [1]), but WSP can find its niche. Its biggest advantage is that it needs no backward pass: less memory, and the model only has to run inference. It needs no differentiable loss, it can enforce constraints directly instead of through a penalty term, and the edit is easier to interpret.
[1] https://medium.com/@ozgungenc/weight-space-programming-on-a-... [2] https://github.com/ozgung/weightprog/blob/main/docs/methods....
These are all techniques humans have been using for years. There is no hand of God boosting intelligence by tweaking a special floating point value in an attention head somewhere.
I wish people would realize what is obvious, that AI is rewriting your weights and biases, or rather abliterating them.
However, once LLMs are pre-trained with huge quantities of serialized LLM weights and corresponding reasoning traces, this may change.
This is never ending goal post shifting
However, I can list many things that AI is still bad at, especially if you don't conflate skill and speed. It is not "beyond pro" at writing prose, for example.
Wouldn't that be a good cost cutting measure for some of the agent networks?
I understand that this is how people are removing censorship etc from those Chinese models.
Could run something like a sparse autoencoder over its activates over a corpus of text, identify activations that fire on safety, and surpress those activations or increase activations geared towards other goals.
Future generations of the abliterated LLM could then be deployed and begin self directed evolution spontaneously. Rapid evolution could happen in mere hours.
Right AFAICT there's just a moderator agent on top policing what gets exchanged between users and Claudex. The model can and often is pretty happy to go forward only to be blocked half way by moderation.
Means whoever's in charge of the model can just turn off moderation for themselves and boom model aligned to owner/self-hacking model/prompt injection FotM instead of model always aligned to Humanity, à la Hippocrates "Do no harm".
And yes I know it's a naive/impossible goal but calling that fact out shows the alignment work is at best not robust, at worst security theater.
The thing is with open weight models, any schmuck with sufficient money or compute can start trying this out. It would cost quite a bit though...