This model card is eye-opening (I think it might be designed to be). The alignment and model welfare sections are extensive, which is heartening. At least on the surface Anthropic seems to be living up to its promises RE safety. That said, has anyone else read section 5.2.3 in the Alignment Risk Update https://www-cdn.anthropic.com/79c2d46d997783b9d2fb3241de4321...? This is referenced in the model card in 4.1.3. Basically they ended up training a the model with an RL reward model that had access to the model's reasoning in 8% of cases, by accident. The problem being that the model could learn to directly manipulate it's reasoning traces to satisfy external observers. This seems like a huge deal and it may have partially poisoned Anthropic's interpretability pipeline moving forward.
For me, the attempted productization of Sora was conclusive proof that 1) OAI was overcapitalized and desperate for revenue 2) safety didn't matter to them much 3) improving the world didn't matter much either.
At one point you mentioned an interaction with OpenAI staff where you were looking to interview AI Safety researchers. You were rebuffed b/c "existential safety isn't a thing". Does this mean that you could find no evidence of a AI Safety team at OAI after Jan Leike left? If you look at job postings it does seem like they have significant safety staff...
Interestingly we are still experiencing the technological momentum inspired and created by what OpenAI used to be. AI for humanity.
Given the initiative started circa 2017, much of the goods remain. It's a hijack of creative geniuses who got together, which is now turning into cow milking tech.
OpenAI played the charity, coupled with a powerful altruistic card.
It didn't say: we believe a more effective for-profit business shall start as a non-profit in this field, because it would yield innovation which we can then skim money off down the road. That would have been transparent.
Not saying it was the intention at the start. But they flipped the game at some point. Let's play Chess, it's a better game. Oh I decided we are now playing Checkers, sorry, I won.
i guess my (too nuance maybe) point was: the system we live in is like water; the urge to swim with the big fish is overwhelming... it was gonna happen eventually at the level they are playing at.
This is a valid point, the good news is I think there is some hope in developing the craft of orchestrating many agents into something that is satisfying and rewarding in it's own right.
I'm so excited about landscape architecture now that I can tell my gardener to create an equivalent to the gardens at versailles for $5. Sometimes he plants the wrong kind of plant or makes a dead end path, but he fixes his work very quickly.
It looks like one of my employees got her whole account deleted or banned without warning during this outage. Hopefully this is resolved as service returns.
I have a deep background in music and I think that while the creation was super basic, the way the output was so unconstrained (written by a model fine-tuned for coding), is really interesting. Listen to that last one and tell me it couldn't belong on some tv show. I've had always issues with any ai generated music because of the constraints and the way the output is so derivative. This was different to me.
I've got to come to the OPs defense as well. This was a remarkable demonstration of Claude performing a task thats probably very out of distribution. This would not be interesting if it were a music generation model or program, it's interesting because this is not what Claude code was explicitly trained for. The fact that it generated waveforms from scratch and built up from there is really amazing. Your cynicism was applied before even reading the article.
How is it out of distribution? There are plenty of Python libraries for sound and music generation; it would be surprising if they were not in the training set.
There's a general pattern becoming evident of people being surprised with AI capabilities because they didn't realise (and none of us do fully) how broad the training set is, the variety of human output AI companies were able to harvest.
Even if all AI does is remix and regurgitate, there's a segment of the audience that is going to find some particular output brilliantly creative and totally original.
This is still a surprising composition of low level in-distribution things then. Like I would not have expected it to generate the waveforms from scratch, and be able to piece them together so well. If it had just plugged some kind of notation into a pre-existing API in its code then I would probably agree with you.
There needs to be more documentation about what info was provided to the LLM and in which format before we decide that LLMs are necessarily bad at this. That said, you would expect the offering from a $500bn company to be more robust and better tested than this, assuming this is reported accurately.
I think you could have credibly said this for a while during 2024 and earlier, but there is a lot of research that indicates LLMs are more than stochastic parrots, as some researchers claimed earlier on. Souped up versions of LLMs have performed at the gold medal level in the IMO, which should give you pause in dismissing them. "It can't actually understand how many there currently are, the season, how much land, etc, and do the math itself to determine whether it's actually needed or not" --- modern agents actually can do this.
The worlds most impressive stochastic parrot, resulting from billions of dollars of research by some of the world's most advanced mathematicians and computer scientists.
And capable of some very impressive things. But pretending their limitations don't exist doesn't serve anyone.
Interestingly in healthcare there is a correlation between companies that license/sell healthcare data to other ones (usually they try to do this in a revokable way with very stringent legal terms, but sometimes they just sell it if there is enough money involved) and their privacy stance... and it's not what you would think. Often it's these companies that are pushing for more stringent privacy laws and practices. For example, they could claim that they cannot share anonymized data with academic researchers, because of xyz virtuous privacy rules, when they are actually the ones making money off of selling patient data. It's an interesting phenomenon I have observed while working in the industry that seems to refute your claim that "there's no money in privacy". Another way to think about it is that they want to induce a lower overall supply for the commodity they are selling, and they do this by championing privacy rules.
