> This whole thread is an overreaction. 302 comments about code that does not work. We haven’t committed to rewriting. There’s a very high chance all this code gets thrown out completely.
Is your reading comprehension poor? Where in that does it say it had no chance of being merged? Do you understand what an experiment and what the purpose of an experiment is ?
Of course he's not writing a legal contract, but to go from saying:
> This whole thread is an overreaction. 302 comments about code that does not work. We haven’t committed to rewriting. There’s a very high chance all this code gets thrown out completely.
And then fully merging into main in under 18 days is quite extreme
>It's clearly not yet a tool that can deliver new math at a scale.
What is at scale here exactly ? This is the most impressive so far, but it is one of several such advances in the last few months, all of which were with publicly accessible models.
LLMs have been trained on a lot more data than any single human (text wise at least) for years now and these sort of results have only been possible for the latest crop of models in the past few months. Models get better as they get better.
The argument is whether models of today, suitably trained on pre-17th century data (if comparable quantity was available) would be able to "invent" calculus et cetera.
If we believe today's models are sufficiently capable to have been able to do so, why are we not getting these types of results today compared to the entire world knowledge and especially math?
Are research mathematicians simply not prompting LLMs in the right way?
No offense but this is just needlessly convoluted fiction. It ultimately tells you nothing (or what it tells you is very wrong) and has no predictive power. Actually it's worse. This sort of rhetoric is much poorer at predicting LLM capabilities than just playing it straight.
Disney bought ESPN in 1996, Marvel in 2009 (literally had 2 movies released here and one of them flopped) and Pixar in 2006.
For Pixar, they and Disney were joined at the hip even before acquisition. Besides distribution rights, Disney had full sequel rights to almost all of Pixar's catalogue at the time. Disney could have made a sequel to Finding Nemo, The Incredibles etc even without Pixar's blessing or involvement. There is quite literally no Pixar without Disney.
Marvel? Their most successful years were under Disney. ESPN did not become the media empire you know it as until well after Disney's acquisition either.
I specifically said reputation. Disney has a history of buying properties and squeezing every last dollar out of them until nothing is left but a husk of what they were at their peak. Disney certainly got their money’s worth out of those purchases. I’m not denying that. But the reputation of all of them has been on a steep decline even if there were temporary spikes after the acquisition.
I’ll grant you that “soon after” might have been a stretch for ESPN, but it’s obviously true for the others. Almost all of Pixar’s most enduring films had their start before Disney bought the company. The same is true with Marvel. Sure, Disney’s fanfare might have played before the Avengers films, but those movies were the brainchild of Kevin Fiege, who was already in charge before the Disney purchase. You can maybe claim Disney has a good eye for finding companies on an upward trajectory, but these are all examples of Disney management’s failure to be a long-term steward of their acquisitions.
>Disney has a history of buying properties and squeezing every last dollar out of them until nothing is left but a husk of what they were at their peak
Only if you don't know what you're talking about.
>But the reputation of all of them has been on a steep decline even if there were temporary spikes after the acquisition.
None of these were 'temporary spikes'. Marvel was on an unprecedented high for a decade. If that is a temporary spike then almost every company’s successful era would count as a temporary spike.
>Almost all of Pixar’s most enduring films had their start before Disney bought the company.
Did you not read what I said?
Pixar’s pre-acquisition slate was already deeply tied to Disney. Disney co-produced and distributed those films, and the relationship was so significant that Disney had rights to continue all but one of their properties even without Pixar’s involvement. That's pretty much unheard of.
Any success you can attribute to Pixar pre-acquisition, you can attribute to Disney as well. Moreover, it would be silly to claim Pixar has been mismanaged simply because arguably the most succesfull animation run in history did not go on forever. Pixar has been managed fine.
>Sure, Disney’s fanfare might have played before the Avengers films, but those movies were the brainchild of Kevin Fiege, who was already in charge before the Disney purchase.
Feige was in charge of Marvel Studios, not all of Marvel Entertainment, and his position was not necessarily secure under the old structure. He had well-known conflicts with Ike Perlmutter, who had significant authority over Marvel at the time.
It was Disney who re-structured Marvel Studios to be semi autonomous, answering only to Alan Horn and the Walt Disney Film Studios Division. Regardless, Kevin is still in charge of Marvel Studios and Disney has left the arrangement largely the same, so blaming the current situation on Disney specific mismanagement would be very strange.
I think videos are a unique thing computers offer. Books I understand. You have them digital or not. But a video ? Without a computer, there is no video. You were present for the initial lecture or you weren't and that's it.
Video is an electronic process for capturing images and displaying them.
Before digital video there was analogue video, and analogue video was perfectly possible without digital sampling, or computers. Heck, video pre-dates silicon chips and used to be done with CRTs and valves.
There's nothing about video that uniquely requires computers. Maybe you meant "streaming video"?
I realize you're probably under 30 and don't remember "ye olden times" but nearly 90% of U.S. homes and every school had an analog VCR long before they had a computer. Widespread consumer video formats included VHS, DVD, Blu-Ray, Laserdisc, etc. I still buy some movies I care about on UHD discs and watch them in a dedicated Blu-ray/UHD player. Even the 'smart' TVs and streaming sticks most people watch streaming channels like Netflix on aren't functionally computers (no meaningful user accessible local storage, input like keyboard/mouse, CLI or windowing GUI).
Personally, I learned an enormous amount from video before I ever touched a computer. In elementary school we learned from 16mm films almost weekly and watched space launches and Carl Sagan's Cosmos series on TV (it was rebroadcast in the mornings specifically for schools). My junior high had a television in every classroom and some classes were planned around shows on PBS, NASA channel, C-SPAN and BBC. In the late 80s there was thousands of hours of educational video programming sent via direct broadcast satellite to 18-inch dishes at schools. In the 90s every grade and subject had hundreds of interactive video DVDs in large notebooks (four discs to a page in plastic sleeves) and multiple DVD players per classroom.
The peak installed base of VCRs in the U.S. was in 1999. Streaming video wasn't common in consumer households until well into the 2000s, YouTube didn't even exist until 2005 and most people had never heard of it until 2007. In 2010 Netflix mailed DVDs in envelopes to 25M homes every week. They didn't even offer a streaming plan until 2011.
As someone who's spent most of my adult life thinking about video technology, with patents ranging from analog days to the streaming tech you use today, computers have been extremely disappointing in terms of enabling any unique "learning from video" features that are computer-specific. In the 90s we realized that computers could make digitized video random access letting us sequence it non-linearly to make it interactive in response to user input. We knew that computer-enabled interactivity, responsiveness and real-time adaptation to learner progress would be incredible for improving video education. Yet the vast majority video content available online today is still linear in form. Even video that's specifically educational is no more interactive or user responsive than a 90s DVD disc.
Sadly, only two things have really changed about consumer video in the last 30 years: quantity and distribution. There's much more video content and it's remotely accessible on-demand instead of being limited by broadcast channels and storage media. But that's far more about communication technologies like broadband than computer technologies. For a few years YouTube even had authoring features like interactive menus and conditional branching but removed them because it didn't increase ad revenue. There are a few dedicated video authoring platforms for education which can apply uniquely 'computer things' to video like dynamic scripting, conditional branching, viewer annotation and timecode-linked threaded Q&A. Unfortunately, such content is rarely found outside high-end corporate training and some university courses. But there are so many other ways we could combine the strengths of advanced wikis with interactive video. Today, the most the public sees is just an HTML link from a wiki to a video clip. Almost none of the learning features computing could uniquely bring to video are widely available to learners. Since ~90% of everyone already had access to linear video playback before they had access to a computer and most online video today is still primarily linear, in my opinion, there's still virtually no uniquely 'computer-enabled video' involved in learning. Computers haven't enabled much that's new in video - just much more, much cheaper and more convenient forms of what we could already do without a computer.
As a broadcast technology architect I agree with you a whole lot on the broader technology statements.
But as a former lecturer, I also think the promise of interactivity is dependent less on the tools than on the people. Authoring interactive learning materials is difficult and while that interactivity is engaging, it's not necessarily great at getting a density of information out there.
The Socratic method is great, but that level of interactivity presumes in advance that you know what questions the student will be asking, otherwise it's just a dumb gate. Branching stories for interactivity are highly labour intensive. I suppose if you use AI you could generate a massive number of videos to cover branching learning, but that's going to still be an intensive operation, especially if you're supervising that.
Squeezing a model like this complete with 'big model smell' into 16GB...Honestly it's not even possible or feasibly possible today.
It'll require some kind of:
- breakthrough in architecture or
- breakthrough in hardware or
- some breakthrough quantisization technique
The problem is that all the parameters need to be in memory, even the ones that aren't active (say for Mixture Of Expert Models) because switching parametrs in and out of ram is far too slow.
We show that EMO – a 1B-active, 14B-total-parameter (8-expert active, 128-expert total) MoE trained on 1 trillion tokens – supports selective expert use: for a given task or domain, we can use only a small subset of experts (just 12.5% of total experts) while retaining near full-model performance."
Yeah he never said that.
reply