This video (with a slightly different title on YouTube) helped me realize that the attention mechanism isn't exactly a specific function so much as it is a meta-function. If I understand it correctly, Attention + learned weights effectively enables a Transformer to learn a semi-arbitrary function, one which involves a matching mechanism (i.e., the scaled dot-product.)
Indeed. The power of attention is that it searches the space of functions and surfaces the best function given the constraints. This is why I think linear attention will never come close to the ability of standard attention, the quadratic term is a necessary feature of searching over all pairs of inputs and outputs.
Er, is it possible that you are using `scipy.spatial.distance.cosine` to compute the similarity? If so, note that this computes the cosine distance, and the cosine similarity is defined as 1-cosine distance.
I tried out your example using the following code:
from sentence_transformers import SentenceTransformer
import scipy.spatial as ssp
model = SentenceTransformer("all-mpnet-base-v2")
A = model.encode(['chocolate chip cookies','PLS6;YJBXSRF&/'])
CosineDistance = ssp.distance.cosine(A[0],A[1])
Where `CosineDistance == 0.953`
This means the model is actually working quite well, were these to be similar to each other we'd expect CosineDistance to be much closer to 0.
The other comments about such distances being useful for relative comparisons also apply: I've used SentenceTransformers quite successfully for nearest-neighbor searches.
I half-expect some sub-branch of the abrahamic religions to issue an edict against AI, arguing that it's a false idol. (Not that I am arguing for this.)
It will get much more anti- when it becomes clearer just how eerily fitting one of the early 'heretical' group's beliefs are to AI and digital twins.
For example, the group believed that it was actually the future but we were in a recreation of the past in the images of an original (and now dead) humanity who had brought forth a still alive intelligence based in light that recreated a twin of the universe pretty much just to resurrect humanity.
They talked about how it was the consumption of one's words that brought them back, not consumption of flesh and blood. How many would be combined into a single one, and that when we one day saw a child not born of woman it would be the creator of this twin universe, which in actuality isn't physical and is just its light in the images of what existed before.
It's a bit ironic as it's the most compelling tradition for Jesus having actually had any prophetic knack, but embracing it means acknowledging that the past 2,000 years of Christian tradition backed the wrong horse.
The Gospel of Thomas ("good news of the twin") and the only sect explicitly recorded following it, the Naassenes as recorded in Pseudo-Hippolytus's Refutations book 5.
The latter picked up a lot of the Valentinian Gnostic ideas absent in the former, but seem to have held on to some rather unique concepts which help to flesh out and ground the concepts in Thomas.
In particular, the way the Naassenes use the language of Lucretius regarding 'seeds' is very revealing in terms of the context for sayings in Thomas historically dismissed as just 'weird' - not weird, just a response to Epicureanism.
Jesus -> illuminated Apostles: Church of Jerusalem -> takeover: new Religion of Paul -> takeover: Roman-Christianity -> secession: Judeo-Christianity
What horse? There was a cart and it has changed hands at least 3 times now. Christianity has always been mainly Paulianism. I mean, if you just read the Gospels and take your religion from the original source it is a different matter altogether from what is the bulk of the verbiage of New Testament, which was a new religion given by Paul, not Jesus.
PG is remarkably consistent in his advice; much of his writing is about developing a nose for "what's missing" and having the chops and resources to attempt a solution.
Also, I think 'Google' in this instance is more for motivation rather than a literal comparison. He's leaving out the part that Google was founded by graduate CS students a) looking for a thesis, b) into node-link graphs, and c) inspired by academic citation metrics. Would PG advise anyone to go to grad school to learn how to find scientific-discovery-based startup ideas these days?
This is what is fundamentally missing from most talk about "tech" startups by VC types these days. They're not actually interested in the nerdy stuff, just in the "disruption" stuff.
L&S were, and they hired other people who were. They didn't start with a business idea, but with a technical one. They filled in the blanks on the business side after they survived the .com crash.
I didn't get to Google until 2011, but it became clear to me after joining that in the past they had gone on a very nerdy mission to hire all the nerdy people and collect them into one place to do nerdy things.
(Unfortunately that nerdy thing ended up being selling ads really efficient, but that's another story.)
My fundamental point is that the Google story is very unlike the kind of stories that YCombinator or a16z like. It started, like you said, as a set of intellectual/technical interests. The other stuff, that VCs today like, came later.
In a way they did the opposite of the usual advice. They started with the hammer (the tech) rather than the nail (the business problem.) They certainly didn't start with a "Like X but for Y" statement like seems to be desired by VC today. And they didn't look like the typical .com story at the time (which was usually: give us lots of VC $$ so we can sell something on the web that is currently not sold on the web, but we'll just use the $$ to buy customers and make no profit...)
I would posit that if today's Google came to YCombinator today they'd be shown the "no thanks" door.
> Unfortunately that nerdy thing ended up being selling ads really efficient, but that's another story.
The irony is that both the thing and its side effects were anticipated very early on.
"Currently, the predominant business model for commercial search engines is advertising. The goals of the advertising business model do not always correspond to providing quality search to users. ... It is clear that a search engine which was taking money for showing cellular phone ads would have difficulty justifying the page that our system returned to its paying advertisers. For this type of reason and historical experience with other media, we expect that advertising funded search engines will be inherently biased towards the advertisers and away from the needs of the consumers."
(The Anatomy of a Large-Scale Hypertextual Web Search Engine, Sergey Brin and Lawrence Page, 1998)
Hopefully 2 Stanford CS PhD students hacking on their project would be funded by YC today :)
You bring up a good point about starting from the tech rather than the problem. Usually the advice from VCs is to start from the problem and iterate on the tech until you solve it. What was very fortunate for Google is that the tech translated into a great business problem. Open up any CS textbook and 'Search' is always a major section. It was also a great business because the problem is important, frequent, and had not been properly solved by the big players in the mid 90s.
"And then we realized that we had a querying tool..." (Page)
> Hopefully 2 Stanford CS PhD students hacking on their project would be funded by YC today :)
Around the dawn of YC, IIRC, when PG did the "summer founders" or something like that, a group of 4 of us Lisp hackers applied as a team. No response.
We were mostly in grad student-like lifestyle modes, and not tied down, were energetic, and already had various applicable experience.
But I think PG was mostly looking for barely-20 year-olds to drop out of college, rent an apartment in Harvard Square together, and sit around hacking in towels.
Since that's what he wrote in one article. Which I guess trumped the article in which he said Lisp hackers are great for startups.
Now his latest article says he's going after 14 and 15 year-olds. :)
I always wonder about the success rate of advice. Do we have any student from this talk of PG who succeeded later in life? Joined an ivy league school? Proceeded to create a startup? Had success with it?
Don’t get me wrong. This folklore is extremely important, if not just to raise the grades of one student. But as a founder, when I give advice, out of inflated ego generally, I also have vertigo from the height of everything that could go wrong about being mistaken about my advice.
I feel that worry too, but I can validate his advice, somewhat.
I'm 44 and spent most of my career working individually and with my closest friends on projects that felt interesting, and it went really well for us in all of the ways. I also met these friends at a selective school, though it was a state-funded high school (massacademy.org), not a university.
Also, someone once came to my very conventional elementary school in 5th grade on career day to talk about a very unconventional career path (being a full-time peace activist) and this had a huge impact on me, both because it validated activism as a career and perhaps more importantly because it validated not having a normal job.
Getting a very short lecture in school from an interesting non-teacher can be at least very memorable and perhaps life-changing.
> Be careful whose advice you buy, but be patient with those who supply it. Advice is a form of nostalgia. Dispensing it is a way of fishing the past from the disposal, wiping it off, painting over the ugly parts and recycling it for more than it’s worth.
But that's the thing. "Search" was not a great business problem. It still isn't. I used Google for years before their IPO and people were always like "how is this thing going to make money" and believe it or not there was a lot of skepticism even at IPO time that they even had a possible reasonable business model. (Same for Facebook at their IPO, too.)
Search is not a good business. Ad sales over top of search turned out to be. AdWords is the thing that catapulted Google towards (insane) profitability. And for the first few years, Google didn't really sell ads, or market themselves as even aiming to sell ads. But AdWords could only be successful because they had already captured the market on search, and so had a captive audience to show ads to (and relevance information based on their searches).
AdWords rolled out in 2000. Many of us had been using Google as a search engine since before the company had even been founded (1998).
Totally agree. What I mean is many research topics can be very interesting from a technical perspective but don't translate into solving problems that are frequent and important enough to build a business around. In the case of Search many business gurus didn't understand at the time that you can make a fortune with a free product. What matters is who has the traffic and who they're selling it to = eyeball meets ad. Page and Brin initially "expected that advertising funded search engines will be inherently biased towards the advertisers and away from the needs of the consumers" (foreshadowing). Also they tried to sell to Yahoo for $1M so it took time to see the full potential, as you describe.
When Google started, you didn't need a business model for a "dotcom" (as we called them, before "tech stocks" became the "tech" that we now call ourselves).
We already realized the necessity of finding information amongst the exponentially growing wealth of Web servers (sites).
And Google obviously worked much better than the existing crawler-based search indexes and curated directories.
And lots of people thought a "portal" was a good place to be, if you could manage it.
> And Google obviously worked much better than the existing crawler-based search indexes and curated directories.
Funny thing is I recall using Google from the very day I first saw it come across Slashdot -- probably late 1997 -- before they were incorporated even... but I also recall that at some point around 1999 I actually switched back to using AltaVista or something similar for a bit. Because I preferred the search results I got.
It was actually a more competitive situation than people might remember in hindsight.
The big difference is that Google made it through the .com crash filter better than anybody else. That and they kept the good will of their customers by keeping minimal and straightforward and (for a while) ad free.
I recall once google came on the scene I ditched altavista and never looked back. I was a teen but my recollection was that Google included the relevant paragraph of text from the page shown in in the search results whereas altavista showed one line that was often indecipherable gobbledygook - perhaps the page title matched a keyword - and it made it so much easier to scroll through ten results and identify the one that was the highest quality
Plus the no fast no clutter homepage for google compared to how many links can we fit in one screen for the portals
There was a brief window where AltaVista improved and I went back to it after being Google since 98. Probably only for a few months. I often preferred the query hints you could give it to get very specific output. No matter, they went out of business.
The turning point that brought me fulltime to Google from then on (until recently switching to Kagi) was when they started having almost live results. It's hard to remember or imagine but search indices were often days, weeks out of sync for critical things like news, etc and Google was really the first to fix that.
I think there's a connection here to the old adage that "small companies make it possible, big companies make it inexpensive".
The purpose of VC can be viewed as trying to take a company from nonexistent to big while skipping the small stage, but small is where the action really is.
Yep, unless some new thing is acquired in a buyout, or if an entire group is spun off with almost no oversight, large companies are horrifically bad at creating new things. It's nearly impossible for them to do it in the case where the new product would affect existing revenue streams.
Very interesting to hear your thoughts and experience.
I wonder if we really shouldn't make a bit more out of the fact that there are different types of startups: those that pick off low-hanging fruit really well, those that combine novel technology with novel problems (for instance, the recent YC LLM meeting note-taking / transcription service), those that are scientific-discovery based (see, for example the NSF Small Business Innovation Research program), etc. I am not experienced enough to create an ontology of startup ideas, but I think it'd be an interesting exercise.
The nsf program basically exists to fill the gap between research and product market fit. Except They focus on commercialization plans which we know that in reality is subject to change (aka meaningless).
The real trick that made Google possible, and the following web 2.0 era startups, was that linux and commodity hardware was all that was needed at the time so engineers could strike out on their own. Comparing it to the current ML era where you need many millions of dollars of data and many millions of dollars of hardware to compete, the guys that wrote the transformers paper are stuck in bigcorp and aren't going to be the next Larry and Sergey. OpenAI might be one of the few new big companies but it's run by a VC/CEO, not the engineers that figured the stuff out, and he had to sell half the company to Microsoft anyway.
These things always happen at intersections. There are lots of smart people with ideas. The magic happens at the intersection of ideas, time (what technology is available) and luck. Apple Newton was too early. Apple iPhone was just right. Doom 3D was just at the right time when commodity hardware could do what the amazing software needed it to do.
The point I was trying to make is what are the situation where an independent small startup can make it big. Your examples are instructive, engineers were able to make id Software big because you didn't need a lot of hardware to invent 3D shooters, but Jobs gets the credit for the iPhone because the engineers needed hundreds of millions of dollars in capital to pull it off.
pg is giving advice based on him getting rich from that period of time where the small startup could make it big. But those conditions are probably rare going forward.
maybe. right now, if a startup has a new ML training technique or whatever, for a specific niche, cloud and VC capital will let them make a model that does way better for their niche than repackaging well funded models from OpenAI.
Wild to think that potentially the most innovative company of the last five years, and next five, was run/founded by a VC. It goes against all of the VC hate
Maybe? YC seems to have an inconsistent view on things that are research vs product. But it’s probably more nuanced than that. Search was a market though at that time. If you pitched an AI LLM before chat-gpt what would the odds have been?
I dunno, I toyed with the starting phase of YC application a couple years ago, and started working my way through what was involved in their application process and reading their materials, even watching some YouTube content, etc.
I didn't really see how a business that was starting from a more R&D angle would make it through.
And a lot of it seemed really pitched to the bizdev "hustler" founder personality, not to engineers/nerds.
The foundational $$ that has gone into the research to make the LLM stuff happen... was inside the existing big corporations (Google, FB, MS, etc.) and only left there once it had been proven.
Ilya Sutskever (OpenAI) for example was employed inside Google doing that kind of work and only left when he ran up against the limits of what he felt he could get done there. But the fundamental R&D had already been done. When I was at Google I saw some of it from a distance before I ever heard of OpenAI.
The VCs have come along at the tail end of the R&D cycle on this stuff in hopes of cashing in. Same as they did with crypto and N number of trends before. They're trailing, not innovating.
They're primarily interested in successful business models capitalizing on existing technology, not the actual development of new technology. Unless one has distorted the meaning of "technology" significantly.
I wouldn't fixate on the Google part too much. It's pretty clear he is using it as a common point of reference for 14 and 15 year olds. "You guys know about Google?, the giant company with the web site? Okay well let me tell you how companies like that are built."
Building a successful company depends on many things. The point made here is that you are unlikely to be expert enough, and motivated enough to build a successful business in an area you are not interested in, and haven't played around in. So to increase the amount of domains that you could create a business in from 0 to a small positive integer you should build things that interest you.
Of course if you have money, or know how to persuade people who have money, you can be something of an idiot and still develop a successful business. Usually you are not really developing the business in this case, more attaching yourself to a business that smart people are running for you.
In a pool of high school freshmen, most probably have a better chance of success following the first path, than the second.
Google in 1999 had the the fortuitous alignment of every possible factor: brains, Stanford connections, timing, extremely scalable and lucrative business model, funding, etc.
It was very different from the punch-the-monkey style ads that had taken over the Internet, though. Google ads were exclusively text based and unobtrusive. They relied on showing you the right ad for what you happened to be searching for at the moment, rather than grabbing your attention and forcing you to think about stuff you didn't really care about.
He should! As I hope to convince people of next month, there’s a wealth of classical AI knowledge that’s just begging to be applied to modern systems. That’s not to speak of the wizardry happening in neuroscience and drug discovery right now…
Isn’t “you shouldn’t focus on science to start a good tech company” a good indicator that our conception of “tech” company is completely broken? That we’re really talking about ways to milk money from gambling billionaires, not change the world with successful products?
The Great Green Wall is ambitious in scope, but composed of relatively simple parts: capturing rain-water with berms, planting the right kinds of plants in the right order, and creating a system of ecologically sound incentives for people to participate in the project. It's wonderful to learn about.
Andrew Millison, the creator of the linked YouTube video, has an interesting channel where he essentially showcases applications of / advances in ecological technology (i.e., permaculture.) I take some solace in the fact that his videos get millions of views; suggesting that permaculture / ecological design principles aren't some fringe-y thing any more.
In some sense, the introductions of scientific papers are supposed to be / are the original 'verification blockchain' albeit one done with human minds and messy research projects.
This is probably preaching to the choir, but in my experience, the best prompt depends on the type of task. The thing that varies is the amount of background knowledge needed to explain the context of what you are asking Chat GPT about.
For instance, there are single-line questions like:
> What is the meaning of the song "What is Hip" by Tower of Power?
> Can you give me an example of verse written in Trochaic heptameter?
These types of question need no additional background as the relevant bits are contained within the question itself.
For more complex tasks, I've found that the best prompting strategy is to approach Chat GPT like it's an engineer who has little background knowledge of your task. There is no set prompt, I just try and give it enough information to help me.
For example:
> I have a dataframe with the following column names: [list of columnnames]. I'd like to use the following R function to Z-score the appropriate columns in terms of certain baseline columns. The baseline columns are: [list of baseline columns]. However, due to {condition}, we need to skip every 4th column. Here is the R code I have, how can I translate it into doing what I want?
{R code goes here}
It's not that they've removed the bar exam and changed nothing else. From TFA:
>The Oregon Supreme Court on Tuesday approved an alternative licensing program that bar exam reformers hope will spur further innovation in other states. After law school, candidates will spend 675 hours working under the supervision of an experienced attorney and create a portfolio of legal work that bar officials will grade as an alternative to the traditional bar exam.
>In addition to completing 675 hours of paid legal work, participants in Oregon’s new program must submit at least eight examples of legal writing, take the lead in at least two initial client interviews or client counseling sessions, and head up two negotiations, among other requirements. The applicants' portfolios would then be graded by Oregon bar examiners, and those with qualifying scores would be sworn into the state bar.
Also relevant:
> Wisconsin allows graduates of the state's two law schools to become licensed without passing the bar in what is known as a diploma privilege, and New Hampshire allows a small cohort of law students who complete a specialized curriculum to bypass the bar.
Funding for the sciences is always welcome; it's going to be interesting to see what SETI does with this money.
Seeing this news makes me wish more donors donated to science at the department level, or to every lab in a given department. Money is so incredibly tight in Academia that the entry level job (being a graduate student) typically gives people around minimum wage or less for five years to make a life with. This is an absolutely terrible incentive for attracting some of the best and the brightest to enter the funnel of knowledge production workers (e.g. grad students, post-docs, researchers, and professors.)
I don't know how this can be fixed systemically, but donors could help change the incentives and improve the quality of science for all.
As an adversarial opinion on this, I don't think that good science is bottlenecked in any way by a dearth of grad students. Conversely, society probably already has enough of the "best and brightest" in academia, and it should do more to funnel them to other, more directly practical, endeavors.
Interesting point; thanks for providing the opposite opinion. It's not that no one is applying to grad school, it's that brilliant, creative people with good job prospects are being "pre-diverted" away from the sciences.
For everyone who complains about the quality of the scientific literature, the replication crisis, the golden quarter, I think part of it ultimately stems from the way the funnel is set up.
Those problems are real, but they aren't caused by a lack of the best and brightest in the sciences. The problems are structural due to misaligned incentives. Intelligence isn't particularly correlated with ethics or conscientiousness.
All issues are definitely influenced by systemic incentive issues for sure.
But is not graduate school in some broad sense competing with the likes of startup accelerators et al.? Don't get me wrong, I think that's a good thing in the abstract. But 5 years of minimum wage while working for someone who likely has paltry managerial skills, with the prospects of an increasingly shaky academic job market. My point is that it's too bad for society that the competition is so drastically one sided.
> I don't think that good science is bottlenecked in any way by a dearth of grad students. Conversely, society probably already has enough of the "best and brightest" in academia
As an adversarial opinion on _that_, what seems to have happened is that the "best and brightest" filter in academia is further filtered to "those who also have the well-off family background to support that". See also the Victorian "gentleman scientist".
Sure, this filter does serve to cut down on the numbers, but I can see problems with using this filter, e.g. the effect on those excluded, and the biases that it will introduce. I would not call it a good thing on the whole.
It's also worth mentioning that many in the academic pipeline leave for Tech or Finance. My point is that the grad students are the ones who go on to become the professors who direct the research our society relies on.
It should be dumped into a trust fund, an endowment that will fund some low level SETI in perpetuity. The goal should be to free SETI from having to worry about funding from one year to the next. Funding such projects needs to be measured in decades, not dollars.
Tax-exempt organizations need to spend 10% of their endowments every year in order to keep their status, pretty much for this reason (avoiding perpetuities).
More seriously, endowments are for specific purposes specified by the donor. Most of the time, this money cannot be spent on anything else, regardless of "need."
They should do it! Desptite centuries of archaeological research we still don't really know how the pyramids were built. A real world test of the various possible techniques would answer a lot of questions. Plus when they're finished maybe the Goa'uld will show up to resolve the SETI issue.
As someone who works in a department that was essentially founded via a gift from some very generous donors, it has been enormously impactful in the nature of our positions, how we approach science, etc. All of them for the better.
The donor class often has very stupid or industrially motivated ideas about what kind of research should be done. In the former case, you're mostly just adding another idiot to your list of pseudobosses. In the latter case, what's really happening is that the rest of the funding is now going to end up subsidizing some private research endeavour. This is a form of capture, and should basically be avoided.
The current state of the system for graduate students is bad, but it is not improved by allowing private investment.
> people who are very good at making money make stupid investments?
Absolutely, particularly when the "investment" is a donation to charity or a university that's more to assuage guilt or plaster a name for posterity than it is for any return.
People that do end up with a great deal of money aren't always the sharpest crayons in the sun - many have luck and good advisors.
No, Walter, it is you who created a false representation of what noqc said when you replied to their comment saying "people who are very good at making money make stupid investments?"
> people who are very good at making money make stupid investments?
Just because someone makes a lot of money doesn't mean they are intelligent or did so upon their own virtue. And it doesn't necessarily mean they're good at investments and capital allocation.
>> it is not improved by allowing private investment
> wow. Maybe we shouldn't allow things like inventing semiconductors, jet engines, television, etc.
An incredibly sizable portion of technological development in the U.S. in the past 100 years is due to government (i.e., public) funding, not private funding.
> An incredibly sizable portion of technological development in the U.S. in the past 100 years is due to government (i.e., public) funding, not private funding.
Never mind semiconductors, jet engines, television, internet phones, AI, electric cars, engines, steel making, typewriters, the whole textile industry, telephones, CDs, electric guitars, autotune, cameras, need I go on?
> Just because someone makes a lot of money doesn't mean they are intelligent or did so upon their own virtue. And it doesn't necessarily mean they're good at investments and capital allocation.
Maybe read biographies of Musk, Jobs, Gates, Bezos, Buffet, Rockefeller, Walton, Sears, etc., then tell me how stupid they are.
I've talked with many professors on the selection committees. They want the best, especially for competitive programs. All of the researchers I've talked with are keen on training the next generation of researchers.
