ChatGTP is amazing. Search engines are dead

[tus666]

I am getting continually more impressed by ChatGPT. But this blew me away. I don't think I will bother with search engines ever again, apart from looking for a company webpage. Look at this. I wanted to know about the introduction of Windows XP and how it replaced Win 98, over what time-frames.

-----------------

> When did WinXP gain widespread adoption among home users? How long did it take?

Windows XP gained widespread adoption among home users in the early to mid-2000s, shortly after its release in 2001. It took a few years for Windows XP to become the dominant operating system in the consumer market, as many home users were still running Windows 98 or Windows 2000 at the time.

> "By 2003, Windows XP had surpassed Windows 98 in market share," - what was the overall market share breakdown in percentage figures?

According to StatCounter Global Stats, which provides data on operating system usage share based on website visits, the worldwide market share breakdown for desktop operating systems in January 2003 was as follows:

Windows 98: 35.02%

Windows XP: 16.98%

Windows 2000: 12.46%

Windows ME: 5.45%

Windows NT: 1.53%

This means that Windows 98 was still the dominant operating system at that time, but Windows XP was gaining ground quickly. Over the next few years, Windows XP continued to gain market share, while Windows 98 and Windows ME declined. By 2007, Windows XP had a market share of over 76%, according to Net Applications.

> Can you give me those figures for Janurary 2002?

Sure! According to StatCounter Global Stats, the worldwide market share breakdown for desktop operating systems in January 2002 was as follows:

Windows 98: 49.38%

Windows 2000: 16.22%

Windows ME: 13.15%

Windows NT: 4.56%

Windows XP: 1.73%

This means that Windows 98 was still the dominant operating system by a wide margin in January 2002, followed by Windows 2000 and Windows ME. Windows XP had only been on the market for a few months at that point and had not yet gained significant traction. Over the course of the year, Windows XP's market share would gradually increase as more users upgraded to the new operating system.

> OK, can you just give me the figures for Win98 and WinXP but from 2002 through 2007?

(GPT now formats result as a nice table)

Certainly! Here are the worldwide market share figures for Windows 98 and Windows XP, according to StatCounter Global Stats, from 2002 through 2007:

Year Windows 98 Windows XP

2002 49.38% 1.73%

2003 35.02% 16.98%

2004 18.32% 47.26%

2005 8.11% 67.87%

2006 3.23% 77.26%

2007 1.54% 76.09%

As you can see, Windows XP gained significant market share over this period, while Windows 98 declined rapidly. By 2005, Windows XP had become the dominant operating system, and its market share continued to increase until it peaked in 2006. From 2007 onward, Windows Vista began to replace Windows XP as the primary operating system for new computers, although Windows XP remained in use by many users for several years after its release.

> So it was really 2003 to 2004 that saw the big migration to XP.

Yes, that's correct. The period from 2003 to 2004 was when Windows XP really began to gain significant market share, largely at the expense of Windows 98 and Windows 2000. During this time, many home users and businesses began to upgrade to Windows XP due to its improved stability, security, and user interface compared to its predecessors.

------------------------------

Now imagine me having to troll through websites trying to get the same information, let alone formatting in such a nice way. I can also ask CGPT for references if I want to go and actually dig in to an article more deeply.

But this really poses a question - why would you ever use a search engine again for any non-trivial research task?

[verdverm]

LLMs cannot search, this is just responding with something it may or may not have read. Have you verified these numbers are in fact correct? Have you tried "searching" for anything outside of its training data? ChatGPT will happily hallucinate references. If you have not experienced this yet, you haven't seen the "true" side of LLMs.

There will be plugins for search, but will still need all the crawling and scraping and serving to be available. LLMs will offer an alternative interface to search, but not always the best one. One thing the LLM does not give you is a spectrum of options. How do you know it didn't miss something. This is where having to "troll" through search results is actually worth the effort.

[tus666 → verdverm]

My understanding is the ChatGPT basically swallowed up the entire internet, although it was a snapshot that might be out of date now. Of course accuracy and correctness are concerns, but that would apply to any kind of summarizer service, and even primary sources can be wrong. Extensive research would always be required to get near 100% confidence of correctness.

[verdverm → tus666]

It only had part of the internet, OpenAI is nowhere near as comprehensive at web scraping as Google, I don't think they actually scraped at all for this, using existing data like CommonCrawl.

The other thing you are not understanding is that it did not memorized these things, it built representations for predicting the most likely next token. This is why it hallucinates and makes up numbers and web links or citations that do not exist.

[hammyhavoc]

You typo'd the title. It's `ChatGPT`.

`/imagine when you discover that ChatGPT hallucinates a lot and is terribly unreliable with facts, especially statistics`

If you don't fact-check, you're going to look like someone who half-assed the assignment very soon.

[verdverm → hammyhavoc]

OP's post is case in point, it took me less time to use traditional search to refute the output of ChatGPT

https://news.ycombinator.com/item?id=35736072

[mtmail]

I tried using search engines to find the 'StatCounter Global Stats, from 2002 through 2007' numbers. Earliest I can find is 2009. It kind of matches up with https://www.w3schools.com/browsers/browsers_os.asp numbers bt I'd ask ChatGPT for reference URLs to be sure.

[tus666 → mtmail]

This was the reply:

Sure! Here are the references to the StatCounter Global Stats website for the worldwide market share of desktop operating systems in January of 2002 and 2003:

January 2002: https://gs.statcounter.com/os-market-share/desktop/worldwide...

January 2003: https://gs.statcounter.com/os-market-share/desktop/worldwide...

Note that these are specific to desktop operating systems and may not include market share for mobile devices or other platforms.

[verdverm → tus666]

Notice how the actual data on these pages is exactly the same and from 2022-2023?

This is what I mean by making up links that don't work. The links further down the page don't even go back that far.

You ought to inspect the links before posting them as proof. By not doing so, you demonstrate the limitations and fallacies of humans putting to much faith in these tools.

This graph made 7 years ago would appear to contradict the numbers ChatGPT gave you: https://www.reddit.com/r/dataisbeautiful/comments/3s3c8o/rel...

Are you ready to rethink your exuberance for ChatGPT?

---

I performed two searches, both from image search, took 1 minute to find a good graph. I suspect ChatGPT took longer to write the combined, inaccurate responses, based on my experience at how slow it streams the answers back.

1. windows version usage by year, 2000 - 2010

2. operating system usage by year, 2000 - 2010

[tus666 → verdverm]

I challenged it on the statcounter figures, and this was the response:

----

I apologize for the confusion. You are correct that StatCounter Global Stats only goes back to 2009. I must have misremembered.

Here are the worldwide market share figures for Windows 98 and Windows XP from NetMarketShare, which provides historical data for desktop operating systems:

Year Windows 98 Windows XP

2002 60.3% 0.6%

2003 45.3% 19.2%

2004 21.8% 50.0%

2005 8.2% 63.9%

2006 3.4% 76.1%

2007 1.6% 74.7%

[verdverm → tus666]

> I must have misremembered.

The fact that ChatGPT says this is very misleading. It does not perform recall, just token generation based on the most likely continuation of the input. It will even finish your questions

[vaidhy → tus666]

Without this thread, how do you know enough to challenge the results? If you have to challenge every output, why is it useful in the first place?

[verdverm → vaidhy]

It's useful for some things but not others.

For example, I used it to help me know how to adjust some things with matplotlib that I hadn't used in years. I wasn't even sure what to type into google to get good results, looking over the docs would have taken longer than the time I spent with ChatGPT.

Like how we all learned how to search with Google early on, the same thing is going to happen with LLMs. They are certainly better at handling the questions and clarification for refinement, something we often do with each other. When they start responding with questions for clarification, rather than just spitting out whatever, that will be a nice step forward.

[verdverm → tus666]

Are you even checking the sources it claims? NetMarketShare only looks to go back to 2016. :facepalm:

At this point, your methodology is looking quite sus

Are ready to accept that ChatGPT is reliably lying to you? The longer your conversation with it goes on, the worse it gets

[tus666 → verdverm]

But the numbers seem credible and realistic when you think about OS usage at that time. If it was hallucinating from web-pages that only go back to 2009, how on earth did it come up with such credible figures?

[verdverm → tus666]

It's not looking things up, it is making things up. This is what it was designed to do... that's why they call it "generative AI"

Half of statistics are made up, so the saying goes. It's easy to make "credible and realistic" numbers if you understand confirmation bias. You want to believe in ChatGPT, so you see its numbers as credible and realistic, despite them not being so.

Also understand that we are in the beginnings of a hype cycle. There was a time when HN was exuberant about the potential for blockchain. I expect that our expectations for LLMs to be made realistic with time. I also believe that LLMs will provide value, unlike blockchain, so the hype cycle is probably closer to dotcom in how it plays out.

The plugin system will be interesting to see play out. It's sort of a hybrid system, using LLMs as an interface for intent while calling out to other systems for accurate retrieval. I'm curious how far it can be pushed, it's hard to see these things being able to interact and navigate all of the tools we have created. Too much variability and opaqueness.

[tus666 → verdverm]

> You want to believe in ChatGPT, so you see its numbers as credible and realistic, despite them not being so.

I didn't really, I had doubts myself. I say the are credible based on my own understanding of how OS usage changed, being there myself at the time.

[verdverm → tus666]

> I say the are credible based on my own understanding of how OS usage changed, being there myself at the time.

This is a classic example of a confirmation bias. How would you know the statistics from that time? Based on anecdotal observations from 20 years ago?

[tus666 → verdverm]

It's not really. I don't know the exact figures, but it seems like common sense to think WinXP would have very low usage in Jan 2022, given it was released in Oct 2021. And anecdotal observations after living through a time and using computers a lot seems like a reasonable basis to gauge the rough correctness of stats.

[verdverm → tus666]

not sure I am following you any more, but pretty sure XP is way older than Oct 2021

Your experiences were biased by being someone technical, windows was widespread, I know of a place still running XP as recently as Oct 2021 tho... there are some business that don't take updating seriously, so what you saw is not evidence of the general population.

Back to the point, it's best not to rely on your feelings when it comes to judging ChatGPT's accuracy

[verdverm → mtmail]

The numbers on that page contradict the output OP was given by ChatGPT, though they are only from W3's logs, so data bias is likely

[bg24]

You do not need to use search engines, unless you have to. For latest information, ChatGPT (and LLMs) are already figuring it out.

It will also have an impact on how user interfaces are being built. Why bother with so much minute design, if you can manage with a Google like search bar, and a voice button.

[ftxbro]

GPT is amazing but when you ask it questions like this it will invent numbers from whole cloth.