I appreciated that, too! FWIW, I could get Claude 3.5 to tell me how many rs a python program would tell you there are in strawberry. It didn't like it, though.
At that point just ask it "Use python to count the number of O's in Broccoli". At least then it's still the one figuring out the "smarts" needed to solve the problem instead of being pure execution.
I'm not sure if your making a joke about the teachers who used to say "you won't have a calculator in your pocket" and now we have cell phones or are not aware that ChatGPT runs the generated Python for you in a built in environment as part of the response. I lean towards the former but in case anyone else strolling by hasn't tried this before:
User: Use python to count the number of O's in Broccoli
ChatGPT: Analyzing... The word "Broccoli" contains 2 'O's. <button to show code>
User: Use python to multiply that by the square root of 20424.2332423
ChatGPT: Analyzing... The result of multiplying the number of 'O's in "Broccoli" by the square root of 20424.2332423 is approximately 285.83.
Yes, the former, trying to satirize cases where people are testing LLMs capabilities by its ability to count characters in a word, do mathematical operations token by token or otherwise. Because LLM is seeing hieroglyphs compared to character by character words that we are seeing. The true test is its ability to solve those problems using tools like somebody is using a calculator. And while it is good to learn and be good at math, it's not because of counting how much change you should receive when buying something. It's to figure out how reasoning works or how to reason in the first place.
Ha! This is a nice easteregg.