Mailbag #3: AI in High School, Math Olympiad, Agents, and more
Plus, be among the first to own the world's worst coloring book
Welcome to the summer mailbag! Before I get into your questions, I want to provide an exciting update. Steampunk Animals is now available for sale on Amazon. It’s an objectively terrible book, but you may find ChatGPT’s hyperbolic description amusing enough to be worth clicking through and reading. Now, onto your questions:
What’s your view on the “Death of Google Search”? It’s clear that there has been a clear shift for people like the readers of your newsletter away from Search to AI chatbots for “informational” purposes, but it’s unclear to me just how much “commercial” Search is being impacted.
When I talk with my typical readers, it’s clear that tech savvy people aren’t searching the way they used to. I rarely use Google to find things anymore. Just yesterday my wife tried to google1 something, and I found the answer on ChatGPT much faster. But how pervasive is it?
We published some data on this topic, so here are a few hard numbers from surveys:
80 percent of search users lean on AI summaries for at least 40 percent of their search queries.
42% of LLM users ask for product recommendations
Roughly 60 percent of Google searches now end without the user clicking out to another site. That’s the so‑called “zero‑click” phenomenon, and Google—via the Gemini answer box—is driving the trend.2
In B2B software, we’re seeing click‑through rates down as much as 30 percent year‑on‑year since those summaries launched.3
This is not the end of the line for Google search, but it’s definitely the most significant headwind Google has faced in 20 years. The big question is how many of the ChatGPT queries are related to commerce, and how that changes when we get to the holidays. I’m working hard to get representative data on ChatGPT searches. When I finally get it, I’ll do a post on what I learn.
What GenAI market intelligence tools are most interesting to you? Is the content or the workflow most important to you?
Sorry, my answer is actually “both,” but let me explain why. There are basically four types of tools that I see these days:
Public‑web synthesis. Tools like Deep Research and Perplexity Pro comb through filings, news, podcasts, and everything else in the public domain. It’s great to get a quick view on a company or brainstorm ideas, but the sourcing can be spotty. The tool isn’t able to always identify when something is reliable, and it’s not able to plug into the really high value third party sources that you subscribe to.
Internal‑data copilots. Sana, Athena, Blue Flame, etc. live behind your firewall, accessing your own internal data. They can let you know that another team already looked at that the same company you’re looking at or that they’ve looked at the sector before. These tools can help you get a proprietary view of a company. Note that with ChatGPT connectors, it’s now possible to combine #1 and #2 in ChatGPT, but it requires you to get your data organized.
Single‑source data chat. Many data providers offer tools to let you chat with that specific data source. Tegus is a good example of this. It’s useful if you use that source a lot, but my dream is a tool that can search across multiple sources.
Workflow‑specific apps. I’ve experimented with a few tools that are focused on specific use cases like building a DCF. At this point, I’m still piloting these, so I’m a little hesitant to recommend anything publicly. The ones I’ve tried are ambitious but don’t yet fully deliver a real workflow.
The holy grail is a tool that can combine the data sources in #1-#3 and turn it into something useful such as an early view on a company. Nobody has nailed that end‑to‑end flow although we are building some promising pilots at Bain. For now, the perfect tool doesn’t exist, but I’ve had more success with tools that use interesting data vs. ones that promise valuable workflows. So, I’d start there.
There’s a lot of models even just within ChatGPT: 4o, o3, o3-pro, connectors, agents, deep research. When am I supposed to use these different things?
I agree that this has gotten very confusing, and it gets worse as each new feature emerges. The good news is that Sam Altman has promised that GPT 5 is coming soon and will automatically select the right model for you. So, in a few months, this problem may disappear. But in the meantime, here’s a quick answer:
GPT‑4o (the default option). This is where I go when I’m trying to look something up or generate images. It’s the fastest, so also good if you are in a hurry.
GPT‑o3 (thinking model). If you need to generate text or need an answer that requires some judgement rather than a straightforward lookup, this is a good choice. Because it “thinks,” it’s also better at math problems or code generation. Note that it will take a minute or two to respond, so it’s easy to get a bit distracted while it’s thinking and get off task.
GPT‑o3‑pro (super thinking model only available on certain tiers). This model is the same as o3, but it thinks for quite a bit longer. This is where to go when you need it right not fast. I use this to draft blog posts (although I still edit them a lot), and it was this model that finally cracked the code on getting the resizing right on my coloring book.4
Deep Research (choose o3 or o3-pro as the model). This is pretty similar to o3-Pro except that it will churn out a 3000-word term paper on the topic of your choice and will consult 100+ sources to do it. It’s a nice way to get smart on a business, but the sourcing can be even more spotty than pro, so you definitely need to go through and double check the links.
Connectors. This allows you to connect to your own data such as Outlook and document repositories like SharePoint. I’ve found it to be quite helpful for questions like, “I met with a company a few months ago that did XYZ. What were they called again?” although it can take a couple of tries to get the correct answer. Document finding is harder if your files aren’t well organized and tagged with metadata.5
Agents (the successor to Operator). Agents spin up a temporary virtual machine, and can log into websites, including some SaaS applications. It seems promising, but like Operator, it still gets into trouble a little to often to run autonomously. Also, when I tested it, it was unable to generate a PowerPoint slide with properly formatted bullets
TLDR: use 4o for images and Google-like searches and o3 for everything else.
How should high‑school students use AI?
One thing about high school work that’s different from work in the real world is that the purpose is the learning rather than the outcome. Probably almost none of the work you did in high school was original work. The math problems have been solved thousands of times, the essays have been written many times before, etc. That’s fine because the point isn’t to find new frontiers in human knowledge, it’s to learn how to do math and write an essay.
So, for high school, we probably don’t want students using ChatGPT to solve math problems because then they aren’t learning. But, there are places where bots can help them learn. ChatGPT can be a tutor that hints at how to solve a problem and then can give you new problems to practice.
For writing, it’s important that people understand how to effectively string sentences together, but I think students could use ChatGPT to read an essay they wrote and suggest edits, explaining why those changes improve the writing.
But I think it is important for HS students to get exposure to the technology, and there are places where it seems genuinely helpful. For assignments like presentations or posters, use GenAI to create the images. That’s going to look better than googling for clip art and require the student to put more thought into it.
I’d like to see schools help kids learn how to use GenAI for these productive use cases. I don’t think an outright ban of the most powerful technology in decades will either work or prepare them well for the next stages of life.
Google and OpenAI just earned gold medals at the International Mathematical Olympiad. Experts seem excited about this. Should I be excited about this?
On July 21, Google DeepMind announced that its Gemini Deep Think model solved five of the six IMO problems—good enough for a gold medal. OpenAI disclosed a similar score the same week, though some researchers quibble that this was not an official administration of the exam. As a reminder, this is a competition for very, very smart high school students.
What’s impressive about the result is that the tools Google and OpenAI use appear to have been general purpose bots not a machine that’s engineered just for math. It’s also tempting to think that if the machine can solve some of the hardest solved problems in math, then they might be able to crack an unsolved math problem and advance human knowledge. That, of course, would be a really big deal. In this case, sixty-five teenagers matched or exceeded the models, so we are not at the frontier yet.
It's hard to know exactly how the models solved the problems since the models themselves are new and unreleased. But, articles described the machines thinking for hours to solve the test. That makes me think that this is still the same old thinking model trick that has yielded previous performance gains. As I’ve mentioned before, I believe we need a true algorithmic break through in order to get to performance at or above the best humans. I don’t think this means we have it yet.
Keep the questions coming—your curiosity fuels the newsletter. Until next time, may your zero‑click bounce rate be low and your AI résumé bullet points be honest.
Note: The opinions expressed in this article are my own and do not represent the views of Bain & Company.
I’m capitalizing Google the website but lowercasing google the verb in case anyone is questioning my grammar
https://www.bain.com/insights/goodbye-clicks-hello-ai-zero-click-search-redefines-marketing/
https://www.bain.com/insights/losing-control-how-zero-click-search-affects-b2b-marketers-snap-chart/
Using this model too much for frivolous tasks, makes me feel vaguely guilty about the energy and water consumption. Sort of the same feeling I get when I throw something in the trash that could be recycled when there’s no recycling bin around.
A lot of my projects are private equity deals, which have code names attached to them. So, I might have a folder called Project Eagle, but there’s no way for ChatGPT to know which company Eagle actually is. If I labelled the files more helpfully, I’d be in better shape.
Former English teacher here. I completely agree with you about how high school students should use AI. Will they use it that way, though? In my experience, they'll need a lot of guidance and guardrails and - even then - it's going to be an incredibly uphill battle. The temptation to complete assignments that would normally take hours in mere minutes is simply too great. Combine that temptation with the strong possibility that the cheating will go undetected, and you've got a recipe for disaster.
We already lost the battle against distraction - TikTok, Snapchat, Instagram, etc. - and the kids have suffered for it. It’s sad to see our parents or grandparents lose the battle, but it can be catastrophic when our children do.
The battle against AI-accompanied screentime will be even harder to win, and the stakes will be higher. I don't think the answer is to try to ban it in schools, since students will just use AI on their phones. And, I also agree that students need to be taught how to use it, leverage it, and learn from it.
There's another layer, though, and it's maybe more of a psychological, moral, or philosophical layer and way of thinking that needs to be taught to students, but the adults need to articulate it first. Challenging - especially because we're still making sense of the technology ourselves. In the meantime, at a basic educational level, students will continue to lose ground in reading, writing, mathematics, and critical thinking as they begin to leverage AI in simple ways to make up the ground we let them lost already. Developing prompting skills won't be a proper substitute for their lack of skills. The only saving grace might be that students will realize they don't have the ability to comprehend the content and complexity of the AI's output. If that realization happens in adulthood, though, then it will be too late all over again.