 |
| AI-generated deliberately cheesy image in the style of a bad stock photo of a frustrated professional working at a laptop |
One of the examples Cal Newport provides in No One Knows Anything About AI has some food for thought:
The AI evaluation company METR recently released the results of a randomized control trial in which a group of experienced open-source software developers were sorted into two groups, one of which would use AI coding tools to complete a collection of tasks, and one of which would not. As the report summarizes: “Surprisingly, we find that when developers use AI tools, they take 19% longer than without—AI makes them slower.”
Even worse, the AI cohort perceived it made them faster.
As an LLM fan and natural skeptic, my gut reaction was doubt, ready to poke holes in the study. But the FAQs addressed my main concerns — other than the fact that it was small (n=19).
It’s worth asking: have I wasted time using AI before? Yeah, probably. Let's think about how this study may pertain to our use of AI: for what tasks and how we engage it.
Is it something you can do yourself?
This study tested one specific scenario — coding, for software developers — and not examples when AI helps me write code I couldn’t create at all without it. For these, using generative AI is the difference between 0 and 1, not 1 and slightly above or below 1.
Will you go to the effort of providing all of the necessary context?
Providing adequate context is a no-brainer for “admin” tasks like synthesizing notes or creating briefs, but solving complex problems efficiently and effectively requires more LLM inputs than Googling.
Lately, I’ve been trying to avoid the “magic” approach — Given A, give me Z — and instead treat AI like a person I’m delegating to.
-
The Magic approach is tempting because of the low cost to give it a shot and see what happens. Then, seeing the code automagically generated in the first response creates the illusion of progress and makes it easy to think, “Maybe it just needs a tweak, and we'll be done.” The last time I did this, I spent 10+ iterations only to end up with something overfitted to my example data, going so far as to hardcode some of the desired results. Well-played, AI.
-
The Measured approach takes longer up front to generate a considerate prompt (less so with voice-to-text), but narrows possible failure points and reduces total iteration time. I explain foreseeable challenges, break the work into steps, and ask for intermediate outputs to test. This way, I can check progress to make sure we're on the right track instead of troubleshooting a giant, all-at-once solution where any step may be the cause of unexpected results.
This study was a good reminder to stay self-aware about AI’s fit for a task for you — and how you use it, although not studied. If you’re already exceptional at something and especially if you're not willing to put the time into providing a detailed prompt, maybe just do it yourself.