Not long ago, we resigned ourselves to the idea that humans would need to examine every line of code that AI generates. We used to do it personally, code reviews will always be part of serious programming practice, and the ability to read and review code will become a more important part of a developer’s skill set. At the same time, I think we all know that this is untenable, and that AI will quickly generate far more code than humans can reasonably review. Understanding someone else’s code is harder than understanding your own, and machine-generated code is even harder to understand. At some point — and this point comes fairly early — all the time you saved by letting an AI write your code will be spent reviewing it. It’s a lesson we’ve learned before; It has been decades since anyone, except a few specialists, has needed to examine compiler-generated assembly code. As did Kellan Elliott McRae writtenit’s not clear that the code review ever justified the cost. While sitting around a table examining lines of code may uncover issues with style or poorly implemented algorithms, code review remains an expensive solution to relatively simple problems.
With this in mind, specification-driven development (SDD) shifts the focus from review to verification, from claim to specification, and from testing to more testing. The goal of software development is not code that passes human review; They are systems whose behavior lives up to well-defined specifications that describe what the customer wants. Knowing what a customer needs and designing an architecture to meet those needs requires human intelligence. Like Ankit Jain points in Latent spacewe need to move from wondering whether the code is written correctly to wondering whether we are solving the problem correctly. Understanding the problem we need to solve is part of the specification process, and is something our industry has historically been He didn’t do well.
Verifying that the system actually works as intended is another important part of the software development process. Does it solve the problem as described in the specifications? Does it meet the requirements of what Neil Ford calls “Architectural characteristicsOr “-ilities”: scalability, auditability, performance, and many other properties that are embodied in software systems but can rarely be inferred from looking at code, and which AI systems cannot yet reason about? These properties must be captured in the specification. The focus of the software development process moves from writing code to defining what the code should do and verifying that it actually does what it is supposed to do. It moves from the middle of the process to the beginning and end. AI can play a role along the way, but specification and verification are the places where human judgment is most Importance.
Want Radar delivered straight to your inbox? Join us on Substack. Register here.
Drew Brewing and others point out that this is inherent It is a circular process, not a linear one. A specification is not something you write down at the beginning of the process and never touch again. It should be updated whenever the desired system behavior changes: the more a bug fix leads to new testing, the more users make clear what they want, and the more deeply developers understand the system’s goals. I’m impressed by how flexible this process is. It is not stand-up sprint agility but incremental development agile. Specifications lead to planning, which leads to implementation, which leads to verification. If the verification fails, we update the specification and repeat it. Drew built Plumb, a command-line tool that can be plugged into Git, to support an automated loop through specification and testing. What sets Plumb apart is its ability to help software developers look at the decisions that led to the current version of the software: the differences, of course, but also conversations with AI, specifications, plans, and tests. As Drew says, Plumb is meant to be an inspiration or starting point, so it’s obviously missing important features, but it’s actually useful.
Can SDD replace code review? probably; Again, code review is an expensive way to do something that may not be useful in the long run. But maybe this is the wrong question. If you don’t listen carefully, SDD sounds like a reinvention of the waterfall process: a linear drive from writing detailed specifications to burning thousands of CDs stored in a warehouse. We need to listen to the SDD itself to ask the right questions: How do we know that the software system solves the right problem? What types of tests can verify that the system solves the right problem? When is automated testing inappropriate, and when do we need human engineers to judge the suitability of a system? How can we express all this knowledge in the specifications that drive the language model to produce working programs?
We do not give as much importance to specifications as we did in the last century; We tend to see writing specifications as an outdated ceremony at the beginning of a project. This is unfortunate, because we have lost a lot of institutional knowledge about how to write good, detailed specifications. The key to making specifications relevant again is to realize that they are the beginning of a circular process that continues through verification. The specification is the repository of the project’s true goals: what it is supposed to do and why—and these goals necessarily change during the course of the project. A software-driven development loop that runs through testing — not just unit testing but fitness testing, acceptance testing, and human judgment of the results — lays the foundation for a new kind of process in which humans won’t be overwhelmed by reviewing AI-generated code.
