Lessons From Tending A Software Garden

Published:

The artifact chain is the audit trail.

Anthropic, The AI-native SDLC playbook

A couple of weeks ago I wrote Gardens & Factories. The TLDR of that post is that we need to be careful, as an industry, that we don’t blindly commit to the idea of a software factory. The factories of the industrial revoloution were hardly great places to be. I think the idea of being software gardeners is a much better one. Maybe not if you’re the executive of a software company trying to cut costs, but for us doing the work. We should all push to be gardeners, not cogs in an software machine shitting out lines of code at an ever faster rate with you, as the ‘human in the loop’ the one who is responsible.

That’s easy to write. But how easy is it in practice? Because it’s pretty clear, at least in my experience, that AI used correctly does speed up the software development process. Even if you’re just using it to save time tapping keps on your keyboard. I love the theory, and have seen some great articles online about ‘doing it’. But what does it actually mean in practice, what does an AIDLC ACTUALLY look like when applied to a real software project whilst still keeping an element of control, quality and craftsmanship.

At Datadog, I own an internal product called TalkForge. It’s a platform I’ve been building that helps speakers find conferences and then brainstorm, structure, rehearse and ship their talks. The main application is a Next.JS frontend, with 4 backend services that perform different functions. Alongside it sits a .NET service that syncs data with an Airtable base. The whole thing runs on AWS using the full spectrum of serverless services. It emits OpenTelemetry, and I like to think it’s pretty well instrumented.

The application is in production, and has real users, it’s by no means a tier 0 service. Which makes it the perfect place to take some of the ideas of an all out AIDLC for a spin.

There were two resources I used as a starting point: Anthropic’s AI-native SDLC playbook, and some internal guidance Datadog has published about our approach to the AIDLC. Whilst they differ slightly, they do agree on one thing. Agents are only as good as their verification loops. If an agent can’t run your system, can’t tell what a passing test actually proves, and can’t see what its change did in production, then all you’ve built is a very fast way to shit out 1000s of lines a code a minute with little to no oversight. And remember, you’re still responsible for those lines of code.

A quote I heard early on in the current AI craze was that:

AI 10x’s your current software development practices”

That 10x can apply positively. It can also apply negatively. And like SO MANY THINGS in the brave new world we are now venturing into, good fundamentals matter more than ever! This time, it’s the fundamentals of continuous delivery and shipping high quality sofrware quickly.

Start With A Baseline, What State Is Your Garden In?

The temptation with this stuff is to start with tools, brand new tools, a SKILL.md file someone generated. Before doing anything though, step back and first understand where you are. I took both resources as context and got an LLM to investigate where the codebase today. This output was a design document what gave a a score against six preconditions:

  1. reproducible local execution - can the code run locally
  2. high-fidelity mocks - are external dependancies well mocked
  3. isolated test deployments - multiple tests can run independenly without stepping on each other
  4. instance-specific telemetry - you can link telemetry to a specific running instance
  5. agent-accessible context - the agent has access to all of the above
  6. production change guidelines - how do you actually apply this to production

And the biggest takeaway from that assessment - man do LLM’s generate a lot of shit.

I’ve used AI heavily to build the entire application, but with what I thought was pretty tight guidance. Ports and adapters, evoloutionary codebase, good design and architecture. I’m as cynical and careful with AI as anyone, and still:

If I put myself in the shoes of a junior developer coming to this codebase for the first time, I’d have felt sorry for them. But then equally, I’d been handing this context to agents every day for months. And there’s me wondering why sometimes they went off the rails.

This is the weeding I talked about last time, and it turns out it’s not glamorous. Most of this first pass was deleting things, renaming things, and tidying things up. This was also the first major lessson I learned:

Even with tight guidance, LLM’s still have a habit of expanding, duplicating and generally making a mess of a codebase. That’s the weeds sprouting!

Is Your Garden Actually Ready To Plant In?

The first precondition is equal parts boring and absolutely fundamental. Can you run your application locally? A useful thing to do LLM or no LLM. But once you start having multiple agents running simultaneously, can you run more than one copy of the app at once? If you want two agents working in two git worktrees at the same time, each needs its own isolated stack. Of course, this would be the same with or without an agent. When it comes to fast development cycles, being able to quickly run the app or different sections of the app locally is fundamental.

I thought I had a working local stack… I did not.

After a bit a lot of hashing aroudn, this got cleaned up. The repo now has the ability to start up to 10 copies of the application simultaneously. When a new worktree gets setup, it gets assigned a slot (0-9). The slot fixes the compose project name and every host port. Slots above 0 are served on sN.localhost so two stacks can run simultaneously, and also be accessed at the same time in a browser.

The repo includes a command that builds the stack, applies database migrations, waits for everything to be healthy, and then prints the URLs to access the application. This is a single command that an LLM can run as part of it’s local verification step, and then grab the URL and use a tool like Playwright to actually go an interact with the app.

The command also includes a couple of switches to control which services get started up, and also to switch between Jaeger (completely local) or Datadog (send it do the Datadog backend). The decision early on to emit OTEL enables this easy switch between fully locally dev cycles and something that gets sent to an external telemetry backend. Telemetry is emitted with a unique environment name, and the current commit hash. That way, the same trace queries an agent would use to check dev or prod telemetry now work locally.

That gives the LLM (and any human for that matter) a really quick development cycle:

  1. Run make local-up to start up a local running instance of the application
  2. Grab the URL from the output
  3. Navigate through the application to test if your new feature works
  4. Go and query the application telemetry to check nothing broke, and that the flow of a request worked as you expected it to

As part of this, I found and fixed a real bug - Whilst trying to verify the end to end flow of a chat message in prod, the actual chat message processing that happened async wasn’t linked to the API request that queued it. The trace context travelled in the message body, but the consumer only looked at message attributes.

Emitting high quality telemetry, and giving the agent the ability to query it, is non-negotiable. Without it, fast feedback loops can’t happen.

Put Up Fences, Don’t Hire Foremen

Humans are really bad at doing the same repetitive task over and over again for hours. That’s the core thesis of Cory Doctorow’s reverse centaur. If your only safety mechanism is “a human reads every diff carefully”, then I have bad news for you. Eventually, you are your employees/colleagues are going to descend into just hit approve. PR’s are coming so fast, change is constant, eventually you’ll hand all control over to the machine and start rubber-stamping. At this point, you are taking all the responsibility with very little understanding of whats actually happening.

So the next step was to stop relying on my attention for the things a machine can check deterministically. Which, shock, is what unit/integration/end to end tests have been trying to do for years. If you’ve used coding agents for a while you’ll have seen some varition of:

  1. the test is failing,
  2. the agent can’t proceed until the test passes
  3. the agent ‘fixes’ the test
  4. It’s fix was to simply delete the test case

To keep pressing the garden analogy, these are all fences. A fence doesn’t tell you how to garden, it doesn’t control what you plant. However it does stop the dog digging up the vegetables, and frees you up to pay attention to the things that need judgement.

The AI DLC, at it’s core, is all the good things Jez Humble and Dave Farley taught us in 2010 with the Continuous Delivery. Or everything described in the Pheonix Project/Accelerate. If you haven’t, go pickup a copy of those three books. Your AI DLC will thank you later.

Teach And Verify, Don’t Rely on Vibes

I thought I’d documented the repo well, and inside Git it included a full set of playbooks: how to add a migration, add a route, add an event, add a deployment workflow. I even wrote up an operator guide in Confluence. They were well written, but only used if an agent happened to stumble across them. And it turns out non-deterministic agents sometimes have a habit of just completely ignoring things. So I took each of these playbooks and turned them into agent skills. These built on top of common skills you will see out in the wild, a reviewer, a ‘start a new piece of work’ which launches a branch and a worktree. But layered some of my project specific concerns on top of that.

Give the agent the skills to perform key tasks

A healthy deploy isn’t a working change

The biggest piece of work, and the one I want to spend the most time on is production acceptance.

TalkForge already had some of this built: each service defines an internal only ‘verify’ endpoint. An in-VPC runner that gates promotion to production calls each service’s verify endpoints, asserting the deployed SHA and environment. The problem with this is that those checks are into a single environment (dev) and static. A PR can’t say “after this ships, I expect to see this new span for this version in prod, and the error rate to stay under X for 30 minutes”. This is the kind of power emitting high quality telemetry can give you.

Quite a few years ago now I worked at an organisation that had a form of this as part of their PR template. To merge a PR, you had to answer the question “how will you know when this piece of code is in production?”. More often than not, that question was answered with a trace query in the telemetry backend.

The intent

Every change should be able to state what “working” looks like in production, and the pipeline should go and check. Not “the deploy succeeded” Not “the health endpoint answered”. Both of which are still important of course. But what you really want to verify is this change, at this version, did the thing it was supposed to do, under real traffic, for long enough to mean something.

The Anthropic AI DLC defines a process of intent → design → plan → code → evidence. The first four steps are pretty straight forward. But that last part… Evidence! At least from what I’ve seen the did it work? step, was either best effort by a human or simply didn’t run at all. And as I find myself saying over and over again at the moment, this is how it’s always been. AI will 10x your current processes remember.

High performing software organisations automate all of this. One of the fundamentals of continuous delivery is to automate as much as possible. History has proven though, that the average organisation, does not. And instead has some form of human in the loop to watch code get released.

The other benefit of a strong evidence based evaluation step, is it starts to change what a PR is. A PR no longer just says “here’s some code” for you to understand and review. One of the problems with this is that you as the reviewer need to try and understand intent. And when you’re getting slopped 10 PR’s an hour this is hard. A PR now also says “here’s how you’ll know it works in production, and I’ll even go ahead and check this for you against an actual deployed environment”. That matters even more when an agent wrote the code, because now the agent has to say up front what success looks like, which you can review at the design phase, and then your CI pipelines are going to verify that. Once deployment has started, you can then even ask your agent to watch it through, running a verify -> fix -> push loop until all CI checks pass.

How it works

Alongside the intent, design and plan for a ticket, a change can now carry an acceptance spec, one small YAML file per service it touches:

schema_version: 1
acceptance_id: adv-123-slides-retry
revision: 1
lifecycle: active
ticket: ADV-123
service: slides
window_seconds: 1800 # watch prod for 30 minutes
poll_interval_seconds: 300
ingestion_delay_seconds: 180 # give telemetry time to land
checks:
  - id: generate-capability
    kind: capability
    capability: slides.generate
  - id: verify-span-observed
    kind: span_presence
    telemetry_service: talkforge-slides-api
    resource_name: slides::verify_slides_generate
    min_count: 1

This YAML file is basically saying for this change to be accepted, the slides.generate capability must pass, and I expect to see a span for service talkforge-slides-api with the name slides::verify_slides_generate, for this version, at least once within 30 minutes.

From there, the flow is:

  1. On the PR, the spec is validated with no credentials and no production access.
  2. On deploy, the pipeline records a deployment receipt: what was deployed, where, when, and whether the existing immediate verification passed. Receipts live in a versioned S3 bucket with conditional writes, so a slow workflow can never overwrite a newer one.
  3. After deploy, a separate job runs a small Rust CLI, talkforge-acceptance, against the deployed version. It runs outside the deploy lock, so a 30-minute observation window never holds up a hotfix. The window is fixed as absolute start and end times.
  4. During the window, it polls. Every trace query is forced to the right service, env:prod and version:<sha>. A spec can’t widen that, so a check can’t accidentally pass on someone else’s traffic. Before each poll it also checks whether a newer deployment has landed. If one has, the run ends as superseded instead of carrying on against code that’s no longer running.
  5. At the end, the result is posted as a GitHub check on the commit and as a Datadog event tagged with the service and SHA. A Datadog event monitor alerts on failed and incomplete results.

There are four potential outcomes:

A few rules sit underneath that, and every one exists because the easy version would lie:

The variation in these rules are important. If I (or an agent I’m guiding) raises a PR and the PR comes back passed. I can give the PR a quick pass, check the intent document, and be reasonably happy everything is ok to merge. Remember, this is also layered on top of more traditional forms of CICD, unit tests, integration tests etc. If the PR comes back inconclusive, that tells me (the human) that I need to have a closer look. Did the code path just not trigger? Or is there something more fundamental going on.

As of right now, rollback doesn’t happen automatically. For a couple of reasons:

  1. I’m yet to figure out what the ‘right’ rollback is and
  2. Depending on the change, this might need some human oversight. Has a breaking database change been applied, which means we might need to roll forwards etc

Where it actually stands

There is still plenty of work I have to do here, not least with Per-PR test-drive deployments. Every PR still deploys into one shared dev environment. This either

a) causes confusion because one pipeline deploys, and another overwrites dev before the checks complete or b) slows down deployments because everything ends up waiting.

This needs updating to have isolated and ephermal test environments.

I’ll also acknowledge, this has been a safe space to experiment. Whilst the TalkForge application has real users, it’s by no means a tier 0 service used by thousands of people every day. This has given me a safe way to try out some of this stuff, whilst not worrying about trashing production for 1000s of users.

I’m looking forward to scaling this up and seeing how it develops though.

What I learned

The agents found more problems than they caused. I’ve been working on this application for ~12 months. A lot of it with what I thought was pretty tight guidance. But that guidance was best effort. Please LLM, could you ensure you use ports and adapters. Yet there was still so. much. slop. All of it pre-existing, all of it mine. This is the most important part of tending the garden. If you’re moving yourself away from tapping out lines of code, ensure you have things in place to stop the shit growing uncontrolably.

The leverage is in the boring parts. Almost all of it is just good software development practices. The kind of stuff that’s been valuable for 20 years. Understand the fundamentals of how to ship high quality software quickly, and then apply an LLM to it. Much of what I’ve seen of ‘new shiny things that you need to do with LLMs’ is basically just good practice. Whether that’s in software architecture, development or shipping. The fundamentals are more important than ever.

The factory version of this week would have measured success in PRs merged. And I do have a rough idea of how much ‘value’ I shipped, if you count value by lines of code, PR’s or features. I didn’t want to include that, because we all know that this isn’t a good metric.

The takeaways? Fundamentals matter more than ever. And ensure you have actual, deterministic guardrails in place that you can use alongside your LLM. Don’t become a reverse centaur, blindly approving LLM generated code whilst also taking responsibility for it. Instead, find the places where LLM’s can make you a ‘better’ developer. Whatever better means in your context.

James