On this page 11 sections
Find out what it can do before you find out how well it does it. Week one is an inventory: what it touches, what it costs, what it is allowed to change, and whether anyone could tell if you made it worse. Fix nothing until that is written down, because the first change to an unmeasured system destroys the only baseline you will ever have.
An inherited system is not a codebase. It is a codebase plus a set of decisions nobody wrote down, and the decisions are the part that will hurt you.
Key takeaways
- Establish the blast radius on day one. What can this thing do that cannot be undone?
- Do not improve anything in week one. Measure first, or you lose the comparison permanently.
- The absence of an artefact tells you as much as its contents. A missing decision log means every prompt line is load-bearing until proven otherwise.
- Credentials in a departed person’s name are the most common thing that stops a rescue on day three.
- Decide rescue or rebuild deliberately, with a test, rather than by drift.
Who this applies to
The vendor relationship ended, the person who built it left, or it arrived with an acquisition. It is in production, people depend on it, and nobody currently in the building can explain why it does what it does.
If you are still negotiating and want to know what should be handed over, that is the other end of the same problem: what you own after an AI project lists the eleven artefacts and the contract language that gets them.
Day one: the blast radius, not the quality
The urgent question is not whether the answers are good. It is what the system is permitted to do without asking anyone.
Three things to establish before anything else:
What can it change? Does it write to your systems, send email, issue refunds, update records, post to customers? Read-only systems can wait a week. Anything that acts needs its permissions known today, and the reasoning for how those should be scoped is in tool-calling agents: what to let them touch.
What does it spend? Find the provider account, the billing owner and the current run rate. Systems inherited without documentation are frequently also inherited without a spend cap.
How would you stop it? If it started behaving badly at 2am, what is the off switch, who has access to it, and does anyone know? If the answer is a deploy pipeline nobody has run, that is the first thing to fix, and it is the only fix allowed in week one.
The inventory, and what an absence means
Spend the next two days finding these. Write down what you find and what you cannot find, because both are findings.
| Look for | If it is missing |
|---|---|
| An evaluation set | Nobody knows whether it works. This is the most consequential absence, and it is the most common |
| The prompts, with history | You cannot tell an accident from a decision, so every line is load-bearing until tested |
| A decision log | Constraints exist that nobody can explain, and some are protecting you from an incident you have not heard about |
| The model and version in use | A floating alias means the behaviour can change without anyone deploying |
| Retrieval sources and their freshness | Answers may be coming from content nobody has updated in a year |
| Integration credentials and their owner | The rescue stops here more often than anywhere else |
| Logs, and how long they are kept | Short retention means the history you need to diagnose is already gone |
| Anything customer-facing that was promised | Disclosure copy, terms, and a public claim about accuracy someone made |
The second column is the point of the exercise. A missing decision log does not mean there were no decisions - it means they are now encoded in a prompt as unexplained sentences, and removing one is a gamble until you can measure.
Do not fix anything yet
This is the instruction that gets resisted, and it is the one that matters most.
An inherited system has no before-number. The moment you edit the prompt, you have permanently lost the ability to say whether the thing improved, and you have joined the people who could only offer impressions. Measure first, even though the measurement is late and the result will be uncomfortable.
The retrofit is a known piece of work: build a set from real cases, score the system as it stands today, and establish run-to-run variance so later changes can be judged. The full sequence is in we shipped without an eval set, and it applies unchanged to a system you did not build.
Two things make the inherited version harder. You do not know which cases the original team tuned against, so anything you find in their test data is compromised as a held-out set. And the historical resolutions in your ticket system may themselves reflect the system’s own answers, so a label pulled from the last six months can be the system marking its own work.
The credential problem
This stops more rescues than any technical issue.
The provider account is in a departed employee’s name. The API key is in a repository nobody has admin on. The integration authenticates as a person who left, so it will fail at the next token refresh with no warning. The billing card belongs to an agency you are no longer paying.
Audit this in week one even though nothing is broken yet, because the failure mode is that everything works until a refresh cycle and then stops at once. Move every credential to an account the organisation controls, with a named owner, before touching anything else. It is unglamorous and it is the difference between a rescue and an outage.
What we usually find
Three things, in almost every audit of this kind.
No evaluation set, so the first honest conversation is that quality is unknown rather than bad. That is a different message from the one the client expects, and it is usually better news than they feared.
A floating model alias in production, meaning the system’s behaviour is subject to a provider’s release schedule and nothing in the repository records which version served last month’s answers. The pattern that follows from this is described in your provider changed the model.
A prompt with no history, several hundred lines, containing instructions in tension with each other, and nobody willing to remove any of them because nobody knows which are holding something up.
Where this costs us work: the first deliverable we propose is a measurement and a written inventory, not a fix. Clients arriving after a failed project want visible progress, and a report reads as delay. We have lost engagements over it. The argument we make is that a fix applied before a baseline cannot be defended to the person who signs off the next invoice, and the second failed rescue is much harder to fund than the first.
Rescue or rebuild
Decide this deliberately in week two, on evidence, rather than letting it happen by attrition.
Rescue is right when the integrations are substantial and working, when the content and data are sound, and when the problems are concentrated in the prompt, the thresholds or the retrieval configuration. Integration work is the expensive part of these systems, and it is the part that survives.
Rebuild is right when the system is mostly a prompt and a thin wrapper, when the data model is wrong in a way that touches everything, or when nobody can explain a single behaviour and the artefacts are entirely absent. If you would have to re-derive every decision anyway, re-deriving them in code you understand is cheaper than archaeology.
The test we use: could a competent engineer, given the inventory, explain why the system behaves as it does in five specific cases? If yes, rescue. If no, you are not maintaining a system, you are maintaining a mystery, and the cost of that never comes down.
When this is not worth doing
When the system is being retired anyway. If it has six months left, keep it running, cap the spend, and spend the effort on what replaces it.
When it is genuinely low stakes. An internal helper used by four people does not need a formal inventory. Check what it can touch, then leave it.
When the original team is still reachable. Two hours with the person who built it is worth more than two weeks of inference. Pay for that conversation if you have to; it is the cheapest week of the project.
Frequently asked questions
How long does a proper handover audit take?
For a single system with a handful of integrations, about a week to inventory and a further week to establish a baseline. Larger estates take longer mainly because of credentials and access rather than analysis.
Can we not just read the code?
The code tells you what happens, not why. The valuable information in these systems is in the prompt, the thresholds and the retrieval content, and none of that explains its own reasoning. Reading the code is necessary and it is not sufficient.
What if there is no documentation at all?
That is the normal case, and it is what the inventory is for. You are reconstructing the documentation, and the first version of it is a list of what you could not find.
Should we tell the client or the users that we inherited it?
Internally, yes, explicitly, because expectations for the next few weeks depend on it. Externally, nothing needs to change unless a public claim was made that you cannot currently support, in which case fix the claim first.
Is it safe to keep it running while we audit?
Usually, provided you have established the blast radius and the spend cap on day one. If it can take irreversible actions and you cannot yet explain the rules, gate those actions behind a person until you can.
Next step
If you have inherited a system nobody can explain, the first deliverable is an inventory and a number, not a fix. AI system rescue starts there, and it sometimes ends with a recommendation to rebuild, which is a cheaper thing to learn in week two than in month six.
Related: What you own after an AI project · We shipped without an eval set · Tool-calling agents: what to let them touch · Your provider changed the model · AI system rescue