Rita Kohli sits as a Judicial Member on the National Company Law Tribunal (NCLT) in Mumbai. Jammu & Kashmir Bank had lent ₹200 crore to a company called Pan India Utilities. A separate company – Essel Infraprojects – had guaranteed that loan, if the borrower couldn’t pay. That put Essel on the hook.
The borrower defaulted on ₹87.43 crore.
The bank came after Essel to collect, asking Kohli’s tribunal to open insolvency proceedings to recover the money. Kohli’s bench sided with the bank – Essel owed the debt. An appeals tribunal agreed the following year.
Then, on July 2, 2026, the Supreme Court set the whole thing aside.
Not because anyone disputed the debt. Because the old court cases the tribunal had leaned on to make its decision didn’t exist. They had been generated by AI, without anyone checking if they were real.
This isn’t a story about legal AI specifically. It’s what happens anywhere a machine hands someone an answer that looks finished enough to skip checking.
Let’s be clear – nobody involved in this case set out to file fake law. The old court case citations read like the real thing because that’s what the AI tool was built to produce. Somewhere between drafting and filing, a plausible-looking citation was trusted without verification.
The Supreme Court did not wave this off as a technical slip. The judges hearing the appeal – Justices P.S. Narasimha and Alok Aradhe – said it went to the heart of whether the courts could be trusted at all, and ordered India’s Bar Council to write new rules for AI use in courts. Their finding was blunt:
A lawyer who cites an AI-generated case without checking it isn’t making an honest mistake. That’s misconduct.
The part everyone will skip past
You will hear the same response every time this happens:
Better AI will catch the fake citations before they cause harm.
No it won’t. Not because the tools won’t improve … they will. The problem is that the tool checking for fakes and the tool making them up are stuck in the same race. By the time a checker gets good enough to catch today’s fakes, the tool generating them has already moved on to fakes that look even more convincing.
AI checking AI will compound the problem.
The fix was never going to be a smarter research tool. It was always going to be a person, named in advance, whose job includes actually reading and validating the citation before it’s filed. Not skimming it. Not trusting that it looks right.
A courtroom has always assumed a human read the law being cited. That assumption held for centuries because a human always had to physically find the case first. AI removed the friction that used to guarantee the check happened.
But it didn’t remove the need for the check.
The court that had the same failure and kept its order anyway
Five months earlier, a different Indian court ran into the identical problem.
A trial judge in Vijayawada was hearing a property dispute. In her ruling, she cited four court cases to back up her decision. All four were made up. When the Andhra Pradesh High Court asked her to explain, she admitted she’d used AI to find the cases, and never checked if they were real.
Same failure. Same tool, doing the same thing.
Justice Ravi Nath Tilhari heard the challenge and asked a simple question: take the fake cases out, does the judge’s actual reasoning still hold up? It did. She had genuinely worked through the legal question herself. The fake citations weren’t holding the decision up. They were just extra decoration sitting on top of one.
That’s why the ruling survived. The mistake was real, but it hadn’t done any of the actual work.
The High Court was blunt about the lesson: AI can help organise and summarise, it said, but it can’t do the judge’s actual reasoning for her. That has to stay a human job, not something AI does while a human nods along.
Kohli’s tribunal didn’t have that same backup. Nothing suggests her bench worked through the legal question on its own and only slipped up on the citation. Here, the fake cases weren’t decoration, they were what the tribunal pointed to as its reason for deciding the way it did. Take them away, and there’s nothing left holding the ruling up.
Same disease, different immune response
Put these two cases side by side and one question stands out.
Fabricated citations entering a judicial order is not, by itself, what determines the outcome.
What determines it is whether a human’s own independent judgment was ever actually exercised underneath the citation, or whether the citation was standing in for judgment that never happened.
In Vijayawada, the judge’s reasoning existed with or without the fake cases. In Mumbai, remove the fabricated precedent and there’s a hole where the reasoning was supposed to be.
In both cases, the fake citations got past the court before anyone caught them. What decided the outcome wasn’t whether the mistake got caught. It was whether a human had actually done the reasoning underneath it.
No fact-checking tool would have caught this difference. Run both rulings through a citation-checker, and they’d both fail the same way; fake cases, flagged … done. Only one of them fails the question that actually matters:
Did a person do their job, or not.
Who reads it before it’s filed
India’s Supreme Court now has the Bar Council of India drafting rules for AI use in courts. That is the all important rule-writing layer.
That is not the same as the real question underneath it: is any actual lawyer, on any actual filing, reading every citation the AI hands them? Or has “the AI found it” quietly started to feel like “someone checked it” … when nobody has?
Two courts ran into the same failure in the same year. One had a human doing real work underneath the citation, and the fabrication cost nothing. One didn’t, and it cost a signed order, an appellate ruling, and two years of litigation, all undone in a single afternoon in Delhi.
The technology was identical in both rooms. This isn’t really a story about legal AI, or about courts. It’s what happens anywhere an AI hands someone an answer that looks finished, whether it be a citation, a score, a recommendation, and it becomes easier to trust it than to check it.
The question worth asking isn’t whether the AI gets better at catching its own mistakes. It’s whether anyone is still required to do the checking themselves, or whether that requirement quietly disappeared the moment it got easy to skip.
References
- Bar and Bench – “Supreme Court sets aside NCLT judgment for relying on hallucinated AI-generated citations”: https://www.barandbench.com/news/litigation/supreme-court-sets-aside-nclt-judgment-for-relying-on-hallucinated-ai-generated-citations
- LiveLaw – “AI-Generated Citations Cannot Replace Judicial Mind, But Order Will Stand If Law Applied Is Correct: Andhra Pradesh High Court”: https://www.livelaw.in/high-court/andhra-pradesh-high-court/andhra-pradesh-high-court-ruling-mere-mention-wrong-citation-cannot-vitiate-order-if-law-correctly-applied-cautionary-note-unregulated-ai-tools-use-520160
Has your organisation sorted which decisions are measurement and which are judgment — or built oversight on top of a map nobody drew? If you’re not sure, that’s usually the answer.
I write Decision Layer Weekly for people working on exactly this gap: Subscribe here. And if it’s live for you right now, reach me directly: mattsheehan@spatialnext.io
Matt Sheehan
Matt is an AI strategist with 25 years at the intersection of geospatial intelligence and decision-making. He maps the architecture connecting three layers most organisations haven’t yet seen together: the sensing layer the geospatial industry has built, the causal reasoning layer now arriving, and the human decision layer nobody is designing. The third layer is where most deployments fail. And where Matt has his primary focus.