I have been trying to build a document review software, DocCheck, using LLMs. This is to streamline my work tools and make them cross platform. I am a licensed architect and professional reviewer - not a programmer. So LLMs are on the technical task. I've encountered all the classic problems of LLM-led development that make software developers less nervous about their future employment. But this software would not exist without LLMs - the market is too small (so far it's 1, though I do want to make it a product eventually).
The most successful bit of LLM wrangling I have done is the adversarial LLM duel. This technique extends the line of thinking in Du et al.'s paper titled Improving Factuality and Reasoning in Language Models through Multiagent Debate.
One LLM, in my use it's Claude Opus, drafts a technical solution to a problem that has been sliced and handed to it. Then another, in my usage Codex, drafts a solution independently. They each hand their solutions to each other and comment on the efficacy of the other LLM's solution. Then both solutions and critiques - four total documents - go to a new instance that synthesizes a response.
This neutral referee is also a classifier that finds "values" calls and surfaces them for me to decide in human-in-the-loop interactions, while the technical decisions are kept within the purview of a future LLM implementer. The referee idea is found in Irving et al.'s AI safety via debate.
I have found this system does not need multiple iterations to be effective. Solutions converge quickly and the use of two LLMs greatly increases the number of mistakes caught in review. In addition, having the referee able to push items back to the queue or raise to the human offers escape hatches for instances when the models do not converge.
https://arxiv.org/abs/2305.14325
https://arxiv.org/abs/1805.00899