Do you have plans to improve the quality of the LLM as judge, in order to achieve better parity with human clinician annotators? For example, fine-tuning models? Thinking that the comparative clinician judgements themselves would make useful fine-tuning material.

yep yep. Its something we have to study and its likely we can improve the LLM as a Judge further.

Same thing for the patient LLM. We can probably fine-tune an LLM to do a better job at simulating patients.

Those two components of our framework have space for improvement