AI outperforms law professors in blind study

I would fail my exams if I was blindfolded

This study doesn’t really improve my impression of LLMs, but it does really hurt my impression of the value of a law degree:

Participants created 40 representative contract law questions that students might ask after class or during office hours, wrote their own answers, and then evaluated responses without knowing whether they came from AI or other participating professors. The AI systems performed comparably to the best human instructor in the study.

Perhaps most striking: professors flagged AI responses as pedagogically harmful only 3.5% of the time, compared to 12% for peer-written answers.

“In most fields where AI gets tested, there’s a right answer. In law, there often isn’t,” said Sarath Sanga, co-author and professor at Yale Law School. “Two opposing arguments can both be good. What we wanted to know is whether AI can meet the latent professional standard that lawyers use to evaluate each other’s arguments. In this case, the answer was yes.”

The article says “…had professors assess whether responses might mislead or confuse students.”

professors flagged AI responses as pedagogically harmful only 3.5% of the time, compared to 12% for peer-written answers.

Which makes me think that real people came to some conclusion, sometimes biased or wrong, but AI could have produced inconclusive inflated perhaps-maybe-sometimes text (which it would be good at) 96.5% of the time. Response not being harmful doesn’t mean it’s good.