Empathetic Language in LLMs under Prompt Engineering: A Comparative Study in the Legal Field

The demand for empathetic conversations increases with conversational AIs' rise and exponentially spreading applications. In areas like law and healthcare, where professional and empathetic conversations are essential, conversational AIs must strive to retain the correctness of information and logic while improving on empathetic language use. When addressing such an issue, we focus on linguistic empathy, relating only to syntactic and rhetoric choices in language while disregarding the emotional aspect of influence. By performing this study, we are interested in finding whether current open-sourced Large Language Models (LLMs) can match human experts in the legal field by using empathetic language while not compromising facts and logic in responses. We compare responses from three open-sourced LLMs under four prompting strategies with the expert responses. In the comparison, we use metrics from three aspects: text and semantic similarity, factual consistency, and ten rules of linguistic empathy from previous research literature. After statistical tests, the comparison results show that language models can use empathetic language without compromising the default knowledge base of LLMs when properly prompt-engineered. To accomplish this, additional domain knowledge is still needed to match factually. The data supporting this study is publicly available at huggingface.co/datasets/RCODI/empathy-prompt and code is available at github.com/RCODI-ConversationalAI/Empathy-Prompt.