Many assessment organizations are experiencing an increased need for items. However, obtaining the necessary number of acceptable items needed for a rigorous high-quality assessment can take a long time and become a costly effort. More recently, using large language models (LLMs) as part of the item writing process is being explored. The potential for generative AI approaches to assist with item writing is very exciting but also introduces practical challenges for organizations.

We plan to discuss two particular approaches that are being investigated at our organization. We are referring to one approach as “variant development”, where existing items are leveraged. A second approach is the use of a tool that leverages generative AI to assist with item writing. It is an extension of variant development because existing items are not needed as a starting point. Rather, item content is created from scratch; generative AI is then leveraged to assist with writing distractors, and creating a set of similar items with different keys. A prototype of this in-house item development tool has been developed. We will discuss our current thoughts, lessons learned so far, and future directions. We would like to invite others to share their own experiences at their organizations.