Pulling Back the Curtain: Demystifying the Test Development Process

If you were to survey clinicians to ask them how psychology assessments are developed, some would have an idea of the process, but most would not know how a test goes from a concept to a reality. It's time to demystify the test development process.

Join us as we explore the primary steps along the test development path, the economics behind this process, data collection components, and how clinicians can participate in the test development process to meet the needs of those they serve.

Where Do Test Ideas Come From?

One of the biggest questions regarding assessments is, "Where do tests or ideas for tests actually originate?” On the Testing Psychologist Podcast, Dr. Carrie Champ Morera, licensed psychologist and Director of Content and Production at PAR, Inc., a leading publisher of psychological assessment products, explains that the test development process is multifaceted. She shares with the host, Dr. Jeremy Sharp, "[Test ideas] come from various places. We have authors who will submit proposals directly to us or we also have an online system where anyone can fill out a form and share their test ideas. The most likely scenario is, our team will come up with great ideas to develop internally."

Dr. Morera adds, "We also conduct market research. So, sometimes, we have ideas about a problem that needs to be solved in the field. We'll send out a questionnaire to customers and obtain feedback from them. Our marketing, customer service, and sales teams may also reach out directly to customers for input which could also spur an idea for the development of a test. Additionally, we may find some experts in the field that are really good at what they do and invite them to develop a test for us."

Melissa A. Messer, MHS, the Vice President of Research and Development and Chief Product Officer for PAR, also shares her thoughts on the test development process on the podcast. She reveals, "Early on at PAR, a lot of tests came to us fully developed, and we were   a commercialization partner to assist with getting their product out in the market, for example, managing additional data collection and some fine-tuning. In the company's second half of its existence (PAR recently celebrated its 45^th anniversary), we transitioned to much more active thinking regarding our product line and finding holes where we wanted certain products to fill."

Messer also adds, "And then we spend much of our time doing revisions. So, it's not all just new product development. It is, as I always say, the care and feeding of the really robust IP that we've established over those 45 years takes up a lot of our R&D time."

Behind the Scenes of Test Development

What happens once test development ideas are realized? And what are the actual next steps after the proposals are approved? Dr. Morera explains that a lot of behind-the-scenes work takes place before the test development can begin. She states during the podcast, "There's a contracting process with the author. Once that is approved, we have a development plan that takes place."

This plan, or SOP (standard operating procedure), at PAR usually encompasses over 20 major steps. However, each project is unique in size and scope and may require additional steps. For example, Dr. Morera reveals, "If we're designing a test measuring reading in Spanish, that requires additional considerations during the development process. For example, we need to make sure we have a set of translators and that we're working with other experts in that field who can help us with those unique challenges for that particular measure."

She also notes, "Our project directors work hard to put a development plan together that outlines the steps regarding who is responsible for what. Who will develop the items? Who will write different parts of the manual or collect the data? It is a guide for what we plan to do at the beginning of the project and all throughout the different steps of the project, and it outlines those responsibilities. That really helps set us up for success."

Creating the budget is another essential piece in the development process that needs attention before a project can get approval. Dr. Morera explains, "It can be very expensive to develop tests. I don't think many people in the field realize how costly it can be and all that goes into it. Data collection alone is very expensive." Finally, she adds, "Once the budget is developed and the project is approved, a project team is created. This cross-functional group usually consists of sales, marketing, and IT teams. Internal and external experts are also part of the group as well."

Test Development Phases

Once the test development proposal is approved and a team is established, it's time to create the actual assessment. This process is not easy, however, as Dr. Morera reveals that it takes about 3 to 5 years on average to bring a new test to market. Why so long? Test development goes through rigorous phases. According to Morera, development stages vary depending on the type of test (i.e., performance vs. rating scale) developed and whether it is a new product, enhancement, or revision. The following is a high-level overview of the process:

• Project planning. It requires a team to develop a test. Test authors work closely with project directors who oversee the project and work with the author to construct a development plan, referred to as a Design Build Document (DBD). The DBD details the phases of the test development, timelines, and outlines the agreed upon author and team responsibilities. Project teams often include many internal PAR staff, including the research and development team, quality assurance, IT, and the marketing and creative communications department.
• Item development. The author and project director typically spend at least several months developing and refining items. External experts, bias reviewers, and PAR staff will provide feedback on items to prepare the test for data collection. Prior to data collection, the staff works together on developing any items that are needed for testing, such as stimulus books, protocols, or other manipulatives in print or digital format.
• Data collection and data analysis. These two phases may be repeated depending upon the type of project. For example, after collecting data in the pilot phase, the data will need to be analyzed, and items may need to be revised, eliminated, or added depending upon the results. Depending on the sample sizes used, this part of the process can take anywhere from months to over two years.
• Requirements and programming. These phases are required for digital products. Requirements include creating test mock-ups, defining report structure, and writing logic. The technology team completes programming tasks so that the test is available on our online platform. This process can take many months.
• Development of final product. This is a collaboration between the author and the PAR project team. Test logos, colors, designs, and the official test name are finalized. This includes a manual which provides an introduction to the measure, administration, scoring, and interpreting procedures, test development, validity, and reliability. All test components, print and digital are finalized after going through quality assurance, editorial, author, and beta reviews.

The final phase includes a comprehensive internal review to ensure all programming and editing changes have been made. Then, the product is released into the world. However, development does not stop there. Customer feedback is important to PAR. We are open to feedback from our customers on our products, and we try to incorporate product updates and additional features as needed.

The Economics of Test Development: What They Don't Tell You

What many clinicians may not know is that test development is costly and precarious. As shown above, assessments can be time-consuming and resource-intensive, requiring frequent data collection, research, programming, and beta testing. Melissa Messer reinforces this point during the podcast by revealing, "We have a huge expense to create a product from start to finish, and there's a lot of risks that go into that. We're developing a new product, having no idea what the customer reaction might be."

For example, though many customers may assume that digital tests would be less expensive to produce, this is not always the case. Messer states, "The [digital] expenses are actually much higher. We have a team of developers that have more work than they can ever get to, and our IT expenses have significantly increased through the years I've been here."

She also adds that as technology evolves and the field continues to adopt more digital administration and scoring of assessments, "We went from desktop software to a digital platform that is HIPAA-compliant and secure. There are a lot of security concerns that go with that, requiring 24-hour constant maintenance and constant updates to not only keep it operating but also make sure it's reliable and available."

Data Collection

As mentioned earlier, test development may include several rounds of data collection, which varies depending on whether the information consists of online or print data. Kathryn Stubleski, LMF and Data Collection Coordinator for PAR, gives a first-hand account of this robust process during the podcast. She reveals, "The project director will come to me, catch me up to speed on the project, the idea, and their ideal sample. For example, how many people they need data collected on, types of populations, and types of  clinical groups. And we work back and forth about the feasibility and any challenges we foresee."

Next, Stubleski and the project director revisit the approved data collection budget. She states, "We start discussing payments to examiners or payments per case that examiners may turn in for data collection. The factors usually considered would be the administrative burden to the examiner, how long the test takes to administer, the difficulty of accessing the particular populations we need, and the inclusionary and exclusionary criteria for the participants. Once the data collection plans are finalized, it's on me to start developing and preparing for data collection."

Applying Census Data

The next step in data collection is taking census data. Stubleski explains that the process includes "Taking the census data and turning it into people that we need to gather data on. So, I would call it cells or 'spots.' I usually refer to spots as representing each participant we want to collect. We typically stratify our samples based on age, sex, race, ethnicity, and educational attainment. We take the census data and make an actual graph of each person or spot we want to collect in our sample. Once we begin collection, an examiner or a data collector would look at that list and see these as people they have access to based on these demographics."

Stubleski gives an example, "They may see a 45-year-old white male with a college education and say, 'I have access to someone like that.' I'm going to reserve this case. Reserve is a word I use that means I intend to collect this case." She then adds, "Once we have that sample, I typically reach out to examiners or data collectors that I know either may have access to those populations, or they've collected data for us recently, or have just expressed interest in data collection, and then they go forth and recruit research participants on our behalf."

Meeting the Needs of Those They Serve

In alignment with the Standards for Educational and Psychological Testing, PAR has always emphasized fairness in all aspects of test development across diverse populations and contexts. However, one real challenge is access to these populations. Stubleski shares during the podcast that she tackles this issue by having a large pool of examiners from which to pull. She notes, "I am always recruiting people interested in data collection with us just to have their contact information on file. So it's a matter of reaching out and seeing who has access to these people."

PAR also invites clinicians, researchers, and other professionals to partner with them to advance the scope of solutions they can provide across a wide range of populations, with a focus on underserved populations. 

How Clinicians Can Participate in the Test Development Process

Test development, whether publishing new psychological assessments, revising older tests, or conducting research, is not possible without the valuable contributions of mental health professionals. How can clinicians participate? Dr. Morera gives insight into the process, "We put calls out for experts and bias reviewers for different products that we're working on, especially in early phases of test development of the particular constructs we're studying. For example, if it's a trauma product we're working on, we may seek out trauma experts for their expertise on item development and review. We want people in the field who will be using these products or have used similar products in various settings to try them out before we release them to the world."

She also reveals, "We have PARtalks webinars that have gained much traction. We incorporate topics that include current events in the field, as well as how to use some of PAR’s assessment instruments. And we really love to partner with clinicians and other experts in the field to speak on some topics related to mental health. "

PAR is currently collecting normative, reliability, and validity data for several new tests in development. Data collectors are responsible for obtaining test subjects based on the specific project needs and are compensated on a "per case" basis that varies with each project. To learn more about participating in the test development process or research opportunities, visit the PAR partnerships page.