Second in a series profiling NCCCO Test Integrity
January 2011 - To ensure its certification exams remain relevant and address the latest technological and regulatory developments, the National Commission for the Certification of Crane Operators (NCCCO) maintains Exam Management Committees for each of its certification programs. Examination committee members are all industry subject matter experts (SMEs) and must meet specific requirements. All are carefully selected by NCCCO committee chairmen and approved by the NCCCO Board of Directors.
The Exam Management Committees are responsible for maintaining the integrity of their respective exams and ensuring their methods, measures, principles, expectations, and outcomes are consistent. The Exam Management Committees monitor and contribute content for the examination, suggest revisions to testing procedures, and monitor the administration of the exams.
At each exam management committee meeting, questions for the examination are also generated by exam committee members and guests. Psychometric consultants train item writers to ensure that questions are written and selected correctly.
“In many ways, constructing a test is the beginning of a process, rather than the end of one,” notes J.R. Williams, chairman of the Mobile Crane Written Exam Management Committee (WEMC), one of the most active exam management committees that meets at least four times a year. “Many people think that their work is done once the test is launched; nothing could be further from the truth.”
At their regular quarterly and semi-annual meetings, NCCCO Exam Management Committee members review exam performance statistics provided by NCCCO staff which maintain statistics on such testing aspects as: the number of candidates tested per CCO examination; the percentage of candidates passing; the average score of candidates; the average standard of deviation for the exams given; and the average reliability statistics.
NCCCO also prepares a comprehensive series of reports for each meeting that analyze the performance of CCO written and practical examinations for all programs, summarize comments from candidates, provide summary pass/fail information, and statistically analyze exam questions and test reliability measures. Any issues relating to the validity or reliability of CCO examinations emerge during this statistical analysis. This analysis is conducted at least every six months to ensure the performance of all examinations meets expectations, maintain the psychometric soundness and quality of the examinations, and verify the tests’ overall integrity.
Using this process the Exam Management Committees harness the synergy of NCCCO staff’s exam development expertise and the committee members’ industry knowledge to develop and maintain relevant exams that reflect best industry practices and the latest required standards.
Revised 022717