I was enjoying a conversation with a friend a few winters ago discussing a mutual passion of skiing. He had earned the highest designation of ski instructor in Canada known as the CSIA IV. It was an accomplishment he held near and dear to his heart even though it was achieved decades ago. He kept in touch with the organization and had mentioned they were struggling with criticism related to current pass rates for this level. Only 3% of test takers were passing the Level IV exam in recent years. The offering led to a 97% failure rate.
Most of those that aspire to achieve this designation are making skiing their vocation. It’s not a hobby to them. It’s their livelihood. The designation is like becoming a golf professional or earning a teaching degree. Years of effort and many thousands of dollars in courses have been spent to get there. No, the pass rate shouldn’t be 100%. A clear, objective standard of excellence should be sought. Those ending up on the right side of the standard should be proud of their accomplishment.
Many chasing a certification do so because of the credibility assumed to be associated with it. They are willing to work for it. They want the standard to be superior evidence of excellence. They are okay with the test being tough. They don’t want a walk in the park. But they do want a fighting chance. They want to know the standards, be able to prepare for them, and have a reasonable chance at success when signing up for the program.
However, a hurdle that only allows 3% of participants to proceed seems too high of a hurdle. Where registrants see the low pass rate the slope to success seems both too steep and slippery. The risk isn’t worth the reward. To sign up for these types of programs is like signing up to be a Kamikaze pilot. Things aren’t going to end well.
Besides serving as a deterrent to participants, what else may a low pass rate say? Does it suggest the program is no good? If the course preparation only prepares three out of a hundred to succeed, the preparation can’t be viewed as productive. Doesn’t it suggest the teaching is poor? Or does it suggest that we don’t want you in our club? Or that those of us already in the club having earned the credential are actively trying to keep others out? Does it suggest that we don’t want you to make it? Are we trying to preserve our position as gatekeepers to a field of play? Are we trying to pad the ego of those already on the other side of the credential?
Whatever the motivation, low pass rates aren’t usually considered as motivating factors for those on the outside looking in. It suggests the program isn’t for the participants, but against them. It’s not about learning. It’s not about improving. It is instead trying to protect those in the club. Our ski instructor group was facing criticism from its members which consist of both those that have achieved this designation as well as those aspiring to it. The suggestion was that a pass rate of 3% was far too low and served as a disincentive to people to pursue this direction. The kinds of questions this organization was likely considering included: What are we trying to accomplish? Are we trying to increase competence? Are we trying to control numbers? Are we trying to flout status? Are we trying to generate revenues for governing organization? Something else? What is a reasonable pass rate we should be delivering?
We can contrast our example of an organization with very low pass rates with another that “achieves” higher and higher pass rates. Public education in the US has been delivering steadily increasing graduation rates in recent decades. These statistics are offered by proponents of public education as proof of progress. Clearly higher graduation rates reflect a higher quality education system is the suggestion. As the US high school graduation rate now exceeds 85% and approaches 90% of kids, this would seem to be substantial progress from the rate of 70% in 1980. Proponents of the education system happily promote these kinds of statistics to suggest that education is working, and we should be grateful for our educators leading the charge pushing the next generation forward. More kids graduating indicates more kids educated which translates into more opportunities and a better society as a whole one would think, or at least hope.
Unfortunately, if we look just beyond graduation rates and at a separate measure of objective ability, we see a problem of considering high pass rates in isolation. SAT scores undertaken by US high school students have been steadily dropping. The relationship between the dropping SAT scores and rising graduation rates is clear in the below graph.
The disappointing conclusion of the above graph isn’t that higher pass rates are positive. It’s that standards seem to be slipping. We’re making it easier for people to pass. Excellence isn’t the end of education. Yes, graduation rates are 20% higher today than forty years ago, but the average test scores on a standardized test like the SAT have dropped 15%. We’re graduating more students, but their average capability appears to be lower. It seems we’ve just passed more people through the system by making things easier to pass. The net result is that high school graduates have diplomas that can’t help but be worth less than those earned forty years ago. In fact, the above chart suggests that the distinction from forty years ago was almost as pronounced in 2005-2010. It’s in the last ten to fifteen years that the graduation rates have grown while the test scores have sunk.
The extreme ends of pass rates pose problems. There are costs to making things too easy or too hard. What’s the sweet spot for a standard to seek? Opening the doors to allow all entrants to easily exit a system ensures that little learning is earned, and the credential is worthless. Keeping the doors tight and heavy allowing only the very strong to push their way forward to the exit also has its challenges. How do we find a balance that ensures a credential that conveys credibility on those that have earned it as well as an incentive for those to give it a go?
Moreover, how does this apply to the insurance industry? What are the pass rates for various insurance licensing programs? How does this dovetail with the employment needs of those in the industry? Are the pass rates low which is making it both unattractive for people to pursue an insurance designation as well as making it tougher for companies in the industry to hire from a smaller pool of potential people? Does the credential provider have a targeted pass rate in mind? Is their testing process delivering results in line with their target? Should there even be a goal pass rate?
Statistics kept by the Alberta Insurance Council record recent pass rates for Level 1, 2, and 3 licensing in the Province. The average pass rate for Level 1 seems to be around 60%. Six out of ten exam takers earn their Level 1 license. This doesn’t take into consideration whether it is someone’s first attempt or not. The Level 2 license pass rate has less test takers and a higher pass rate. Almost eight out of ten test takers achieve their Level 2 license. Those taking a stab at their Level 3 license are fewer in numbers and have a lower pass rate in the neighborhood of 50%.
What’s interesting is that the Alberta Insurance Council also tracks the pass rate for the Level 1 exam by education provider. Students come to the exam having registered and attended a training program for the Level 1 exam offered by an education provider. Presumably, providers that deliver pass rates for their students higher than the average can claim credibility for their program offering. Whereas those providers whose students pass at rates lower than the overall pass rate may have some work to do to improve their education so that it better prepares participants for the exam.
Conferrers of credentials should consider: For whom are you? Are you for your own organization and generating as much revenue as possible for yourself as a credential giver? Are you for those that are seeking those with the credential your organization offers, for example employers? Are you trying to ensure your pass rates produce people with desired capabilities in the numbers that are needed by employers? Or is your focus those signing up to earn the credential? Are you trying to provide them with the most meaningful information? Finally, are you seeking to protect an end user of a service by ensuring service providers can demonstrate competence? Having clarity as to the core purpose of the credential being bestowed will be a key factor in targeting a pass rate.
Moreover, deliberating these questions and producing a pass rate plan as a result isn’t a one and done process. It’s iterative. Pass rates should be regularly reviewed in the context of current market conditions. Is the program delivering the right number of quality and prepared candidates to the finish line? The ultimate test of a credentialling program should be based on positively answering this question.
