“How would you rate yourself, on a scale from one to ten, in technology XYZ?”
Ignorance more frequently begets confidence than does knowledge. – Charles Darwin, Descent of Man, 1871
There is a paradox that presents itself as people learn about a new subject. Only when they get somewhere past the halfway point to mastery, however that might be measured, do they even begin to realize how very little they actually know. Until they reach that point, the unwise and immature will consistently overestimate their abilities. The wise should realize that “they don’t know what they don’t know” all along.
This isn’t some sensei/grasshopper Karate Kid cliché. There are studies to back this claim up, and the claim happens to be one of the favorite subjects of Alan Holub, an object-oriented programming guru (my title, not his). From the American Psychological Association’s selected article from Dunning and Kruger:
Across 4 studies, the authors found that participants scoring in the bottom quartile on tests of humor, grammar, and logic grossly overestimated their test performance and ability. Although their test scores put them in the 12th percentile, they estimated themselves to be in the 62nd. Several analyses linked this miscalibration to deficits in metacognitive skill, or the capacity to distinguish accuracy from error. Paradoxically, improving the skills of participants, and thus increasing their metacognitive competence, helped them recognize the limitations of their abilities.
These ideas have been formalized in the Dunning-Kruger effect.
When answering the question above, any self-reported rating above a seven is immediately suspect for your average business application developer, per the system I will expound on below. The good news is the higher ratings are easily objectively verified – including everything above seven in my ranking system.
Let’s start with some assumptions and ground rules:
- The ratings are one to ten, inclusive. This represents the entire spectrum for everyone – completely ignorant to the world expert. Note that completely ignorant != zero, completely ignorant = one.
- To make it easier, and to match the implication when this question is typically asked, we’ll use whole number ratings only. Because of that, we won’t try to divide those ten different levels up perfectly so that each interval represents the same magnitude (a move from 3 to 4 may not represent the same improvement of skill as a move from 6 to 7). We’ll have to arbitrarily set some levels based on reasonable, real-world changes in skill. This one is best explained by just looking at the examples below.
- A “local-market business application developer” is any employee or consultant working in software development in your local area. For the Microsoft Windows Communication Foundation (WCF) example used below, we are not talking about someone who works for a product team at Microsoft (although he or she might work for Microsoft Consulting Services, which is a different thing entirely, or work in MS evangelism/sales support). For reference, my local market is the Milwaukee, Wisconsin metro area. In my mind these example ratings would apply to anyone in the Milwaukee/Chicago corridor, or the Great Lakes Megalopolis, or anywhere really.
- Because I am taunting – regardless of the intended humor – I’ll direct personal pronouns to my own gender and avoid the whole “he or she” thing.
With that out of the way, here are reasonable meanings and interpretations, using the WCF library as a technological example (because it is the easiest for me, really):
|Rating||Actual Meaning||Example for Microsoft WCF||Meaning When Self-Reported by a Local-Market Business Application Developer|
|1||No real understanding of the technology or its appropriate use, may have heard of it, but that is all||A salesperson or my Mom||Accurate, or humble|
|2||Has an understanding of when and where it would be used, but no theoretical, practical, or technical experience with it||Someone who describes it as, “A unified API for distributed application development in .NET, superseding past libraries for remoting/MSMQ/web services, with emphasis on the WS-* standards”||If he can provide the explanation in the example, he is telling the truth. If he responds with only “the new library for web service development,” he is overestimating himself ***|
|3||Has a bit of practical experience with the technology, may have done some prototyping or maintenance work||Someone who realizes configuration is a beast for WCF||Probably true|
|4||Has practical, real-world experience with the technology on a development project, has worked with the technology daily for around three months||Someone who has experience taming the WCF configuration beast||Probably true, could be verified pretty easily in a technical screening|
|5||(somewhere between 4 and 6 )|
|6||(somewhere between 5 and 7 )|
|7||Has worked with the technology for years, with a track record of delivering solid solutions leveraging the technology correctly, understanding the larger contexts in which the technology lives||Someone who not only understands WCF, but understands distributed applications in general, and understands Service-Oriented Architecture (SOA) well||May be true, could be verified with work and personal references in combination with a technical screening|
|8||A local leader in the technology (one of only a handful at most), may blog extensively on the subject, has experience extending and customizing the technology, may speak on the subject locally, may have written a book||Possibly true, but easily verifiable either way via public artifacts from the developer on the subject|
|9||Internationally recognized leader, may have written a premier book, may speak on the subject internationally, may provide premium training, may sit on an advisory board for the technology||Juval Lowy||Self-deluded…May soften his response with, “I realize there is always someone better out there,” which is the understatement of the century.|
|10||Technical lead and/or senior developer on the development project||Clemens Vasters||Self-deluded, possibly crazy.|
*** Personal WCF axe to grind
Like the previous interview question I proposed, there is a litmus test quality to one aspect of this question. The tendency for rating above my magical 7 (you can choose to set your magical break wherever you wish, but you should acknowledge it exists) indicates the interviewee’s ignorance on the subject (per the Dunning-Kruger effect, they have no ability to accurately measure their own abilities and will overestimate), ability to delude himself, and/or his desire to deceive the interviewer.
For the record, I’d rate myself a 6 or 7 on WCF, and it has been my primary focus of the .NET 3.0+ technologies. I’d like to move towards some of the things listed under number 8 – specifically blogging and speaking – but I’m not there yet. I don’t doubt that people I’ve worked with – and not just salespeople – would call me a “WCF expert,” but personally I realize that is pretty silly. Also for the record, I’ll say that passing the .NET 3.5 and .NET 4.0 certification exams for WCF only proves that I answer multiple choice questions really well – the technical knowledge required for those exams would be possessed by a 5, IMHO. I believe that all IT certifications have ever proven is a base knowledge in a particular subject, and I am well versed in certification exams. It’s 80/20, really: I have about 80% of the skills, with only 20% of the attitude.
The question above should be asked separately for all important technologies in the job description, as a bare minimum. Better still, ask the question for all of the keyword/acronym craziness the candidate throws onto his resume! The questions could also be asked before a technical screening, so the screener could compare subjectively perceived skills against the actual skills revealed in the screening. Finally, there should be some level setting during the first few answers – the interviewer should make the assumptions and ground rules I enumerate above well known. My next post in this category will cover this “level setting” and the associated assumption that everyone is bragging about themselves in an interview situation…