3D Spherical Feedback – Redux


SphereLast month, I wrote a short blog post outlining a new technique for getting personal improvement feedback that I called 3D Spherical Feedback. I directly solicited several hundred of my friends, family, and professional colleagues from across the years to answer four short questions about me. The results have been enlightening, insightful, and yes, quite funny at times. More importantly, the process helped me reconnect with a lot of people that I had lost touch with. In this post, I will analyze both the process itself as well as the results for me personally. If you are more interested in knowing what I learned about myself, skip ahead to the section entitled Personal Results.

Design and Method

I used Google Forms to create a six page survey. On the first page, the respondent was asked for their name (optional), as well as primed to remember the most salient experiences they shared with the subject (me). The next four pages corresponded to the four questions from Want Great Feedback? Ask These 4 Questions, also shown below. The last page asked whether the respondent would like to have a follow-up conversation, as well as whether they had any further questions, comments, concerns. In a handful of cases, the questions were asked in person instead of online.

The Four Questions

  1. What do you think is Chris’s greatest strength?
  2. What do/did you appreciate most in your interactions with Chris?
  3. What do/did you wish Chris did more or less of?
  4. What gift would you like to give Chris?

Respondents were solicited in three ways, in the following order: first via posts to social media sites (LinkedIn, Facebook, G+), then via bulk emails to a list selected by the subject, and finally, via direct emails to a shorter list culled from the bulk list by the subject. As would be expected, response rates improved as solicitations became more personal and directed. All in all, somewhere in the neighborhood of 500 participants were solicited, well over 200 viewed the survey, and 123 responded. Each respondent received a personal note of thanks from me.

The online survey was designed as a step-by-step survey to better mimic an in-person interaction, where a question is asked and answered before the next question is presented. Likewise, the question about identity was front-loaded to mimic an in-person interaction. A small number of respondents expressed dissatisfaction with this design, wanting instead to see all of the questions and being able to make the anonymity decision after answering.

A number of solicited participants also expressed dissatisfaction with either the specific questions used, the impersonal nature of the survey, their lack of sufficient knowledge of the subject to provide an informed opinion, or the implication that they should not remain anonymous. These reasons (and presumably others) account for the drop-off between the number of people viewing the survey and the number who completed it.

Results & Discussion

Nearly all respondents answered all four questions. 15 of the 123 decided to remain anonymous. Only 5 specifically wanted a follow-up conversation, 16 specifically did not want it (this corresponded strongly with anonymity), and the rest left it up to the subject.

Half of the answers were ten words or less, with only 6 of the 492 (=123 x 4) answers exceeding 100 words, and the median answer length was 10 words. This is likely partially a result of the questionnaire, which specifically instructed the respondent to answer quickly, although a large text box was provided for answers of any length (the longest received was over 600 words).

Roughly 40% of the respondents were friends or family of the subject, 48% were colleagues, and 12% were both.

As a rule, the responses were overwhelmingly positive in tone. There are at least four large sources of bias that help explain this. First, the questions themselves are phrased in such a way as to elicit a positive response. Second, there was a selection bias introduced by having the subject choose which participants to poll and follow up with. Third, a self-selection bias was also likely in play, in which only respondents who had something positive to say bothered to respond at all. Finally, because the respondents were a mix of friends, family, and colleagues, yet the questions were more geared towards a work environment, it was not uncommon for friends and family to elect to not give a substantive response. This happened particularly with question #3, “What do/did you wish Chris did more or less of?”, which was often answered “Not sure” or “Don’t know” by personal acquaintances.

Respondents were also coded by relative strength of relationship to the subject: a function of how many years they had spent together, how long it had been since they had been in close contact, and how close their relationship had been. This relationship strength was later used in the textual analysis of the answers (below) to weight how much to count the text of the response.

In terms of design, I see a few opportunities for improvement. Moving the “Name:” question to the end of the survey would likely assuage respondent’s discomfort and may improve response rate. Choosing a different set of questions, more geared towards personal relationships, would also likely improve responses from that segment of respondents. Most importantly, however, a more personalized plea for participation, along with a deadline, would likely go the furthest to improving the quantity and quality of responses.

Personal Results

My goals for this exercise were twofold: one was to experiment with a new technique for gaining feedback. Fundamentally, I was trying to see if short responses from a large number of people from many relationship categories and phases of life could provide a more textured, nuanced picture than the typical low-quantity, detailed feedback from current co-workers only. The second goal was to get feedback so that I could get a more clear picture of who others thought I was, and to use that feedback to help guide my personal development, and even my career path going forward.

I received over 10,000 words of feedback from people who have known me since I was born through people I’ve only first met in the last year. From people whom I’ve worked closely with for years to people who only knew me briefly and in passing. I’m still going through them and will likely be doing so for months or years to come. As mentioned above, the feedback was overwhelmingly positive, to the point of making me feel almost narcissistic in having undertaken the exercise. If I were to do it again, I’d definitely change the questions to try to elicit more constructive feedback.

To try to get a gist of the most important points, I’ve run the feedback through some textual analysis and produced some “word clouds” (below, thanks to Word Cloud Generator – Jason Davies). Two of these are summaries of colleagues’ comments, and two are from friends and family. They also come in two flavors: single words or sentence fragments (these are cut off in the middle of words). I was amazed at what some people said, and that some of what I considered my more hidden strengths were the first things to come to mind for some.

Word Cloud - Colleagues
Phrase Cloud - Colleagues
Word Cloud - Friends
Phrase Cloud - Friends

Reading through the responses, there is no doubt in my mind that this exercise has been eye-opening and immensely useful for me. In terms of the methodology, I don’t think this would replace the typical 360 feedback used in corporate settings, as it does not address the particulars that was designed to address. Nevertheless, it is a powerful way to expand the feedback net to gain insights that are more about the whole person, and not just the professional persona we put on at work.

Thanks to all who took the time to respond — I will be forever grateful for both your thoughtful responses and for the “gifts” you gave. I was overwhelmed by by your kindness. By far the most common gifts were some combination of health, happiness, love, peace, and friendship. I also received at least a dozen books, plenty of wine, several job offers, and a bunch of vacations. Not to mention the child(!), 3 puppies, grain of Tibetan sand, Japanese vase, and book of Persian poems. But the gifts that made me pause and think the most were the numerous offers of time, the wishes that I find a suitable challenge, and the hopes that I could find my voice and a become the leader some of you seem to think I am. It is these gifts that I will be focusing on making a reality in the coming days.

And thanks for the one box of chocolates — of all my vices, this is one I hope to never give up!