So Now You’re a Wise Guy, eh?

The Wisdom 2.0 conference just wrapped up. If you haven’t heard of it, the goal of the conference is to address the question: “How can we live with wisdom, awareness, and compassion in the digital age?” This is probably the premier event for the mindfulness community in America. The speakers (and, for that matter, the attendees) are an eclectic mix of monks, neuroscientists, media moguls, psychologists, yogis, teachers, corporate leaders, and even an NFL trainer and a U.S. Congressman. The full schedule can be found here.

The premise of the conference is pretty simple: there are certain practices like meditation, yoga, and mindfulness based stress reduction (MBSR), that we have borrowed or adapted from ancient traditions (that’s the “wisdom”) and are now able to scientifically prove their effectiveness on physical, mental, and emotional health. That’s a really good thing, because our health is getting worse and worse as we adopt more and more technology (smartphones, monoculture food, etc.), and the pharmaceutical path to health is fraught with innumerable pitfalls and traps. The hard part is getting the word out about these new, old techniques. The gathering was focused on strengthening the mindfulness community and encouraging people to take the actions to spread the good news.

As a data scientist, attending the conference was a bit of a departure for me, but I heeded my own oft-used advice: when in doubt, take the action that you would regret not taking later. My self-proclaimed goals for attending the conference were to: Listen, Learn, Connect, and Act. I did pretty well on the first two, and not so well on the second two. I’d love to be able to report that the conference was amazing and life-changing, and maybe eventually as some of it sinks in more deeply and I take actions based on what I follow up on, it will have been. But in the near term, I can only say that it was interesting, and worth my time.

There were a few talks that really shined for me, and made the conference especially worthwhile. Luckily, they were all videoed and made public on the conference website. Here are my quick-picks:

Here are the handful of quotes I chose to tweet, that also summarize the conference:

  • “You check your watch, and my God, it’s ‘now’ again!” Jon Kabat-Zin
  • “A team’s inability to focus often stems from ego.” Irene Au
  • “What are we giving our eulogizers to work with?” Arianna Huffington
  • “It’s much more fun to be curious than judgmental.” Jonathan Rosenfeld
  • “With all … that we are doing, wrap the ‘doing’ in ‘being’…and it will bring us back to being fully human.” Jon Kabat-Zin
  • “Onward, upward, and inward!” Arianna Huffington

Finally, my tweetable summary of the message I took away from the conference:

  • “We spend too much time living in rewind and fast-forward. Just hit play. And listen.” Chris LuVogt

I hope to expand on the importance of listening in a follow-up post soon. In the meantime, may you find peace and happiness. And yes, I do feel a little more wise after having attended, but “don’t call me Curly!” 😉


The Power…It’s Electrifying

A couple months ago, we decided that the timing and circumstances were finally right for us to take the plunge and start using an electric vehicle.  A number of friends and family have asked about it, so I thought I’d put my thoughts down here for all to read.  Long story short: We decided to lease a Nissan Leaf.  We love the car and have even considered getting a second one.  But to be clear, EVs are not currently for everyone – yet!  We had been watching the Leaf since before it had even been released for sale in the U.S. in 2010, but a lot went into the decision to get an all-electric car, and specifically get the Leaf in particular.

Fundamentally, and incontrovertibly, gasoline is a limited resource that pollutes the air.  At some point, it will need to be replaced.  There are lots of opinions about how and when that will happen, but regardless, we need to find an alternative, so why not get ahead of the curve and do it now?  I won’t go into all of the advantages (or disadvantages) of using electricity as an energy storage and delivery mechanism, but some of the obvious ones include an existing delivery infrastructure and the ability to be cleanly generated at scale.

We could have gone with a hybrid car, or even a plug-in hybrid, but deep down we felt the right thing to do was to commit to the cleanest vehicle we could manage.  We wanted to show our support for what we felt was an important emerging technology.

Driving an all-electric is a surprisingly awesome experience.  First, it is noticeably incredibly quiet.  We had relatives visiting, and my 6 year-old nephew commented on this (without prompting) as soon as we started moving.  It’s so quiet that they have to add sounds (via external speakers) so pedestrians don’t get startled.  When we test-drove a Prius plug-in hybrid, this was a big difference – what a noisy car.  Second, it accelerates better than most gasmobiles.  Yes, it really kicks it out of the gate.  Third, it has no gears, so there are no funny shifting hesitations and all  the power is instantly available.  Fourth, it is much cheaper to refuel than gas cars, roughly a third the cost, and this drops even more if you have access to charging stations that are free to use.  Finally, it gives you an incredible sense of satisfaction to know that you, personally, are not spewing harmful gases into the air (especially if your electric company offers clean energy).

Clearly, an all-electric vehicle is not currently for everyone because the state of the art technology has some reduced functionality compared to gasmobiles: less range, longer refuel times, and a still-nascent refueling infrastructure.  Even with government incentives, the current EVs on the market do not quite make a compelling case when you look at total cost of ownership, although they are close.  All of these factors will change with time.  My guess is that either for your next vehicle purchase or the one after that, an all-electric vehicle will be a contender.  Right now, EVs are best as commuter cars (if your commute is less than about 35 miles each way, or, if you have access to a charging station near work, then about 70 miles), and for running errands around town.  Taking a long trip in an EV would be a real hassle.  But given that most households have over two cars, chances are good that at least one can be replaced by an EV, and the gasmobile can be used for any long trips (this is the route we’ve currently taken).  EVs are also mostly for homeowners, because of proximity to electrical outlets when the vehicle is parked at home.  We, of course, met all of these criteria, as do a lot of households.  I am also very fortunate to get free, fast charging from my employer, which makes it even easier to stay “topped-off” at all times.  Once we decided that it was time to replace one of our aging gasmobiles, an EV became the obvious choice.

As a car, the Leaf in particular is an excellent vehicle, even if you ignore the fact that it is all-electric.  It has all of the safety, reliability, and convenience features you’d expect from a new vehicle.  It also has a greatly designed  interior space (we like it much better than the Prius or the Focus), and some other cool features (like being able to control some functions like climate-control via an app).  It is solid, reliable, and drives just like a regular car.  Plus, it looks nice, is eminently practical, and priced right. Nissan has been a leader in the development of EVs, and has taken a big risk and put in a huge investment to be a pioneer. It’s clear they believe in the technology, and we wanted to show them our support.  Taken together, all of these factors made the Leaf the best choice for us.  We decided to lease the vehicle in anticipation of improvements in range and charging speed in the next few years.

The great thing is, a lot of manufacturers are following Nissan’s (and to be fair, Tesla’s) lead.  The Honda Fit EV, the Ford Focus EV, and the Mitsubishi MiEV are all currently available in some U.S. markets, with offerings from Chevy (Spark), Smart, and others just around the corner and new EVs getting announced all the time.  Not only will this mean more choices for the consumer, but it will also put downward pressure on the pricetags, upward pressure on proliferation of charging stations, and drive innovation around range and charging times (e.g., battery-swapping facilities).

In short, EVs are a technology whose time has come.  To be sure, it will be a very long time before they are more prevalent than gasmobiles, and will likely never entirely replace them, but I see their widespread adoption as inevitable.  We’re still on the low-end of the technology adoption curve, with less than 100,000 on the road in the U.S. today, but I liken them to cell-phones circa 1997.  They are a proven, superior technology with support from the government and business.  In a decade or two, all of the technical downsides to EVs will have been addressed, and public opinion will have turned against gasmobiles to the point where people who still hang on to them will be considered quaint, anachronistic, or just plain irresponsible.  If you’re currently in the market for a car, I’d encourage you to consider if an EV is right for you.  Take a few test drives – I think you’ll be impressed.

You Are Not an Average, Joe

When I was applying for scholarships for college, my father said something that I thought was nonsensical at the time.  I was bemoaning the acceptance rate statistics, and Dad said something like, “It doesn’t matter how many scholarships are awarded or how many people apply, you only need one scholarship.”  Being a  mathematically talented teenager, I “knew” this kind of advice was utter nonsense.  Of course it mattered!  If the acceptance rate was only 5%, then naturally I only had a 5% chance of getting the scholarship, and if the supply was limited, then the more people that applied, the worse my chances.  Right?

With time, I’ve come to realize the wisdom in what Dad said, and even from a mathematical standpoint, that he was actually correct.  It is a subtle but critically important distinction that we all would do well to internalize.  In statistical language, it means not confusing summary statistics with point observations or samples.  In layman’s terms it means: you are not an average anything (at least, not necessarily).

It is all too common to hear advice like “you should work out 3 times per week for 30 minutes,” or “the average person needs 8 hours of sleep,” or “you should eat like a caveman.”  The problem with these broad prescriptions is that they are not based on your particular needs, and they may or may not actually apply to you. You always need to examine where these recommendations come from, what assumptions they make, and whether they will likely be beneficial for you personally.  Statistically speaking, there is a very good chance that as long as they are based on solid science across a large sample, they actually will apply to you.  But you should be prepared to “reject the null hypothesis,” or to say it more concretely: “your mileage may vary.”

In other words, don’t assume that what works for the average person will necessarily work for you, but do try it as a starting point.  There is a very good chance it will work for you – after all, there is definitely something to be said for the “wisdom of the crowds.”  On the other hand, you may have to modify it a bit to suit you, and maybe even throw it out and try something else that your gut tells you is right.  More importantly, actively look out for situations where you are being treated or acting like everyone else when you shouldn’t be.  Yes, “most” people can tolerate gluten, but that doesn’t mean you can.  “Most” people happily guzzle soda made with GMO-based corn syrup, but that doesn’t mean you should.  “Most” people aspire to get a decent job working for a company, buy a house, get married, and have kids, but you should carefully consider whether that’s the best path for you.  “Most” people blindly follow in the footsteps of their parents when it comes to religion, but why would you make such an important decision by accepting the default?

If you only do what the average person does, you will probably only have an average life (or worse).  To be happy and healthy, you need to be productive and strive towards fulfilling your best true self.  That involves finding your strengths (i.e., those aspects of your self where you don’t fall into the middle of the bell curve, i.e., you are not average), and doubling-down on them – not just settling for what is “normal” or “expected.”  Since you only get one life, chances are you’d like for it to be a bit more than average – it’s up to you to discover how to make it great and to march resolutely down your own path.

In the end, I worked hard on my applications, and I did get that one acceptance to a very selective college and that one scholarship I needed, and the experiences I had because of those “lucky breaks” have had a positive lasting impression on who I am and how I approach life.  But to think of them as lucky breaks isn’t right.  It’s clear now that I was not an average applicant. Rather, I was an outstanding one – I just didn’t realize it.  But on some level, Dad did.

P.S. Many of the concepts here are based on a basic understanding of statistics, a subject I and others have suggested should replace Calculus in high school curricula.   If you want to learn more about statistics in a fun way, I heartily recommend The Cartoon Guide to Statistics.


WhatIf…You and Your Friends Held a Conference?

Last Saturday, a dozen or so friends gathered at my home for the first edition of what I hope to be a recurring tradition – a sort of mini-conference for the members of a bookless book club I co-founded with Jesse Bridgewater and Chad Carson.  We called the gathering IfCon*, after the sentence prefix we use to inspire conversations at our regular meetings, “What if…”

The format of the conference was simple: every member was invited to give a 20-45 minute talk on a topic of their choice, with the only rule being “don’t be boring.” Ten talks and three meals later, I had learned a lot, laughed a lot, gotten to know them all a bit better, and was not for a single moment bored.

The full agenda is on our website, along with links to videos and slides for those speakers who agreed to have them made public.  To give a taste of what we talked about, here are the titles of the talks:

  • Tomorrow is a new day…what if you really believed it?
  • Video games as a storytelling medium
  • Divine Experience – On the Neural Basis of Revelation
  • The Piraha: A People Without Numbers
  • Life lessons from backgammon
  • An iPhone fit for a Queen: Thoughts on American consumerism from Andy Warhol
  • Longevity, Health, and Happiness : How to Live the SWEET Life
  • A Perspective on Understanding the Arts and Humanities as Demonstrated by Differential Image Analysis Bingo!
  • Reflections on man vs machine and digital vs. analog
  • Environmental, Educational, Health, & Economic Sustainability – Lessons from Cambodia

My talk was about how to live a longer, happier life by tracking the physical, mental, and social aspects of your health on a daily basis.  My friends brought up some interesting questions and challenges to my technique that still have me re-thinking my approach.  Giving the talk was invaluable for me – it’s so hard to challenge your own assumptions, but presenting your ideas to someone you know and trust can teach you a lot about your blind spots!

Aside from the actual content of the talks, one of the most gratifying aspects of the day was that nearly every speaker shared something personal.  Whether it was their love of video games or backgammon, their family trip to Cambodia to visit the schools they helped build, their fascination with language, or their personal trick for starting each day anew, I learned a little more about everyone and connected with them at a deeper level than I had before.  And if you watch my talk (Longevity, Health, and Happiness), you’ll learn how making those kinds of connections is critical to a long, happy life.

What if you and your friends held a conference?

* We subsequently discovered that there is another conference with the name, so we’ll likely change it next time around to avoid confusion.



Beale was sweating. Not uncontrollably. Not so much that you could see it when he raised his tuxedo-clad arm to retrieve a serving tray from an overhead cupboard. But noticeably, and with a different kind of nervousness. Ironically, this should have been a piece of cake for him. He had the record time on the Gauntlet, after all. Now he just had to repeat something he’d already done flawlessly, as trained, for over a month.

That first day two years ago, he had arrived as instructed at the square, with high hopes of changing the world. A minute later, he stood befuddled, holding the rucksack his old friend – the one who had recruited him – had shoved into his hands, and contemplating the cryptic instructions she had breathlessly rattled off from rote before snatching his glasses and dashing away. “Start at Salsa and Seventh, then make a right somewhere between 4th and 5th. Oh, and you’d better eat before you start.” He learned later that last part was her own, unauthorized addition, and while it didn’t strictly give away much, it had made all the difference. Looking around, there had been only Joints in view. He couldn’t stomach printed food even on a regular day, and since this was apparently some kind of orienteering test, he really couldn’t afford to be dumbed down. Desperately, he opened the sack and found, unbelievably, a full, natural lunch, complete with what was obviously a tree-grown, organic apple. These guys must have a helluva budget, he thought as he sat down on some steps to eat and plan his strategy. Without the aid of his glasses, he had to use only his un-augmented brain and bio-senses to get to the rendezvous point. Luckily, he knew there was no street call Salsa, and with that as the starting point, he had unraveled the puzzle in less than 2 hours.

Now, stepping in front of the dumb-waiter, Beale paused for a millisecond before tripping the camouflaged button that would remarkably, silently switch out its contents, for the last time. Getting a design spec for this single slice of carrot cake had been hard, but as they say, you can find anything on the internet. Especially if you have the support and skills of a global terrorist network supporting you. Although it had taken some time, he had come to accept that what he was doing was indeed terrorism. Slow, subtle terrorism. Necessary terrorism for the good of mankind. But terrorism nonetheless. He liked to think that because he was now on an all-natural diet of precisely what his employer was supposed to be eating, swapped out from the dumbwaiter, that he now had the clarity of mind to especially appreciate the irony of poisoning a mogul over the course of two weeks using genetically therapeutic food manufactured by the same robotic printer/assembler the mogul had promulgated across the entire world. But truthfully, even the “dummies” that normally ate that crap could probably understand, and hopefully approve of, his actions. It was, after all, for their own good, even if they didn’t or couldn’t understand it yet.

“He’s on his last bite,” said the maid-servant as she raced past on her way back to the kitchen. That was the signal for dessert. According to Beale’s glasses, the mogul typically paused for 47.3 seconds between main course and dessert when lunching at home on Mondays by himself. But it also predicted that because of an appointment, he would be rushed today, and there was a 32% chance he would skip dessert altogether. Beale couldn’t afford for that to happen. The penultimate dose had to be delivered today for publicity reasons, and the mogul had evening plans that required his physical presence and wouldn’t be dining at home. When moguls died, authorities always did a thorough autopsy – a complete nanoscopic workup.  After all, they weren’t supposed to die, except in accidents. And if Beale’s counterpart had indeed infiltrated the coroner’s office as planned, the results wouldn’t be squirreled away, but broadcast to the world. In the best case, the “dummies” would rise up.  Beale was smart enough to know that was a very unlikely outcome. But the news of the death could lead to an influx of new recruits, and once the Movement had de-toxed them, they would be able to accomplish even more covert operations, and, he hoped, some day truly incite a revolution.

Reaching into the dumbwaiter, his gloved hand brushed the icing of the cake. Most moguls had a distinct lack of tolerance for imperfection, even though their food was paradoxically prepared using natural ingredients with an inconvenient tendency for just such imperfections. Beale’s mogul wasn’t quite so fussy, but he couldn’t take the chance of rejection, and hurriedly repaired the damage using his crumb sweeper. Normally, such an intimate interaction with food would elicit a visceral reaction, even mild salivation, but Beale knew how this “food” was made, and stoically placed the plate on the serving tray.

“No dessert today, Beale,” said the mogul as Beale approached, “Got a meeting, and want to take it upstairs.” He started pushing back his chair.

“But, sir.” Beale knew this was the wrong response. The right response came from the set, “Yes, sir”, “Very good, sir”, or “As you wish, sir.” But even with thirteen days of slow dosing, there was no guarantee the genetic therapy would take. This slice of cake had an especially high concentration of the triggering agent that put the odds of the mogul’s immune system failure at nearly 100%.

“Beg your pardon, sir. Just wanted to be sure your caloric intake was sufficient for peak performance at tonight’s Gala.”

The mogul thought for a second. “Good point, Beale. Wonder why my glasses didn’t point that out? Bring it up to the office, and I’ll try to squeeze in a few bites while on visual mute. Can’t guarantee anything though – those guys in Omaha usually let me do all the talking!”



(This piece was written as an entry to the Big Think Short Fiction Contest #1 : 1000 words of fiction around the theme “Future Food”)

Bespeak Skillfully and Carry a Nanoscopic Stick

When I was about 13 or so, my father read an article somewhere about the best careers for the future.  Near the top of the list of recommendations were computers.  My dad had the foresight to get me in front of a machine (a Commodore Pet) at work as soon as he could, and also bought a TI 99/4-A home computer for my brother and me to bang away on, which we summarily proceeded to do.  This is one reason why we both landed in Silicon Valley some 15-20 years later, and flourished.  I’ve often wondered, if that article were written today, what advice it would give to parents…

Knowledge of computers and programming is still a useful skill today, and will likely remain so for quite some time.  But those kinds of skills are just table stakes now.  So yes, make sure your kids learn something about the inner workings of computer systems, get them a solid foundation in science and engineering, absolutely have them master their math and statistics, and above all, prepare them for a lifetime of learning.  But to really set them up to succeed as adults, I’d also suggest something you may not have thought of: make sure they have a firm grounding in the liberal arts.  Because they will most likely live their life as “designers,” “makers,” and “bespeakers.”

Looking out 20 years is difficult in this age of accelerating change.  If you believe, as I do, many of the tenants of the Singularitarians, the year 2032 will be so utterly different than the present that it may seem daunting to try to predict a viable educational strategy for a young adult coming of age in those times.  In order to make my case for liberal arts, I therefore need to don my futurist cap and take a brief detour to describe the most probable state of the world in 20 years.

Some technological advancements that will likely happen by 2032 (hold on to your hat, this is a wild ride):

  • Robots will be ubiquitous, and will have taken over many of the skilled chores we now outsource to cheap labor markets (either at home or abroad), including producing our physical goods, growing our food, discovering and extracting natural resources, recycling our waste, and piloting our vehicles (although the need to transport both humans and goods will be significantly curtailed).
  • Low cost, local, on demand manufacturing will be commonplace.  Need a new gadget?  Just place your order, and it will be 3D-printed and assembled at a nearby convenience store for pickup or delivery in hours or minutes.  Some staples like disposable towels or razor blades or, well, staples may even be “printable” using an at-home “replicator” (chalk up another correct prediction to Star Trek!)
  • Sustainable energy will be ubiquitous and cheap.  Why?  Because we’ll be way past peak oil at that point, so we will have no choice but to solve the problem.
  • Many of our common gadgets today will be woefully obsolete.  Smartphone?  TV?  Camera?  All replaced by glasses/contacts/implants/neural interfaces that can beam images directly to the eyes or retina or neocortex, have an array of built-in sensors, and have access to 1000’s of times the computing power and storage available in today’s gadgets.
  • We will spend the vast majority of our waking hours (which will likely be most of our hours, as the need for sleep will have been largely eliminated) in either virtual reality or augmented reality.  These artificial worlds, populated by lifelike avatars of our own design that represent us, will seem as real as the real world, and will be just as important.  And we won’t be the only inhabitants – there will also be artificial intelligences to act as assistants, companions, and compadres.  And the augmented world will be populated by real-world objects that hook in to the global info net, using a variety of sensors to add to the unfathomable stream of messages on the global communications network.
  • Nanotechnology and biotechnology will have solved most, if not all, of the major sources of disease and illness, including aging.  People will begin to correct and augment their physical and mental abilities with technology, and true cyborgs and bionic humans will walk the streets.
  • Many mundane mental skills will effectively be outsourced, and thus would be “downloadable” on demand.  For example, real-time universal translation between any two languages, in any context, in any modality (verbal, written, Braille) will be ubiquitous.

All that in 20 years?  We’ll see, but even if we don’t quite make it to that point by then, we’ll at least be well on our way.  Of course, there are many other aspects to life on earth (politics, climate change, population growth or decline) that will define the culture and zeitgeist of the time, but one thing is for sure – it will be vastly different than how we live today. And keep in mind, even if things aren’t quite as I describe in 20 years, they will be at some point in your child’s lifetime, especially considering that they will have a very, very long life.

So what happens when physical items are commoditized, energy is cheap, everyone and everything around the world is connected at all times, manual labor is minimized, and it doesn’t pay to learn shallow mental skills?  Intellectual Property happens.  In this brave new world, knowledge will truly be power, and manifestations of thought will be the new currency.  Which brings me back around to the original topic of this article.  To be a part of this new economy, your children will need to be creators – people who use their brains to gather knowledge, synthesize it, and create something of value for others to use or experience.  And the artifacts that they directly create will be almost exclusively digital in nature.

It’s nearly impossible for me to know in any detail what exactly these jobs will be, but I can try to imagine a few.  For example, new kinds of musicians will come into existence, who will be more like composers, combining all sorts of music on the fly, in response to their live audience, and who use algorithms to adapt the music based on the “vibe” coming from their listeners – even if there’s only one person in the audience.  (Modern DJs and music-recommendation services both presage this concept).  New kinds of product designers will custom-build designs of a wide range of products for individuals, and then be able to turn around and re-sell those designs to wider audiences or negotiate higher prices with their patrons for design exclusivity rights.  Virtual world designers will be in high demand, and will need a wide range of skills from visual design to narrative skills (Modern video game designers are their precursors).  “Beamers” (as Ray Kurzweil calls them) will make their living by allowing their customers to live through them vicariously in ways that were never previously available, by literally streaming their sensory inputs in a way that makes the end-consumer feel as if they were inside the the beamer’s body (like in the film “Brainstorm”).  Nanotechnicians will design and build machines for specific purposes on the atomic level (like in the novel “The Diamond Age”).  Yes, it all sounds like something out of a SciFi novel.  In fact it sounds a lot like the civilization described by Arthur C. Clarke in 1956 in “The City and the Stars,” except this world is only decades away instead of a billion years in the future.

Some things have always been true about humans, and these will, of course, not change anytime soon.  People love a good story, they need social interaction, they enjoy beauty and elegance, they enjoy experiencing the world through their senses, and they have an overwhelming need to both learn and teach.  As our physical needs and desires become easier to satisfy, our emphasis will naturally shift towards producing other, less tangible things that fulfill these essential human needs.  And, more importantly, both the supply and demand will move into the “tail” of the distribution.  In other words, not only will people demand more customized, even bespoke, products and services, but by the laws of supply and demand there will naturally emerge more people to supply them.  Those will be our children – both the producers and consumers.

What skills would such designers, makers, and “bespeakers” need?  Clearly they’ll need a command of the tools – computers, nanotech, biotech, and their derivatives.  More importantly, though, they’ll need to know how to manage the creative process, relate to others, communicate their ideas, tell a good story, draw a good picture, relate everything to the rich history we have as humans, and build a philosophy of living in this changed world.  In short, they will need a solid grounding in the liberal arts.

A second renaissance is coming – will your child thrive?  Will you?


2012-01-19 P.S. This article was just published – my favorite quote: “being more fully human is what individuals will need to stay one step ahead of computers”.  The Career Of The Future Doesn’t Include A 20-Year Plan. It’s More Like Four. | Fast Company

2013-02-07 I just finished reading Daniel Pink’s A Whole New Mind, which comes to the same conclusions as I do here, but using different sources of evidence.  He also develops the types of “right-brain” skills that will be necessary, and ways to sharpen them.  Recommended.


Don’t Fear the Filter

reaper (rē’pər) n.  1. One that reaps, especially a machine for harvesting grain”

grain (grān) n. 13b. An essential quality or characteristic.” –

There’s been a lot of  FUD spreading around the net about a “filter bubble.”  In my year-end review of TED videos, I even noted that Eli Pariser gave a talk on the topic in that highly regarded forum.  As someone deeply involved with designing and implementing one instance of such a filter, I’d like to send a simple message – don’t fear the filter.  After all, you’ve been living in a filtered media world since you were born – it was just filtered by humans, not algorithms.  Neither humans, nor the algorithms they design, are infallible, and the assumption that humans will necessarily provide better curation of content is not only an untested hypothesis, but completely disregards the impracticality of an only-human approach to the information glut we now face.  While there are legitimate reasons to be concerned, the important thing to realize is that the designers of the algorithms are already acutely aware of these potential problems, and attempt to address them when solving the very hard problem of algorithmic content curation.

For example, Pariser expresses deep concern about the “invisible editing” done “without my permission” that results in “showing us what it thinks we want to see instead of what we need to see”.  Of course, he’s talking about “the internet” (or, more precisely, the major players in content on the internet), but those same statements could equally apply to the editorial staff in any traditionally run media outlet.  Yes, humans are typically a better judge of quality and it’s true that algorithms don’t yet have the embedded ethics that editors have.  But imagine if we lived in a world where every time you needed to find something on the internet, you had to rely on a human to find it for you instead of a search engine.  In reality, the internet has been in a “filter bubble” ever since the first web search engines came online over 15 years ago.  And while it’s true that more and more content is being run through filters, that’s because the naive approach to using humans doesn’t scale, and the amount of information is certainly growing.

Unfortunately, the typical characterization of personalization technology, i.e. that the algorithms are mainly looking at what you click on first, is a gross simplification that doesn’t do justice to the complex algorithms used in the field of content recommendation.  Yes, clicks are a tremendously important input signal to the algorithms, but they are by no means the only signal and most importantly, they are not sufficient in and of themselves to build a system that could compete with the likes of a human editor.  And yes, it is often the case that what the algorithm is trying to optimize is clicks.  But not always, and not only.  More importantly, any decent algorithm has to take into account the “wisdom of the crowds” – in other words, it’s not just what you click on, but what others click on, especially others that are similar to you in some way (your friends, followers, or other cohorts, however that’s defined).  So, in reality, there are humans in the mix.

At the core, the concern seems to be about what Pariser calls the “self-looping and fragmenting effects” that can result from the use of learning too well what a person likes.  But any Machine Learning practitioner and filter builder is so acutely aware of this effect (variously known as “overfitting” or “local optima” or “explore vs. exploit” or by a number of other technical terms in a myriad of flavors depending on what particular aspect of the problem you are looking at), that it hardly even bears mentioning.  It’s kind of like asking a truck driver to make sure he doesn’t run out of gas on his cross-country trip.  Those of us who work on this are always trying to avoid getting “stuck in a rut” by making sure we throw in enough variety and diversity to promote discovery.

At Yahoo (where I currently work), we do take a hybrid approach, at many levels, of incorporating humans into the algorithm.  But we are also always looking for ways to take humans out of the loop when that makes sense, which is often.  We can’t possibly serve up relevant content to hundreds of millions of people across the globe without some big data science and heavy lifting done by machines.

To their credit, the filter-fear-mongers do make a few points that I particularly like – for example,  the suggestion that the algorithms need to be transparent enough and that people need control over how it works.  This part is particularly challenging for those of us doing Machine Learning for recommendation, primarily because the techniques we use are often not readily amenable to transparency and explicit control.  Regardless, I certainly agree  – although the vast majority of people will be more than happy with the default settings of the algorithm, we always need, like in Star Trek, a manual override.  Presenting the user with the thousands of words, phrases, and other features we use to model their preferences just won’t work (especially after we’ve projected their “features” into a low-dimensional subspaces in an effort to divine their latent semantics).

I’m mostly not worried about filters because their designers will naturally be kept on their toes by the people using the filters.  If your filter only suggests a narrow range of content, then people will stop using it or at least complain.  Pariser used the example having his conservative friends filtered from his Facebook stream.  Well, he wasn’t the only one to complain, and Facebook re-instituted the option to get an unfiltered time-sorted newsfeed.

In the end, you are your own best filter.  As information gets easier and easier to publish, and as it gets more central to all of our lives and careers, the volume becomes unmanageable, and it behooves us to become active in our quest for relevant information, and not just swallow whatever the “big guys” decide to publish.  Filters will be one of the indispensable tools for helping us do just that.

Get Inspired – LuVogt’s Selections from TED 2011

Last year I created a list of interesting videos from the TED conference. It was such a hit that HuffPo decided to copy me this year! And while they point out many great talks, of course, I have my own take.  So as a way to avoid most of the items on my end-of-the-year to-do list, and of course gain some inspiration for the coming year, I’ve once again spent a few hours over the holidays combing through the TED videos that were posted this year, and picked out a few that I found particularly noteworthy. (You’ll note that only a few overlap with the HuffPo list.)

These first two were so inspiring that I’ve decided to act on both – my first personal 30 day challenge is to smile more (yes, I’ve established specific numerical goals on a per-day basis).  So if you see me smiling a lot this month, don’t be weirded out, just smile back 🙂

Before I jump into the techie stuff, here are a few nice talks from the world of art and design:

Here are a couple that fall into the “self-help” category:

  • On being wrong – Kathryn Schulz – A great reminder to move out of our ever-present bubble of right-eous-ness
  • Doodlers, unite! – Sunni Brown – Exercise that right brain!

Now on to grander things – some “change the world” ideas, literally!

And last, but by no means least, some advances in technology that should leave you gaping:

I could spend a lot of time “getting inspired”, but the new year has begun, and it’s time to act!  One of my resolutions is to publish more to this blog, so you should hear from me again soon.

For a complete list of 2011 TED videos, visit here.

A Few Selections from

I spent the afternoon yesterday browsing – I hope to make it an annual ritual. Such a high density of good stuff, thanks Chris (Anderson)! If you’ve never spent some time on the site, I highly recommend it. Instead of indulging your regular TV habit, spend the evening with some of the world’s great minds – it’s inspirational and entertaining!

Here were some of the ones that caught my eye – your mileage may vary:

  • Jamie Oliver’s TED Prize wish: Teach every child about food Jamie’s so passionate that he’s a bit all over the place, but his message is spot-on.
  • Peter Molyneux demos Milo, the virtual boy One demo of how the Kinect can be used. I just tried it out for the first time yesterday at a store, and gotta say, this changes everything.
  • Arthur Benjamin’s formula for changing math education A famous math prof from my alma mater (Harvey Mudd) says it best to forget calculus, we should be aiming towards stats and probability as the “apex” of the math curriculum. I’ve been saying this for years, so of course I agree.
  • Mark Roth: Suspended animation is within our grasp Mark takes his time getting to the point, but the idea is pretty big, and I love how it demonstrates an “aha” moment.

And these are for the data geeks in the crowd:

Happy New Year!