Don’t Fear the Filter


reaper (rē’pər) n.  1. One that reaps, especially a machine for harvesting grain”

grain (grān) n. 13b. An essential quality or characteristic.” –

There’s been a lot of  FUD spreading around the net about a “filter bubble.”  In my year-end review of TED videos, I even noted that Eli Pariser gave a talk on the topic in that highly regarded forum.  As someone deeply involved with designing and implementing one instance of such a filter, I’d like to send a simple message – don’t fear the filter.  After all, you’ve been living in a filtered media world since you were born – it was just filtered by humans, not algorithms.  Neither humans, nor the algorithms they design, are infallible, and the assumption that humans will necessarily provide better curation of content is not only an untested hypothesis, but completely disregards the impracticality of an only-human approach to the information glut we now face.  While there are legitimate reasons to be concerned, the important thing to realize is that the designers of the algorithms are already acutely aware of these potential problems, and attempt to address them when solving the very hard problem of algorithmic content curation.

For example, Pariser expresses deep concern about the “invisible editing” done “without my permission” that results in “showing us what it thinks we want to see instead of what we need to see”.  Of course, he’s talking about “the internet” (or, more precisely, the major players in content on the internet), but those same statements could equally apply to the editorial staff in any traditionally run media outlet.  Yes, humans are typically a better judge of quality and it’s true that algorithms don’t yet have the embedded ethics that editors have.  But imagine if we lived in a world where every time you needed to find something on the internet, you had to rely on a human to find it for you instead of a search engine.  In reality, the internet has been in a “filter bubble” ever since the first web search engines came online over 15 years ago.  And while it’s true that more and more content is being run through filters, that’s because the naive approach to using humans doesn’t scale, and the amount of information is certainly growing.

Unfortunately, the typical characterization of personalization technology, i.e. that the algorithms are mainly looking at what you click on first, is a gross simplification that doesn’t do justice to the complex algorithms used in the field of content recommendation.  Yes, clicks are a tremendously important input signal to the algorithms, but they are by no means the only signal and most importantly, they are not sufficient in and of themselves to build a system that could compete with the likes of a human editor.  And yes, it is often the case that what the algorithm is trying to optimize is clicks.  But not always, and not only.  More importantly, any decent algorithm has to take into account the “wisdom of the crowds” – in other words, it’s not just what you click on, but what others click on, especially others that are similar to you in some way (your friends, followers, or other cohorts, however that’s defined).  So, in reality, there are humans in the mix.

At the core, the concern seems to be about what Pariser calls the “self-looping and fragmenting effects” that can result from the use of learning too well what a person likes.  But any Machine Learning practitioner and filter builder is so acutely aware of this effect (variously known as “overfitting” or “local optima” or “explore vs. exploit” or by a number of other technical terms in a myriad of flavors depending on what particular aspect of the problem you are looking at), that it hardly even bears mentioning.  It’s kind of like asking a truck driver to make sure he doesn’t run out of gas on his cross-country trip.  Those of us who work on this are always trying to avoid getting “stuck in a rut” by making sure we throw in enough variety and diversity to promote discovery.

At Yahoo (where I currently work), we do take a hybrid approach, at many levels, of incorporating humans into the algorithm.  But we are also always looking for ways to take humans out of the loop when that makes sense, which is often.  We can’t possibly serve up relevant content to hundreds of millions of people across the globe without some big data science and heavy lifting done by machines.

To their credit, the filter-fear-mongers do make a few points that I particularly like – for example,  the suggestion that the algorithms need to be transparent enough and that people need control over how it works.  This part is particularly challenging for those of us doing Machine Learning for recommendation, primarily because the techniques we use are often not readily amenable to transparency and explicit control.  Regardless, I certainly agree  – although the vast majority of people will be more than happy with the default settings of the algorithm, we always need, like in Star Trek, a manual override.  Presenting the user with the thousands of words, phrases, and other features we use to model their preferences just won’t work (especially after we’ve projected their “features” into a low-dimensional subspaces in an effort to divine their latent semantics).

I’m mostly not worried about filters because their designers will naturally be kept on their toes by the people using the filters.  If your filter only suggests a narrow range of content, then people will stop using it or at least complain.  Pariser used the example having his conservative friends filtered from his Facebook stream.  Well, he wasn’t the only one to complain, and Facebook re-instituted the option to get an unfiltered time-sorted newsfeed.

In the end, you are your own best filter.  As information gets easier and easier to publish, and as it gets more central to all of our lives and careers, the volume becomes unmanageable, and it behooves us to become active in our quest for relevant information, and not just swallow whatever the “big guys” decide to publish.  Filters will be one of the indispensable tools for helping us do just that.


Get Inspired – LuVogt’s Selections from TED 2011


Last year I created a list of interesting videos from the TED conference. It was such a hit that HuffPo decided to copy me this year! And while they point out many great talks, of course, I have my own take.  So as a way to avoid most of the items on my end-of-the-year to-do list, and of course gain some inspiration for the coming year, I’ve once again spent a few hours over the holidays combing through the TED videos that were posted this year, and picked out a few that I found particularly noteworthy. (You’ll note that only a few overlap with the HuffPo list.)

These first two were so inspiring that I’ve decided to act on both – my first personal 30 day challenge is to smile more (yes, I’ve established specific numerical goals on a per-day basis).  So if you see me smiling a lot this month, don’t be weirded out, just smile back 🙂

Before I jump into the techie stuff, here are a few nice talks from the world of art and design:

Here are a couple that fall into the “self-help” category:

  • On being wrong – Kathryn Schulz – A great reminder to move out of our ever-present bubble of right-eous-ness
  • Doodlers, unite! – Sunni Brown – Exercise that right brain!

Now on to grander things – some “change the world” ideas, literally!

And last, but by no means least, some advances in technology that should leave you gaping:

I could spend a lot of time “getting inspired”, but the new year has begun, and it’s time to act!  One of my resolutions is to publish more to this blog, so you should hear from me again soon.

For a complete list of 2011 TED videos, visit here.


Keeping your Finger on the Pulse of your Social Feeds


There’s a really cool Yahoo product out there that I find immensely useful for the average internet user, and I don’t think gets enough love – Yahoo Pulse. *

If I were to describe Pulse to you, you’d probably say, “That sounds a lot like my Facebook News feed.” And you’d be right. In fact, Pulse incorporates your Facebook feed, and your Twitter feed, AND a feed of activity from your friends on Yahoo. All rolled up into one “uber-feed.” Here’s how it works, and why you should use it:

  • Go to and sign in with your Yahoo id (or, you can sign in with your Google id or Facebook id – isn’t that cool?).
  • You should see a couple of links and a message to link your Facebook and Twitter accounts. Do that.
  • Now, when you go to, you should now see a combined stream of your Facebook and Twitter streams.
  • You should also connect with your friends on Yahoo. You’ll then start getting updates in your Pulse stream about what they are doing on Yahoo sites (like, Buzzing up articles, bookmarking things on Delicious, posting pictures to Flickr, answering questions on Answers, etc. etc.). Just go to the Contacts tab on Pulse, and start inviting folks by importing from your other contact lists.
  • Finally, as always, you should adjust your settings to make sure you’re only sharing the things you want with the people you want. Go to the Settings page and click through all of the links there and set things up in a way you feel comfortable – and do this again later after you’ve used the product a little while and have a better understanding of what’s going on.

Here are the really great things about using Pulse:

  • No annoying ads! That’s right, by bypassing the Facebook site itself, you also bypass the ads (at least, for now!). You also bypass the overly aggressive tracking that Facebook does.
  • All of your social updates in one place. No more visiting multiple sites to see what your friend are up to – it’s all right there. Personally, I have friends who refuse to use Facebook, and only use Twitter. This way, I get to see their updates right in line with all of the Facebook updates.
  • Likewise, you’ll now have one place from which to post your updates – and it will get cross-posted to Facebook and Twitter.
  • You can also Comment, Like, Reply, Retweet right from the Pulse interface.
  • The updates aren’t just on this one page – you’ll see them a lot of places around Yahoo. Are you a Yahoo Mail user? You’ll get the updates there. Are you a Yahoo Messenger user? Yup, there’s a tab in Messenger AND whenever there’s a new post in any of your networks, there’s a pop-up window (make sure you’re using Messenger 10). A My Yahoo user? Yup, you can add the Yahoo Updates module to your My page and get your fix that way too. None of those describe you? I bet you’re on some site of Yahoo at least once a week reading an article, in which case, you’ll see the site-specific updates from your friends in modules along the side. And, of course, you can always just go directly to to check and post your updates.
  • Lots of other nifty things (like apps, and a built-in blog, etc.)

Here’s a screenshot of what a Pulse feed looks like on My Yahoo:

I’ve been using Pulse as my primary social feed reader for nearly a year now. There are still some occasional hiccups with it, but overall, it works like a charm, and I get to leave myself logged out of Facebook (which, as you can see from my other blog posts, is important to me).

Give it a whirl – and don’t forget to invite me to connect!


* Disclaimer: As a long time employee, I make a conscious effort to not be too much of a shill for Yahoo – nothing’s worse than having a friend who’s always shouting about the same thing over and over, especially when it’s got something to do with their job. In fact, aside from the occasional tweet, I don’t think I’ve every actively promoted a Yahoo product, and certainly not in this blog.


A Few Selections from


I spent the afternoon yesterday browsing – I hope to make it an annual ritual. Such a high density of good stuff, thanks Chris (Anderson)! If you’ve never spent some time on the site, I highly recommend it. Instead of indulging your regular TV habit, spend the evening with some of the world’s great minds – it’s inspirational and entertaining!

Here were some of the ones that caught my eye – your mileage may vary:

  • Jamie Oliver’s TED Prize wish: Teach every child about food Jamie’s so passionate that he’s a bit all over the place, but his message is spot-on.
  • Peter Molyneux demos Milo, the virtual boy One demo of how the Kinect can be used. I just tried it out for the first time yesterday at a store, and gotta say, this changes everything.
  • Arthur Benjamin’s formula for changing math education A famous math prof from my alma mater (Harvey Mudd) says it best to forget calculus, we should be aiming towards stats and probability as the “apex” of the math curriculum. I’ve been saying this for years, so of course I agree.
  • Mark Roth: Suspended animation is within our grasp Mark takes his time getting to the point, but the idea is pretty big, and I love how it demonstrates an “aha” moment.

And these are for the data geeks in the crowd:

Happy New Year!


The Search Tools They are A-Changing


This evening, Yahoo is announcing some cool enhancements to their Search functionality. You should check them out, and I’m not just saying that because I work for Yahoo Search, but because after several years of posturing by the big search engines, the paradigms for search on the web are finally actually shifting.

The big three engines (Google, Yahoo, and Bing) have finally started to diverge, much like the auto industry did in its early years. At first there was a plethora of small companies, mostly making custom vehicles, with varying levels of quality. Similarly, in the early days of the web, there arose a plethora of search engines (Lycos, HotBot, Alta Vista, Infoseek, etc. ad infinitum). Eventually, when Ford showed the superiority of the assembly line, consolidation began, and there became a correct and accepted definition of what a car was. And likewise, Google led the way in ushering in high levels of relevance, comprehensiveness, and speed – with Yahoo (and later Bing) following in tow, and at times even surpassing the leader. But those features are just table stakes now. It’s the natural progression in a product development life cycle in a competitive market: first match your competition, then differentiate.

I’ve purposely used the term “Search Tools” in the title of this article instead of “Search Engines” because now that we’ve finally got the underlying engines (or, platforms, to use the technical term) in place for search, the fun is just now starting! Google, Yahoo, and Bing are all starting to really innovate and offer different ways of searching. Just like cars, each has its own personality. (And, just like cars, sometimes they share some of the core elements, like Yahoo and Bing are doing for their “algorithmic web” and advertising content.)

To continue the auto metaphor, if you had an SUV, an all-electric sedan, and a sports car sitting in your driveway, which would you use? The answer is: it depends. Going on a ski trip? Heading out to pick up the kids? Need to get out and blow off some steam? It depends. The same will soon be true of your search tool – and since we all have all of these tools at our disposal, why not use them all?

Google recently launched “Google Instant”, which shows you search results as you type. It is really slick and really fast. Yahoo doesn’t have it, and neither does Bing. It is a fundamentally different way of interacting with a search tool. Some people love it, some hate it. And you can choose whether you want to use it. Personally, I find it too distracting – I don’t always want to live my life like I’m hopped up on caffeine. But I am not you, and you may find it a godsend. Give it a whirl and see.

Likewise, Yahoo just launched some pretty cool features around entertainment searches (like searching for movies, actors, musicians, TV shows, famous people, etc.) as well as searches about newsworthy topics. We’re able to recognize these real-world entities and give you all of the most relevant information and the ability to get things done right there at the top of the search results page. As an example, try searching for “The Social Network” on Yahoo. Not only do you immediately see ratings, showtimes in your area, and a link to a trailer, but you can also buy tickets and if you are a Netflix user, you can immediately add it to your queue (and if available, you can Instant Watch it – I especially like this last feature – so cool!) And, as you flip through the “accordion” tabs of Stories and Twitter, you get see see the most recent and relevant content – at your own pace. We’ve collected all of the “good stuff” in one place for you to browse at your leisure. If you do another search like “Lady Gaga“, the accordion changes accordingly to include News, Events (Shows), Albums, Videos, and Twitter. And get this: if you search for “Lady Gaga albums“, we’re smart enough to take you directly to the right tab. Collecting all the good, trusted stuff in one place – isn’t that what Yahoo was always known for?

Just like you wouldn’t drive your sports car on a camping trip, you can’t expect to always get the best results from your trusty old search engine anymore. So I invite you to explore. If you’re a Google user, try Yahoo for a week. If you’re a die-hard Yahoo fan, give Bing or Google a try. And keep exploring, because this is just the beginning of some exciting stuff in search.


I Quit Facebook, I’m Rejoining Facebook


Last month I quit Facebook. Really, the full deal – complete account deletion – wiped clean. And I made a big deal about it, in multiple venues, including individual messages to all my friends and family. It was the right thing to do, and for the past month, my mental life has been a bit quieter.

But, when the factors leading to a decision change, one must re-evaluate the decision. That’s why some of my friends have started to see posts by me, and others have received invitations to connect, on Facebook. No, this is not the work of some “fake Chris LuVogt”, it’s really me.

Why the flip-flop? In short, three reasons:
1) Facebook changed their policies,
2) Yahoo Pulse launched, and
3) I am in the internet services business.

Facebook Policy Changes
Last month, FB made a major revamp of both privacy controls and privacy defaults. This was in large part due to pressure from industry leaders, Congress, and the press. My quitting was one drop in that tsunami. To be honest I didn’t think they had it in them, but I have to give them credit for doing the right thing. Sort of. To be honest, they pulled a typical “two steps backwards, one-and-a-half steps forward” move. By itself, the improvements were not enough to get me to rejoin, and I still don’t really trust them, but other factors came into play. The Electronic Frontier Foundation summed things up nicely – if you’re still a Facebook user, this is a must read: Facebook’s New Privacy Improvements Are a Positive Step, But There’s Still More Work to Be Done

Yahoo Pulse
Just recently, Yahoo launched a revamped social networking platform (called Pulse). To be honest, it is functionally very similar to Facebook, but with better privacy controls . But more importantly, it includes features that let you cross-post between the two networks, so that (at your discretion) a post you make on Facebook will show on Pulse, and vice-versa. [As a Yahoo employee, I invite you to give it a try – it’s pretty cool.] This marks a subtle change in how you can use Facebook – namely, it is now possible to use a non-Facebook client application (Pulse) to manage and view your Facebook stream. In other words, Facebook’s walled-garden is still walled, but the wall separating it from Yahoo has been largely broken down. And, more importantly, as an individual user, I am now able to use Facebook’s backend without contributing directly to their revenue stream (of course, I am contributing indirectly in a number of ways just by having an account and friends). This allows me to get many of the benefits of using FB, without leaving myself logged in and exposed to some of features FB has introduced that have me worried about my privacy.

My Job
I work as a minor thought leader at an large internet services company. As such, I have a responsibility to both keep abreast of developments in the industry, as well as support my company’s efforts (what some refer to as “eating your own dog food”). This, of course, was true when I first made my decision to quit, but at that time I felt the problems I had with FB outweighed this point. This balance has shifted now, and especially given the significance of the Facebook-Yahoo integration, I feel my obligation to have a working understanding of a frenemy’s product is even more important, and I cannot truly have a good understanding without actively using the product.

Ramping up Slowly
I enjoyed my vacation from Facebook too much to jump back in with both feet. One of the nice side-effects of my quitting is that I have re-examined how I use the service, and I’ll be slowly and carefully ramping up my usage and friends list over the coming months. If you’re a friend or colleague, you can look forward to invites from me on Facebook, Pulse, and/or LinkedIn.



I love Facebook, I’m quitting Facebook


Two years ago, I debuted on Facebook with the unassuming status update ‘i been hearing bout this “so-shull networkin” thingiemabob – what’s it all about anyhoo?’, to which my first friends replied ‘uh oh – you’re going to get sucked in!!’ and ‘watch out for the ads. they’re evil’. Both were right. I have immensely enjoyed the ability to connect with friends, both current and long-lost, as well as actively participating in the “social web” revolution.

But now I need to join another revolution – Facebook users revolting against the abuse of power and clear lack of respect for privacy Facebook has shown repeatedly and increasingly. So, I’m deactivating my account. I don’t take this decision lightly – I get immense value out of Facebook. But I liken it to fast food – sure it tastes great, but at what cost, and do i really want to support a company that does not respect me as a customer and a person? I am not the only one doing this – some Googlers are also in on the movement: “Google engineering gaggle flees Facebook – The Register”

Luckily, I have been very careful to not store all of my data in Facebook’s cloud, so this is not so hard. Also luckily, there are alternative ways to connect online. I actively use Flickr for photos, Yahoo Updates for social networking, LinkedIn for professional networking, Youtube for videos, and Twitter for technology and political ranting. I don’t want to lose touch with you all, so please do connect with me on those services where it makes sense:
I don’t use Google Buzz, mostly on principle – Google released a product called “Buzz” many months after Yahoo already had a similar product with the same exact name – I consider that the ultimate in bad form and poor sportsmanship!

I do not expect many, or any, of you to join me on this quixotic journey. But, here are some articles from the past couple of years for you to take into consideration (reverse chronological). You make your own decision:


Common Sense


FINALLY! A replacement for the MPAA has emerged…

For those of you with kids, I thought I’d share this web site I just discovered – it’s called Common Sense Media.

Basically, it is a non-profit, non-partisan organization that rates and reviews all sorts of media (Movies, TV, Games, Books, Music, even Web Sites) with respect to age appropriateness. It is similar to the MPAA (the folks who rate movies G, PG, etc.), but Common Sense is not hand-in-hand with the industry like MPAA is*. Also, the Common Sense reviews go into much more detail about the many dimensions that affect whether you want your child consuming the media (sexuality, violence, language, social behavior, commercialism, drug use, etc.), allowing you to make a much more informed choice.

And, for those of you who use Netflix, the ratings are included directly on the Netflix site (that’s where I found out about them).


* As the documentary film “This Film is Not Yet Rated” exposes, MPAA has close ties with the big studios and is actually a very secretive, private organization with very strange biases (against sex and for violence). BTW, before you go and watch this film, please read about it on Common Sense – despite its title, it actually is rated (NC-17) due to prolific inclusion of clips from censored movies.