How did you initially develop an interest in CS and what was the
point in your life when you knew you wanted to pursue a career in CS?
I suspect this
story is similar to millions of other computer geeks. I read a book as a kid
that included code to make the computer say funny things. Then I figured out
how to make it ask questions, and differentiate its responses based on the
answers. I was hooked from that moment on. It’s an amazing moment when you
first understand that a computer is an artifact of embodied intelligence. You
pour into it some set of instructions, some vision of the world, and then you
turn it loose to deal with situations you couldn’t possibly anticipate. It’s
like a logic puzzle where you can’t know all of the rules. What could be more
fun than that?
Was there a specific person who influenced your interest in
computer science? If yes, please describe.
My high school
computer science teacher, Richard Watson, was a huge influence. I not only took
his classes, I also did what was called a ‘co-op’, where students work at real
jobs for course credit. Because I grew up in what was then a small town, there
weren’t a lot of co-op opportunities within walking distance, so I was allowed
to work with Mr. Watson as his teaching assistant. He took great pleasure in
standing me at the front of the room and announcing to the class what the topic
of today’s lecture would be, and handing it over – with absolutely no
preparation. You sure learn to think on your feet quickly in that situation!
Like many schools, there was no one who had the job of system administrator,
Mr. Watson had a whole cadre of co-op students, who were each trusted to
administer the school’s network, run its website, fix the computers etc. We all
worked with confidential information, and a lot of trust. That program
graduated whole generations of computer scientists who work all over Canada and
around the world. A bunch of kids from a small town in southern Ontario
accomplished amazing things when given the right opportunity and guidance. The
power of education is staggering.
What led you to specialize in HCI?
When I was a
teenager, I was given the gift of a book called “the Psychology of Everyday
Things”; it helped me to realize that all of design is about communication.
Physical objects communicate with as simply as by having a particular physical form
and software communicates in its appearance. Most of computer science is about
the creation of artifacts which exist in a very real sense. A great deal of
computer science is dedicated to achieving particular aims of capability,
capacity, and efficiency. It all happens within an implicit understanding that
these properties are desirable and will yield a better user experience. The
natural extension of this is in determining the capabilities and capacity of
the person using the artifact, and engineering it to maximize the potential for
In high school you participated in a DCS outreach event. How
did this experience influence you? As an educator, how important do you feel
outreach events / programs are and why?
Yes, I was a
“Computing Insights” camper. This is an absolutely amazing program. There is a
lot of power in programs like this: they help otherwise isolated geeks to
connect with one another and explore their passions, and they give kids
exposure to the whole breadth and depth of computer science as a discipline.
Before I participated in that program, I thought that computer science was
another name for programming, by the end, I had been exposed to all manner of
disciplines in the field. When Professor Rackoff taught us about computational
complexity, that there exist classes of problems to which no one has ever
found an efficient solution, it opened my eyes to the reality that computer
science isn’t about computers at all, but rather about all manner of our
interaction with information, with one another, with technology, and with the
world around us. I was fortunate enough to work as a TA for that program one
summer years later, and seeing other kids make the same connections was
Describe your experience as a CS student at the undergraduate and
graduate level. Do you have a specific memory that stands out and exemplifies
your years as a student?
I had a bit of
an odd undergraduate experience, because I was given the opportunity to be a
teaching assistant, and then a course instructor. By time I graduated, I had
taught half a dozen courses to a few hundred fellow undergraduate students. It
was a huge responsibility and opportunity, which, looking back now, seems
really risky for the department. It taught me an incredible amount, but in a
way it was also isolating, since I wasn’t really a ‘normal’ undergraduate
student, nor was I a faculty member. I couldn’t hang out in the labs with my
classmates, because my students would also be there.
How did you develop your career at MS?
works at Microsoft is, in a technical sense, among the best in the world. Raw
talent is just the table stakes. Making a career at Microsoft is about finding
the genius of the people around you, building a great team, and developing and
pressing a vision. I started working at Microsoft with a list of “14 things”
that I believed needed to be done to build fundamentally new user interfaces
that would truly leverage the capabilities of touch and gesture. Every day was
about moving that agenda forward, not by myself, but by empowering the people
around me and infusing the teams with that vision. In a certain sense I was
lucky, because Natural User Interfaces was emerging as a major theme for the
company, and I was able to be a big part of where that vision is now. My book
is as much about the experience as it is about the technical details of
creating these interfaces.
Describe your experience at MS. What’s your best memory of your
career at MS?
Microsoft is a
wonderful place, filled with incredibly creative and talented people. The
company has turned 180-degrees from the version I read about before I went there.
Its focus is on individual empowerment, and every single member of a team has a
voice and an opportunity to influence a product. This can be both a blessing
and a curse: it creates huge opportunities for all, but for someone with the
job of spreading knowledge of HCI across the company, it meant there was no
higher authority to focus on – everything had to be bottom-up. I had to meet
with literally hundreds of people and give dozens of talks in order to make
progress in achieving our vision for the future of HCI. A rather popular
training class at Microsoft is “influence without authority”, which basically
means how do you get people to do things when you have no authority to make
them do it. That this course is so popular really tells the story of what life
is like at the company.
In terms of a
best memory, it’s hard to pick a particular favourite moment. Steve Balmer
using my material in a keynote speech seen by hundreds of thousands of viewers
was pretty wild. So was the first time I showed my friends and family the
Surface toolkit that my team built. Really, though, it’s in the messages I
still receive from people from my team who have gone on to do amazing things
both within and outside the company. A few are considering coming to Toronto to
do graduate work in my lab, which is incredibly gratifying.
Describe your role at MS and some of the projects you have been
In 3 years at
Microsoft, I had 11 different job titles (no kidding) - it was a bit of a
whirlwind. Throughout, though, the core goal was the same. There is a
deficiency in computers as they exist today in meeting the whole of human
capacity. Computers’ understanding of the world is limited to tracking the
position of a mouse and the presses of keys – everything else going on around
them is missed. We are at an exciting time where technologies are becoming
cheaply available that vastly increase the machine’s understanding of the user,
enabling things like gestures, touch, speech, and so on that vastly increase the
potential bandwidth. The challenge for HCI is in crafting user experiences
which enable users to take advantage of this huge bandwidth. We’re just
scratching the surface of this now, but it’s happening really quickly.
I started out
as a designer, which was an odd position for a computer scientist, but it was
an opportunity to do HCI work on Microsoft’s Surface table, which I really
wanted to do. >From there, my role expanded to a team lead, then a manager,
then the architect of user experiences for Microsoft Surface. After that I was
asked to take-on a broader role consulting across the company for products with
innovative user experiences. Some of the products I consulted on have shipped
already, like Surface, Windows 7, Windows Phone 7, and Kinect, others haven’t
yet, which I can’t talk about. After I told Microsoft that I was planning to
leave to join U of T, I was offered the position of Researcher at Microsoft
Research for a 6-month stint, where I have had the opportunity to work with
some incredibly talented researchers and interns from universities around the
world. One of my favourite research projects is a tool, AnatOnMe, developed
with Tao Ni and Amy Karlson, which doctors use to project medical imagery onto
patients, so that patients can ‘see into’ their bodies to better understand
their medical conditions. Another is “Rock and Rails”, developed by a whole
team of folks, which is a gesture system to enable computer animators to use
large drafting-table sized touch computers to do their work. Around the
company, I’m best known for a set of design rules I dubbed “No Touch Left
Behind”, which was basically a way to make touch systems seem more reliable and
consistent. The term has been used in a few press interviews by executives,
which is kind of funny – I meant it to be catchy, but perhaps it was a tad
impolitic. I guess it’s what happens when a Canadian gets to name stuff
at a big American company.
Why did you decide to leave a successful career in industry to
work in academia? (I know that we talked about this when we met but I would
like to include additional details). At what moment did you make this decision
or was it a long-term plan?
Two reasons: I’ve
known since high school that I would be an educator. That was a big part of my
job as a manager at Microsoft, and I took great pleasure in teaching classes as
an affiliate professor at the University of Washington. But I always knew that
if the right opportunity came along, I would take it. The University of Toronto
and a handful of others were exactly the caliber of school that I had my eye
on. That it happened after just a few years after I began my career in industry
was as much lucky timing as anything.
That there is a
huge need for education in HCI became apparent to me when I was growing my team
of designers. I would interview recruits, each of whom would show me their
portfolio, which was always a collection of websites or applications. There’d
be screen shots, and the designer would say “here’s the first screen, here’s
the second screen,…”. I’d interrupt and ask “how did the user get from the first
screen to the second screen?” They’d
answer “they clicked the link”. I’d then ask them “what’s a click?” I was blown
away that of the 20 designers we interviewed for that job, all from absolutely
top design schools, some with decades of experience, only 1 was able to fully
explain a “click” to me. They’d know the user presses a mouse button, but what
happens if the mouse pointer is over a link, they press down the button, then
slide off the link before they lift the button? To these 19 designers, this was
so basic, so fundamental, it had never come up. Needless to say, we hired the 1
who had thought deeply about it, and he’s been incredibly successful at
building fundamentally new interfaces, because he understood the mechanics, the
relationship between physical interface and software. I’m back in academia for
the sake of those other 19.
What do you feel are the most unique characteristics of the
department that make DCS stand out from other CS programs?
To someone on
the outside, computer science might seem like a single field, but in reality,
it’s a collection of widely divergent disciplines. Most university departments
have a specialty that they cater to. It might be software engineering,
artificial intelligence, HCI, databases, whatever. U of T’s DCS has such
incredible depth – there are absolutely top people in every area of CS. It’s
hard not to be inspired by coming to work every day with such incredible
Why did you choose to come to UofT?
I always knew
that academia was ultimately in my future. U of T, as a university, is such an
active part of the community in Toronto and in Canada. It’s not an ivory tower,
but a hub of innovation and opportunity. I had the opportunity to do several
months of my PhD work at Harvard, which has a similar relationship with the US
that U of T has to Canada. I knew that, after having had the kind of influence
I had the opportunity to wield at a company like Microsoft, I would only be
happy at that kind of a school. I don’t think of my position as limited to
‘academia’. I am excited to be an active participant in the Canadian tech
industry, and look forward to seeing my students flourish there.
to me about U of T is the sheer number of undergraduate students who are given
the opportunity to study here. It can be daunting, to be sure, and a huge
challenge to distinguish yourself, but the opportunities to work with some of
the world’s leading researchers in just about every field is made available to
tens of thousands of undergraduate students every day. In that way, U of T is
unique in the world.
As an educator what is your biggest hope for your students? What
are the primary messages you would like to impart to your students?
said “Never doubt that a small group of thoughtful, committed citizens can
change the world. Indeed, it is the only thing that ever has”. I believe
absolutely in this. All great accomplishments are achieved through equal parts
opportunity, capacity, and drive. It’s my job to provide my students with the
opportunities and direction to enable them to accomplish their goals. Their
responsibility is to meet this opportunity with the drive to see it through,
and to understand that there is no reason that their project can’t be the
beginning of the next Google (which itself grew out of a computer science
department). Simply having a good idea is only a necessary condition, not a
sufficient one. The world is full of great ideas - the ones backed by drive and
ambition are the ones that change the world.
What are you looking forward to the most when you officially begin
your position in January?
with my incredible colleagues both at both St. George and at Mississauga,
participating in graduate admissions for 2011 to start building my team, and
connecting with the great tech industry in Canada. There are some amazing
technologies coming out of Toronto and Canada, and I’m really looking forward
to being a part of this culture of innovation. I recently have been giving a
few guest talks at industry events and at other great institutions across
Canada. The energy here is palpable, and U of T is at the centre of it all.
Are you nervous about any elements of your teaching career?
It’s funny, but
being on teams developing the next-generation of technologies can also be
isolating from technology trends. I was using my iPad at a restaurant a few
weeks ago and the waitress asked me what it was. I was floored. When you live
and breathe this stuff, it’s very easy to forget that it’s not always a major
part of people’s lives. I was talking with someone recently who pointed out to
me that kids don’t wear wristwatches anymore because they all check the time on
their cell phones. At the same time, students now live and breathe these
technologies. A frosh in my classroom next year was born the year my family
bought its first handheld cell phone. I remember being in high school and
teaching friends about the Internet – now, every single one of my students will
have a web presence on Facebook, YouTube, Twitter, and others. They are all so
facile with technology. HCI used to be so easy to sell to students – I’d show
classrooms a few concept videos of gestural or speech input, and they’d be
blown away. Now, that’s all just par for the course – HCI is mainstream.
Products actually differentiate themselves based on their user experiences.
These students in a world where the next technological breakthrough is only a few
months behind the last. I suppose this is all by way of saying that I’m
starting to feel a little old! But I look forward to the challenge of engaging
with these students.
Which courses will you be teaching? How have you made your mark on
I’ll be teaching a graduate course which basically follows the lessons I was
teaching around Microsoft. I currently am also planning out an
undergraduate course using my book as a frame to teach the design of
next-generation user interfaces. I’m excited to be creating classes that really
deeply engage the students to work in design teams, like they would in
industry, and to think really deeply about the marriage of hardware and
software in creating user experiences.
Though I love
teaching HCI, my absolute favourite course to teach is “ The how and why of
computing”, which is a first-year course intended for non-computer scientists.
It’s a blast to teach, because it’s an opportunity to turn non-techies into
converts. I have a small collection of letters from students who changed their
major to computer science after taking that class, and I treasure each and
every one of them.
opinion, what direction is HCI technology headed in?
future of technology will be increasingly specialized. If you look at a laptop
today, the form is a function of the UI – it is the shape and size that it is
because of the screen, keyboard, and Trackpad, the other electronics have
become so small that they can fit almost anywhere. The rise of the e-book
reader and slate computers like the iPad, which exist alongside and rarely
instead of a primary computer, demonstrate the society of devices we will each
soon interact with. The era of generic UI design for mouse input will soon be
behind us - the best user experiences will be marriages of hardware and
software, with the software specifically designed for the form factor.
input and display technologies are all about increasing and aligning the
bandwidth of the communication channel between user and computer. At the most
basic level, every computer can achieve the same states, as described by Alan
Turing 80 years ago – I’m not talking about making computer do things they
couldn’t be told to do with a mouse, I’m talking about doing it in a new way.
Gestural UI will never replace the mouse and keyboard. Instead, they will all
exist together, with increasingly specialized software enabling unique user
experiences which leverage a given technology.
What are your hobbies / interests?
In Seattle, we
lived on a floating home; just down the lake from the one Tom Hanks lived in
Sleepless in Seattle. We’ve become such water rats, spending every non-working
daylight hour on the lake. It’s hard not to be a kayaker when your boat is
floating off the back porch; I’m looking forward to spending time on Lake
Ontario. I’m also a bit of a baseball nut. I can’t believe it’s not called the
“Sky Dome” anymore! I’m looking forward to taking in a whole lot of Jays / Red
Sox games with my Bostonian wife.