MIT Media Lab Fluid Interfaces-Prof Maes Team 2022 Top Research Powering Humans

Professor Pattie Maes deep insights working with her research team of Joanne Leong, Pat Pataranutaporn, Valdemar Danry are world leading in their translational research on tech-human interaction. Their highly interdisciplinary work covering decades of MIT Lab pioneering inventions integrates human computer interaction (HCI), sensor technologies, AI / machine learning, nano-tech, brain computer interfaces, design and HCI, psychology, neuroscience and much more. I participated in their day-long workshop and followed-up with more than three hours of interviews of which over an hour is transcribed in this article.

All insights in this article stem from my daily pro bono work with (now) more than 400,000 CEOs, investors, scientists/experts.

MIT Media Lab Fluid Interfaces research team work is particularly key with the June 21 announcement of the Metaverse Standards Forum, a open standards group, with big tech supporting such as Microsoft and Meta, chaired by Neil Trevett, Khronos President and VP Developer Ecosystems at NVIDIA. I have a follow-up interview with Neil and Forbes article in the works.

In addition, these recent announcements also highlight why Pattie Maes work is so important: Deep Mind’s Gato multi-modal, multi-task, single generalist agent foundational to artificial general intelligence (AGI); Google’s LaMDA Language Model for Dialogue Applications which can engage in free-flowing dialogue; Microsoft’s Build Conference announcements on Azure AI and OpenAI practical tools / solutions and responsible AI; OpenAI’s DALL-E 2 producing realistic images and art from natural language descriptions.

MIT Media Lab Fluid Interfaces Research Team

Pattie Maes

PATTIE MAES is a professor in MIT’s Program in Media Arts and Sciences and until recently the chair of the Media Lab’s executive committee. She runs the Media Lab’s Fluid Interfaces research group, which does research at the intersection of Human Computer Interaction and Artificial Intelligence for health, wellbeing and learning applications. Maes is also one of the faculty at MIT’s center for Neuro-Biological Engineering. She is particularly interested in the topic of “cognitive enhancement”, or how immersive and wearable systems can actively assist people with issues such as memory, attention, learning, decision making, communication, wellbeing, and sleep.

Maes is the editor of three books, and is an editorial board member and reviewer for numerous professional journals and conferences. She has received several awards: Time Magazine has included several of her designs in its annual list of inventions of the year; Fast Company named her one of 50 most influential designers (2011); Newsweek picked her as one of the “100 Americans to watch for” in the year 2000; TIME Digital selected her as a member of the “Cyber Elite,” the top 50 technological pioneers of the high-tech world; the World Economic Forum honored her with the title “Global Leader for Tomorrow”; Ars Electronica awarded her the 1995 World Wide Web category prize; and in 2000 she was recognized with the “Lifetime Achievement Award” by the Massachusetts Interactive Media Council. She has also received an honorary doctorate from the Vrije Universiteit Brussel in Belgium and has given several TED talks.

In addition to her academic endeavors, Maes has been an active entrepreneur as co-founder of several venture-backed companies, including Firefly Networks (sold to Microsoft), Open Ratings (sold to Dun & Bradstreet) and Tulip Co (privately held). She is an advisor to several early stage companies, including Earable, Inc, and Spatial, Inc. Prior to joining the Media Lab, Maes was a visiting professor and a research scientist at the MIT Artificial Intelligence Lab. She holds a bachelor’s degree in computer science and a in artificial intelligence from the Vrije Universiteit Brussel in Belgium.

Joanne Leong

JOANNE LEONG is a PhD Student in the Fluid interfaces Group at the MIT Media Lab who is passionate about research in the field of Human-Computer Interaction (HCI). Her recent research examines how AI technologies can be used to synthesize new perspectives, of oneself and others, for self-growth and cognitive augmentation. Her approach involves prototyping novel systems and experiences and evaluating their effects through user studies.

Joanne’s work has been published at a number of top-tier HCI conferences, including two best papers in UIST 2016 and TEI 2017 respectively. She has also interned at multiple companies, such as Snap Inc. and Sony. Joanne holds a master’s degree in Media Arts and Sciences from MIT, a master’s degree in Interactive Media from the University of Applied Sciences Upper Austria, and a bachelor’s degree in Systems Design Engineering from the University of Waterloo, Canada.

Valdemar Danry

VALDEMAR DANRY is a researcher, artist and tech humanist at Massachusetts Institute of Technology. He is part of the Fluid Interfaces research group at MIT Media Lab led by Professor Pattie Maes, which specializes in designing technology for cognitive enhancement. With his background in philosophy of mind and artificial intelligence, he seeks to explore how the augmented body mediates our identity and experiences of the world not just as something physical but as something which is “lived”.

He has worked with international collaborators from MIT, Harvard, Georgia Tech, IBM Research, and University of California, and has exhibited his work at Ars Electronica, IDFA, Electrical Artifacts and at the MIT Museum. His research has been published in impactful journals and conferences like “Nature: Machine Intelligence”, “ACM CHI”, “IEEE Computer Graphics and Applications”, and “ACM Augmented Humans” with more research under review.

Pat Pataranutaporn

PAT PATARANUTAPORN is an antidisciplinary technologist/scientist/artist at the Massachusetts Institute of Technology (MIT). He is part of the Fluid Interfaces research group at MIT Media Lab led by Professor Pattie Maes, which specializes in designing on-body technology for human enhancement. Pat’s research is at the intersection of biotechnology and wearable computing, specifically at the interface between biological and digital systems.

Pat had worked with global collaborators such a NASA TRISH, IBM Research, Bose, Harvard, UCSB, ASU, NTU, and the Futuristic Research Cluster of Thailand (FREAK Lab) to examine the symbiotic relationships between human and technology. His interdisciplinary research ranges from investigating human-AI interactions, developing wearable lab on the body with programmable bio-digital organ for space exploration, developing machine learning model to detect linguistic markers related to mental health issues, developing, and designing mind-controlled 3D printer.

Pat’s research has been published in IEEE, ACM SIGCHI, ACM SIGGRAPH, ACM ISWC, ACM Augmented Humans, Royal Society of Chemistry, etc. He also serves as reviewers and editors for IEEE and ACM publications. Pat’s artistic projects had been exhibited at National Museum of Singapore (Singapore), Essex Peabody Museum (USA), London Design Festival (UK), Transmediale Festival (Germany), National Taiwan Science Education Center (Taiwan), IDEA Museum (Arizona), Mesa Arts Center (Arizona), Autodesk Gallery (California), and more.

Pat and his team’s projects have been featured on FastCompany, Time, Disruptive Innovation Festival, National Geographic, The Guardian, and UNEP. Pat believes that innovation must converge aesthetics, functionality, and community to create a sustainable future. Pat is the co-founder of several tech startups, and Freaklab (Futuristic Research in Enigmatic and Aesthetics), an open research lab in Thailand focusing on designing the future and beyond.

MIT Media Lab Fluid Interfaces Team Interview

The video interviews are found with the non-profit IEEE TEMS (interviews series – Stephen Ibaraki). IEEE is the largest non-profit technology engineering organization known for their conferences, journals, and standards work. AI is employed to generate the transcript which is then edited for brevity, clarity, summarized in parts. AI has about an 80% accuracy so going to the full video interviews are recommended for full precision. Due to the length of the interviews, only portions are provided in this article. Time stamps are provided however with the caveat that they are approximate.

Stephen Ibaraki 00:00

Thank you for coming in. MIT Media Labs, you’re number one in the world, in innovation and transformation; you take on the impossible. And then it becomes the possible or the near impossible. Patti, as the director, thank you for coming in. And then three members of your team; Pat, who I’ve already talked to before; Valdemar, all the great work you’re doing. And then I find out Joanne, she’s comes from Canada. So again, I really appreciate your sharing your experiences, and this just wonderful workshop I attended << Full Day Workshop on Virtual Beings & Being Virtual >>, that you did earlier. It’s just so much great content, ways to interact, and so on, and so many insights, it’s just amazing. So again, really appreciate you coming in. Thank you.

Stephen Ibaraki 01:17

I’ll probably start the questioning, first of all with Patty, because your career, it just goes back so many years, many awards, many accomplishments and continuing so that you can set the framework of this conversation, the work you’re doing at the lab at Fluid Interfaces; setting the context for the work that your team is doing as well. What is the purpose of the lab? And what are some of your immediate and medium term goals?

Pattie Maes 01:50

Yes, well, let me tell you briefly first about the Media Lab, and then about my lab within the Media Lab. So the Media Lab really looks at emerging technologies, and how they will shape society. And we’re especially interested in emerging technologies that can empower people and empower people to get to know themselves and really work on themselves and increase their own potential, as well as their communities and their environments. So a lot of what the lab does, is looking at new technologies, new tools that can benefit people and their communities and their environments. Now, within the Media Lab, we have about 25 faculty or so. And I run a research group where we look at the intersection between artificial intelligence and human computer interaction. So we’ve for many years now, 30 years or more, been looking at how AI technologies will make interfaces smarter, and we will change the relationship that we have with machines basically. Now, one of the topics within that sort of field of AI and human computer interaction is that of virtual humans, virtual characters, actually, for many years been interested in this concept of agents and natural interfaces that you can communicate with, converse with, in a natural language, the way you communicate with another person. And while many years ago, it wasn’t really practical to build this yet. We actually feel that right now. There’s just been so much progress in AI, and specifically generative AI, but also these language models, that we can start thinking about creating virtual humans that people will interact with, to maybe as their teachers. I mean, the virtual humans will act as teachers, they will be salespeople, they will interact with us in virtual games, in for interactive entertainment, forums, and more, so that technology is really at the cusp now; I think of where it’s gonna have a major impact on society. And so we are exploring both the positive use cases but also the warnings; or the things we should be careful about, of all of this technology being used in all these different applications.

Stephen Ibaraki 04:51

Pattie, that’s sort of amazing context of the work you’re doing and to give the audience additional context: the penetration of the internet will be over 70% of the world this year. It was in the 60s last year and people spend over six hours now on their mobile. Mobile has AI built into it often, right? Or like the Apple, A15 bionic chip with 15.8 trillion transistors or something like that; 15 trillion operations per second by AI is sort of an integral part of that. Or this idea of the metaverse which foundationally is built based on AI and this idea of avatars. And I think Goldman just recently said, it’s a 12 trillion dollar marketplace. Others have said it’s in the hundreds of billions, so it’s an amazing sort of journey that we’re going into and transitioning to. And really, you’re leading that work. Kudos to your team, and you’re leading it for Tech for Good, right, because you’re looking at all of the aspects of this work. Okay, let’s go to some of the members of your team here. Let’s go to Joanne. Joanne and I just had a conversation—just before this session here today. I learned that you were an undergrad at the University of Waterloo, and I’ve got some great friends there. Then you went on and do two masters. Can you talk about how you fit into this philosophy? Or this theme that Patti outlined?

Joanne Leong 06:32

I would say that my entry point into this work was thinking about the areas of self development. And for me, I think, in my international travels, I had an interest in learning languages, and how to communicate well with people, how to work well with different people. So when I started on this topic, and actually I was saying before, in our earlier conversation, I began with Pat, I was thinking, well, how can we use this technology to develop new perspectives on ourselves. I’ve done two studies within this area, looking at what it means to show us different possibilities of who we can be. One by changing our visual appearance using real time camera filters, and another method where I tried to take the abilities of one person and kind of superimpose it on our own self image, to try and see if that can inspire us to see ourselves with having improved abilities, or just giving us this kind of freedom to be more creative.

Stephen Ibaraki 07:48

That’s really fascinating research. Are you finding then that there’s some positive aspects to taking on enhanced personas as within this digital world as an avatar, for example?

Joanne Leong 08:03

Yes, so actually, we’re also building on a lot of work that comes in virtual reality. And there’s a lot of evidence to suggest that if we see ourselves in a different kind of body in VR, that we can be influenced in terms of how we make decisions or how we perform on tests. So in the studies that we’ve conducted in our lab, I would say they’re quite preliminary, where we’re trying to take that same principle and see if it applies. When we use camera filters, say in this very Zoom call, I’d say our early findings suggest a tendency that people for example, if they see themselves as a child, or if they see themselves as a kind of inventor figure, that they seem to have a tendency to give more creative responses. But I would also add that we haven’t found statistical significance for that yet. But there is an exciting suggestion that these filters can play a part, not just for entertainment, but they might actually help us to be productive in this sense.

Stephen Ibaraki 09:12

I’m freeform thinking there. There’s the epigenetic effects of your environment, which can be generational in terms of its impact. And that’s because there’s these methylation markers that appear in your DNA. And that can even influence aging. But so if you have this idea of a superposition of what’s occurring on the digital side on you as an individual, and that can influence you; maybe can influence you in a positive way, for example, on an epigenetic side, on these methylation markers on your genetic structure and the expression of genes. I just think it’s amazing work. Okay, let’s go to Valdemar. Valdemar we just had an immediate earlier discussion just before the live interview here, and you talked about, the philosophical side and how that integrates into the work you’re doing. Can you talk about that how that fits into Patty’s overall theme of the purpose of the Fluid Interfaces Lab?

Valdemar Danry 10:21

My background in philosophy made me kind of have a lot of interactions with previous philosophers that have passed away. And I got a lot of exposure to logic and a whole tradition of using logic as a method to process information and look at certain structures and get a better idea of truth. And when here in working in Pattie’s group, so what I’m really interested about in the context of AI generated characters, is looking at how we can; how we are processing information with these kinds of new technologies, like, you know, deep-fake technologies, and also these big language models. Some of the things that we found with that is that, or that we’re really interested in looking at is, what if this information that we’re that we get is delivered by a person with a certain appearance, or like an AI agent that has a certain appearance? What if this AI agent, the way that it tells you certain information, if that makes it more believable, and especially if this AI agent is trying to deceive you? You know, if it’s a malicious AI agent, that spreads information, right now, using these big generative models, we can produce very convincing lies. And so one thing that we’ve been looking at a lot recently is also when these AI agents lie to you, how do people respond to that?

Stephen Ibaraki 12:02

That’s fascinating. Are you able to identify markers when when people are lying? Then you could do safeguards in some way? That’s in real time?

Valdemar Danry 12:16

Yes, so what we did for the work we’ve done on AI systems lying is that we had the AI and AI system using language models that exist right now generate explanations for why a certain news headline was false, or why it was true. So that means; then we would give this to people, and we would see how they responded to this, these explanations of why a headline was true or false. What we found was that when people get feedback where the AI system is honest, so where it says that a headline is true, when it actually is true, or when a headline is false, it says it’s false, when it’s actually false. We see that the accuracy increases in people being able to tell what is true and what is false. But then when the AI system makes a lie, when it deceives people and says this headline is actually false, when it’s actually true. Or it says this headline is actually true. When it’s a false news headline, then people have significant decrease in their ability to tell if it’s true or not; showing that people actually seem to rely on this AI system, when it gives very elaborate explanations. And that’s something we find quite terrifying.

Pattie Maes 13:40

Yes, so basically, people trust, sometimes trust these AI systems too much. And so they will also trust them, when they give us bad advice, especially if they give some fancy explanation for why that advice is supposed to be good advice. So that’s what we learned that people will actually then make less good decisions than in the case where they were not assisted by this AI system in making their decision.

Stephen Ibaraki 14:16

That’s really fascinating, because you know, that this idea of the ethical aspects and the ethical frameworks around AI have been around for some time, there’s over, I would measure now, over 300, frameworks and principles, and they’re operationalizing. This operationalizing has been driven now for a number of years and you see the IEEE with their P 7000 work; the ISO with their work; ACM with the code of ethics, and then even government’s looking at saying, you know, should we put some kind of regulatory framework and how do we do that, but your work could be key to that, because you’re looking at this in a very concrete way of Explainable AI and seeing some of the problems…Are you working with governments right now then, on this work?

Pattie Maes 15:00

We’re not working with government governments actually, I mean, we’re aware of this other work that is happening on AI and ethics. But I think that often the approach that is taken is limiting, actually, because people are mostly focused on making sure that AI systems are not biased, that the data isn’t biased, or that the AI system is more accurate, or that it provides explanations or that they can be tweaked by a human etc. But they don’t look at the combined system of an AI and a human or human being assisted by an AI. And what happens in terms of that combined system making decisions. So that’s really where we where our work is focused; we look at what happens when you put an AI in a context where a person is using it to make decisions. And what we are learning is that even if say, the AI, comes up with great explanations and doesn’t have biased data ..etc…, that there are many human factors that really determine the outcome of that combined system, the decisions made by the human assisted with the AI. And depending on what that AI looks like, or how that AI provides its explanations, and so on, all of that matters in terms of how the person will take that advice and what they will do with it. And that interface of the AI can really be manipulated, to make people believe and trust the AI more or trusted less, and so on. So we think it’s very important to look at these human elements, basically, these human factors of what will happen when these AIs are making recommendations. And they already are doing this to people in very critical contexts, and where can that go wrong and ultimately, result in the combined decision making of the human plus the AI being worse than that of a human by themselves, or the AI by itself.

Stephen Ibaraki 17:26

This is really profound, and really something that industry should be looking at it… more deeply all of the work that you’re doing, I’m just reminded that the World Economic Forum in 2020, came out with a future of jobs report, where they’re predicting over 80 percentile of the adoption of AI in just a few years. And you’re seeing similar kind of work from the big consulting firms. And in fact, I just saw a recent report where they’re saying even in an enterprise, it’s now it’s over 70%, at least some kind of pilot adoption, something like that. So you know, it’s going to proliferate. So definitely, industry has to be looking at what you’re doing, because that’s the immediate sort of outcome is going to be this interface between humans and an AI and your work is at the center of that, and government should be looking at that as well. And I’m just reminded of an interview I just did earlier this month with Tom Park of the Canadian government, Deep Tech Fund. They’re actually looking at Explainable AI…Okay, Pat, we did an (IEEE TEMS) earlier interview, because I caught your <<day 1>> keynote with the Translucia Metaverse Unlimited Forum <<here is day 2>>…175,000 live participants. I was just so impressed. I was amazed at what you’re trying … Can you talk further now and sort of treat this as a separate discussion on Patty’s framework of the importance of your lab and Fluid Interfaces. How are you fitting into that theme, the nature of your work, and where you see it going?

Pat Pataranutaporn 19:16

Yes, thank you. I mean, when Pattie talk about the idea of using technology to sort of help people and augment people; it’s always sort of made me think of Doug Engelbart, who is the pioneer in our field. Doug Engelbart was the creator of the computer mouse in 1963. One thing that he said, that inspire me and kind of made me think of this work a lot is that he said that, a little computer mouse is a part of a bigger project for augmenting human intellect. Like the computer mouse is part of a bigger ideas, which I think is very profound, how visionary he is at the time. Now we have the technology to actually recreate ourselves, making something more sophisticated than making a computer mouse. We are using this virtual human for mediocre tasks … help them use as a tool to answer phone and things like that. So I think the question that I’m always asking is like, what is the potential of this? Now we have something much more powerful than computer mouse, can we use it to even, augment, or help us imagine a sort of broader vision of how we and AI can sort of lead to more, prosperity, and, more fulfilling life. I think one area of research that we do a lot in our group is learning; because I think learning is important, for all of us, we learn all the time. At the Media Lab, we always think about, magic, how can we make learning more magical? I think the Media Lab is always about magic and possibility and imagination, and, with the possibility of AI generated characters, we can use AI to allow us to learn from people that we admire, like, I really love Einstein. I want to dress up like him when I was younger, with white hair and put a mustache on. You know, or even Pattie, one of the reasons why I’m here at the Media Lab was because I watched Pattie’s TED talk, when I was in high school and I wrote an email to Pattie and she was kind enough to to wrote me back. But you can see that inspiration; travel through internet, you learning from someone or seeing somebody you admire, you get to change your life. With the possibility of AI generated characters, Joanne Wong and I, we have done a study to show that when you are actually learning from a virtual hero or virtual person that you admire, it lead to significantly improved learning motivation and learning experience, which I think is very profound, especially during the pandemic, where … Zoom…, meeting the teacher in this square grid, we can make that more magical. I think that will be very powerful. And I think, as Doug Engelbart says, this is a small project but talking about a bigger idea of how we and virtual humans can coexist and augment one another; at the end for a more meaningful human life experience.

Stephen Ibaraki 22:46

Let’s go back now to Pattie. Where do you see this progressing? Are there specific areas now that you want to address beyond the things that we already talked about?

Pattie Maes 22:59

So yes, maybe to make it a little bit more coherent, of what we’re talking about here, and what the different people on the screen here are working on; we are really looking at two different use cases for all this technology, where we can create these realistic looking characters that also increasingly can just have natural conversations that are seemingly intelligence with us. So two use cases; one of them is a use case where you interact with virtual character, for example, getting advice or the system helps you with a problem and so on. The other use case, which is what Joanne talked about earlier, is embodiment, a person can become another character, for example, on a screen with filters, or in VR, with the so called deep fake technology and the virtual avatar technology. So we’re really looking at both of these use cases. They both have a lot of positive use cases, positive potential in areas ranging from not just entertainment and customer service and things like that, but also learning health, helping people with sort of personal growth, like Joanne talked about. And a more controversial topic maybe would be digital immortality. How can we work towards one of these characters may be representing us after our death based on all the data that we left behind, not just videos and pictures and so on, but also things we said, the way we expressed ourselves and what we wrote and talked about and so on. Can if we have sufficient data about people, can we start creating, basically a representation of a person that is really like a clone, or it’s a new type of portraits, really, of a person that becomes an interactive portraits that is so much more deep than, say, a picture of a person or a painting of a person, or even a documentary or a biography about a person, it becomes this living interactive artifacts, really, that embodies a lot of the nature of the person.

Stephen Ibaraki 25:54

You know, so Pattie, I mean, again, that’s fascinating this idea that the two use cases, the one where it’s giving advice, but the other one where you’re the embodiment of something and another character could be somebody from your past and you’re bringing life to that. And in your workshop, which I thought was amazing, you had the VP of D-ID, and, of course, My Heritage and the application of that, and that technology was quite amazing and a lot of adoption. It’s also interesting that I was asked to keynote earlier this month, all the colleges…there was an interest in the metaverse and these ideas of how does this influence education, this aspect of advising but also this embodiment of characters. And there’s a high school district called, Anaheim Union High School District, in California, which is looking at the metaverse and really teaching this within their high school curriculum…they’re going to hit these two points that you’re talking about, right? That’s the embodiment for the education side, but also the advice side, and then Jeremy Bailenson, and his work at the lab in Stanford. So where the students have built a Metaverse. Are you actually, on the metaverse side, looking at how that works? Or are collaborating with people who are actually looking at the metaverse because it’s going to be interesting and used for advice? But it’s also going to be used for this embodiment that you mentioned as well, and the impact that it is going to have?

Pattie Maes 27:32

Yes, I mean, that word of the metaverse first was be resurrected recently, of course, by Zuckerberg. But of course, that concept has been around for 25 or more 30 years and was not at all invented by Meta the company, but rather by science fiction writers like Neil Stephenson. We have been doing work on virtual reality and learning for many years from before all … here joined my group. So there is an enormous potential to use what we now call the metaverse, but these technologies of virtual reality and so on. For learning, of course, learning can be a lot more engaging to a person when they can embody one of the founding fathers rather than learning about them in a book or something. Or you can be in a famous battle as one of the soldiers and I bet that you will remember all the facts and so on about that battle really well, when you’ve been there yourself, embodied as one of these characters. So there’s not just potential like that for increased engagement, but people learn with their entire bodies. And right now, we will either have them sit down in a chair in the classroom, or we have them look at a little screen at a video on a little screen to learn. And using sort of the whole space around us. All of our different senses and modalities has the potential really to again, make all of that learning stick more. So there’s many reasons why using these immersive environments and virtual reality—Metaverse are really interesting. Now I talked about sort of the embodiment but again, you have the same other use case there where you could be interacting with virtual characters, so that you can go into some Metaverse and interview Einstein or interact with other characters or maybe Einstein gives you a lesson there and shows you his trip passing trains and how he came up with the idea of relativity and so on. So there’s so much potential there. And what we try to do is explore that potential by creating prototypes, and testing them; doing user tests to really controlled experiments, where we have some people do something one way and some people learn something another way, and we can really evaluate, what was their experience? What were the learning outcomes, would they prefer to have material presented like this in the future, etc? So that’s the type of work that we do…it’s really trying to understand the potential of these technologies more so that then the businesses that many of which work with us and our members or sponsors here at the Media Lab, they can learn from the lessons that we have learned by creating a little bit of this future and testing it on their behalf.

Stephen Ibaraki 31:10

That’s really fascinating in terms of the work, and it really illustrates the importance. Also this idea of the devices that you’re creating, and you’re working with, as well. And just comes to mind that Apple is on the verge of releasing their AR VR sort of set of tools <device(s)>…you have all the other big tech companies are doing iterations, there’s work on contact lenses, which have embedded AR capability. Elon is potentially going to do some clinical trials … on his Neurolink mesh. Let’s delve into this sort of device area of this vision of your lab. And this time, I’ll go in reverse order. So we ended with Pat before. So we’ll start with Pat. From a bio medical engineering standpoint, and this intersection of the mind and neuroscience and virtual reality and all of the themes that Pattie talked about, where do you see this going?

Pat Pataranutaporn 32:21

I think when people talk about virtual character, or the idea of entering the metaverse, the device that we wear today, is still not very comfortable. That’s why people are not wearing them for a long period of time. But as you’ve seen, all technology is going to get better, and, much more compact and more, user friendly. Right? I think one thing that we can see in the future. And I think in our group, we try to kind of go beyond that and think about other modalities that you can do. Another PhD student…has done a study where she had another modality; was the smell, to the virtual reality, right? Because you’re in the metaverse and you’re in the restaurant, you don’t just want to see it, you want to smell it or even, you know, consume it yourself. So adding modality to to virtual experience, I think would be really important. Or another project that people in the Tangible Media Group at MIT Media Lab have been working on is adding the tangible, or the tactile feedback, make things graspable and tangible, right? So not just the visual part, I think, is really critical. Because at the end, if you want to really be in the metaverse, you need to sort of connect with all the senses, not just the visual, but the physical, olfactory, and other things as well. I think I mentioned it in the previous interview that a lot of our work also is about wearable computing, and wearable device that we can put on the body, from the wake state to sleep state, from things that can sense physiology to even biomarkers like molecules that are released from the body; so something that we have been working on with NASA, for example. And I think all of these are kind of helping humans and this virtual experience become much more seamless; as the name Fluid Interface, right? We want to make things more fluid and more integrated. I think that’s the future that we will see. And I think many times that we heard about new gadget or new device in the news, I think, our group, we always ask, what kind of evaluation or what kind of study have they done. And I think that’s also one of our strengths that we don’t just build the future or invent or imagine the future, but we do study to understand, how to actually improve people. And go beyond the hype into sort of trying to understand what is the hope for that to improve human condition? I think that’s sort of our strength.

Stephen Ibaraki 34:54

Again, this triggers some ideas as I’m interacting with you. Clothing manufacturers are doing work in this area of the integration, all sorts of sensors and the impact that’s going to have in clothing, so you don’t notice it. What impact will that have, or I have a Forbes piece on the Terasaki Institute for Biomedical Innovation (TIBI) , where they have all of these bio nano size kind of interfaces they’re working with the body and it’s so small that they are basically non-invasive. Bidirectional communication sensing that occurs; I can see your work supporting all of these different communities that are out there. Let’s go to Valdemar and your work. You’re interested in devices for sensing and evaluating reasoning patterns. You look at the sort of intersection of the mind and neuroscience and extended reality and artificial intelligence from a device standpoint. What are your thoughts on on this particular theme that we’re talking about right now?

Valdemar Danry 36:19

Yes, I think that’s a very interesting question. One thing I’ve learned is that it’s very hard to predict the future. But what we can do is we can try to plant some seeds. And we can try to see if, and imagine potential futures and see what happens. And one thing that I want to say specifically, in terms of what we’ve done research on that I’ve been involved in, is looking at these kinds of virtual reality systems, these very immersive systems. One thing that we found with the current devices that are there is that when people spend a significant amount of time in these metaverses, or whatever we want to call it, is that we see that when people then go out of these environments, and they go back to the real environments, they tend to get these strange, like bodily hallucinations, almost, where they feel as if they can walk through things, they feel like their hands are unreal. And they feel that other people are like bots, or like, you know, these kind of artificial agents. And so one thing, one question that let’s think is really interesting, is the affordances, that the technical or the technological limitations that we have right now, in terms of, as Pat talked about the sensory modalities that they afford? What like the lack of a consistency between the real and the metaverse world where you can do anything to these inconsistencies? And new possibilities? Do they leak into the real world? And do they actually cause us, you know, pain, or causes a lot of confusion in the real world? I think that’s a very interesting sort of a question that I think a lot about. And in terms of virtual humans, in these kinds of environments, and metaverses, one thing that we found, if you can, personally become a different avatar, and you can change your appearance, like you’re changing clothing, right? One thing that we found in this study was that people, they start to to dissociate their physical appearance from their identity, and their identity does not become their shape, or if they have hands or claws, or whatever, but their identity becomes their voice and their behavior more than their appearance. And so I think, in terms of, of devices, I think devices is interesting, but it’s also interesting to think about how these devices structure, the virtual environments, and how these virtual environments feed into a real environments, because I think there’s a really significant difference there that’s important to investigate.

Stephen Ibaraki 39:07

You know, this transitioning, almost like this morphing from your physical sense, because you’re living in this altered reality and then extends into your real life, then I can think of this and the implications in space travel, where there’s going to long times, outside of the confines of the planet and gravity and or those kinds of influences and especially if we have plans to go back to the moon again, and but also habitats on the moon and also Mars. I can see your research influencing all of that as well, because there’s these big questions. And when people are in these confined environments, again, just fascinating in the work that you’re doing. And now, Joanne, again, with all the things that you’re doing, and your thoughts on this theme that we’re on now?

Joanne Leong 40:01

Definitely. So I think it’s kind of interesting to bring up maybe one more use case we didn’t touch upon yet with the virtual characters. And that is, like applying deep fake technology to actually protect our identities. And I’m bringing this up in this conversation, because one aspect that we’ve discussed is what is the role of sensors in helping people to have more empathetic exchanges and more authentic exchanges with people. So in the context of tele mental health, one application that we’ve thought about is being able to change how you look and how you sound, but still give the right cues to someone who might be able to help you such as a counselor or a therapist. And I think when we’re talking about devices, one opportunity is for devices to actually pick up on certain signals that might be very subtle. So if we have like a change in heart rate, or we have a change in our electrical skin conductance, devices can help to read that. And we as researchers can also think about how can this subtle information be put through in conversations like these to help people connect on that deeper level. During our Virtual Being and Being Virtual Workshop, we listened to some researchers from Facebook, and they’re talking about how technology is actually advancing in this way that we can come across extremely realistically to someone in the metaverse and they also talked about how there’s a certain kind of connection that we still have yet to attain, which is like, when we’re together in person, we can pick up on such amazing subtle cues, just the exchange of a glance, can change how you feel, can change the feeling in a room. And so thinking about these kind of subtle signals, I think is the next frontier that devices can help us kind of go through and follow through with.

Stephen Ibaraki 42:15

And I guess this, again, reminds me of the this idea that digital transformation or digital reshaping that occurred because of the pandemic has forced people to be online. And then there’s this comparison, is that the same as face to face, and, people at one time thought, wow, it could be a substitute. But now we’re finding there’s so many nonverbal, really micro cues that only occur if you’re with that person face to face. And there could be even things like pheromones and so on smell, and others that that you can’t get from a digital experience. Can we mine that further? Pattie? I mean, what do you think?

Pattie Maes 42:58

Yes, well, I was going to say, I mean, you’re absolutely right, when two people in person are having a good conversation, often their physiological signals start getting in sync, become synchronized, actually. And it’s really almost a measure of how good a conversation or a rapport between two people is. And of course, these types of things are not possible in remote communications technologies today, however, I mean, we always strive to reproduce real, say, face to face communication in our online technologies. And maybe it would be a smarter approach to actually look at some things that we can do in remote technologies that we can not do in human, in just face to face communication. So we’ve started to do some of these experiments actually, in our group, Valdemar was involved in one of them, where two people basically can communicate remotely. And we actually convey the sensation of the breathing of one person to the other person as well as the heart rate. So we sort of make this syncing up of these signals happen in a more artificial way, basically. So we think it’s very interesting to explore whether there are some things we could start doing, new technologies that we could employ, that actually can make remote conversations almost richer than in person experiences.

Stephen Ibaraki 44:50

I mean, I’m just thinking again, Pattie, I believe this in your workshop, you actually showed where you can have a digital representation which is different from what happening physically where that person is mirroring some of the mannerisms of the person that I’m talking with as well?

Pattie Maes 45:08

Yes, that’s what Joanne was mentioning, I think that you can basically anonymize a person, but still maintain their facial expressions, their tone of voice, even though the voice itself may be lowered, or, maybe changed, so that it’s not recognizable who the person is. But we can keep all that emotional content basically, in the voice, in the artificial voice. And that becomes very exciting that you can basically have a conversation with another person anonymously, while conveying a lot of these emotional signals still, without conveying your identity or giving away your identity.

Stephen Ibaraki 45:56

And I guess this sort of relates to this work of mirror neurons, right? And you have a lot of development of mirror neurons, then what happens is you can exhibit more empathy, more inclusion and more active listening. But you can synthesize that right?

Pattie Maes 46:03

Yes, sure.

Stephen Ibaraki 46:20

And that again, because you can give additional feedback digitally, like you said, heart rate, breathing, that again, synchronizes the two minds who are in a digital format, and maybe even getting some kind of synchronization even in the brainwaves if you measured what was happening. Right? So have you done that already looked at those kinds of things?

Pattie Maes 46:46

Valdemar (Vald)—you want to talk a little bit more about … ? Vald was one of the people involved in this study, actually, that we did. Yes, we didn’t do brainwave. We didn’t measure brainwaves, but that’s a very interesting idea. Actually, we should include that as well, as a measure.

Valdemar Danry 47:03

Yes, I’m happy to talk about a study. So what we did in this study was that we had people sit down and listen to stories of people who had some significant life event. So it could be somebody who committed crime, because he had a community in the prison. And that was how he had a life and so on, or we had a hacker who had access to a lot of people’s private information. And he was talking about the whole kind of ethics around having all this information and how to handle it, and so on. And so what happened in this study was that we were artificially, we’re sending the physiological signals of these narratives, these people like their heart rate, or their breathing rate, we were sending that into a wearable wristband. And breathing was sort of coming out of the headphones of people in the study. And then what happened was that after watching these videos, we asked them and conducted very in depth interviews of their experience of the other person. And one thing that we found was that the breathing, especially like listening to this kind of breathing in a very subtle way, they didn’t notice it, particularly. But it led to an increased kind of physiological synchrony between the signals coming from the narrators, and from the people listening to it, which is extremely interesting. And we also learned a lot about how people perceive other people using these signals, and how sometimes these devices that we wear can actually disrupt people from being empathetic towards a person because they all of a sudden have to pay attention to the device instead of the person. So those were just some of the findings.

Stephen Ibaraki 49:13

So I can see this sort of multi modal approach and even skin conductance as well, and how that would change. And you could even measure not just the respiration rate, but really even that the gas is given off. Maybe there’s a difference in absorption of oxygen or more carbon dioxide may be given off and things like that. And just perspiration could also change, I think, together with the different brainwaves, you know, maybe more alpha or something like that. During if there’s synchronicity happening in real time, and I guess you can even take it further into doing some active MRI, maybe functional MRI imaging or something like that, right? So, which is very granular. Okay, what about if we go, let’s say, five years into the future? I know that’s really, really difficult now like, again, I’ll start with Pattie, do you have some sense of what this is going to look like? And where the research is going to lead, let’s say five years into the future; or it’s just too hard to go there?

Pattie Maes 50:33

No, I, personally, I think that our research is looking five or even 10 years into the future. Even though all of us here probably think this could be commercialized and used today or tomorrow. In practice, it takes that long for our work to influence what people really experience in terms of technology day to day. So that certainly is my sort of long term perspective from having been involved in many sorts of inventions, innovations here at the Media Lab from wearable computers before there were smartphones, we were working on those actually. Other things like recommendation systems, often it takes the work that we’re doing; only makes it into the real world 10 years later. So I think everything that we’ve talked about today is probably more like a five, five or 10 year types of sort of horizon.

Pat Pataranutaporn 51:41

Maybe it would be nice to share a little bit about, your sort of journey. I mean, I’m always fascinated by your journey …

Pattie Maes 51:52

I don’t want to talk about it too much. But I have been here for over 30 years. And so that involves a lot of the development of personalized web pages. And a lot of the things that we now all I mean, there’s all of these kids, the other ones they grew up, or they were born into that world, but a lot of that is actually has a Media Lab legacy. Social media recommendations is all of that. These are all things that not just me, but also others that Media Lab developed in the 90s.

Stephen Ibaraki 52:39

<<I overview Pattie’s global contributions and innovations. They are in Pattie’s profile shared earlier in the article.

I ask each researcher their perspective on commercialization and their plans to having their work scaling globally.

I provide Pattie’s response below since she summarizes the Lab’s opportunities. >>

Pattie Maes 58:19

Yes, there’s really multiple ways in which our work influences or ultimately gets deployed. And maybe we heard already about, well, the startup and about a third of our students start a company and we even have a fund that helps very early stage, helps the students commercialize their ideas. But then the second one is open sourcing, like Vald was saying, we open source a lot of the tools that we develop. And that third way is really working with the sponsors of the Media Lab in commercializing some of the technology that we develop. So there’s those three paths, really to real, real world deployments.

Pat Pataranutaporn 59:37

<< Pat talks about collaborations, the interdisciplinary work that is international and the value it brings to the international communities. Extracting portions below … >>

And, you know, you came to our workshop, that was a collaboration between NTT DATA which is a member company in the Media Lab. And that’s one way that we can collaborate and partner with industry to expand the impact of our work…one story that I always tell people is that in the early day, I guess, even before the Media Lab that was the invention of the early prototype of touchscreen, and people … and even around the world call it the most stupid invention ever. And I think that, really speak of sort of the spirit of this place, to be inventive, to be bold, think of things that other people might think that, Oh, why would you do that? Or you say, it’s almost impossible. We do that kind of work…And, you know, in Thailand, …, I’m always trying to connect back to the my country. And I think all of us, we are international here, I think our group have, you know, many, many people from around the world. We always kind of contribute back. And yes, Translucia and the companies from Thailand that are now connected to the Lab, I think it’s going to hopefully make the Media Lab richer in terms of, the companies that we work with, and different perspective, different culture that they bring in. And hopefully, the Media Lab spirit can also travel, not just in the US, but you know, even in Southeast Asia and Thailand; that I think that’s my dream also.

Stephen Ibaraki 1:02:06

So quite a commendable dream, because all of you represent role models; Patty’s the pioneer, the icon. So she’s been 30 years of being a role model and icon. But all of you are still on the rest of yours, still fairly early. And you’re putting your footprints on the global stage, right? Sharing your work, talking about it here, spending time with me today. So I’m going to ask just a few more questions. And then we’ll end this dialogue. And so let’s now go outside of that Media Lab, and you personally, so now start with Pattie, first, outside of the Media Lab, where do you like to go? Personally, it could be personal professional, but not connected to the lab? Do you have any sort of dreams that you still want to conquer?

Pattie Maes 1:02:57

I think, well, to be honest, my life is mostly Media Lab. It keeps me very busy, it’s more like, two full time or three full time jobs rather than one. But the one reason that I am at the Media Lab is that it’s very interdisciplinary, and very free sort of, we have a lot of freedom to really work on whatever we find interesting. And so while other people may be feel restricted in terms of what they can accomplish professionally, I feel that I can easily bring in other interests that I have into the work that we do in our lab. So for example, I love nature and plants, the plant world and I had a student who was looking at how we can use plants, as sensors, and as displays can we actually use living plants to sense movement to sense pollution, and so on, but also send signals to people so that we have less silicon out there, and instead we use plants to do our sensing and our displays for us so, so it’s one of the wonderful things about the lab that we have the freedom to take on crazy ideas like that related to our own passions

Stephen Ibaraki 1:04:31

You know, I guess this triggers this idea that plants, even different species in a forest will actually communicate or even that’s infusing plants with nanoparticles, so they fluoresce instead of using …

Pattie Maes 1:04:59

… some of the type of work that we did also; making them absorb certain things so that we can use them as a display. And so without killing them off, of course.

Stephen Ibaraki 1:05:11

Yes. And their detection of the environment is much finer. What much more granular than what we could do, right?

<<I ask the same questions to the remainder of the team. I close off this transcription for this article as we go on for another 35 minutes. We discuss topics such as biomedical research and aging, more on symbolic AI and logical reasoning / artificial general intelligence, trillion parameters AI models and what this means, the impacts of our microbiome, and diverse topic areas. I ask each of the research team their final recommendations to the audience. I summarize the implications of their work.

There’s the separate interview with Pat where the link was shared earlier.>>