Episode Transcript

Charlie Melcher:

Hi, I'm Charlie Melcher, Founder and Director of the Future of StoryTelling, and I'm delighted to welcome you to the FoST podcast.

Charlie Melcher:

Our guest today, Nonny de la Peña, has been called the "Godmother of virtual reality." Her journey in new media grew out of her successful career as an award-winning print and broadcast journalist and her desire to get people to feel more viscerally the story she was covering. She earned her BA from Harvard University, an MA from the USC Annenberg School for Communication and Journalism, and a PhD in Media Arts and Practice from the USC School of Cinematic Arts.

Charlie Melcher:

As founder and CEO of her own company, Emblematic Group, De la Peña uses cutting edge technologies to tell stories, both fictional and news-based, that create intense, empathetic engagement from viewers via virtual, mixed, and augmented reality. She created the first VR piece ever showcased at the Sundance Film Festival and is widely considered to be the founder of the field of immersive journalism.

Charlie Melcher:

She was named one of CNET en Español's 20 most influential Latinos in tech in 2017, Wall Street Journal's technology innovator of the year in 2018, was inducted into the South by Southwest innovation awards hall of fame in 2020, and just this year, was honored with a coveted Peabody award. She's a New American national fellow, a Yale Poynter media fellow, and a senior research fellow at the USCS School of Communication and Journalism. And she was just recently tapped to be the founding director of Arizona State University's Narrative and Emerging Media program, based in Los Angeles.

Charlie Melcher:

Please join me in welcoming my dear friend, Nonny de la Peña to the FoST podcast. Nonny, it's such an honor to have you on the FoST podcast. Welcome.

Nonny de la Peña:

Charlie, you know you're one of my favorite people on the planet. I feel lucky to spend the time with you.

Charlie Melcher:

Mm, that is a feeling that is very mutual. So, thank you for being here.

Charlie Melcher:

In thinking about this conversation, I started to remember some of the journalism in my life that was incredibly moving to me. There was a cover photo from the New York Times some years ago that showed a picture of a father and son. The father had been shot. It was an exchange of fire between Palestinians and Israelis, and the boy had been throwing rocks. The father had come. He was telling his dad, "Hold on, the ambulance will come and everything will be okay."

Charlie Melcher:

And I just saw that, read the caption, and bawled. I just was crying uncontrollably that morning in front of my paper. And it reminded me of what's so true about really powerful journalism that it can touch us and take us out of our daily life and move us to not just powerful emotion, but to action. And it made me wonder if you had an experience like that. Were there pictures or stories that you had experienced that led you to want to become a journalist?

Nonny de la Peña:

So, there are certainly many images that were brought into my home really by both my parents. My father helped start the film series at UCLA, and he often would screen both documentary and plenty of films, but also there were things like the collection from Life Magazine of world war II, including the liberation of the camps. And then add on top of that, the experience that my mother particularly would describe growing up in Texas as a Mexican-American. I mean, when my dad and my mom first tried to rent an apartment together, they couldn't get one. They just wouldn't lease to them.

Nonny de la Peña:

My mom would describe going to coffee shops with her best friends and they would never serve them. And the way that it would work is they'd sit there and go... They'd walk in, they'd take their seat, and they'd go, "Okay, we're ready to order." And they would just go, "Okay, be right there." And they would never come. There were just so many times my family oriented me toward issues that matter. And I think the combination of that and a fierce desire to do something about it led me to journalism.

Charlie Melcher:

It's so interesting to me too that you emphasize this idea of the information that leads to change, right? If we could get them the information, if you can cover the story correctly and communicate the injustice, then people would pay attention. They would be moved to do the right thing.

Charlie Melcher:

You started working with Associated Press, the Newsweek, Time Magazine, even with New York Times, some of the greatest names in journalism, but you're also talking now about documentary film and the power of that kind of storytelling. And I'm just trying to kind of understand your evolution from things that might have been more fact-based reporting to documentary filmmaking, and then ultimately moving us towards your early work with immersive journalism, and wondering if there was a desire to move towards things that created greater empathy or greater emotional impact for your audience.

Nonny de la Peña:

I think 1000%. So print is very powerful, right? Scritches on a page, right? They're just little dark lines on a page.

Charlie Melcher:

Dots actually.

Nonny de la Peña:

Yeah. I mean, really literally, right?

Charlie Melcher:

Yeah.

Nonny de la Peña:

And yet, the meaning can be so profound and so important to connect with story. I moved to photos, documentary, all with the idea of trying to enhance the way that I could present the story and the way that I could connect the audiences to the story. When I read about virtual reality, I didn't have the coding skills yet. I slowly taught myself HTML and it began the journey for me of approaching this stuff. And then ultimately, one of my documentaries, it was called Unconstitutional, and it had a big piece on Guantanamo Bay prison.

Nonny de la Peña:

I saw this grant the MacArthur Foundation was offering to take a serious documentary, and turned to the digital world. And I called up a friend of mine, an incredible digital artist named Peggy Weil, and I said, "Let's apply for this grant." And we had 24 hours. We decided to apply to do Gitmo and Second Life, and we got the grant. And after that piece was made, so many visitors came from many universities all around the world. And that's when I was like, hang on a second, these kind of technologies offer a way to do what I started calling "immersive journalism," journalism that puts you on scene.

Charlie Melcher:

By the way, we've had Philip Rosedale on the podcast who was the creator of Second Life. And-

Nonny de la Peña:

That's right.

Charlie Melcher:

His vision of that was really to be able to, in a way, offer a kind of intimate freedom. You could build a house, you could go anywhere, you could travel. It was like a whole another universe that he was creating of opportunity and expression and creativity.

Charlie Melcher:

Describe what you did in Second Life because I think it couldn't have been more in a way the opposite of that.

Nonny de la Peña:

What we did is we took the photos, video, all the bits and pieces that I'd gathered from Freedom of Information Act material, stuff from the Department of Defense, and we used it to make a one-to-one recreation, first of Camp X-Ray and then Camp Delta.

Nonny de la Peña:

You would first walk into a C-17 transport plane, and then we took control of your screen in Second Life. We hacked in. And you had agreed that you would allow us to do this, but then it was as if a hood came over your avatar, like what happened to the real people being transported to Gitmo, and you heard sounds. I used dialogue that real soldiers had said that they had heard and done. And then when the hood comes off, you find that you're literally in a stress position inside the Gitmo cage.

Nonny de la Peña:

After that, you're allowed to get up and walk around. But as you do, first you trigger off video and photographs of the real way that prisoners were being brought to the camps and what was happening inside the camps. And then when you went into Camp X-Ray, we actually had a little, I mean, you could call it a mini game, and you would query the "guard." It was all text-based, but the guard would...

Nonny de la Peña:

You could say to the guard, "Why am I here?" "I'm not allowed to give you that information." "Can I call my lawyer?" "No, I'm sorry. You're not allowed to do that." "Can I alert my family?" "That isn't a possibility, sorry." So, you get this idea of what it means to lose your habeas corpus rights. I mean, I think people just still don't understand what a mockery of our legal system Gitmo is.

Charlie Melcher:

And so, you're using this set of avatars and this kind of gaming engine world to let someone have access to something that in fact no journalists had access to. No one was going in to watch interrogations at Gitmo.

Nonny de la Peña:

That's correct. We tried to make a virtual but accessible version of a place that was inaccessible. And we were careful. It's not like we showed any waterboarding or anything. We integrated a lot of real images, sounds, videos very carefully, architecturally exact to the real camps. Achieve a level of accuracy and integrity that would also have informed any kind of documentary or print piece I might have done in the past.

Charlie Melcher:

And did you take some criticism for having to take some liberties in terms of telling that story?

Nonny de la Peña:

Did I take some criticism? Was that a capital C? I refused to call that "enhanced interrogation technique." It was just torture. So sometimes, I got grief. I was told that I was doing opinion pieces. Mostly, I feel like there's a lot of support from journalists now. I mean, when you hear the New York Times calling it immersive journalism... I mean, I just won the Peabody, right? For a field builder award for the work that I've been doing. So I think-

Charlie Melcher:

That's me clapping, yeah.

Nonny de la Peña:

After years of basically fingers in my face and saying things to me like, "You can't do this," "This isn't journalism," "These are games," I had a pretty rough road trying to make these pieces and piece after piece to show people, guide people that this in fact is technology that's totally appropriate to youth for non-gaming practices.

Charlie Melcher:

And so after doing the Gitmo piece, your next one, was that the work you did with Hunger in Los Angeles?

Nonny de la Peña:

I had this incredible material that I had obtained through a Freedom of Information Act request. All this FOIA work that I'd collected on people being put in stress positions and there was that Donald Rumsfeld's comment that, "I sat on my feet all day long. It's not that bad for these prisoners." We asked people to sit in a chair and hold their hands behind their back. And people were wearing a breathing strap and when they went into the headset and they would look in a mirror, the individual's breathing that they saw in the mirror was hunched over like they're in a stress position, in an orange jumpsuit.

Nonny de la Peña:

And because the breathing strap was on you, they were breathing in the same rate you were. And if that you turned your head, the character you saw would turn your head in its virtual mirror. And you heard interrogation sounds playing in the background. And the interrogation sounds were from an actor. I had an actor yell, "Sit down. Stand up. Sit down. Stand up." Right? And what was crazy is when people pulled off the headset, we said, "What was your body like?" And everybody reported being hunched over in a stress position when they were just sitting upright. It was clear to me this was an incredibly powerful medium.

Charlie Melcher:

I'm just so fascinated by how people, the emotional response they had to being put into a stress position, being able to also see themselves or experience being in that interrogation, in that position and how it changed their understanding of the kind of torture that people were being exposed to. What was some of the feedback you received from people as they came out of it? Was it terrifying? Was it upsetting?

Nonny de la Peña:

Yeah. I mean, everybody was like, "Whoa, I didn't realize." There's a lot of that "I didn't realize." The information that we gathered on that piece led to the immersive journalism paper and that immersive journalism paper ended up being published by the MIT Journal, PRESENCE, and became the second most downloaded article in the journal's history, using virtual reality for the first person experience of the news. So you're talking about the first person experience of the news and that's what we were trying to do with immersive journalism. After we finished that piece is when we began to work on Hunger in Los Angeles.

Charlie Melcher:

Hunger in Los Angeles, I remember that so well. Really powerful piece. Here, you actually had the real audio that you were able to work from, right? Not a transcript that you're having actors read, but actually, audio recordings of this scene that was taking place in Los Angeles. Would you describe it?

Nonny de la Peña:

Yeah, that was a remarkable project. I really wanted to do something about the fact that the hungry were invisible. This is again right in the middle of the downturn and food banks were literally overwhelmed and running out of food. And I thought initially that I wanted to be there, have people be on scene in the moment when a parent has to return to their child, because the food bank's run out of food and you have to leave.

Nonny de la Peña:

But as we started doing these recordings, we found that at that moment, parents would just go silent. And we didn't even have GoPro cameras at that point. And I knew I was going to have to build it all on CG and we needed good audio. This man with diabetes who was waiting in a long line for food, his blood sugar dropped too low and he collapsed into a coma. And it was crazy chaos. Paramedics took forever to come. And when they did, they were so horrible to all the people waiting in line for food. Then in the crazy chaos, a woman tries to start stealing food. So, the audio by itself was extraordinary.

Charlie Melcher:

Nonny, can you share a piece of Hunger in LA for us? Let's listen to that.

[Audio from Hunger in LA]

Charlie Melcher:

I remember when I saw that the first time, my very initial thought is, "Gee, this is so low res. The graphics aren't super high quality. It's a game engine, but it feels a little bit rough." And then by the end of the piece, I didn't think about that at all. I was just totally in the piece, I was in the drama of the scene, and I was surprised by how I had forgotten about the quality of the visuals or that it had game characters that you had sort of adapted.

Nonny de la Peña:

Yeah, hodgepodged together.

Charlie Melcher:

Yeah.

Nonny de la Peña:

Hodgepodged together. Donated, I spent 700 bucks of my own money to buy characters. I mean, it was just a ridiculous hodgepodge of people. Truly, I didn't know how people were going to react. We got into Sundance and then people were taking the headset off and weeping, and they were down at the ground with the seizure victim trying to hold his head and talk to him and get people to do something. Believe me, we were completely shocked, all of us, to the reactions of the audience or the participants.

Charlie Melcher:

That's a thing to really try to understand better is how much more powerful you believe or you have found that these things are when you're giving people a first person perspective on that real life experience. You're not out of it as a voyeur, you're in it as a character.

Nonny de la Peña:

So I call that "duality of presence," you feeling that you're there and you're here at the same time. And it's amazing that the mind could be activated that way with such rudimentary imagery and rudimentary tools really at that time. We're with this crazy, hacked together headset mostly put together by Palmer Luckey, who at this point was the lab intern at USC, and came to Sundance, crashed in my hotel room, drove the truck around for us, the van, and then drove all the equipment back. He was just a kid, right? And then nine months later, started Oculus Rift.

Charlie Melcher:

And two years later, sold it for $2 billion to Facebook.

Nonny de la Peña:

Yeah. Actually, $3 billion.

Charlie Melcher:

3 billion, okay.

Nonny de la Peña:

It's a pretty extraordinary story. Anyway, that was kind of a pivotal moment. And after that, cobbled together grants and did stuff on border patrol violence and then I was invited by the World Economic Forum to do a piece about Syrian refugees and a piece that puts you on the street in Aleppo. When a bomb goes off, that's the one I got to share with you at FoST.

Charlie Melcher:

Let's listen to a piece from Project Syria.

[Audio from Project Syria]

Charlie Melcher:

I mean, I just want to say a lot of people are using these tools for games, for entertainment, for corporate marketing, but you were really the trailblazer, at least that I knew of, who was using them for impact storytelling, for social change, for moving people on topics that frankly they maybe didn't want to see or have to be in or face.

Charlie Melcher:

But I think we should talk for a second just about scale, right? The one downside, if you will, of a VR experience is it's one person at a time. The upside of a photo on the cover of the New York Times is it's millions of people all at once. And VR is not like that, not yet.

Nonny de la Peña:

Not yet, but getting closer, right? Whatever issues Meta has, Facebook, the Oculus quest has made it much more accessible. 300 bucks in your end? That's not bad. We know that Apple moved their top hardware guy into the AR/VR team. Something's going to percolate where we are all going to be able to see this material. And every smartphone now has a lighter camera on it, which means that everybody can capture our world in three dimensions in how we're going to share that with each other.

Nonny de la Peña:

So one of the things that I've been trying to do, because I really want to democratize the ability for people to make and share this content to actually scale it, is I've been building a button-based software called Reach.love that lets anybody start to make their own stuff. It's no code software, based in the browser that doesn't require you to learn how to be a C-Sharp coder like I had to.

Charlie Melcher:

There's a lot of criticism these days of journalism, that it is too subjective, that it's... People are telling, have their own set of facts, that it's all partisan and biased. And there's also some hopelessness around that. Do you think that this kind of journalism perhaps offers us a way through this time of crisis in the world of journalism?

Nonny de la Peña:

I can't say that I'm the one who's got a solution beyond can we use this technology for good, for empathy, for connection? And I have to say, Arizona State University is really starting some very extraordinary initiatives thinking about policy and using immersive content for change.

Charlie Melcher:

And you're now the director of this new graduate program that you're doing with ASU in Los Angeles, correct?

Nonny de la Peña:

Yeah. So, I was lucky enough to be offered a position with Arizona State University to be the founding director for a new center on narrative and emerging media. They bought the old Herald Examiner building in downtown Los Angeles. Talk about scale, it's a very inexpensive university, but we have taught people working there. And the goal is to really try to diversify and create a much changed demographic of who's getting to tell stories, create stories, whose narratives are being told in these new emerging technologies. It's virtual reality, augmented reality, extended reality, virtual production. So, it's really about narrative and emerging media.

Nonny de la Peña:

Arizona State University also has a new center in Mesa we call the "MIX center." Literally over a dozen of people have been hired to think about immersive technologies and create and learn within those new technologies. We have a new, we call "XRTS Fellowship." So, look for that. It's a very well paid fellowship that people can apply for. XRTS Fellowship. And we're just about to announce our opening for application for our fall cohort, for the Master's we're going to have here in LA. And we have some scholarship money for that too, particularly for this first cohort, for people to apply. So we may not be able to scale yet in terms of being together to look at stuff or everybody looking at this stuff through a headset, but we're beginning to scale the creators in the field with a real lens on trying to diversify who gets to be a participant in this new field.

Charlie Melcher:

I think about this problem that we have with the polarization of journalism and people not believing things. And none of us have the answer, but it does seem to me that one place to turn is to train more people in the fundamental principles of good journalism. So it feels to me like you are addressing the problem by helping to teach and bring up a new generation of journalists who are both digitally savvy and have fundamentals of what it means to have well researched, well fact checked, the rigor of proper journalism, and then on top of that, add those skills to a much more diverse group of people than have normally had access to those tools. It does seem to me like that's at least a good start in terms of getting us out of where we are right now in the world of fact versus fiction.

Nonny de la Peña:

Oh, God. I recently had a colleague say to me, "Well, what you're doing is not journalism because you're teaching students how to make deep fakes." And this is recently. So we're not out of the woods yet, Charlie, with in terms of acceptance.

Charlie Melcher:

It touches on something that I think storytellers have always known, that a really well told story can be more truthful and get to the essence of something perhaps even more than a set of facts or a more clinical document of something. There's often that tension in journalism between, "Just the facts, ma'am," and here's the arc of that story in a way that you can really connect to and relate to as a human being.

Nonny de la Peña:

Story is a real glue for all humans, right? It allows us to understand the other and have empathy for the other. And I think my son really got it right when he first said to me... He was just a young guy, 10 years ago, but he said, "I think the reason why it makes you care is because if you're standing there, you feel like it could happen to you too."

Charlie Melcher:

Do you imagine a new kind of newspaper or news show? I've always tried to imagine what would it be like if Nonny got to make a new form for conveying the news. And you had unlimited budgets and would it be a program in VR? Would it be AR? What is that immersive first person journalism look like if it gets up to scale I guess is what I'm saying.

Nonny de la Peña:

Well, even journalism sites still have categories. I could really imagine getting to publish an immersive journalism publication, paper, whatever you want to call it. It won't be paper, but a publication that has those sections where I would really like to see some of the reporters who are on the ground right now in the Ukraine using their smartphone to give us the ability to walk through some of the rubble.

Nonny de la Peña:

I would also love for a home and garden section where you could walk through a garden and the plants are immediately identified for you. I think that this is still a possibility. I think these verticals can be brought together in a way that would make a publication hole that people would like to subscribe to. That would be my dream. I'm hoping actually that I'm going to start doing some prototyping around that next year.

Charlie Melcher:

Sign me up. I'm in.

Nonny de la Peña:

And listen, there's use cases for AR, there's use cases for VR. And it will depend on the piece. Sometimes, your publication will not just be, "Okay, I'm getting up and I'm reading it in the morning," but you may need an AR warning from the journalism organization, "Hey, an incident has just happened," the way that local news always helps you find out that there's been a car accident or it's an incident in your neighborhood or, "Here's where you can donate." I mean, there's no reason why you couldn't be getting an alert on your phone while you're driving and you're wearing your air glasses and quick detour here. All those things would be that the journalism organization is with you.

Charlie Melcher:

You just made me think about what was in the news, just this most recent mass murder, mass shooting, and the new trend that's out there for these crazy, terrible human beings who try to live stream their attacks and make it so that people can watch them in real time. And I guess all of that just reminds us that these are just tools in the end and we can use them for good or for terrible evil.

Nonny de la Peña:

1000%. My son last night said, "Mom, everybody I talked to about the incident was 18." He said, "Everybody I know had watched the video or part of it." He said, "I started watching it and I turned it off." That's a separate can of worms. And that's also why I think people might gravitate more towards a publication that does some filtering. These are not simple problems, but that doesn't mean we should just be trying to tackle them all the time. How do we filter for this stuff? How will AI be effective? It's complicated, but we have to keep trying every day to figure out how to stop that kind of material from being spread everywhere, because it's pretty awful. It's more than pretty awful, it's terrible.

Charlie Melcher:

Certainly, it's the hope of things like virtual reality, which we both know has been referred to as the "empathy machine," that we could get to a place where people getting to understand the world through each other's eyes and through that, to be able to empathize and feel connected to one another in a way that would help us become a more loving and supportive world. And certainly, a lot of the trends right now are pushing us in the opposite direction. But I too remain hopeful that technologies can and will be used to help not take us apart or push us apart but actually, as you say, to bring us together.

Nonny de la Peña:

And there will always be a tug and a pull. And don't forget that the Rosetta stone, this early written document with different translations on it, it was a tax collection document. We celebrate this amazing document when actually it was just trying to skim the money off the people. Technologies are always going to be used for good and bad. We know that. We know about propaganda. We know that. But in much the way Rwanda went from a radio show that spewed so much hate that led to-

Charlie Melcher:

Genocide.

Nonny de la Peña:

Terrible massacres. Yeah, genocide. They turned it around and then started communicating with each other through a soap that was intergenerational marriages and love. I mean, they literally just changed the script and it really helped unify people. And I just think that we have to be the ones always fighting to change the script. I'm delighted I got the opportunity to say that on your show, Charlie. Thank you.

Charlie Melcher:

Thank you, Nonny. It's such a pleasure to spend time with you. Big hug to you and thank you for everything you do.

Nonny de la Peña:

Thanks, Charlie. Hopefully, it'll be in person soon. Thanks so much.

Charlie Melcher:

My sincere thanks to Nonny de la Peña for joining me on today's show. You can learn more about her work and the new Narrative and Emerging Media program she's leading for ASU by visiting the links in this episode's description. Thank you for listening to the FoST podcast. I hope you enjoyed it. Please leave a review wherever you get your podcasts, and follow us on Facebook, Instagram, and Twitter at Future of StoryTelling.

Charlie Melcher:

FoST also produces a monthly newsletter filled with valuable information for storytellers of all kinds. You can subscribe for free by visiting the link in this episode's description or on our website at fost.org. The FoST podcast is produced by Melcher Media in collaboration with our talented production partner, Charts & Leisure. I hope we'll see you again soon for another deep dive into the world of storytelling. Until then, please be safe, stay strong, and story on.