Walmart's SVP of Global Technology Anshu Bhardwaj and VP of Emerging Technology Desirée Gosby join Sama CEO Wendy Gonzalez for a roundtable discussion about representation in AI, explainable & ethical AI, and how representative teams are a key way to reduce biases in AI technology.
Walmart's SVP of Global Technology Anshu Bhardwaj and VP of Emerging Technology Desirée Gosby join Sama CEO Wendy Gonzalez for a roundtable discussion about representation in AI, explainable & ethical AI, and how representative teams are a key way to reduce biases in AI technology.
[music]
0:00:04.5 Rob Stevenson: Welcome to How AI Happens, a podcast where experts explain their work at the cutting edge of artificial intelligence. You'll hear from AI researchers, data scientists and machine learning engineers as they get technical about the most exciting developments in their field and the challenges they're facing along the way. I'm your host, Rob Stevenson, and we are about to learn how AI happens. We have a special treat for you here today on How AI Happens, rather than our typical weekly investigation into AI applications, replete with my own ham-fisted narration, we are bringing you three exceptional experts, two of the senior most technology officers from Walmart, joined by Sama's very own CEO, Wendy Gonzalez. These three sat down for a roundtable all about representation in AI, explainable and ethical AI, and the impact representative teams can have in rooting out the biases our algorithms can sometimes contain. Without further ado, I give you Wendy Gonzalez, Desi Gosby and Anshu Bhardwaj.
0:01:17.4 Wendy Gonzalez: Hi. Welcome one and all to an extra special edition of How AI Happens. I am today's host and the CEO of Sama, Wendy Gonzalez, and today on the show, we are featuring two exceptional AI executives from Walmart, the SVP of Global Technology Anshu Bhardwaj. Anshu, welcome. And also joining us today is Walmart's, VP of emerging technology Desi Gosby. Desi, thank you for being here today. What is very exciting, first of all, is we've got three powerful women in AI, so I just wanna say, I love that. It's a pleasure to be here with both of you. You're doing amazing things in the world. And I wanted to bring us together to really have a wide-ranging discussion about your expertise, some of the exciting things going on in AI over Walmart, but mainly about the state of representation in our industry, and also reviews on explainable and ethical AI and the crucial nature of eliminating bias and the technologies that we developed. First though, before we start, I would love to get to know both of you a little bit better. Desi, do you mind sharing a little bit about your background?
0:02:18.8 Desi Gosby: Thanks for the intro, Wendy. I'm an engineer at heart. So I've been working in the tech industry for about 20 years, and I started... I went to school at NMIT and I studied math for computer science, which of course included some foundation in machine learning and AI, and I started my career at IBM working on document management and knowledge management systems. And as part of that, as you can probably imagine, I did a lot of machine learning, document classification using NLP and NLU techniques, but from there, I actually made a pretty significant shift and started to work in mobile and started to get into mobile software, I was really interested in how you could take advantage of computer in your hand. And I ended up working on a couple of startups on the East Coast, which crashed and burned, I learned a lot from that, and eventually ended up going back to a larger company on the West Coast, company called Intuit, started there in their innovation and advanced technology group, still with a focus on mobile and building a lot of mobile applications.
0:03:30.4 DG: One of the most interesting projects that I worked on was actually a project where we wanted to leverage the camera to actually allow you to do your taxes by taking a picture of your W-2, and extracting information off of that. And that's a really hard problem. So I've got really back into and it's leveraging machine learning, AI, computer vision techniques, OCR techniques, which really was a lot of fun, but I'm now at Walmart. What we do here is we really take a look at some of the emerging technology trends for Walmart and really look at what are the capabilities that we can build. It's really unlock and unleashed at scale. And we're gonna number different things, but most notably, and something that many folks might have heard about is our conversation on AI platform, that is a platform that allows our teams across the company to create voice and chat experiences, leading our shopping assistance for customer care. And at the heart of that, of course, this is taking me back to my roots, and we use a lot of natural language understanding, and a lot of that AI-ML models and some of the newer techniques that are out there to really drive what we call sort of retail NLU that's driving and underlying all of those experiences. So in many ways, I've come full circle.
0:04:49.4 WG: It's amazing. It's an incredible, incredible journey. And I've been at some of those, I've also been to start the discretion about, so we share that in common, lots of learning for sure. [chuckle] Anshu, I would love to learn more about your background, your journey in AI and your role at Walmart.
0:05:05.6 Anshu Bhardwaj: Awesome, thanks Wendy, thanks for having us here. I do have to say we probably have a little bit of a self-selection bias in selecting the three of us to be here on the panel, but that's great. I think you have to lead by positive example, and I also didn't realize that Desi, you and I, share a love for math because my undergrad was in mathematics, and at some point I wanted to do a Master's and then a PhD, but the I don't ever reaching its destination just got to me after a while. And I changed the course, I didn't really start off being in the field of AI. I have actually been at Walmart now for 12 years, and in the retail industry for about 22. My entire 12 years have been at on the e-commerce side within Walmart. So I've really seen the evolution of e-commerce and the role that technology plays. I think from just being an enabler of helping you put things up on a site and serving up recommendations that are most logical on the basis of the catalog that you have, to now where we are, where for instance, back in the day, analysis is a root force, but now, not so much.
0:06:08.0 AB: So truly unlocking that potential that AI bring to the table of analyzing with data sources, providing recommendations, which is really huge, and for me, my first foray was of what AI really brings to the table was about... I'm seeing like five, six years ago, maybe at this point, where we actually organize an AI-ML Summit for the engineering talent at Sam's club, and we also pulled in a lot of leaders from Walmart, and we had a huge panel of external speakers who came in to talk to us about how they are applying AI and ML to every day of what they are doing and that to me was a big aha moment when I was used to dealing for instance, on the finance side, I was used to dealing with a big finance team that was working on various different kinds of models and big teams of analysts who were working on bringing out the Product Insights. And here was this one person whose entire team was just two people, and for a ridesharing company, they did the entire financial analysis for that whole company across the world, and orders of magnitude, which were probably 50, 70 times more complex than what we did, and it was all done to just two people, so that to me was a big aha moment. And then from there on, so I used to lead product for Sam's Club, and I was on this learning quest, how can we make things better, faster, simpler, more productive?
0:07:29.4 AB: And my role funny enough, Desi do, I had the earlier version of what Desi is doing, so before a lot of these things became popular I used to also lead the product team for innovation... Sorry, I had marketing membership, all of e-commerce, and then this little thing called innovation, we basically just dealt very deep into what use cases can be solved through applications of AI, which were unique for Sam's Club because the business model is very different. So if you think about computer vision applications where you're trying to detect an item in a cart, so that you don't have to stand in a checkout line, we experimented with that quite a bit, chatbots for our call center and for our site online, we worked on that quite a bit, which became fairly successful, and then there was this product that we had launched called Ask Sam, which was around, how do you basically understand what associate in the store is asking and serve them a good relevant recommendations or rather answers, so it could be anything from, How do I cut meat to, What is my schedule?
0:08:32.6 AB: So I think since then it's been obviously from five years ago, it's been a very, very long journey and from a maturity curve standpoint, Walmart is much farther ahead than where we were five years ago when we started out, and it's been a lot of fun, and I can't wait to see... Get to the evolution of everything that Desi and team are building and infuse that into our app, into our every day experiences, which will be really awesome.
0:08:55.9 WG: Yeah, that's an amazing foundation. And I also cannot wait to see how this transforms the experience. Let me think about that all of Walmart's massive global customer base, it's really incredible. So we do have a self-selection, Anshu, as you mentioned earlier, that's absolutely 100% right. With that said, do you believe female representation in AI is the same or differently compared to other fields in the tech industry?
0:09:21.2 AB: I'll go first and Desi jump in. So I think Wendy, this is from a data standpoint, and I don't have exact numbers in front of me, but I know that overall STEM careers as a whole needs more women and more people in underrepresented populations, and I think that goes without saying. As I think about Walmart overall, and a few of the women leaders who are very active in the field of AI, I won't be able to cover all of them, but I do wanna give a shout out to a few folks, Monika Shrivastav, Vicky Wen, Swati Kirti, Namita Collins, Lauren Shores. These are just a handful of women, who are really trailblazing, some of their stories I can share, some of theirs I can't. So I think for the purpose of what's happening, specifically from Walmart standpoint, I'm gonna pick, let's say Monica who joined Walmart two years ago, and I think what's remarkable about Monica's work in the AI is not just her work, what she does today, but the path that she took.
0:10:21.4 AB: So she actually took a break for a period of time, and as you can imagine, Wendy, it gets really difficult for women or for anybody actually who's taken a break to come back into the corporate world, so Monica joined Walmart two years ago through this program that we have, which is called a Path Forward Program, and it's specifically designed for women who are experienced, but who've taken a break from work to prioritize other aspects of life. And she is now a principal, technical program manager. In fact, I remember having met her two years ago before COVID, when we were still meeting people. She's focused on AI strategy and operations, and she worked on a really, really important program for us, which is around Markdown optimization, and as I alluded to earlier, when we were just chatting about through in our introductions, we moved from rules-based to actually not being rules-based. So the work that she and her team have done, they basically have taken Markdown and really changed it, so instead of applying a rule of Okay for every item which has X percent unsold inventory, let's make a 25%. They basically said, Let's not do that because we are sub-optimizing for the customer, we are sub-optimizing for ourselves.
0:11:35.1 AB: So they run several experiments, and in a period of, I would say, like five to seven weeks, they basically took a progressive Markdown approach and each store, instead of receiving a fixed discount without any consideration, they basically used a bunch of other factors around speed, elasticity, location, velocity, etcetera, and replaced this rules based pricing strategy with a dynamic personalized Markdown pricing approach. I think what's also remarkable about this is, I'm sure a lot of companies are doing that, but this is just one small example that I could actually share with you today, it's great that others are doing it because I can talk about it, if it's something that others are not doing, I can't talk about. But this just goes to show that for a company our size, there's a lot of work that can be done in the everyday single nitty-gritty, the fabric of operations that we have. So that's one example. Another example is of Vicky, who's a principal data scientist, and she's also been here with us three years in the Bay Area, in her words, when we were talking to her, she was like, I really love shopping for clothes, and I love this thing of where you can complete the look.
0:12:40.0 AB: So she basically worked with the Aster team to create this complete the look opportunity where instead of you having to do the work, basically the recommendations are outfitted for you up front. It's a very, very, very hard problem to solve. I know the bug keeps moving on this one. Easier said than done by the time we figure out something is done, the world has moved on, something else has come. Maybe there is TikTok shopping, there's something else, right? That you have to integrate. But there's a bunch of work that's happening across the board within Walmart. In fact, our machine learning team that worked on supply chain actually, in the INFORMS award, they basically got a notable mention, which was actually very, very exciting for us. So there's a lot of external validation and recognition that's also coming our way for the teams that are working through this.
0:13:26.3 WG: I am not a fashion maven, I could really use complete the look. So, I can't wait to see more about that application, I could definitely use the help. I would love to hear from you too, Desi, about your experience, and is AI any different than other fields in the tech industry in terms of representation of women?
0:13:44.3 DG: Yeah, I echo the things that Anshu talked about. Since I've been here, I've had the privilege of meeting so many great data scientists, and in particular Anshu has actually mentioned one of the data scientists that I had the opportunity to spend a lot of time with, is Lauren Shores. One of the reasons I pick her out, is because her story is really kind of interesting, when you talk about sort of the representation. Her story for how she came to Walmart is interesting, her background is one of... It's not a traditional sort of AI background, she had a background in economics. And then from there, studied predictive analytics. She spent a lot of time in the finance world... A little time in the finance world before coming to Walmart. And she actually was recruited from a Kaggle competition, and which I thought was awesome. And just sort of shows where we can go to get that diverse talent. And she's been at Walmart now as a senior data scientist for about five years now, and has worked on so many different types of problems. And she started off in working in supply chain and really focusing on basically operations research, but really... How it is that we optimize a team as it comes into the Data Center and improve the productivity within the Data Center? And it's work that she did five years ago, and it's still in use today.
0:15:08.3 DG: And she's also working with the real estate team and looking at ways for how it is in using AI and ML for optimizing how we find different locations for our manufacturing and our real estate plants. And it's just really great work, and she's also working on some other things that I can't talk about. But I think the fact that we have such a diverse set of problems that we solve here at Walmart, it really helps to attract that great talent that we have.
0:15:35.6 WG: Amazing. I love Lauren's story and I love the recruitment from a Kaggle competition, that is super cool. Do you think that increasing representation in the AI field will necessarily result in less bias in the tools that are developed? The AI, right, is a representation, of course, of human judgment or human knowledge, so very curious. Do you think this will make a difference and basically, how can bias be measured?
0:16:02.5 AB: I think the operative word, Wendy, is necessarily... I don't know if I would say necessarily result, because then I'd be a betting person, right? But I do think diversity in any field is very important. And corporate America has learnt it the hard way, and that applies to AI as well. So if you have a diverse workforce, then hopefully the outcomes will also ensure that the systems are not biased, because every different person is bringing in their own perspective. That's probably only a part of the cure, because it has to be a multi-pronged approach, and one approach is to basically have a diverse workforce. But internally also, I think companies need to kind of do a little bit more introspection and think through the implication. So for instance, in our case, we have a digital citizenship team where they are the stewards for us; however, they don't just take it upon themselves to become the stewards. However, they actually work with a very broad cross-section of partners in technology, in business, in supply chain, in legal, and lots of various different areas where they proactively identify and try to address issues, or at least raise awareness so that the engineers on the ground or anybody else who is working on operationalizing different elements of AI are more sensitized to it.
0:17:26.2 AB: From a system stand point, we wanna make sure that we create, again, to the best of our knowledge and to the best of readily/proactively available information. Our systems are regularly monitored and tested so that we don't let bias slip in, and worst case scenario, if it does; then you actually also take a step back and correct them. We all know that if you train an AI system with a non-diverse data set, the output will definitely be skewed. But at the end of the day, this is also just about basic data science approach and hygiene. And for us, it basically starts with fairness being top of mind, so that when you're actually building out a system, it's not just enough to ask if our output is getting better business accuracy, we take it a layer down. And I'm not sure, Wendy, if you've heard, but Walmart is committed to now becoming a regenerative company, and a lot of other companies have followed suit. I think the whole premise of being a regenerative company is, we don't want to just be in a taking mode. So we don't want to just have a better business for the sake of having a better business. We want to contribute back to our associates, to our customers, to our environment, to our stakeholders, to our shareholders. And that's the basic premise of being a regenerative company. And I think the same applies to AI as well.
0:18:38.3 AB: So as we think about growing the pie and growing the business for everybody, we basically think about the impact it creates for other diverse groups as well. So if needed, if something fails from a system standpoint, do we have enough human intervention that's needed, so that we come back on track? As an example, if there's a system that's making a prediction about fraudulent transactions. And let's say, it's basically somehow in testing it's determined that me as Anshu is going to be a fraudster, because I was in Miami today and I'm in the Bay Area tomorrow and I'm in New York the day after, and somehow it's calculated; it's not really fast enough. You can't even fly fast enough to all these locations to be using your credit card, right? Which may very well be true if you wanna flag something. But, if you're looking at the data and there's a certain pattern that's coming with me as a person. Then is there a way for us to actually dig deep into what's really going wrong if I'm a false positive that's come up here. Is it a trend that can then be extracted from my behavior, apply to other people so that you don't end up penalizing people like Anshu for something where I actually haven't done anything wrong?
0:19:45.7 AB: So just keep going deep into a specific area and making sure that everything that we've learned, whether it is from a system standpoint, working with a cross-functional team standpoint, all of the above that we just talked about and actually saying, how do we bring up more fairness and if there is disparity, what do we do about it? But that's exactly what we do. We continuously keep asking the why and the how, and which is why people on the fraud team, for instance, I really sympathize with them, and I remember when I was launching Scan & Go, the first three months, all I did... I knew the product worked beautifully, but I only worried about fraud because the false positive, as well as the false negative can really skew numbers, right? So I think that's where the power of AI can really be harnessed, but it's a continuous process, and again, the puck keeps moving and you have to keep moving with the times, making sure that the information you get is extracted and applied all at the same time.
0:20:40.1 WG: Yeah, I couldn't agree with you more. It's never one and done. It's about keeping your eye on the ball constantly to evaluate those patterns, and I love what you said a moment ago. Really developing that AI for everybody, I think that's fantastic. Desi, did you wanna add anything to Anshu's feedback?
0:21:01.3 DG: If I were to add anything, it was really just to emphasize something that she said around, it's just good data science to make sure that you are applying fairness across the board. I know in engineering, we talk a lot of times when you have a problem, like did you do the five whys? Are you going in and digging in and asking those questions and applies to data science as well. When you're looking at the data, asking yourself why, why, why and really making sure that you understand it and if it's impacting a particular core disproportionately, really digging into that and then making sure that you are using those principles of fairness to make sure that you're creating the best predictive solutions as possible.
0:21:46.7 WG: Yeah, we practice the same thing. We have the five whys, we kind have posted right front and center, 'cause you both said it very well, it's the hygiene, right? It's really all about having good practice, so I think that's fantastic. So one more... The things that's really interesting, certainly about AI is as it gets smarter and smarter, and from Ask Sam to Complete the Look, to all these amazing applications that are being built, how important is it to be explainable? So I was gonna ask the question really, what is meant by explainable AI, and what about non-explainable AI?
0:22:24.7 DG: Yeah. It's a great question and this term explainable AI is definitely coming up because businesses are using AI-ML to make more and more and more of their business decisions, and we wanna understand the inner workings of how those decisions are being made. You'll also often hear the term sort of a black box, and what that really means is that we're now at the point where even the technologist can't really fully explain how those decisions are actually getting made because of the nature in each of the AI systems that we're dealing with. That simply just doesn't work anymore, and it certainly doesn't jive with Walmart's commitment in terms of creating trust and in our commitments around trust, and so we really need to continue to push to evolve to how it is that we can provide some transparency in the logic and the decision-making that's being made. I would say it's definitely a journey. It's a process, and again, it'd probably sound a little bit like a broken record, but it comes right back down to that really asking those questions around the why and how something is happening, and then finding really creative ways for how you surface those things within the experience of that so that either our customers or whoever it is that is the recipient of that decision understands what's happening. I think we're gonna see this evolve a lot in the coming years as we go through it, but again, it's gonna just become part of our practice as it evolves.
0:23:57.4 WG: Yeah, completely makes sense. Not that this is the same parallel but it's part of how the transition works. I was having a conversation with a friend about Thomas Guides, and then I used to print out things from MapQuest, and all of a sudden the GPS tells you where to go, and you follow it, even if it puts you on a loop. So understanding what's happening, what the algorithms are, I've done that before, by the way. Taken that 25-minute detour just listening to the GPS, so I think that notion of explainability makes total sense, and trust at the end of the day is gonna be critical. And another sort of aspect of that is really ethical AI. So I was curious, you know, what considerations really comprise ethical AI in your opinion, Anshu?
0:24:40.6 AB: I think Desi dashed upon this, but I think at the end of the day, Wendy, I think for us, we are a responsible company and we're committed to being a destination that our customers, our suppliers, our stakeholders can trust, and from that standpoint, I think it was last year, probably the year prior where we established our Digital Citizenship Team, and their mission is to support and advance Walmart's ethical use of data and responsible use of technology, and I think that's a very, very powerful statement for the company to make, because at the end of the day, we are saying we don't want to use data where a customer may feel it's not the best use of data, an associate may feel, that's not what they signed up for, and therefore, all of this then permeates into making sure that our data practices and our technology treat people fairly, with dignity and with respect. So in a nutshell, I'd say that's ethical AI at Walmart, and every single day that's getting put into use by all the different teams working in cohesion with each other and also watching out for each other because you don't always know everything that's happening.
0:26:00.2 AB: The reason we all have biases is, we are kind of a tribal society, right? Which has now become a modern society, but our instincts have been built over many, many, many, many decades, thousands of years. So making sure that we have this one central body which then is educating and helping everybody else come across the finish line is very critical, and we're fully invested in making sure that the data practices and the technology that we have, treat people fairly, and with dignity and respect.
0:26:29.7 WG: I love that. I think dignity, respect... Fairness, dignity and respect are things that are really keying in on. Just my two additional cents on that front is that dignity and respect is key. So even when we were talking about things like data annotation and labeling, that's a practice and approach that we've taken as a B Corp, that we have transparency, we pay living wages, and we make that visible with a purposeful hiring model that includes diversity, 50% women is part of our hiring guideline, and that is all about creating better AI through treating people with dignity and respect and fairness, so I really appreciate that commentary. This has been incredible. I have learned so much, not only about the incredible things that Walmart is doing, but the regenerative approach that Walmart is taking, Digital Citizenship. I am thrilled to have spent the time with you and to hear your insights and the incredible work that you're doing, so thank you so much, and really appreciate your time.
0:27:29.4 AB: Thank you, Wendy. This was great.
0:27:31.2 DG: Thanks so much for having us. This is awesome.
[music]
0:27:40.3 RS: How AI Happens is brought to you by Sama. Sama provides accurate data for Ambitious AI, specializing in image, video and sensor data annotation and validation for machine learning algorithms in industries such as transportation, retail, E-commerce, media, MedTech, robotics and agriculture. For more information, head to sama.com.