RI Seminar: James McBride : AI, Robotics, and Autonomous Vehicle Development at Ford Motor Company

RI Seminar: James McBride : AI, Robotics, and Autonomous Vehicle Development at Ford Motor Company


it was actually first met him never goes and now times change here’s the story for Wow thank you for the introduction red so I have to confess I was out of town on a last-minute business trip one of those kinds where the chief technical officer says can you be somewhere on a plane in an hour and I had this Charter to come here and to talk about three things sort of what Ford’s doing an autonomy historically and now given that this is the tenure of the DARPA challenge throw in a few slides you know giving you an image of what things look like back then and a bit of recruiting because you know I come here once or twice a year to say hey you know we like your school oh you know we like your students you know come hang out with us and do some projects so what I did last night when I got home at midnight from San Francisco is I went to the closet and I looked for the dozen or more laptops in you know there’s probably a good video on there from the DARPA challenge but which one of the laptops and in which file folder so if you’ll bear with me what I’ve done is I’ve just dragged a whole bunch of slides there some of them have words we don’t need to read but on those are some cool pictures and some videos and I’ll talk a little bit about that in the context of DARPA challenge Ford and recruiting so let me just get started here ever since my childhood and in a reg childhood we’ve been promised that you know we’d have this car that drives itself just right around the corner and I have to say Ford’s partly guilty for that because the famous Batmobile came out of her own design studio and the answer to this always seemed to be you know ten years in the future and ten years never ever came and as a matter of fact when I joined Ford Motor Company the state-of-the-art and autonomous vehicle technology pretty much looked like this Dilbert cartoon from 1984 and so what happened so how many of you guys would believe me if I told you I had half a billion dollars in my pocket oh I got one taker on that too takers yeah so in 1984 a megabyte of memory cost you about a thousand bucks and I think I have two iPhones and a couple of thumb drives in my pocket and I probably got you know at least a gigabyte of memory in my pocket and that’s worth you know quite a bit of money in 1984 so what happened to change the story between 10 years is never going to seem to come is there was a vast improvement in computational power and remote sensing capability and that’s essentially why we went to the DARPA challenges and I assume this is Carnegie Mellon most of you guys know what the DARPA challenges is but here’s a slide where I usually explain to people in 2001 Congress that had this mandate saying you know we’d like to remove people from the battlefield and it’s just not happening fast enough so why don’t we get some people from academia industry you know some crackpots and and get them all together in the desert and see if somebody can drive across this drain and help kick-start this effort so there was a series of increasingly difficult challenges I don’t want you to read this slide this is sort of the history of it but basically the things you should take away from it here is that the winners of each of these events were universities paired with some auto companies and that Ford was one of six teams that made it to both of the finals so that’s the ford message here we had a very small team we maybe a dozen people at most we came to the game rather late I’ll talk about that in a second but there was a problem here so if you went to any of the early briefings on the DARPA challenge the guys from DARPA said you know you’re gonna drive through stuff like this and so when I saw terrain like this and like this and like this I thought well you know if I go to this race and I fail the failure mode cannot be a Ford pickup right so the failure mode has to be something other than that so we went to the race with the attitude that we really were going to drive a cost drain like this so we had you’ll see the picture here I think it’s well let me let me back up I don’t want to show you that one yet but you’ll see the picture of our vehicle we had one of our larger Super Duty pickups and it had an exoskeleton on it welded to the frame with skid plates locking differentials Kevlar tires inner run-flat liners you know we had the computers with their own air condition unit shock towers in a box I mean you know we weren’t going to break the platform itself if we were going to fail it was going to be the software and as it turns out most of the course looked like that and you could have got across it in a golf cart so I think there were only three teams perhaps it took DARPA school seriously one would be Reds team here sandstorm and Highlander and such our team had ironically the least maneuverable vehicle in the race now you might call me crazy because you see Terra max this vehicle it’s you know bigger than this room but it had rear wheel steer and in our case our vehicle if you turn a circle in the desert or a dry lakebed it was a 54 foot diameter now our vehicle could go in Reverse and it did but in both the races ultimately that 54 foot diameter cost us because of some of our own stupidity and the way the course was laid out and I’ll talk a little bit about that later but at the first challenge I didn’t actually participate I only learned about it a few months in a day and I volunteered to help DARPA stage the race and so I went out there and one of my job’s is to go visit every team and read their technical papers and make sure they had some credibility and they weren’t gonna kill people out on the course and so I managed to meet everyone in the industry and that was a real boon for me because now I know all these people that are the heads of all these self-driving car teams and such but one of the things that struck me is a lot of the people that showed up missed the problem statement the problem wasn’t to invent a new vehicle the problem was to put the brains in the vehicle and so you saw all these entrants like this show up and in the end date they really didn’t do so well it was the wrong problem and as I said it wasn’t a lot of advance notice before this race so unless you were a well-established robotics program or somebody that was working on this over you know for a long period of time you only had a few months to put together your machine in your software so you saw a lot of things that were cobbled together and you know just bolt it in place here and there and and since this time we’ve seen a lot of refinement and what what platforms look like I only want to talk about a couple of these notable results sandstorm went the farthest at the first race a little over seven miles and they got to the hard part of the course that was something that DARPA hadn’t really thought through too well they put the hard part at the beginning and the shortest distance traveled was the motorcycle their ghost rider made it four feet tara max was afraid of tumbleweeds and it would go forward and back and forward and back and forward and back and it logged over you know two-tenths of a mile going backwards before they finally decided to call it quits I’d say one of the surprising teams there was team Gollum they actually passed ten other people on the course and the red team only knocked down two fence posts they lost it to CMR but what really happened at that event wasn’t so much the cars it was the worldwide media as they’re watching it’s kind of a double-edged sword because on one hand the best team in the world only made it seven miles and most the rest of them were stranded at the finish line more or less and so you had a bunch of people writing what a colossal failure this was and there was a lot of ridicule about it and on the other hand it you know really got everyone’s attention that this was a problem and it had implications beyond just driving in the desert for the military and it really did kickstart the entire field of ground robotics in the automotive industry as we know it and so DARPA rather rapidly said you know we better to this again and you know improve on the media coverage because media is everything and I work for a company and they know that so at the second race I decided that you know I should show up and actually not run the race but be in the race and so this was our vehicle we called it the desert tortoise we had two vehicles they were identical we called one the desert hare and we had this crazy idea that we’d program one of them to be really aggressive and the other one to be not so aggressive and we decided on race day which one to race and one of the things that I want to talk a little bit about here is the sensing sweet on here and so at this date and time there was nothing moving on the course aside from some desert tortoises which we named the truck after so the way that people got around was largely to drive roads that were devoid of you know any transient or moving obstacles and so it made the problem a lot simpler and you’ll see on this vehicle here we okay that worked well Wow we had to really high precision airborne lidar so these things of 0.2 degree per per shot of the laser so really finally resolution we could put two lines across the road and if you put a cross section across the road you can see the crown in the road you could see the ditches you could kind of infer where the middle of it was so in between whenever we get a GPS Waypoint fix this was a really convenient way to drive the RO we need a couple extra down here and an array of radars in the front some stereo cameras you know all the usual sort of suspects but what you don’t see here is you don’t see any live imaging that would enable you to handle real-time dynamic moving objects and that’ll be the big difference between the desert race and the urban race and there’s a little bit of a backstory here in the in the lasers as well so one of my favorite teams in the first grand challenge the team from Bella dine it was two brothers and they just showed up in their pickup truck not knowing that the vehicle would get impounded at the race and they had no way to get to their hotel but the owner of Bella dine this guy Dave Holly’s really brilliant eccentric genius inventor he had one of the more creative stereo systems I ever saw and if you recall you know I had to go around and do the technical analysis of each team and the safety assessment and I had lots of arguments with Dave about you know how he was going to improve his system and in my background I don’t know if this if we mentioned this earlier but I’m a physics guy so you know as physics guys we smash things together and in my life I smashed photons together and so did a lot work with lasers and optics and so I said to Dave you know if I were gonna come back to this race eventually I’d just glue a hundred lasers together and spin them around really fast and that’s in fact what Dave did and I worked with Dave over the years and he’s created all the products that you see on most everybody’s car that’s under development now the valladon lidar systems and you’ll see that on the on the urban vehicles the only one at the desert vehicle that had it was Dave because I told Dave to build it so why did we go to the DARPA challenge well we didn’t go there to build robots the premise that we had was if you could go there and learn about the rapidly evolving sensing hardware and algorithms you could probably take that knowledge back and make some better active safety features on mass production vehicles and we thought that was a noble effort and we came away from the race however with the conclusion that you know the idea of a self-driving vehicle was entirely crazy that we could do it and you know a lot of people also had that idea and that’s where you’ve seen this proliferation of autonomous vehicles now I’ll explain this top statement here in a second let’s just leave this aside for the moment but the obvious safety reason that I just stated is borne out by the fact that over a million people worldwide die in automotive accidents every year and the vast majority of them certainly in the 90 some percent centage range are human induced so there’s a huge opportunity here for taking these technologies in improving safety and so from our point of view safety was always the first first goal of the mission and wasn’t until later on that we just discovered all these ancillary benefits like you know added mobility for disabled and elderly and young people and you know freeing up some of your time in the car and ride sharing and conserving fuel and you know all the other things that you see in the media every day talked about but initially it was all about safety and and if you look at the accident statistics in the United States about 3540 thousand people a year die in a car crash actually equivalent of one of these a day if that happened once a day in the United States you’d have everyone up in arms but somehow we seem to tolerate these car crashes so anyway we went to the DARPA challenge and we did in fact drive our truck on roads that looked like what you know DARPA scared us with some pretty rough stuff was our test terrain we had a small crew like I said this was our you know this was our flotilla of support and a couple of highlights of the race we drove 50 miles an hour during qualifying out on the course we were the I think we were the first bot to ever pass somebody would driving 50 miles an hour there on that dry lakebed when we passed T maxi on but we had a low point here we ended our race day right here when this military photographer stepped out in the road when we had to make a turn and I’ll show you in a little more detail where that was at we were actually driving down this road here doing exactly what I was telling you using the cameras and the laser scanner to identify what the road was and the way our computer architecture worked at that time we had a bunch of different sensors coming in and the sensors kind of got some voting scheme about saying who thought you know they had the right to have the correct answer and there was actually a GPS coordinate so I guess my laser is given out there’s a GPS coordinate down this road here this road was only a foot wider than our truck and it was at about a hundred and ten degree angle remember I said it took 54 feet for us to turn that truck around we had literally zero margin for error to make that corner so we had this voting scheme going on the cameras and lasers say go straight down this big road the GPS says maybe you should make a left-hand turn so eventually we get to this point we start to make a left-hand turn and we see this photographer standing where this guy here is in the road picked him up as an obstacle and we said well obstacle and road turn right so we went left right left he ended up in us going just about two feet to the right of the road and we got behind a wall of bushes just like you see here and there was just this infinite wall of bushes for the next few miles and you know we could never get back to the road so that was the end of our race day we eventually dropped a hard drive and then in the in the vehicle as I said you know lidar came to this race and so this is the first valid EIN sensor that dave built you know based on me nagging him to do it this is what the output looked like then wasn’t as crisp and clear as today but it still did the job and some other people had ideas about how to multiple lidar beams I wouldn’t necessarily recommend this one but these guys did pretty good in the race in fact they finished first so fast forward to the urban challenge and take all the desert stuff that we just showed you and add to it a closed air force base that sort of looks like a city that’s been bombed with the you know a few air strikes so the streets looked like this fairly barren except when other vehicles were on them and they did put other vehicles on them and then they put some network through the city and you had to you know randomly connect the dots and figure out how to do that again don’t read the words here our truck looked a little bit different at this point in time you can see we’ve added okay I’m going to give up on this here at some point we’ve added a bunch of radar sensors to the outside for proximity radar around the vehicle but the big thing here is now the rotating chicken bucket that people call it the sixty four beam valid eye naitch’ DL 64 one of the first ones in the world was sitting on my desk and it’s been highly leveraged ever since and that made all the difference here the only other noteworthy thing here is this is our computer box in the middle here it was air conditioned sat on some shock towers and what ran the vehicle at that time was a whole stack full of server Class C on computers this is a the start line on the morning of the race we’re the white truck here you guys from Carnegie Mellon or the caterpillar here and gosh gosh down here is obviously the biggest looking truck but it was quite maneuverable actually this is a view from the other side and it’s even got some signatures on it one of the things that most people don’t know is we also supported the teams from Virginia Tech and MIT that made it to the finals and collaboration was pretty important back then and it continues to remain so as people try to solve some of these hard problems to move forward in autonomy so I think I have a video clip here to show just a little snippet of the car driving at the to start with challenges it’s kind of what a typical Mojave Desert Road looked like that we drove down typically we drive these about 30 miles an hour this was a particularly tough challenge because there’s a big car compared to a big truck compared to those cars we’ve only got a couple centimeters or a couple inches of clearance on either side to not hit that wall and stay inside the lines and again that was thanks to really great sensing we got out of the lighter so how did we screw up on the urban challenge this one’s a little more difficult to explain so if I were to go back to that map where I showed you the city layout one of the things they had you do in qualifying was they barricade a road and you had to do a u-turn in the road or a K Point turn or in the case of our vehicle K equals 32 you guys ever seen the Austin Powers movie where the guy gets his cart stuck in the hallway and he’s going like this 54 foot diameter remember so so generally what happened when you’d see some sort of false positive in the race and this is what humans do to they go what does that false positive really mean anything so like if I’m driving down the road and there’s a car in my lane that’s not moving there’s a yellow line to my left I usually so he’d say well you know screw the yellow line I’m just gonna pass that stalled car right we don’t think much about it but when you’ve written lines of code to say hey you know you know thou shalt not cross double yellow lines or things for us it was a problem to do this 32 point turn on this barricaded Road and to get around these sort of problems what we do is we’d say well you know if you wait a certain amount of time maybe you might consider not following all the rules so you know you cross that double yellow line you pass the stalled car well in our case it was like that kerb is nothing to an f250 let’s just go through the neighbors lawn so what every programmer knows or at least one that works on a product is you never change code and they before you had to use the code but nonetheless we thought we’re going to solve that problem so we went out in the desert at midnight and we were solving this problem about how to do this kay point turn work you know Kay was going to be a very large number and not hit the time out where we’d say you know screw it we’re going to go through the neighbor’s yard and around the barricade so we fixed it but what we were fixing it we changed that countdown timer from 2 minutes to 20 minutes we added a zero so we’d have plenty of time for testing the code and not getting invoked well two in the morning somebody forgot to turn the twenty back to a two and so the first time we saw a false-positive it’s not very easy to see here but there’s a fairly deep rain gutter here it’s deeper than your normal curb you can see it lit up as this green line in front of us showing that we’ve got an empath curb and one of the rules says don’t drive over curbs so we sat there at that stop sign and DARPA came out and said you know excessive delay at a stop sign and we opened the door to the vehicle had a valid route plan and we’re like oh the timer’s clicking down from 20 when it’s not 2 minutes so again in an offhanded manner and manner that the size of the vehicle bit us that and the fact you should never change code you know the night before you’re gonna use the code so I want to touch on this topic a little bit going again why did we do this so how reliable are human drivers and this conversation comes up in almost every interview I do you can go to nits or any governments website and find out how many miles the average person drives and how long they live and you know such things and find out how many miles are going to drive in a lifetime it turns out that the key number I want you to take away from here is there’s about 14 and a half fatalities per every billion miles driven or to put that in another fashion roughly you know ignore the round off error here it’s roughly 100 million miles between fatalities so humans on on average drive 100 million miles between fatalities now you say wait a minute you just told me like a lot of you died right well that’s because there’s billions of us on the planet we all drive a lot and so you can actually calculate what your probability of fatality is it’s just you know 1 minus e to the minus alpha X where alphas the failures per per unit distance and and you can accurately just by sticking in these numbers that come out of the database you can say your probability dying in the car accident and a lifetime for the average person is about 1% and that totally agrees with the other accident statistics now why do I throw that out there because I want to compare that to robots so again I don’t want to go through too much arithmetic here but if we look at instantaneous failure there’s a probability of failure in a short amount of time is just proportional to that amount of time so the probability of failure is just alpha DX and okay I’m going to give up on the laser player unless somebody else has one but then the probability of success is 1 minus the probability of failure and you get that that’s e to the minus alpha acts where alpha again is the Nemean failure rate and the interesting thing about that is I turned that equation around and I said okay let me just apply that to the two DARPA challenges that we did so in this case success was 5 of the 23 vehicles finished the second challenge and six of the 11 vehicles finished the urban challenge so if I calculate you know what’s the sort of mean failure rate at this point in time for robots of this vintage I got about point zero one in other words about a hundred miles between failure and that was exactly what we were more or less measuring when we were testing prior to the race and our stupid screw-ups with the software but this is an important point a hundred miles between failure humans are a hundred million so every time we talk about doing a level four autonomous vehicle you have to in the back of your mind realize that in the court of public opinion and in the court of law you’re gonna get sued if you don’t achieve something that resembles human or better performance and you hand it to a consumer who has the expectation that you’re giving them that something is good or better than they are so we have to close this gap between once in a hundred miles – once in a hundred million miles so this is why this problem is so daunting so here’s what our test fleet looked like in 2013 and we’re still driving these cars I’ll play a little video clip here it’s really boring so autonomous vehicles from a consumer point of view should be consistent reliable robust comfortable they should be predictable and so we put people in this car and they’re just bored numb and that’s success now this is a proving ground right across the street from my office and I’m just gonna skip forward to the hope it’s the next slide here I guess maybe I can let this play out there’s only a second left see if I can fix my laser pointer anyway we don’t always drive that way so sometimes we’ll take people around that track it’ll be strewn with obstacles stalled cars cars to pass cars to merge with pedestrians not to hit you know all the normal things but once in a while when they go yeah you know I’ve seen that elsewhere we go okay well well ours goes to 11 and how many how many of you are old enough to get that will drive the vehicle to where it gets airborne so on this track it gets airborne at 60 miles an hour and or six tenths of a G there sound with this you can hear the engine screaming but the sound doesn’t work on my computer here today so the point of this wasn’t just to say we could drive six tenths of the G it was when we were actually working on the longitudinal and lateral control algorithms but so we’d give people a ride on this track you know and be like grandma and then we take them out here and you’ll see here why it suddenly gets a little nauseating it’s a lot better with the sound cuz you hear tires squealing and such but in any event the vehicle can perform at the operating m envelope of what a human would ever want to do with it so what are the sensing suite look like on the car today looks pretty much like the DARPA car only M everything’s gotten a little bit smaller a little bit cheaper you know a little bit less expensive our vehicles are typically in the in this generation of fleet they’re typically for light our units on the roof for mapping and localization and obstacle detection we have a suite of radars under the skin and we have some machine grade cameras and that’s about it so how do you drive into town on this vehicle pretty much everyone has a notion of this it all starts with knowing where you are that’s what we call localization so you take some combination of your prior knowledge of the world maybe you have a map maybe you have some other sort of prior information you take the information coming from your sensors and you know maybe you’re lucky enough to have you know an inertial measurement unit with some gyros and you know wheel speed encoders and stuff figure out where you’re at and then you use your sensors to figure out where the other stuff’s at then you plan a path that doesn’t hit that stuff and then you control the vehicle that you know drive like you just saw in that previous video it’s pretty simple so let me show you an example of two of the sensing modes so on the top here you’ll see the rings of the laser scan in the ground this is just one scan this happens you know 10 times a second but this is the corresponding camera imagery outside the research building afford if you put the two together you will see the individual cameras up here I didn’t stitch them I overlaid the laser strikes with the camera and I’ve removed those in the ground plane because if I covered the ground plane you couldn’t even see the rest of sure and then and then then the problem is down here so then you go out driving and so you see the power of sensor fusion right both the camera and the laser agree where where these objects are these things sticking out of the ground plane that shouldn’t be there they both agree and if I overlaid the radar in any other sensors the idea is to try to put them all in the same reference frame so that we can get the most robust information about tracking the obstacles so I’m not going to play that video it won’t play on this machine it’s got a bad codec so I want to talk a little bit about this one I think I maybe even pause it here this is my favorite video this is a big debate right now about whether you need a map to drive autonomously and I have an opinion on most everything in the world and I have an opinion on this as well I think as we go forward you know the artificial intelligence and some of the machine learning techniques you know deep neural Nets will do quite a bit to help us drive in areas that we don’t have a lot of prior knowledge but I make the counter-argument that if I’m going to say you know go ahead you put your kids in the car and let them go to school or you know you go ahead and take a nap in your car I’m not going to put you on that road now unless I know that roads traversable so it means I’m going to drive it once to know that a vehicle can even get down it and while I’m driving it I just keep the data so I keep the data I keep it like this it’s 3d data and I make a map out of it I don’t throw it away I just don’t quite understand why you want to throw all the data away so what you’re looking at here it’s kind of bright in this room it’s going to be hard to see but everything above the ground plane I’ve colored cyan and everything in the ground plane is the reflectivity of the road so that’s actually from those laser beams on top the car measuring the reflectance in the ground plane by the way that other thing that was Michigan Stadium that went by so anyway so here’s the trick right so we’ve driven it once we made this fancy map here so now we come back and the idea is just to answer the question where am I in this map so now I’m gonna read rive this thing and only the live data I’m going to color it so now you can see I just snap the two to a line and now I know to within a few centimeters where I am in the world and any of the stuff sticking out of the ground plane like these cars here just don’t hit them and in a nutshell that’s how the vehicle works it’s a gross oversimplification but and I’m only showing the light our data here we have corresponding you know camera tracks and radar tracks and you know all that sort of stuff but it not to clutter up the problem too much here if it were darker in the room here you can see every crack in the road and every tire strip and you know every tiny little feature might not seem obvious but almost every piece of pavement in the world is different at this scale it’s like you know your thumb looks like mine except when we look at the fingerprints they’re really different and and that’s actually it’s not the only localization algorithm we use but it’s a really it’s a really good one and when the ground gets covered with snow we omit the ground plane and then we just localized to the 3d structure above it yes question yeah yep I’ll answer that so I didn’t put those slides in here but you know I made a map in the winter of 2009 and I drove it in the summer of 2011 winter summer snow banks leaves write your question roughly 30% of the scene content changed you can easily reject those as outliers and fit to the inliers okay sinkhole it’s about three slides from now okay we’ll get to that it literally is so let me just summarize that again so we make a map and everyone’s going to disagree about what features they put in their map but this is just one example right ground plane intensity and then you and I agree is a society we say hey we’re gonna call these things roads that we can drive on you don’t drive on the rest of it just where we put the green lines and they have names you know and they have some rules you don’t like go only go one way on this one and you know this speed here and stop there and your favorite Starbucks is over here and we just put all the metadata in the map and then the idea is really we just come back and we just see one little piece of that puzzle and go well where is it and we Slough it around in the map a little bit actually we don’t even go that far we just go plus or minus about 20 meters and we get a couple centimeters localization lock so not to belabor that point but it’s just one of the algorithms that we use to solve that big problem right in the middle of everything where am I because once you know where you are then you can segment out the stuff that shouldn’t be there and not hit it so I’m not hit it front I want to talk a little bit about sensing and how that’s evolving how many of you guys let’s show a vote of hands here how many you guys sitting at this intersection would make that left turn yeah okay so this car and that car are a hundred meters away so at highway speed they’re going about 30 meters a second they’re going to be there in three seconds you don’t clear the intersection you’re gonna get t-boned it takes you about six to ten seconds even aggressively to go from a stop through a left turn up to highway speed so you better be looking six to ten seconds on the time horizon for those of those obstacles and what that really corresponds to is more like these cars in that car so how big are those cars so now that gets to there and what resolution you need to look at so you know you look out in the horizon what might it be it might be a motorcycle might only be half a meter wide and a meter tall and it’s 200 meters out there you know what angle does that subtend on the horizon so if I don’t have that resolution in my sensing at least one mode that’s reliable I’m not gonna see that and I risk you know and this is just one example I can probably give you dozens of examples like this where you’re sensing horizon can get to be quite long and the reason that we’re having challenges right now meeting this is because lasers like this one and it’s pretty close to the frequency of the laser on those scanners your eyeballs can see them and your eyeballs heat up and so we have rules about how much power you can put out there and so now we’re trying to put you know that power is efficiently as we can out there and return it similarly for our people say well why don’t you plaster a bunch of radars around the car well you can only put a radar beam plus or minus 10 degrees out to 200 meters you’re limited in how much power you can put out if you want to put it 360 degrees around the vehicle the FCC if you know the Federal Communications Commission’s says you have to reduce the range it’s the power that they mandates so if you do 360 radar you only need up to 80 meters so right now you see a lot of startup companies working in these spaces of you know getting the cutting edge of the sensing technology to be able to get out to these time horizons that you need for some of these difficult driving tasks yeah sure anyone know what the half-life of ok I have three answers for you the deer don’t talk okay sure but they do jump out of the woods fast that the flyff of a vehicle in the United States is 11 years so if I’m if I waved a magic one it said tomorrow federal government every new car has DSRC radio 11 years from not only 50% of the cars would have it 22 years from now you know it’s another half-life so we’re talking like decades and decades and even if I had an even super fancy magic wand and I said I’m gonna retrofit all the cars today I don’t know that yours is working today yours could have gotten busted you know the fuse might be blown so I still have to stand on my own merit and the answer I always give people is I’ll take any data you give me because it’ll make the answer more robust but I can’t rely on it always being there I have to stand on my own merit so yeah we’d love DSRC you know we’d love all that stuff they’re not talking yet so right now our cars are driving in these scenarios and you know we’re managing to get by it’s it could be more robust and that goes back to this question can you meet human performance of 10 to the 8th miles between fatality right so that’s part of the part of the answer but in and of itself risk slide I just threw in here to show this just doesn’t happen on rural roads that happens you know this is right near our office in Palo Alto you know they have these little stoplights in California got to wait and get on the freeway and you’re waiting here and you got to go from zero to 80 miles an hour and like these things can occur at any angle any weird angle so you really do need 360 degrees fencing these people are thinking and put a sensor out the front well they’re obviously not going to drive anymore with a merge ramp or two roads across each other or anything but you know 90 degrees so let me jump jump ahead this is a car that we’re developing on right now it’s also a fusion platform but this one was production this one has what I would call is moving in the direction of airliners so we’ll have like dual cam buses dual electrical systems dual epass steering racks dual brake systems can’t have any single point of failure right now if you’re driving the car and the power steering goes out you as the human can still steer it you know you’re the backup if you’re a level four autonomous vehicle you have no backup so the platform of this vehicle is designed to have no single point of failure and you see see that progression from the you know the 29 pound laser down to you know these are probably three or four pounds and these are now you know like hockey puck size and it’s a continuous progression I’ll just skip these slides here I threw this slide in here to switch topics so the slide doesn’t really mean much so I basically kind of talked a little bit about the DARPA challenge and why we got into driving autonomously and sort of where Ford’s at and some of the algorithms and sensing challenges out there so a lot of you have read in the news and I’m sure you’ve got all know what you’re here in Pittsburgh we joined together with this subsidiary company that we helped form called Argo dot AI how many of you heard of Argo day I and we basically had announced that in 2021 we’ll have a commercial product doing level 4 autonomy and it’s a very defined in goal it will likely be a fleet service doing rideshare or package delivery something where the vehicles go out and back every day they can be serviced and calibrated and the routes well understood that’s the kind of thing you can expect on that timeframe and you can’t expect everything in 2021 the media doesn’t seem to understand that and so the the sole focus of our go day eye is to take the experience of the guys that they form the company and the funding and the experience of day to day Rob most of my team from Ford or well it’s a partnership that is to take their experience our experience at Ford and get up from the bureaucracy of a major corporation and go off and just you know hammer on making that first production product feasible in that timeframe so that’s a blessed met a mixed blessing and curse for me I’m no longer on the hook for you know meeting production deadlines I don’t like I’m not a product guy I’m a research guy on the other hand you know I don’t get to every day go out there and say geez you know I kind of do it this way versus that way but but what I’m doing now is we’ve taken all the people in research that didn’t go off with the agro endeavor and started to look at the broader mobility picture it’s not just self-driving cars that we’re interested in in the mobility space and it’s not just these limited use cases like driving 25 miles an hour around in a rideshare so a couple examples here that are just immediately obvious I’m not going to go into all of them how do you drive in you know bad weather right that’s out of scope in the next few years you know you’re not going to drive in in you know really heavy snow or rain you’re not going to drive really complex topologies like this you’re not going to drive at Audubon speed and pick up road debris I’m going to show you some slides about road debris and then there’s something like I said I just pulled these slides randomly so this would be I’m going to show you some road debris on a road like this but I want to vote here which one of these roads do you guys think is the harder to drive how many of you pick California how many of you pick the Bolivian death Road I’ve driven both of them this one’s trivial to drive for robot there’s a huge discontinuity there’s negative infinity there and positive infinity there and there’s only one flat spot there’s only one lane there’s not much decision-making there that’s trivial this road on the other hand is the most dangerous road in the United States it’s the boring section of road between Los Angeles and Las Vegas and most of the road has virtually no shoulder and very loose gravel if you put a tire there you’re gonna roll the vehicle and it’s almost certain death I’ve seen it in person so the kind of the point of this slide it’s kind of a legacy slide in my slide deck is the newest people and you know sort of the public perception very very often handicaps the easy and the hard problems in reverse reverse order and and it’s easy to see why they do so so speaking of road debris I often ask people you know there’s a lot of weird things that can happen to you in a lifetime of driving and if I went around the room here and ask each of you what’s the weirdest one that happened to you it wouldn’t be the same as me so this is one of the challenges with getting to the 10 to the 8th miles reliability how do you understand all these weird things that can happen right you know for me and one of the stories I tell this I was actually driving behind a horse trailer and the prize horses fell through the bottom when it rested out they got ground into chunks and thrown on the hood of my car so that’s why I have this picture here I don’t have a horse getting flown through the air but you say you know this is pretty rare but you go to Google and what do you know there’s more of them and for any of these problems that you think of so you can just go to Google Images and think of any weird thing that you say will never happen and you can find pages of them so here’s another one pole through windshield that’s a Google search so I find this one here to be pretty insidious and this goes back to the sensing set so in a camera image you have no idea if that’s you know imagine before it got to you you really have no idea how big that log is right I mean these cameras a two dimensional imager and a radar sees right through wood it’s transparent to radar so if you don’t have a lidar system on your car that can resolve something that size you’re never gonna see that pole coming out of your head and these poles can be any size and literally I’m telling you you can go through pages and pages and pages and pages on google images and find scene after scene like this where people you know they they fail to realize that things are sticking out the back of vehicles and and for every one of these scenarios and there’s trust me I’ve watched tons of Russian dashcam videos and you you can find amazingly weird things just a couple of them here that are challenging so they’re not like that one I can hit so why can I hit that one but not that one so okay so now here’s where cameras come in right because the you know camera kind of can maybe classify that as something yeah and that’s so good that one you can ignore so each of these sensors has you know some strengths and weaknesses surprisingly there’s about a million mattresses flying off of cars you can find just literally hundreds of videos of them whizzing by things this guy on the motorcycle I had to do some fancy slalom turns to get around that I know where that hatchet came from but there’s your sinkhole and this this one’s a pretty difficult problem in general any any negative obstacle in the road is basically a line-of-sight issue and to some extent the first car is going to be the probe vehicle and report it back to the cloud for all the other cars that’s a fairly intractable problem at the moment I throw this one in here because this is our friends Tesla they’re Auto Park it illustrates what happens with today’s radar so people always say well you know why do you use lighter just you just spray some radar out there well radar doesn’t have any vertical discrimination so it basically goes out to a very you know narrow field of view in the vertical plane and so you really need to put all these sensors together so let me just close with a few slides here about some of the broader scope things that we’re doing in the research lab right now so we’ve renamed you know after we’ve shed this first launch and and let Argo assume responsibility for it we’ve renamed our departments the robotics and AI department and we’re thinking about transportation and mobility and robotics in a broader sense and so that could include things like moving into three dimensions and so you can imagine you know package delivery this way because you know you have that truck that Argo made that drives to somebody’s address but somehow you got to get the package from the truck to the front door or you know there’s that last little 100 meter problem there’s also a bunch of people out there rethinking now that electric batteries or now that batteries make electric propulsion feasible for flying aircraft they’re thinking of for seat drones like this it can do vertical takeoff and landing these wings rotate so just lift straight up and then turns into cruise flight at 200 knots so now suddenly if you’re in the Bay Area and it takes you two hours to go from San Francisco to you know your office and Cupertino where you work at Apple or something you can be there in five minutes so there’s a whole bunch of business models around thinking about adding these so you can imagine your autonomous vehicle takes you from your garage over to you know where the nearest one of these little helipads are you know a mile from your house and you hop in that and whisk off to the inner city or you know maybe the Steelers game or you know over your favorite sports team is there’s also a lot of that can be done in the world of personal mobility and assistance devices you know obvious ones out there are things like the Segway we’ve all seen variants of this this is a popular one for you know not just models and high-heeled shoes but elderly people who want to get their groceries as you know through the mall or the shopping store you know I saw somebody in the elevator today on one something like this you have people working on robotic prosthetics to help you know elderly a whole bunch of works going on and bipedal robots you can imagine this thing could be your the last hundred meter pipe problem to get your package from the end of the driveway to the doorstep because not every driveway allows you to walk right there might be some stairs involved or some cobblestones or something or I mean not every drive it allows you to roll like this so and then another thing that’s become popular just only recently is rethinking actually how we make everything in this country and in the world and just making all the factories you know completely robotic never used to be sexy to do you know a factory robotics but but now people are thinking you know I don’t know do we do it this way or do we just create some humanoids that you know can can do it so there’s all these new ideas that are opening up where you use the same techniques you use in autonomous vehicles right you got a know where you are got a localize you got to sense the world around you in 3d and you’re gonna take some control action to get through it without hitting this stuff and so pretty much at the core a lot of the key elements are the same and that’s where we’re moving in the future at Ford generally broadening the scope of robotics and artificial intelligence to not only include the driving cars but other things and I think I have a way extended my stay here and I’ll take as many questions as you guys want to answer or one I have answered yes [Applause] encouraged to me I failed to show you the most boring video evolved because it won’t play in here I’ll play that in the background while we what we answer questions if I can open this file you know it sucks when you get old and you got to pull out your reading glasses to see the file names so we had this question at one point in time can we make a large-scale map using that intensity the ground plane intensity that I showed you so we drove this route in California well basically it’s kind of more or less from Phoenix to Palm Springs and back and I’m probably missing the play bar at the bottom here right yeah okay I just let this play in the background this is a time-lapse of what it looks like to drive on the freeway it’s totally boring except for me and Cheetos but that’s how it should be okay questions go for it I’m not in front of the news media I can be a little bit more up front so that’s fuzzy logic part two yeah yeah okay so you actually hit on one of the sore points that I like to harp about a lot so you know the pendulum swings right now the flavor of the day is deep neural Nets and in deep learning of all sorts I think it’s great for some problems like classification I don’t see any reason why you’ll use such a thing to learn what the speed limit is because you know it’s posted on the road or what the name of the street is it’s in the map database or or how you run the control algorithm on the engine of your car you know I mean we understand those things there’s mathematical equations for them so in my world I see a mix of properly using you know artificial intelligence you know DNS with rule-based logic other people disagree with me they say you know just stick a camera into an Nvidia card and let it turn the steering wheel however you know a hundred million drivers did and I kind of questioned whether you’re going to get the New York City taxi cab driver from that model or you’re going to get a good driver but there’s another problem with it that you bring up it’s the traceability so when you write rule-based logic in equations and as a physics guy I’d like to ever have everything be just a simple equation it’s pretty easy to debug what you’ve done and understand where things went right and where they went wrong it’s a lot more difficult with any sort of deep net and that’s that’s an issue it really is how you well everything you’re seeing here is completely rule-based at the moment so we’re just now adding to some of these scenarios particularly the things that you do need to classify and I’m also of the belief that people worry too much about classifying things that they don’t have to if I’m driving down the street and you’re having coffee at the cafe on the side of the street I don’t really need to know that because your momentum vectors never going to be in the street you know so there’s there’s this tool that people like to use and they want to classify everything under the Sun if it’s sticking out of the ground plane where I know you and I have agreed to Road I just don’t want to hit it after the lowest order at a higher order there are different rules for things like a FedEx truck and a school bus right they might look the same size and shape but to school bus you know you can’t pass it when the kids are getting off and so you do know need to know some things and classify some you need to classify and learn everything I tend to think not but I might be ruled wrong by history yeah brother patient validation of cases yeah self your DMV team so that’s what Argo is about I get to go back to research no but to answer that question a little bit more honestly we data log everything we do so terabytes of data an hour come off this vehicle and anytime we make an algorithm change we can play it against all the previous data logs to see if things got better or worse we can use learning networks on the prior data and you can make generalizations right that horsemeat that got thrown on the front of my car I didn’t really need to know that that was horsemeat right it was just debris it was debris of a certain size and you know volumetric size you had different momentum vectors than me I can make a decision in this general sense about what I should do about debris so lots of verification and validation you know lots of replaying against previous data lots of acquiring data and resem you lating it you know a lot of work goes into it and I should also mention things like there are more mundane to guys like us is you know like the ISO 26262 requirements and you know systems engineering and failure failure modes analysis and you know all of that kind of stuff oh well go ahead I’ll stay as long as you guys want so this is this is another difficult question because a lot of states want to jump in and say well you know if you pass our driving test you know we’ll give you a thumbs up and but we don’t want you know 50 states to have 50 different driving tests that doesn’t make sense so nips has stepped up to start to write some sensible rules about minimum sort of requirements and testing not at the level of detail of telling you what specifically to do but at the larger scale of saying what the performance at the endpoint should be you know like you should not hit pedestrians they’re not going to tell you how not to hit the destron for example okay I got a question over here I’m sorry the people moving then when a car wreck happens how fast what so what went okay so if I understand it right when a car wreck happens how fast am I gonna respond to it in front of me so one of the things that humans do let me get rid of this distraction here you saw a boring the freeway driving less so one of the things that humans do is they tailgate and we violate you know thank you windows closed the program we we violate all sorts of rules to tail gating being one of them so if you’re if you’re a robot you can say you know I know what the deceleration capabilities of my vehicle is I know the stopping distance don’t get closer to the anything in front of me than the stopping distance so theoretically you should never be in that situation now if you do what ends up happening in the reality out on the road is people get pissed at you and they start leapfrogging because they want to tailgate and so you have to play this game with fitting in with society and and you know fitting in with giving a guaranteed safe path now we all do it every day you know we all crest the hill and we don’t know what’s over the far side you know we we pull out when we don’t have line of sight there’s just a million examples I can give you you couldn’t get home without taking some leaps of faith but generally when we’re out on the freeway driving we don’t get in those scenarios we we always leave a gap that we can decelerate to to zero at a comfortable pace and without you know fully locking up the brakes something that a human would find comfortable yeah look at the ground plane in the in that map okay so we have probably at least half a dozen localization algorithms and in that one we’re simply looking at the reflectivity of the road at the wavelength of the laser and the laser will give you range elevation in azimuth and you can back out you know what the height of the ground plane was you know based on that and we we tile in that particular map is tiled into ten centimeter tiles we search about 20 meters around the vehicle we get a pretty good lock I don’t know how much more granularity you want me to talk about it on it’s not as simple as I’m saying okay because there’s actually four lighters on the car each with 32 beams so there’s now a hundred and twenty-eight beams sweep in the ground and they’re all hitting at a different angle of incidence from a different direction so we have to backward reconstruct where they came from make sure they all observe the same piece of ground with the same intensity so we have to correct for the you know the spot size and the angle of incidence and all of that kind of gut stuff goes into the nitty-gritty details and some threshold on the noise and things like that but essentially we’re just taking a black-and-white photograph of the ground at 905 nanometers we don’t we don’t we don’t consider it a plane I just generically say ground plane to mean the road right I mean it’s not a plane right there’s ripples in it and we see ruts and and features and yeah we see hills and yeah it in fact if I had more time here I’d show you how we fit two three dimensions and and we have some very interesting algorithms for seeing things like you know wires overhead and tree canopies and you know stop signals it just turns out that that’s the simplest algorithm to run robustly it’s not the only one we run yeah so I kind of mentioned this earlier I there’s a debate about there about how much you can do with learning and how much prior knowledge of the world you need and I cannot predict how fast computational power and sensing capability will get you to the point where you have lesson reliance on that I can only make the argument that I’m going to drive those roads before I let you drive them taking a nap as long as I’m driving them I’m going to exploit the data and make a map it cost me nothing so you know can we make the maps what people will call lighter weight you know less data intensive yeah highly likely how fast we’ll do that I don’t know it may be some some stretches of road the like in an inner city like this there’s lots of features right corners of buildings things that you can just extract a few features from and know where you’re at if you’re on a road say in Iowa or Nebraska or Nevada where there’s no features whatsoever there’s just corn corn corn corn corn corn corn and Road the only features you have are the paint lines on the road and the cracks and things like that that’s the only features is the ground plane itself you’re not going to find any above-ground features to you know to to put in a different feature space that’s not as intense as a intensity map that would actually be probably easier in some aspects than driving on on the road right because there’s a well you know everyone’s following the same rules all the cars have the same sort of performance you’d have to up the you know the refresh rate on the sensors right now most of our sensors run at 10 or 20 Hertz which kind of limits how close you can get to the vehicle in front of you because at that speed you know it takes us finite amount of time to close the brake calipers and accelerate and stuff so you need faster sensing and mechanics mechanicals on the car but I think it’s totally possible right now especially if you have the V 2 V right like the race cars are talking to each other then it’s it’s probably a containable problem you got these big barricaded walls too so you know you can hit stuff yes so if you go back two years literally the only company in the world unless you built it yourself it was offering lidar that was suitable for these test vehicles was validating if you look in the marketplace now there’s probably two dozen or more of them and there are some interesting ideas out there but ultimately it all comes back to traffic and scenarios that I need to pay attention to it can happen 360 degrees around the vehicle likewise if you’re in a city like San Francisco or Pittsburgh you know you can have roads with discontinuities in the road grade right so imagine a road comes down like this crosses the intersection goes down like that you know San Francisco I not only have to look this way but I you know I have to look up at whatever you know that road grade is so you need to expand the field of view that you’re looking at right right now the lighters we don’t have enough beams to afford to throw them away and the valid I next product is 128 beam 360-degree lidar there are other people out there doing similar things by taking single beams and rastering them around with mirrors or phased arrays or things like that I think it’s too early now to determine who’s going to win out in the end on the one hand the rotating units they do see 360 degrees people will argue it rotates you know it’s moving but you know the wheels kind of rotate on the car as well and pistons go up and down and I don’t know they don’t seem to fail the solid state units people call them solid state but they’re all really made out of a solid so that’s kind of a misnomer to that the solid state units typically have 50 degree field of views at best so you’re talking about you know maybe put an 8 or 12 of them around the car to do the same thing that one valid I’m would do and yes their package smaller but you got a multiplex a whole bunch of them together and you know I don’t know where that’s going to come out in the wash and I actually point out the reason this is really challenging to get out to that 200 meters is the eye safety requirements if you move out to the mid-infrared at 1550 nanometers you can put a laser beam as far out as you want but then you move to a different semiconductor you’re out in the indium gallium arsenide and things get expensive so can people drive down that semiconductor cost to be similar to you know silicon or ái at work here yeah so I don’t know it’s a the marketplace is still open there as is it is for the 3d imaging radar and you know better cameras the sensors can be improved across the board yes all right so if I went back I think these things came out ten years ago right if 10 years ago you said everyone would be walking around looking at tablets like this I wouldn’t have believed you right I’m not gonna really be able to predict very well 10 years out I think what you’re going to see is in some of these fronts people are going to work together to solve some of the hard problems because some of those problems are going to involve a lot of data and so there might be some data sharing you might get some groups coalesce and work together obviously not everyone is going to have the financial and intellectual wherewithal to get a product to market who knows what’s going to happen with the economy you know the federal regulations the regulations in China or you know they don’t want you to look at anything with a sensor in China right Europe there’s a lot of a lot of questions out there it’s really hard to predict I think though that the first movers do have a big advantage because they get people into the vehicles to get them to build confidence and so there is there is quite a race on to be among the first who isn’t okay no no well I don’t know if GM said it I know Volvo said it press I don’t read every press clipping out there by saying no I’m not saying that that won’t happen I just you know I haven’t seen them indemnity you know Boeing or you know things that fall out of the sky or you know all sorts of you know other things so I don’t know it’s a speculative question right and the the carrot in all of this though you go back to this the carrot and all of this is a million people a year die worldwide and car accidents and we can put an end to that can’t get rid of all of them but a lot of them so we’ll see how how much regulation pushes to help us along with this problem yeah yeah so that’s a good question so you know a lot of people come to the conclusion that by doing this we want to take driving away from people I love to drive a lot of people love to drive and so when we get past these early you know commercial fleet operations and we get into personal ownership models my vision for the way the autonomous vehicle works it’s just like cruise control today right when you like driving you drive when you don’t you just push the on button and you let the car drive so and maybe you you can choose modes right maybe you want to do that six tenths of the G driving or maybe you want to do as a grandma like driving it’s it’s all programmable so yeah we don’t plan on taking the driving experience away from people maybe yes I’ll take a couple more questions here because I don’t want to trap people in the room that want to leave yes yeah I did there was a slide in there I skipped through it really quickly like you know what are other things people might want to do with their cars I had a whole bunch of really cute videos one of them was platooning a trucks that certainly is a business model and you’ll see that being developed plenty of people working on that right now there was a project in Europe concluded maybe two or three years ago called Sartre safe autonomous road trains for Europe where a Volvo and a bunch of other companies demonstrated a string of vehicles platooning on the their equivalent of interstates and I know three or four of the EU countries no presently we’re looking at Fleet Services in urban areas I think that’s a fair assessment of what you’re going to see from anyone who’s telling you the truth through the industry you know it’s kind of we’re way most editing or ubers heading you know a lot of people are heading for that same space because they know they can’t do everything at once and so they’re going to geofence the operation into areas that they understand well and can contain yes not necessarily what you owe but you make it okay just so you’re making it something everyone’s going to be driving on with autonomous turned on all the time so you don’t know where the problem can come from right like it is a if I’m driving in someone in in an intersection violates a red light and t-boned me you know I might have had the right away and I might not have even seen him right like say I’m the car here and there’s this tractor-trailer next to me in Lane we’re both waiting to go through the intersection I can’t see the oncoming cars you know and light turns green I go and boom I get t-boned right this is where the vehicle the vehicle Communications really helps is where you don’t have line-of-sight viewing well certainly yeah and I’m quite surprised to be honest with you that there wasn’t more backlash after the few Tesla accidents in addition in the one in Florida there was a really really terrible one in China it actually predated the Florida one so yeah that’s the question let me give you an analogy here that I find helpful we introduced anti-lock brakes on cars and airbags on cars and both of those had really negative public perception right if people were like I don’t want any damn computer pumping the brakes on my car and I don’t want an airbag blowing up in my face and I can’t see the steer I’m gonna go in the ditch well as soon as people learned that those antilock brakes help them stop on ice and that the airbags are saving you know lives and serious injury we got sued for it not being standard equipment in the passenger seat and then it got mandated by the federal government so now all cars have a knee lock brakes you know some sort of roll stability control and airbags and so the bottom line is that you know people have to get used to the technology and then when the data comes back and and sort of speaks for itself then then you get a lot more leverage on the public perception you know there are standards that the most common wants something known as DSRC there’s been more than one iteration of what that acronym stands for but typically it’s dedicated short-range radio communications transmit basically your location and your heading and your mass of the vehicle a very limited amount of data about a hundred meters to the cars around you that standards been under works for more than two decades and they keep saying that next quarter next quarter next quarter Congress is going to act on it never seems to happen it may eventually happen but it’s kind of a moot point just a juncture in time because you’ve got things like 5g and LTE and other other communication standards it may have already leapfrog that standard now that doesn’t mean you don’t still have the ability to wireless communication it just might not be the DSRC that was proposed 20 years ago we’ll see where that goes I don’t know as I said earlier we do have to stand on our own merit and be able to drive without it [Applause] thank all of you for your interest in the project and in attending today


Leave a Reply

Your email address will not be published. Required fields are marked *