LAS16-409: Time Sensitive Networking kernel modifications for automotive industrial

LAS16-409: Time Sensitive Networking kernel modifications for automotive industrial


so okay so let me give her so they’re asking do a little bit of a presentation or or I’ve been and okay let’s start Becca Juarez from Texas Instruments with some background in networking also outside also before joining T I so I was gonna give a little bit of a talk at the kind of the requirements or a use case level on a kind of a trendy trendy topic of time-sensitive and networking or and so so just to set the expectation so it’s kind of a Rick describing the use case not trying to propose that this is the solution to build I’m kind of looking for that for the I guess the folks in this room and then in general to see if Leonardo is interested in picking up and picking up on this and sort of book coming up with solutions so it’s kind of a mix between the okay let’s see let’s see what its a mix between okay so what is time-sensitive networking it’s most people approach familiar with Ethernet a VB or audio/video over Ethernet from from about over ten years ago which was kind of an OK solution it’s the example pictures just pictures I guess make it more concrete so the middle picture here’s a audio/video mixer not Ethernet avb so each individual cable is an individual stream so you can imagine that this is that’s a lot of cables a big mess so replace it with one big fat Ethernet and and the world is good that’s that’s kind of the the motivation and and that that’s of course a very clean and elegant set up to the it doesn’t mean it all look that clean okay so then sort of at the same time in the automotive world of course people are using their canned buses and so on it’s not using Ethernet but and then in the industrial market they’ve kind of moved over from proprietary serial bus based proprietary networks to Ethernet that were a little bit hacked up in one way or another and they were either like ether cats or prophy nets or anyway kind of variations of Ethernet in one way or another okay so what’s happened there is kind of the the a VB version – or or is where these the interest of these kind of three domains kind of started to merge that they are automotive guys and industrial guys sort of noticed that okay Ethernet it’s winning everywhere so how can we jump on board and get all the goodness of Ethernet but we still want our our guaranteed delivery type type type features okay so most of the TSN is really related to two ethernet mac layer but but but there’s a few items which I believe at least show up in the software and kind of migrate towards that that part yeah and then here’s a pointer to an excellent presentation for for for an introduction on on on kind of all angles of TSN it’s by university of new hampshire so so it’s kind of neutral and it’s a nice description of 25 slides on standardization bodies and various control rooms who are involved and then kind of like what are they trying to solve and then the the end part of it is how it’s being solved so that’s an excellent presentation so I’ve stolen a whole bunch from there so it’s just gave credit to Bob Noseworthy so this is then the the kind of the hardware centric view of what is time-sensitive networking well in reality it’s a bunch of I Triple E standards they’re just bundled in and into this I don’t know semi fuzzy DSN so what exactly is it do you implement all of these and so on that’s not really I don’t think that’s defined so you can do time-sensitive networking and only do like three of these I at least in my understanding you some people might still call it TSN so alright so if this without digging too deep just describe it so the first thing to solve is of course you got to make everyone have the common agreement of time you can do it by putting a dedicated wire to distribute a clock of course that’ll work or you do it with a conveyor belt like with an old factory I mean everyone is not the same time if the conveyor belt is your your every everything is moving with with the belt so time synchronization has to be solved of course time synchronization needs to be solved for other reasons as well so so that’s not necessarily unique to TSN okay then there’s a bunch of I think the next four there’s q q AV s and Q B B’s is there’s a queueing model where really the unique new stuff I would say are the time aware queuing or time aware shaper and frame preemption so let me just kind of focus on those a little bit here just of that so the the so the time of where queuing is is basically in an ugly way if we said it’s TDM over Ethernet so they’re recreating TDM over Ethernet and if you look at it it’s it actually kind of goes to the level that I think some of the people involved in the standardization are the same people that did it in atm so it’s it’s kind of a back to the back to ATM but just over Ethernet which is not necessarily a bad thing it’s and I don’t think it’s yeah and then they do have included now approval it’s frame preemption in Ethernet so frames larger than you think 128 bytes there is a true frame preemption and and continue after pre-empting so that the best-ever traffic can send their jumbo frames or fifteen hundred frames while the the the real-time traffic gets there little packets in their time slots through okay okay then there’s a couple okay redundancy comes into play as well of kind of indirectly in but it’s also not just time-sensitive network but usually redundancy matters in in TSN as well okay at the end there the the second from last they had to to get the preemption going you gotta touch the lower levels of Ethernet so that that’s included so touching them yeah but all that’s approved so it’s it’s no it’s no longer like a unique brands its standard Ethernet okay yep a little bit more on the motivation so it’s pretty broad I talked about this but if you know I just listed a bunch of companies to kind of generate that okay this is something that it has a broad set of interest if I listed some company I they they’re not endorsing me listing them or I I’m not saying they are really good at this but I’ve just that in a public forum you can find in the kind of the consumer audio-video contributions or or conference presentations by Broadcom Intel apples some of the they support some of the advanced features and in some of the Mac books and people have kind of referenced those on the professional audio video site they have the there’s a if you if you look you’ll find some pretty there’s a very nice example of the ESPN’s for those who are not from the US that’s the big sports channel so their main main Center in the East Coast in Bristol is called DC – it’s it’s built from scratch it’s it’s yeah it’s Ethernet ABB across the board like a couple hundred million dollars worth of stuff Sports Center is comes from there so they’re promoting it as an example and and there are other these professional refereed Oh Harmon and other professional audio video guys and Cisco of course Cisco’s in many of these domains but but for example in that domain they’ve shown up the car world it’s everything from the driver assist infotainment chassis control powertrain body so today they’re kind of different technologies for different if you open the windows it doesn’t go over the same bus as if you if you engage the emergency braking in the future it might so yeah so they’re both the end producers like Audie and BMW and then these massive mass massive kind of the technology providers like Continental and then the semiconductor guy is like like Austin Broadcom are active on I know in the industrial side although all the big names are there so basically nobody’s missing that that’s kinda I guess what I what I’m trying to say okay then going a little bit to why should why should Leonardo care well it’s this is a why maybe why you know its limits networking arm is dominant this stuff is kind of some of it is very old so it’s kind of predates the dominance of arm so a lot of people are looking at migrating to arm and and and it’s at the same time it’s our tours all over the place or even bare metal but but if you built your serial serial port based communication mechanism in the 80s you might have used serial port now you’re finally upgrading you’re looking at Linux at the same time so so I got this I update my networking and I got a update my OS seems like a good good good intersection with Linares doing the IOT or industrial IOT industry for all in Germany whatever you want to call it I think the one unifying thing there is they they want the full connectivity and security so and the easiest way to get there is just put Linux there and run your full-blown stacks the same there don’t try to go squeeze it into a microcontroller I mean the separate stuff is all fine but but an easy way to get to a full-featured connectivity and update mechanism its booting Linux ok and then you also have to this is this this is true embedded space so it they need to be cheap the silicon that needs to run this is it kind of needs to go down to the sensor level that that I have a sensor that the Box sells for a couple hundred bucks so the silicon that’s able to do this has to fit in that budget so so nowadays you can get Linux capable arms so cortex a devices in such kind of an MCU type price range okay and then all these markets also want their development cycles to accelerate I mean traditionally it’s a new product like every 10 years or something so now the the at least the management there is dreaming of can we move a little quicker and then they have a whole bunch of new new engineers who kind of expect better and so there’s a lot of things driving towards that yeah so okay then I gotta get a couple pictures just to reiterate why this is different so this this one I took it from from Michael Tina’s presentation but it’s it’s just showing time so we we have the we need the timing over packet to get everyone to run at the same clock if you don’t have that forget about it I mean but it’s also that that’s kind of basic timing over packet needs to be there then I need scheduled cubes and preemption which is the part which I think maps to the to the possible software implications because I own the wire I want it to look like like this so at these synchronized TDM instances the black line black vertical line I have agreed with the resource reservation protocol that this this guy can transmit and it’s guaranteed to get a slot across the entire network and and and I create these kind of what’s colored red here I create these kind of a tedium like time slots in the Ethernet traffic that guarantee deterministic latency for those typically a control traffic or sensor traffic telling how fast the how fast the liquid is flowing you want that goal that going to the valve that’s controlling the how much liquid do I let through so that that’s the small packets then but then at the same time I want to blues on the blue stuff which is disproportional of course the blue is usually a lot larger I want to I want to I want my regular Ethernet to work so I want my whatever software update firmware update stuff all that traffic to just just work so that that how do I make these two things coexist and I how do I of course with a switch it’s a big deal but at the at the end point how do i how do I make make that work okay hey by the way Oh feel free to fire away questions or comments so I something’s not clear yes yes exactly my thoughts but I am I am I’m presenting I have the standardization and the motivation and how it came to be that’s exactly my thoughts it’s no longer really Ethernet but they but the I view it from this use case perspective the the the the real-time the guaranteed delivery use case is valid now Ethernet has been hacked up to solve the problem and because the these semi proprietary solutions with deployed today have shown that yes it works now the I Triple E has standardized kind of a set of those I mean it’s a little ugly they call it hacks but it’s really ATM over or token ring over Ethernet oh okay okay yes so it has so what is address there is is its compatibility is a big deal so so of course with a non TSM switch in the network you won’t get the the guarantees of delivery but it will work so they’ve stuck to the the principle of Ethernet is the backwards compatibility they’ve kept that and and some of the that’s why some of the things look so kind of weird is to perhaps when you go to a detail is they look weird but if the backwards compatibility is there and and it is designed in a way that even with layer 2 switches that are not aware I can get to some some types of levels of service but but the most extreme cases are always really closed closed networks or like a closed networks probably the wrong way to say but kind of firewalls bubbles like the easy one to think about everyone knows what a car is so if you think that a car has a hundred sensors and a couple engines and it’s an electric car so then and then you kind of network that that’s a that’s a closed Network you can kind of predetermine engineer it but it might also talk to a kind of a regular regular reason it is yeah but but I I completely agree I it’s not yeah okay yeah so okay let me I’ll come back to that with with this picture this wasn’t I didn’t put any add any numbers and again Oscar sighs I found a nice picture so I didn’t draw anything so I put that’s a credit from but but I’ll come back to the number after I kind of describe this this picture it it’s kind of crap but think of the the this this part as the application that that the application is I have the real world and I have a computer that’s trying to control it so I gotta get readings of the computer doesn’t know anything about the real world so I gotta get sensors to tell me what the real world looks like how fast is the wheel spinning what’s the air pressure I gotta get all these readings in then I make decisions and I move some actuators that that that affect the real world so that’s like a I guess a fancy it’s a it’s a cyber physical system or whatever that if you want to put some other fancy words but that that’s my loop but but the problem is the physical processes might be over I don’t know what the factory for it’s not just right next to the computer so that that’s where how do i how do i talk to these so that’s the application and and how tight can i make this loop that that comes to the number so so okay so I’ll get so I think the tightest loops people run today are like about I think there are these magic numbers of it it’s 31 31 microseconds for the control loop which includes read the inputs make decisions modify the the the motion controller like a like a factory robot I think the tightest loops would in commercial use today or something like that but there’s a whole bunch of systems where that is you don’t need it 250 milliseconds is fine for the control loop so I think where the the so there’s a little you don’t always need to target the tightest loops but then all these budget of how inaccurate are my clocks there out of that loop its jitter at the transmission of an input and output side so you kind of calculate that budget out and to guarantee the the performance so yeah so the moving on something either at least I I find useful to keep in my head in when thinking in that spirit of what are they really trying to solve here and why does it look so different this is a typical topology here in the middle is a daisy chain so I got the computer up top I got a daisy chain which is potentially a hundred note it’s a with with the switch and a small computer basically a sensor or actuator or maybe a couple at each node so I have a daisy chain like this and because maybe the physical form of the system doesn’t allow me to do a star network I’m doing a daisy chain so so then this is where that TDM kind of comes in and these colors are trying to show it in and they in the that if I agree on the transmit moments on everyone nobody’s nobody’s colliding with anyone else everyone gets gets their own own TDM slot and then I’ll fill the rest is usable with the traffic that doesn’t meet those guarantees yeah so this is just trying to draw out the the flow of the packets and and yeah basically it’s traffic traffic engineer to minimize the latency and because it’s again in the application it’s this guy didn’t spontaneously start to transmit it you bye bye bye TDM multiplexing that the moments you kind of get the minimum latency to each of those hops and here is where you can see why the pre-empting the frame makes a big deal because even it has the 1500 and if you use jumbo even more but but if at any of these sports a best-effort 1500 comes in it’ll add up with a 100 hops so that that’s where they came up with the preemption not everyone is a fan of the preemption so when I said earlier that TSN is sort of these it doesn’t mean it’s all of these it does not mean everyone’s gonna do preemption but but at least it’s standardized so there is a possibility that you can do it okay yeah so so that was the part of the talk which is the where I sort of think I know what I’m talking about now I’m getting to the part where I it’s like I’m not sure I know some know as much about what I’m talking about it’s like okay what how does this link to the Linux and kernel so so ok let’s see so the whole point I’ve been repeating all the time is it’s there was there needs to be a guarantee of maximum latency and most of these applications average does not matter everything is budgeted to be always on and very very rarely do anyone take advantage of the average performance so so the outlier packet determines the system of course that’s not later only true there’s nothing 100% they always have a at the next level of the protocol they’ll take care of that one missing packet but but but the amount of outliers has to be really really small okay so the stuff that I think is common with other use cases is definitely huge pages the the no dickless dickless because if I have that control loop I I even if I get that one take every seven minutes I don’t stop the production or I don’t stop the car every seven minutes to to to allow for that or disable the emergency braking while I’m gonna update some thick so that comes into play the RT patches again to get to the control latency the isolation I mean approach should have written it’s like usually you have a couple course so isolating two to a given core features so so that it’s truly isolated and I threw in the PCIe pass-through as in because in some of these systems the for example switch the sensor might be talking to the switch chip using PCIe and it might might make sense to have our architectures where the the latency critical application is directly using the switch okay III listed that there’s a there there’s a there is some activity that you can easily find so I think the first idea is to just map what’s already out there what are what’s missing so that’s that’s definitely make sense then we’ve done a little bit of this kind of pre work and the already in LNG because some of the parts we have today they include these hardware capabilities that are similar to some of the approved standards you know and then then the bottom part here that I I wouldn’t worry about starting with the worst case latency so don’t worry about the thirty one microsecond loop I mean it’s at a gigahertz processor that’s you gotta be counting like assembly instructions to do that kind of a loop so they’ll keep using there are toasts for that that’s this display I think if finding a kind of a good enough good enough solution is is the point where where I can I can get all the other goodies that that running Linux gets me not necessary in the context of TSN yeah all these products the networking is not less surely the primary application networking is there to let me talk to the sensor the application is is the it’s not networking so so good enough for the networking is actually the goal yeah I guess but and maybe maybe things like jailhouse or open app to do partitioning would would kind of be the way to go or maybe even ODP like like applications yeah I promise you about it I’ll add it before I upload it to the slide okay yeah sorry for the text heavy slides I that was kind of yeah should have but big room should I put more pictures so okay so since there’s data plane and control and management I’ve switched things like that I think the controller management somehow I don’t necessarily see that as a first priority or something it’s not really that special it’s just regular control stacks so I wouldn’t but then on the data plane that the the transmit side or or as the specs call call it a talker if the ethernet hardware on board is able to do time aware shaping and and frame preemption exposing that to the application so that my which I guess usually it’s it’s a it’s a hardware cue of some sort or or or structuring multiple structures in memory that that I have the I’m able to send and receive actually I should have put what but okay as a first step probably the talker side but but those are the receive side how do I get this separation of when when when when a jitter of microseconds matters how do i how do i and it’s all about latency how do I get that latency value out to the application yeah the the preemption because I think at least in theory that that kind of could also be behind the same kind of cues if the hardware itself actually I think even at the talker you can preempt if the best efforts I’d was sending a fifteen hundred packet and now my time aware shaper tick a time I were say per se fire fires that I now sent me to send this control frame do the preemption that that kind of similar future then I put in bold in the middle testing and benchmarking maybe III don’t think maybe it’s just me but there’s no what if I just run the kernel stack what kind of weight with the RT patches and a few other things like that what kind of latency profile do I get so establishing the baseline and then also maybe another aware of measurement techniques for well let’s I’m not aware about the wrong way to say it but but because the it’s not enough that it’s logically correct the kind of the point where the time of our shaper is if I’m talking did it show up on the wire at what point of time it’s not enough that I see it ended up at the at the receiver but but it’s like how do i how do I measure how do I have that that latency or the concept of time per packet yeah I think that that’s something be great if there’s a or yeah do you do you need to meet a 300k x yeah put on the wire to measure that or how do you address that yeah and then the time synchronization area that’s seems like there’s some some pretty good activity on that that area already but but just just because it’s so fundamental and so that’d be another one to look at because for those I don’t know if how familiar people are with the timing over packet but really the key hardware feature is a simple thing to that as close to the physical wire as I can I need the time stamp both a packet going out and a packet coming in and and now how close to the wire can I do the time stamping is the jitter goes down the closer I can do it so so how do ability to leverage the capabilities of hardware makes sense there the software solution is inherently weaker because it’s it’s so much latency is a latency from software the wire is always higher than than the time stampers which basically sit next to the physical pin yeah but there seems to be pretty good activity on that but it’s it’s again just to map it and then then the other one here is if you go back think back to my daisy chain topology so these if there is a small switch integrated to a device how do how would that change the picture or would it or would it or would would you is this is that that don’t be something something okay and I think I think that’s it so fire away questions yeah yeah that so so they there have been so I Triple E 1588 is kind of the acronym sorry the the number but then the there has been there has been improvements to that in this 8o 2.1 a s so there are have been tweaks tweaks to that done by kind of the TSN in the TSN work so so there it’s it’s not nothing revolutionary but it’s two tweaks to that and and now I convergence time and and and and several yeah so so in principle it’s nothing new but it’s a it’s incremental work on that side yeah okay as they honor I would say that in the theoretical standardization side it is not restricted to wired but in all practical drivers the the commercial drivers for that that I’m aware of are are restricted to wired but but you have to think of it there that it’s it’s like a lot of these areas are these ultra-conservative I mean they’re just they’ve just moved from a serial port to an Ethernet like two years ago so it’s it’s it’s a little too much but it yeah so in principle it’s done in that layered I Triple E manner that they they should work with with wireless as well yeah yeah okay so so that timing soon yeah yeah so okay sometime soon this is a typical typical it so it’s also a cost trade-off right in the sense that the if you with more expensive so usually about about 100 nanoseconds between between nodes is sufficient for for a but then but then some some systems require higher synchrony higher but typically the the precision timing side what what is looked for is about a 100 nanosecond accuracy across the everyone so and then then because that timing synchronization is then fundamental to the everything else it created as jitter to everything else because of the inaccuracies of the time but but then you just throw in a little bit more again it’s looking for good enough networking there’s no value in being any better also so it’s you want the light in C to be as good as it’s needed but if it’s any better that’s a zero value and it’s the same with the accuracy it’s kind of it’s a true embedded thinking it’s not yep I yeah I think so I yeah this is because yeah that’s a good I will say that if we do what’s been happening now and it’s kind of like if nothing is done what happens is people will will have a two core device they will boot up boot up Linux they will do all the RT patches and and and and core core isolation they’ll put one process on the other core try to shield everything out of it open a window into the ethernet hardware and run their own school artists type application in that box I think that’s the default choice and and then they’ll solve the how do I get the regular Ethernet packets over to the Linux via some sort of a driver that takes it from the side I think that’s the that’s the solution solutions of today so that’s kind of where I’m bringing it up that is that is that kind of I think we could do a little better that that’s that’s kind of and you know you you can see some of the like the the jailhouse jailhouse is already a little bit better but it’s kind of trying to look at exactly that same same same approach it’s – it’s to make it shorter but it’s to make it shorter for the longest one if it makes sense because if I have a if I have a hundred sensors I need information from all of them it and there’s no you I need all of it before I can start processing – typically they won’t take advantage of I had I got information earlier from these centers it’s like I need a snapshot in time of what the physical world looks like before I can start my ladder logic to control the robot arm so yeah okay I think you’re saying exactly what is time aware shaper kind of I’m not sure if I but-but-but I was separated in a way that the the PTP synchronization is a nice isolated feature the only reason you needed is because I don’t have a dedicated wire going to every single entity if I have a dedicated clock going to everyone I don’t need PTP because everyone gets that same thing and everyone is completely synchronized but in the real world that’s very expensive because you essentially need a cable so the PTP is just to replicate the clocks so I would kind of isolate the that’s a good problem to solve but it needs to be solved and it can be solved kind of has to be solved before any any of the others make sense so no matter how I recover the clock to pt/ptt or a true clock signal that should not make the difference between the time of our side well well that that that’s kind of my well that’s kind of what I’m here’s the application that’s why that’s my question what’s a good way to do it is it and because and if I if I do it in if I just do it in Linux because if it’s good enough people will not do these custom solutions so what is the good enough that I how what kind of jitter can I get to can I get a with a 30 microsecond jitter in Linux with RT patches can I get it to work that’s great I can solve like 80% of the use cases with that or something like that yeah but that’s kind of my first first suggestion of the let’s get those numbers where is it to start with and and with these improvements or kind of incremental things I’ll get it to this point and what is that point yeah yeah yeah I yes that’s that’s what some of these proprietary Ethan its I call them proprietary Ethernet it’s it’s probably not it’s not some other word their their niche Ethernet they do things like that they say that you’re Minu maximum size is 512 now suddenly you’re daisy chain they do things like that but that is not yeah and yeah it’s so the that I would not get too much stuck on that preemption because my feeling is majority of the people are not gonna actually the preemption part is rather go like what you said the time of our saber I’ve I believe is gonna that that time we’re shapers you can’t kind of get away from it TDM over Ethernet you’re gonna end up with that one way or another but the preemption is maybe some folks had some patents on it or something and they’re really fond of it but yeah yeah yeah the other the the other one which I did not throw anything on that the redundancy comes into the picture and actually coming back like the precision timing it’s actually make the newer one is mixed in with the redundancy so you use both both networks and and so like so if if you physically separate the two Ethernet and you do redundancy now you got four and it’s you yeah it’s those are all valid points and and and like like standards are never well I I don’t think they’re good but they’re never perfect and they kind of show some people some interest which is not always justified but but I think at least it there is a standard I think that’s that’s maybe better than the seven different hacks of Ethernet that that there used to be or shipped today TSN is really new these are most of these are approved in basically this year this year okay any more questions or I have no idea what I’m doing on time but I’m guessing I am okay yeah it’s just about right so if there’s no more questions thanks thanks hopefully it was useful [Applause] you


Leave a Reply

Your email address will not be published. Required fields are marked *