Former Tesla Programmer's anecdotes about problems
**** I've added some more ****hj9快充网络
I have no way of proving any of this to be true, but I thought it was worth sharing. Enjoy.hj9快充网络
i used to work for tesla writing infotainment firmware and backend services - all of which runs in a single bottom tier Datacenter in a single location on the worst VMware deployment known to man.hj9快充网络
fun fact: a jenkins pipeline once caused almost the entire fleet to reboot loop for about an hourhj9快充网络
model s and x use openvpn to talk to their backend. inside that backend there are metadata services that feed info to the system, one of those things being a ~20MB+ (generated by the worst erp system) json payload that describes supercharger shit for the map in the touchscreen. somebody was smart enough to do automated linting but forgot to validate against the custom parser the car runs which caused a segfault in the qt app that runs the ui, which in turn for a variety of reasons forces a reboot of that component. I think we clocked about 15 seconds before it read the file and faulted after boot. it was doing that for an hour before everyone panicked and got me and qa on the phone to fix it. i wrote a quick python/fabric script that ssh’d to as many cars as possible at a time to rm the filehj9快充网络
why do the cars run a cluster of ubuntu vms? used to be centos 6 and Ruby on Rails. I haven’t worked there in 3 years, but last I heard it hadn’t changed much for s and x. model 3 uses newer tech, but still based out of a single Datacenter
some of what I wrote runs on the factory line - at the time we started the model s program, which has not changed to this day, we fake the backend to install and validate firmware as the car moves down the line. a tech runs over to the car, plugs an eth cable in diag and dumps an image on the car using curl and a tui app I wrote using python. as the car moves down the line it is installing firmware for about an hour. if that station for any reason can’t talk to the PKI system, erp, or a ruby webapp it halts the linehj9快充网络
can't you flash the storage before its installed in a car?
yes and no. the firmware update process in a car is complicated because you have a bunch of dumb components hanging off of CAN or LIN and they have to updated in very specific order and sometimes you have to retry 10s of times to get it to take. ( fuck you Bosch). Tesla never bothered to flash those things ahead of time before assembly so that gets done the first time as it rolls down the line. the infotainment system and gateway arbitrate that stuff. typically any update that tuned voltages becomes a one way - no downgrade is possible without frying somethinghj9快充网络
this is the thing, like i work with boards that have many devices on them that have firmware and they're all flashed well before the >board is installed in anything if not before even being soldered down they got smart eventually - model 3 does do this now, but doing that at scale with all the components for a car is a challenge when you have it being done with stations running yocto images and perlhj9快充网络
like, for all the lols @ tesla, have they literally never heard of a process engineer?
like everyone else who was smart they either quit or were fired through no fault of their own so what you’re left with are people fearing for their job who desperately don’t want to change status quo for fear it will break somethinghj9快充网络
they forgot that the unspoken part of "move fast and break things" is that you're supposed to fix what's broken
exactly this. we never really had time to address critical issues and were constantly short on staff because people were quitting or they just wouldn't give candidates competitive offers. this is why you hear about people burning out - they've managed to chase everyone awayhj9快充网络
more fun facts:hj9快充网络
the infotainment system and gateway don't have a battery-backed rtc. when the system reboots (sleep, deep sleep, reboot, whatever) the car is at tyool 1970 until it gets ntp again. the logs themselves are written in a binary ring buffer format and when they come in they used to end up in a giant 700TB single mysql database after they were expanded. all of production after-sales service and engineering relies on that single log interpretation system which ran on centos 5 and python 2.4 until hbase/hadoop and friends were brought in.hj9快充网络
the supercharger system uses ssh dss keys to "vpn" back to the datacenter to a single server over 2G wireless with very limited resources. the connection is essentially simplex for various reasons so getting data to and from the supercharger is usually a 1KB/s operation unless that site has had connection aggregation done. at one point i looked at the system and to pull data out for analysis, somebody had written a bash script that was printf'ing in a for loop across ~5k devices. it would usually take about 3 days to do a successful firmware update on any single supercharger.hj9快充网络
we once patched openssl to ignore client cert expiry because somebody forgot to create a process to update keys in the field and all the customer cars started falling offline because their certs had expired. the quick and dirty was to just patch openssl quickly and make openvpn on the server side use that one while we created those processes for about 2 weeks.hj9快充网络
most of the time me and the other firmware folks were chasing elon's whims about what to do with firmware. where i should have been fixing critical issues in the system i was pulled off to do shit like add farting unicornshj9快充网络
uh we literally do the same thing; well, yocto images and pythonhj9快充网络
tesla isn't the first to solder down SOMs running embedded linux and a bunch of MCUs hanging off an i2c/canbus/whatever line
they aren't the first - for what we were doing at the time it made sense and helped us get the program off the ground quickly. lots of room for improvement and in 8 years, they should have done so.hj9快充网络
my issue was the fact that the systems doing the flashing were running the yocto images and perl and the guy writing the perl was also responsible for writing the thing that actually updates the car. that thing (the car-side updater) is about ~100k lines of C in a single file. code reviews were always a laugh riothj9快充网络
i am SO GLAD your nda expired
99% of what i'm talking about is "public" anyway. tesla isn't encrypting their firmware and it's really easy to glean information from the vpn with a packet cap because nothing inside the vpn (was) encrypted. dumping tegra 3 model s and x is trivial and tesla's cars are nowhere near as secure as they'd have you believe.hj9快充网络
for example, at one time you were able to root a model s with a usb stick and a gstreamer exploit.hj9快充网络
while tesla should be given credit for updating the car over the air to fix issues, that's also any connected car's biggest weakness - you're one exploit away (or malicious employee with access) from remote root.hj9快充网络
more fun stuff: there's limited space on the emmc in the touchscreen system so updating maps can't be done using an image or a binary diff. so the thing rsync's map updates (all 2GB of them) from various places. they may have fixed that in the newer intel-based boards, but who knows.hj9快充网络
autopilot had really high turnover at one point before release because some guy from space x came in and gave the entire dept a C pointer/memory test because Elon said they were "late" to ship.hj9快充网络
There's the story online of that hacker who was pulling software images off through the door Ethernet port and found that his car's >firmware was remotely downgraded after he uncovered and posted the first references to the P100 models.hj9快充网络
Does that sound plausible to you?
yup, i'm the guy that installed the older versions. this was a marketing mistake really. if i recall correctly, he ended up getting a marketing car or his car got tagged in the update system as a trusted car and he ended up getting pre-release stuff. this happened from time to time - sometimes marketing would sell off a car and the shit erp system wouldn't record the change. that car would then get prerelease and sometimes very broken firmware. i seem to recall another case where we just forgot to remove the prerelease materials from the official build, so all you had to do was look around.hj9快充网络
the early days of tesla, post-roadster, early model s and the start of model x were good times - everyone was trying to prove the technology worked, we were innovating and making something that hadn't been done before. things really started to shit the bed around the time we pivoted from model 3 plans to shipping model x first. the falcon wing doors were such a shitshow. they ended up delaying the program almost a year, hence why model 3 basically skipped all the usual phases a car goes through for validation. i mean, come on - you have bumpers falling off in the rain, the interior is a disaster, there's no instrument cluster which takes your eyes off the road - this list just goes on.hj9快充网络
tesla basically runs their entire business like a just in time compiler only they don't treat warnings or errors as failures. most groups in the company don't cross-communicate so there's a lot of duplication of effort.hj9快充网络
i once got pulled into a meeting because a car burned down when it was attached to a supercharger and we didn't get a log out of the car. normally under some emergency circumstances the car will try to upload a log when it thinks shit has gone really badly, but in this particular case it was far enough away from a tower it had half 3G connection and had to upload a 30MB log via HTTPS POST. the car burned down before it even got to 10MB and the system was only designed for exponential backoff retries, not resumption of in-progress. elon was calm about it, but we had to justify why we never had time to address it - maybe it was because we were all busy making unsafe features work?hj9快充网络
also on the supercharger note - you can get blacklisted from using them if you charge on them all the time. that's because the supercharger bypasses the charging regulator boards and dumps directly into the pack at 300A/450v which creates a ton of wear on the battery. want to keep your range high? don't supercharge often.hj9快充网络
do they define “too often”?
algorithm-based now - the ai shit i was working on took into account a lot of factors to determine if you were abusing it before i left. the criteria takes into account the state of many components in the car, your driving patterns and other details. or it did anyway. not even sure that stuff is running still - they rotated projects in and out of existence pretty rapidly.hj9快充网络
what is elon like when stuff goes wrong due to his idiotic micromanagement and big stupid ideas?
he's never wrong. his "open door policy" was an invitation to catch you breaking rank.hj9快充网络
tesla was also in the news because they were doing cute shit like spinning up k8s clusters which had AWS IAM access to sensitive S3 buckets but wasn't ssl'd and the k8s mgmt api was available publicly. there were other teams running industrial control equipment with centos 7 an no hardening at all.hj9快充网络
there was one time where a canadian kid stole the domain and redirected emails and managed to take over slack and a bunch of other shit because the idiot IT team didn't hide the registrar information or use something like markmonitor. the car-side stuff at least did full mtls at the time so it was ok, but lol did that kid get a lot of info.hj9快充网络
**** the new stuff:hj9快充网络
Some more:hj9快充网络
thats just what i want, the car manufacturer monitoring how i drive the car i own and deciding that features should be turned off after i >have purchased it, that's a good feature. you have no idea. any connected car is ripe for data harvesting and you (the consumer) should expect it going forward. on that note, china has a law in place that mandates all electric cars send real time telemetry to their government servers - model s/x/3, NIO cars and any other electric car if they're driving already complies with that law to be road certified. don't be surprised if that becomes a mandate in other countries
for all the shit that went down at tesla, there were some positive aspects. everyone i worked with really cared about physical safety and we put a lot of effort into making sure the engineering was sound so nobody got hurt. if you subtract autopilot, and that's a big if, the car is generally well designed minus the fit and finish issues + interior, but i'd argue that's never been tesla's strong point anyway. the cars are fast, the 2013-2014 model s lines were really good, solid, basic cars. my last straw was the summon feature - i strongly believe a car you are not in, backing out on its own from a parking space with the current sensors is super dangerous.hj9快充网络
i was making jokes with the tesla expats when ol' musky launched his roadster into space that you could see the gaps in the fit and finish without a telescopehj9快充网络
just remembered some bits of triviahj9快充网络
they took away our free snacks in deer creek and replaced them with shitty vendors
said vendors food poisoned people often enough osha or whatever the body is shut them down
people were so mad about the free cereal being gone they'd intra-office snail mail bowls of cereal from the factory and post pictures in slack
deer creek's parking got so bad (too many people, not enough space) they hired permanent valets
they were cited for the shitshow parking for fire safety violations (unconfirmed, but i believe it)
elon publicly being a shitbag to trans people
the first time we turned on real time telemetry for the dev fleet we caught somebody going 130mph over the san mateo bridge
it networking so bad the company had permanent 5~8% consistent packet loss between various places (like, next rack)
firmware git repo so large they had to mirror it (something like 2TB)
depending on when and what features you got (and if you got a marketing used car) they could go as low at $40k after incentives - but totally agree with you. fit/finish issues have been a thorn in their side foreverhj9快充网络
the touchscreen is kind of a safety issue in that you have to look at it to touch it, stealing focus. tactile buttons for some functions would have been betterhj9快充网络
the firmware repo was that size if you take into account a huge company, many devices in the car at play and incremental updates to firmware across all those devices + branches for people to do work in. i contributed to that mess by policy, not by choice, but whatever. i'd imagine they'd be smart enough to move to something like git lfs so it isn't as much of a painhj9快充网络
scale stuff:hj9快充网络
tesla has a real thundering herd problem at this point. if you factor in common peak drive times for any region (bay area CA being the largest by pop) they have to weather something like 100k+ cars slamming servers all at once during rush hours. i saw this play out on some of the cj dashboards, it was fun to watch the production shit come to a grinding halt before they figured out they couldn't just-in-time the autoscale and had to provision ahead of time for peakshj9快充网络
i had to deal with marketing people sincerely asking me why we weren't going to run containers on the car in firmware. no, marketing, i don't care that the car would "update faster" or "features would release faster"hj9快充网络
a web front-end (we'll say it's a cms that's php-based) that needed $500k in WAF bullshit just so we didn't get pwned every 5 minuteshj9快充网络
fragmented installs of splunk. i think i counted well over 20 installs for various departments before they finally hired a decent data scientist that cleaned it uphj9快充网络
so many random java, django, .net services from various places, more than i could count and i had to touch a lot of them with firmware. ActiveRecord controlling way way way too much. i consider this probably one of tesla's biggest scale problems - i don't think they actually know or can track exactly what they're running server side at all - so you end up with teams running vmware, nsx, k8s, openstack, hyper-v.hj9快充网络
a car that has a json parser implemented in bash 3 because <interpreted language> is dangerous in the car. there are some seriously magic shell scripts on that thing that probably 3 people in the company understand in fullhj9快充网络
nodejs was a thing for a while but quickly broke down once we reached the 20k car mark - ended up replacing a bunch of that stuff with a Go varianthj9快充网络
bets on whether the fire was due to incompetence, act of nature, or deliberately set?
never attribute to malice what can more easily be explained by incompetencehj9快充网络
not surprised at all. earlier in Falcon 9 lifecycle at SpaceX, they kept having helium problems because the QC team kept signing off on >defective bottles and valves. do you think that attitude might have scared them into not saying anything?
absolutely. taking advantage of the "open door policy" was the fastest way to lose your job at tesla and from what i'm told, spacex, being run by the same guy was no different. there is so much pressure to ship on time they push people to work 14 hour days, 7 days a week - i did that for a while before i just couldn't take it anymore and just accepted being marked down in employee review for being latehj9快充网络
the openvpn problem is easy to get around thundering herd/scale issues if you design it correctly and know how to run a network. in theory, you could get around a lot openvpn scale issues if you use bridged networking, ipv6 on the inside, and some redundant dhcp servers to hand out leases - that kind of shit won't work in most cloud providers though so you stuck at running that crap in a datacenter.hj9快充网络
tesla's issues around the services were many fold - the specifics would give away too much, but i'll say this: when you make all of your services depend on a single rdbms while simultaneously using the world's worst ORM, you get what's coming to you.hj9快充网络
i poked around on a 3 a friend has and after looking at a packet cap it looks like they're doing ssl'd amqp - i didn't see any openvpn packets so i suspect they got wise to how shitty it can be, but lol at running connected car stuff directly over the internet outside a private apn or a tunnelhj9快充网络
The staggering level of internal fragmentation reminds me of how PayPal was when I worked there in '09-15. They experimented for a few months with an "agile product solutions" team that basically >took "we need a widget that does this" orders and cranked out custom Java shit that never worked.
that's basically tesla in a nutshell only, i guess it kinda works. every different team has some kind of different service where you can get data but none of it published anywhere, there are no standards, and everyone just loves to write their own client implementations because they don't trust you to do it right (sorry that we don't have a client in C++ which is mandated by policy for the car)hj9快充网络
poking holes in the firewall was always super fun - i would describe, in full detail all ports, sources, destinations, have security assessments done, etc and somehow, still, the firewall cj's would fuck up the ports. i once spent, and this is not a joke, 3 weeks chasing a single port down - i think that email thread had 100 reply-all's, two video confs and me visiting the firewall cj in fremont before it was finally fixedhj9快充网络
was there any sort of accountability for the devs there, or was it if you knew how to talk the talk you could bs your way through the ranks while producing nothing of value? was there any noticeable increase in the absurdity of musk's requests as time went on? anything particularly absurd he called for that was flat out shot down?
no, if you didn't do work it was really really obvious and they purged you quickly. that didn't mean it was any good but if you produced you were generally left to your own devices as long as you weren't breaking builds - this seemed to be true of most engineering teams.hj9快充网络
ol' musky did increasingly weird shit, but i wouldn't necessarily call it out of the ordinary for silicon valley - many folks, me included, for a time, viewed him as a bit of a Jobs-type. his behavior became really erratic around the time we wrapped up X and headed for 3 full steam - the more stuff piling on about autopilot, the more issues with the factory, the ongoing issues with X and then with 3 mfg, his ongoing spacex work - the dude really needs a nap and to just walk away from tesla at this point. its arguable he isn't running it successfully considering all the issueshj9快充网络
edit - running it successfully by silicon valley standards. too many issues to reach profitability because of really poor strategy and execution. too many people get wrapped up in his celebrity without really asking 'can he pull this off' which is the difference between him and Jobs - Jobs actually did shit
yeah, i get that, it's just they make a product that will probably shit itself when the back end goes dark, and that product costs $65k-$120k so it's an outlier by sv standards.
the product shouldn't shit itself when the backend eventually goes dark - autopilot won't work, updates won't, remote phone shit won't but otherwise the driving and infotainment part of the car should still function if you pull the sim and put your own in. given how shit the firmware security is it'd be pretty easy to dump the firmware, compile up some statically linked tools for shits and just patch in your own services. there's been a few clever people on twitter who figured out you can run Go arm bins on the thing - after that it's just figuring out what crap you care about on CAN (if anything).hj9快充网络
all that said, tesla did sell cars explicitly with the sim pulled and no network ever - service was always complaining to us because the ring logs on those cars would take hours to parse.hj9快充网络
speaking of the ring logs - because there was no battery backed rtc, we had to stitch and best-guess times based on the intervals when the car did have valid time and patch that into the logs serially before they could be imported. inaccuracies in the signal data could and did lead to all kinds of bullshit when somebody needed to be debug issueshj9快充网络
