ford
timeline
january 2026 - april 2026
role
ford motor company is a global fortune 20 automotive leader. i joined the vehicle root cause analysis team as a software engineer intern, where i contributed to development and testing of machine learning systems that enhanced vehicle functionality, reliability, and automation capabilities. i also worked on safety-critical software systems in c/c++.
work product
one of the projects i worked on involved simplifying the defect root-cause analysis process and algorithm. this process was usually done
by gathering context(historical and testing data) from many different sources and then using a well trained model to complete the root cause analysis.
i worked on the system that unified this whole process. the whole thing from start to finish takes about 30-90 seconds depending on
how much context got stuffed into the model prompt and how busy that model was.
the spinner problem
the naive way to build this is a normal request to response. the client posts the current context, waits, and the server
replies once with a finished analysis. it pretty simple and it works, but for anything that takes more than a couple of seconds,
it feels broken.
heres the thing about spinners: a spinner that runs for 2 seconds and a spinner that runs for 90 seconds look identical
for the first 2 seconds. the user has no way to tell between progress and no progress. so what actually happens is people
wait for lets say 10 seconds, assume its dead, and send another request. now you have got 2 90 seconds jobs running for
one user which then makes everything even slower. the underlying idea here is that there are 2 different types of latencies:
1. precieved latency: how long it feels you waited
2. actual latency: how long you actually waited
actual latency is bounded by physics or model size, and you can't really do much about it. i briefly talked about this in my previous
experience, see here for that reading. so yea,
the model takes as long as it takes. precieved latency on the other hand is a ux problem and you can absolutely change that
without touching the model itself. the trick is to stop hiding the work and start showing it to the user.
time to first token
the metric that matters for this is time to first token(ttft). so basically how long until the user sees anything come back as opposed
to how long until the whole thing is done. if you can get the first words on the screen in a sec or 2, the user isntantly knows the system is
functional, even though the response might be x seconds away. the ambiguity disappears because they literally see the system in action. this is the
same reason why llms stream their responses to you instead of dropping a large chunk of text at once. nothing is made faster here, its only done so you as the
user dont add more load to the system and end up waiting even longer. so now the goal becomes stream the models output to the client token by token
as its generated instead of buffering the whole thing and then sending it all at once.
using sse for this, but alternatives?
once i decided i wanted to push the data from the server to the client incrementally, i had to decide between a few options.
1. polling: the client repeatdly asks "done?" every x seconds. you either end up polling too fast or too slow, and you would
need "middle" memory to store the partial outputs between the polls. so not really a good option.
2. websockets: the duplex bidirectional channel over a single tcp connection. this is pretty good when both sides need to communication
constantly. like multiplayer games or multi-user editors. but this option requires a protocol update where we start as a http then switches.
3. server-sent events(sse): its one directional, from the server to client, over a single long lived http connection. the server keeps the connection
open and sends events down the pipe whenever it wants.
and so the deciding question is if we need bidirectional communication or not. and in token streaming, the answer is no. the client only makes one
request here with the context, and from then onwards its just the server doing its thing. theres nothing the client needs to say mid-stream. so websockets
dont make sense at all. kind of overkill for this case actually.
how does sse work?
its just a normal http response with a specific content type that never closes. the server responds with something like
"content-type: text/event-stream" and then instead of writing the body once and closing, it keeps the connection open and
writes little text framed messages whenever it has something. each message looks like this:
data: hello\n\n data: world\n\n
the format is literally data: <payload> followed by a blank line. the double newline (\n\n) is what
tells the client "this event is complete, fire it." that's basically the entire protocol.
the payload itself is just text, so the standard move is to shove json in there and parse it client-side. that's exactly what i
did and each event carried a small json object so i could tag what kind of message it was.
the implementation
the backend piece hangs on one flag. the llm client call gets stream=True, which changes the return type from "one
finished string" to "a generator that yields chunks (deltas) as the model produces them." each delta is usually a token or a
few characters.
so the endpoint became a loop: pull each delta off the model's generator, wrap it as an sse event, flush it down the connection.
i used a type field on each event so the client knew how to handle it:
data: {"type":"token","text":"Root"}\n\n
data: {"type":"token","text":" cause"}\n\n
data: {"type":"token","text":":"}\n\n
...
data: {"type":"result", ...}\n\n
token events are the live stream. the final result event is the signal that
generation finished and carries the cleaned-up structured payload.
on the client side, the logic is just: read each event as it arrives, and if it's a token, append its text to an
accumulating string and re-render a little monospace preview box so the user watches it type out in real time. one small but
important detail — i only rendered the last ~1200 characters of that accumulating string at any time:
accumulated: [.......... 8000 chars of report so far ..........] rendered: [ last 1200 shown ]
the reason is that, in worst case scenario, a 90-second generation produces a lot of text, and if you naively re-render the entire growing string
on every single token, the dom (or in my case the rendered preview) gets heavier and heavier and the ui starts to chug right when
you want it smooth. capping the visible window keeps re-renders cheap and constant no matter how long the response gets. the full
text is still accumulating underneath
then when the result event lands, the client wipes the raw preview box and swaps in the final structured sections,
nicely formatted. the messy live stream was just a progress indicator and the real output is the parsed result.
the actually hard part: parsing structure out of a stream
this was annoying and an obstacle that i had to get help on.
i needed structured output. specifically a json object with exactly six keys, one per report section, so i could render each into
its own styled card. the prompt told the model in no uncertain terms: return a json object, these six keys, nothing else, etc etc.
llms do not reliably listen to this.
even with a clear instruction, the model would sometimes do things like:
extra text here
{
"problem_description": "...",
"root_cause": "...",
...
}
extra text here
so now my "valid json" has random text glued to the front and back. if you just hand that
whole blob to json.loads(), it throws immediately, because that string is not valid json. it's prose with some json
in it. and you cannot just retry and hope, because it's non-deterministic; it'll do it again on its own schedule.
the principle i landed on, which generalizes way beyond llms: never trust the output format of a non-deterministic upstream.
if something can hand you garbage-adjacent output, you parse defensively and you have a fallback path. so the parsing went two layers:
1. try json.loads(raw) <- works when the model behaves 2. on failure, regex out the json <- works when it wraps it in prose
the fallback was a regex that grabs everything from the first { to the last }:
pythonre.search(r'\{.*\}', raw, re.DOTALL)
pull that substring out, parse that, and you recover the json from inside the prose sandwich. then on top of both layers,
any section that came back empty ("" or null) just got dropped from the display instead of rendering an
empty card.
the limitation in this approach is the interesting part.
\{.*\} is greedy. the .* matches as much as it possibly can, so it grabs from the first opening
brace all the way to the last closing brace in the whole string. for the common "prose, then one json object, then more
prose" case, that's exactly what you want. it will skip the leading prose and trailing prose and snag the object. but if the model
emitted two separate json-ish blocks, or there were stray braces in the prose, greedy matching would happily swallow everything
between them and hand you something malformed.
so it's a pragmatic fix, but not a bulletproof one. it solved the actual failure mode i was seeing (prose wrapped around a single
object) cheaply and reliably. the more robust version would be a balanced-brace parser that counts { and }
to find a properly closed object, or better yet using the model's constrained/structured output mode so it can't emit prose in the
first place. but for the failure i actually had, the greedy regex was the right amount of engineering, which my mentor agreed to as well. i think that knowing why it works
and where it'd break is more valuable than pretending it's perfect.
takeaways
a few things i learned:
- perceived latency is a real lever when you cant fix the actual latency
- pick the transport that matches the data flow
- defensive parsing at every non-deterministic boundary