Fault injection
Faults injected in flight.
The flights below are recordings. ArduPilot runs on an emulated Pixracer and flies a quadrotor plant, and partway through each flight we break something.
A chip that stops answering
Pick a chip on the board's buses, such as either IMU, the barometer, a compass, the GPS or the microSD card, and it drops off the bus like a cut trace, between two transactions or halfway through one. The firmware finds out the same way it would on real hardware.
One sensor that reads wrong
Freeze a reading, bias it by an amount you choose or add noise to it, on one chip while its neighbor keeps reading true. Noise comes from the run's seed. You can also flip or drop a single word the chip sends, at whichever instruction you pick.
The world around it changes
The battery sags, the GPS position jumps or the ground station goes quiet, and the controller only learns about it from what a real one would sense.
A rotor that gives out
Limit one rotor to part of its command, or cut it entirely in a hover. The motor failure and the noisy IMU below are older flights, recorded on September 19 with a demo tool that applied the fault to the plant from a file. Both are test cases now.
A test case sets each fault at the moment it chooses, so a sensor can die six seconds into a flight without restarting anything, and running the case again tomorrow breaks the flight identically. We'll add more, and each new one goes up here with a recorded flight.
Where the fault goes in
Faults go in below the driver.
They're applied at the board, one chip at a time. A dead chip is gone from its bus. A bad reading is the value the board hands that chip's model, and from there it goes through the same device model and registers a good one would, so ArduPilot’s driver reads it, scales it, runs its consistency checks and passes it to the EKF like any sample from real hardware.
The firmware isn’t patched or stubbed anywhere, and it has no way to tell it isn’t on a real board.
The flights
What the aircraft did and what ArduPilot thought it did.
The same flight with nothing broken. Read the other two against this one.
Loading the recording
What the flights showed
EKF3 stayed accurate while the aircraft shook.
None of these runs pulled the estimate away from the truth. We expected at least one to, and the fact that none did turned out to be the more useful result. In the noise run EKF3 holds attitude to within a third of a degree and ArduPilot raises no variance message or failsafe, so your telemetry screen would tell you the aircraft is fine. Meanwhile the true pitch rate has gone from a sixth of a degree per second of noise to nearly ten, and the drift in the hover has gone from one centimeter per second to nineteen. The filter is tracking a shaking aircraft accurately, and because it’s that good, the shaking never shows up in anything the autopilot reports.
You can shake a board on a bench, but you won’t see this there, because a bench has no ground truth. The only body rates you have are the ones from the sensor you’re corrupting, so you’d be measuring the fault with the faulty instrument, and the aircraft would look as calm to you as it did to the autopilot.
This rig, built around your board.
Everything above is an emulated Pixracer flying a quadrotor plant we wrote. In an engagement we build the same setup around your hardware. The unmodified image boots on the MCU you fly, the chips on its buses answer the way their datasheets say, and we write the vehicle model against the plant trait and couple it in.
Then you can set the failures on this page yourself, from your own test cases, on your own runners, for each merge request. See how an engagement runs.