The pieces for rolling out software to a fleet of devices have been open source, mature and running in production for years. Most companies still cannot do it reliably, and the reason is not technical.
Disclosure: I worked in the Bosch group until May 2026 and have been independent since. Bosch IoT Rollouts, named below, is a Bosch product. No vendor named in this article pays me anything, in any form.
I have been working on IoT since 2001, at ProSyst Software in Cologne. Everything we built rested on OSGi, a Java standard first published in 2000 for running software on small devices in modules that can be installed, updated and removed while the device keeps running. Our runtimes went into home gateways, set top boxes and telematics units, you name it. What we were building was not firmware delivery. It was an extensible ecosystem, a shipped device that could take new software from more than one vendor, years after it left the factory, without being taken down.
That idea reached further than the device world. Among many others, Deutsche Telekom built its smart home platform on it. So did the Eclipse IDE, whose plugin mechanism has been OSGi since version 3.0 in 2004, with millions of users and tens of millions of downloads a year by the Foundation's own count when the IDE turned twenty.
Remote update was in that first specification, May 2000: install and update those modules without bringing the system down, administered by a remote service provider. Nearly every customer we had used it. Not as an advanced feature, as the normal way you keep a shipped product working.
So when people describe over the air update as a recent capability, my reaction is that the hard part was never the technology.
What it costs when this goes wrong
Three cases from connected devices, each failing at a different point.
A rollout that reached the wrong devices. On 13 November 2023 Rivian pushed an over the air update to R1T and R1S vehicles. By Rivian's own account at the time, the wrong build went out carrying the wrong security certificates. Infotainment screens went black, and with them the in vehicle defroster controls, which is what turned a software fault into a formal safety recall for reduced visibility. Rivian caught it inside two hours and cancelled the campaign, which is why 1,463 vehicles installed it rather than the whole fleet. NHTSA recall 23V-783. Every one of them was remediated by 19 November, over the air.
Both halves matter. The update mechanism caused the defect, and the update mechanism is also what contained it and then fixed it within six days. Without one, that is a workshop visit for every car.
The firmware write path used as a weapon. Over three days at the end of October 2023, an unidentified actor got into consumer routers on a single internet provider’s network, installed the Chalubo remote access trojan, and left ActionTec devices permanently inoperable; Lumen assesses the attacker likely corrupted their firmware. Lumen’s Black Lotus Labs, which documented the incident, never established how the actor got in and suspects weak credentials or an exposed administrative interface. It counts the loss in hundreds of thousands of devices, and public scan data showed forty nine per cent of the modems on that network gone inside 72 hours. The devices could not be reflashed. They had to be physically replaced.
That is what a device without a root of trust is worth to somebody who gets code execution on it. Signing, key management and revocation are not compliance paperwork. They are the difference between an update channel and a remote kill switch that you built and handed over.
A channel that did not cover the part that broke. In May 2026 Hyundai recalled more than 421,000 Tucson and Santa Cruz vehicles from model years 2025 and 2026 because the front camera software made the collision avoidance system too sensitive and it engaged earlier than drivers expected: 376 reports between October 2024 and April 2026, four rear-end crashes. The fix is a software update and nothing else. Every owner still gets a letter, mailed from 22 June, and drives to a dealer where a technician reflashes the camera. More than 421,000 appointments for one software update, and the vehicles that never make it to the workshop keep the old one. Some of these cars can be reached: five weeks later Hyundai filed a second recall on Tucsons of the same model years and offered that fix over the air as well, for owners who had opted in through Bluelink, a path it had already used for the same instrument cluster in January. The camera got no such path. Hyundai does not say why, and no update server decides that for them. Eleven years after the Jeep Cherokee recall that shipped its fix on USB sticks in the post, a pure software defect is still a logistics problem.
None of these three is exotic. They are the ordinary ways this goes wrong.
An update is not just a file
That is not the same as saying an update is simple. Ask what has to be true before one device accepts one new build, and the list gets long.
A fleet is rarely one device. It is generations of a device, and they differ: another CPU architecture, another operating system, another amount of flash and RAM. The build that fits the hardware you ship today may not fit the one you shipped in 2019, and the one from 2019 is exactly the one still in the field.
Then the dependencies. Your application does not run alone. It sits on a logging framework, a TLS library, a bootloader, often firmware on a second processor beside the Linux one. Update the application against a newer logging framework and it stops working on every device still carrying the old one. Nothing in the campaign shows it: the download succeeded, the install succeeded, and the device is broken in the only way that counts.
Then the physical limits. A device without enough free flash cannot hold a second image while it writes the first. A device without headroom in RAM or CPU cannot verify a signature and unpack an image while doing the job it was sold for. On a metered cellular link the size of that image is a line in somebody's operating budget, and the devices that need the fix most are often the ones on the worst connection.
Then the customer. Every device out there carries somebody's configuration, somebody's application, somebody's integration. None of it may be worse after the update than before. An update that resets a setting is not a successful update, whatever the campaign dashboard says.
And then multiply. Hundreds of thousands of devices across several regions, under different rules about what a device may do, on a radio stack whose certification a firmware change can invalidate. Some of them will be offline for a week. Some updates cannot be rolled back at all, because the fix rotated a key or migrated a data schema, and going back means going back to the vulnerability.
Every one of these has a known answer. None of them has an automatic one. They are decisions about one specific fleet, and they are cheapest to make before the first campaign rather than during it.
What Eclipse hawkBit already does
If you want to push a firmware update to a hundred thousand devices in stages, with automatic abort when the failure rate climbs, every component exists in open source. Eclipse hawkBit™ reached version 1.0 in March 2026 after a decade of development, and 1.1 in July, licensed under EPL 2.0 and carrying the Eclipse Foundation's highest maturity classification. Its rollout engine splits a target list into deployment groups, advances on a success threshold you define, and shuts down the entire campaign when an error threshold is hit. The shipped defaults are five hundred groups per rollout and twenty thousand devices per group, both configurable. The commercial service built on the same code fixes them as quotas and offers a million devices per tenant on its top plan.
I have worked with both since 2015 and in several roles: with Eclipse hawkBit as an open source project, and with the commercial service built on it, Bosch IoT Rollouts. So I know the people.
On the device, RAUC and SWUpdate both handle A/B partitions, bootloader coordination and rollback on failed boot. SWUpdate speaks hawkBit's device protocol in tree; RAUC speaks it through a separate daemon, which is a second component to build, package and configure. Zephyr ships hawkBit support in tree, and Nordic carries it in the nRF Connect SDK.
The rollout engine is the part worth understanding, because it is where a campaign becomes something you can defend in a meeting. You select devices with a filter written in a query language over the attributes the devices themselves report: hardware revision, installed version, region, customer. hawkBit splits the matching set into deployment groups. Each group carries a success condition, a share of installs that has to succeed, and an error condition, an absolute count or a share. Meet the success condition and the rollout moves to the next group, or waits for a person if you told it to. Hit the error condition and it shuts the whole campaign down. A rollout can require approval before it starts, and who gave it is recorded.
Three things follow from that which matter more in a factory than in a car park.
Downloading is not installing. hawkBit separates them. The device may fetch the image now and install it only inside a maintenance window you define with a cron schedule, a duration and a time zone; it asks, and the server answers whether the window is open. That is the difference between an update regime a plant manager will sign and one he will refuse, because the window stays his.
Generations are a first-class concept. Devices carry a type, distribution sets carry the types they are compatible with, and hawkBit refuses an assignment that does not match. The hardware you shipped in 2019 cannot be handed the image built for the 2026 board because somebody wrote a filter badly.
The fleet converges without you. An auto-assignment rule attaches a distribution set to every device matching a filter, so a machine that was offline for a month, or one that left the factory yesterday, arrives at the right version without anyone starting a campaign for it. Dynamic groups do the same inside a rollout that is already running.
And an update can wait for consent: hawkBit can require a confirmation before installation, which is the mechanism you need when the machine is not yours.
None of that is clever engineering. It is a decade of other people's scar tissue, available under EPL 2.0, and most of the arguments I hear about update servers are conducted by people who have not read this far into one.
Which raises the question this piece is actually about. If all of it is free, and has been for years, why do so few industrial companies have an update path that works? Five reasons, and not one of them is a missing feature.
1. They evaluate the server, and the server is the easy half
The first mistake is a category error, and it happens before anyone writes code.
Teams treat OTA as a platform decision. They compare update servers, build a feature matrix, argue about hosting. Meanwhile the hard requirements sit somewhere else entirely.
hawkBit is unusually honest about this. Its original project proposal, filed in September 2015 by Bosch Software Innovations, says:
"It is out of scope to provide a full blown device management and it is also out of scope to provide client solutions for handling software updates on the device."
Read the second half. The server does not install anything, and it does not inspect what is inside an artifact it was handed either. A/B partitions, bootloader coordination, what happens when power fails during the write, whether the device comes back at all: none of that is a server problem. It is a device problem, and it is where the safety requirements live.
The order that works feels backwards. Choose the device side first, because that is where you can actually hurt someone, and the server turns out to be whichever one your updater already speaks to. Start with a platform evaluation instead and the device work arrives afterwards, unbudgeted, because it was never in the budget that got approved.
2. The demo is not the system
Type one command and you have a running hawkBit. That is genuinely impressive, and it is also where the work starts.
What that command gives you, from the project's own shipped configuration: an in memory database, credentials of admin and admin, TLS not required, no device authentication configured, so nothing can talk to the device interface until you pick a scheme and provision it, and remote events disabled with a comment in the file noting that this means no cluster support.
It is a demo. The problem is that a working demo creates the impression of a solved problem, and the difference between the two is where the budget goes.
The work between the two, all of it verifiable in the repository:
- Database. MySQL or MariaDB, the two marked production grade. PostgreSQL is still classified for test and development use.
- Artifact storage. The filesystem is the maintained backend. Object storage is yours to build: the extensions repository holding the S3, Azure, Google Cloud and MongoDB backends was deprecated in September 2025, its last functional commit more than a year before that.
- Deployment. Your own manifests or chart, since there is no official one.
- Observability. Micrometer and Actuator are in the build, undocumented and not exposed in the shipped configuration. Exposing them, choosing the metrics and building the dashboards is your side.
- Operations. Backup, restore and retention are policies you write; upstream documents none of them.
- Scale. More than one instance needs a message broker, and clustering carries constraints the documentation states plainly, including no shared cache and all schedulers running on every node.
Every one of those is ordinary infrastructure work. Together they are a project, not a configuration afternoon. And because nobody publishes what that project costs, it gets estimated by someone who has only seen the demo.
3. Nobody owns it
This is the one I keep meeting, and I think it explains more than the other four combined.
OTA sits between four groups and belongs to none of them. Product management wants the capability and treats it as a feature request. Embedded engineering owns the device and correctly regards the server as somebody else's layer. IT or platform would have to operate the server, but it is a product system rather than a corporate one, so it falls outside their remit. Security and compliance now carry a legal duty attached to it and have no way to discharge that duty themselves.
Each of those groups is behaving rationally. The result is that OTA is everyone's second priority, and second priorities in industrial companies do not get staffed.
This is not a technology observation, which is precisely why it does not show up in technology comparisons. Every organisation I have watched ship updates reliably had a named person who owned the whole path from build server to installed base, with enough authority to make both the device team and the platform team answer the same questions.
If you take one thing from this, take that. Name the person before you choose the tool.
4. The trust question arrives last, from a different department
Here is the technical point that most often lands late, and it lands hardest.
hawkBit stores MD5, SHA1 and SHA256 hashes for every artifact, and you can upload a detached signature as just another artifact file beside the image. It never reads it. It does not sign anything and it does not verify anything. There is no key management, no signing roles, no thresholds, no revocation. Asked directly about artifact signing in January 2024, a hawkBit committer answered within a day: signing belongs to the device integrator, not to the server. That is a design decision rather than an oversight, and it is the right one. It is also not what a compliance questionnaire assumes.
The trust chain belongs on the device, in RAUC or SWUpdate or your own agent, because that is the only place a verification decision means anything. But it has a consequence that surprises people at exactly the wrong moment.
From 11 September 2026 the Cyber Resilience Act's reporting obligations apply, with the full requirements following in December 2027. Manufacturers must provide security updates for a support period that reflects how long the product is expected to be in use, with a floor of five years, and Annex I, Part II, point 8 requires that security updates be disseminated without delay and, unless a manufacturer and a business user have agreed otherwise for a tailor-made product, free of charge. For vehicles in Europe, UN Regulation 156 covers both the manufacturer's software update management system and the vehicle itself, and the vehicle requirements include protecting the authenticity and integrity of updates.
A team that has got distribution working will reasonably answer a compliance question with "we use hawkBit". That sentence addresses distribution. It does not address authenticity. Whether the update the device installed is the update you built is a separate question with a separate answer, and it lives in a part of the system a different team owns.
This is why the commercial offerings built on hawkBit add signing on top, and why Bosch IoT Rollouts sells it as a named service.
How much it matters is not abstract. RAUC shipped a fix in March 2026 for an integer overflow in its legacy plain bundle format, where a bundle larger than two gibibytes ended up with a signature covering only the leading bytes of the file. The verity and crypt formats it has recommended for years were never affected.
5. It is an operating capability before it is a build or buy question
The last reason is the one that catches companies who did everything else right.
OTA gets funded once, as a project, with a launch date. But an update channel has to work for the service life of the product, which in industrial equipment is ten to twenty years, and under the Cyber Resilience Act for a support period that has to reflect that expected time in use, with five years as the floor rather than the answer.
There is a number that settles this, and it is not a cost. During my time as Business Owner at Bosch.IO I was responsible for a customer running FOTA across several million vehicles. The return was in double-digit euros per vehicle per year. An update channel at that scale keeps earning for as long as it runs, which is a different budget conversation from a project with a launch date.
That means upgrades. hawkBit's own path illustrates the point: version 0.10.0 is a mandatory stepping stone before 1.0, and its list of breaking changes is long, including removed roles, a removed REST API and restructured module paths. That is normal for infrastructure software. It is not normal for something a company budgeted as a one off.
Which brings me to the choice between running it yourself and buying it. There is no general answer to that, only conditions.
hawkBit is a strong option when several of these are true. You want the roadmap under your own control. Your devices sit in networks that never reach a public cloud, or your data may not leave your own infrastructure. You are prototyping, or starting with volumes too small to justify a contract. You already run databases and services at this level, so one more is marginal. And your update channel has to outlive your supplier's interest in the category. A server you host yourself is a mechanism you can still operate when a commercial relationship ends.
What comes with that is a bill paid in engineering rather than in licence fees. Integration to build, the production work listed earlier, then maintenance, version upgrades, monitoring and somebody reachable when a campaign misbehaves, for as many years as the product is in the field. Self hosting is not free once that is counted honestly, and it may well be more expensive than a managed service.
Where those conditions do not hold, a commercial service is the better route. Bosch IoT Rollouts, which I have worked with, takes the operating work off you and offers what hawkBit deliberately leaves to the integrator, artifact signing among it, as a separately licensed extension. It is not the only one: Mender, Memfault and the hyperscalers’ own device update services address the same buyer. Whichever you pick, ask where the signing key lives, because a service that signs on your behalf with a key it holds has taken the root of your product’s trust with it. The same company founded the open source project and still drives it, first as Bosch Software Innovations and now as Bosch Digital, so the expertise behind both is the same. An open licence leaves you the option to fork and keep operating.
What I would actually do
Whether it is a car, an industrial machine, a consumer device or a single sensor, being able to ship a software or firmware update stops being a choice. In Europe, UN Regulation 156 requires a certified process behind every update a car maker ships, and a further set of requirements on the vehicle itself once those updates go out over the air. For most other connected products the Cyber Resilience Act’s essential requirements apply from 11 December 2027.
Beyond the regulation there is a plainer reason. Anyone building connected products has an interest in designing the hardware and the software for the longest life they can carry, and an update path is what keeps that life usable.
Four things, in this order.
Name the owner first, before any tool evaluation, with authority over both the device and the server side. If you cannot name that person, nothing else on this list will hold.
Choose the device side next. Your safety requirements, your rollback behaviour and your recovery path live there, and they will constrain the server choice more than the server choice constrains them.
Ask where your trust chain is, explicitly and in writing. Who signs, with which key, stored where, verified by what on the device, and what happens when a key has to be revoked. If the answer is a hash, you have integrity in transit and nothing else.
Budget it as an operating capability with a five year floor and a fifteen year expectation, including version upgrades, and decide consciously whether you are self hosting or buying. Both are defensible. Assuming the demo was the hard part is not.
The technology has been ready for a decade. What is usually missing is somebody whose job it is to care.
Sources
- OSGi Service Gateway Specification Release 1.0, May 2000, docs.osgi.org
- Eclipse Foundation, "The Eclipse IDE Turns 20", 2 November 2021, for the user and download figures; Eclipse Equinox replaced the Eclipse plug-in runtime with an OSGi based one in Eclipse 3.0, June 2004
- NHTSA recall 23V-783 (Rivian, November 2023), Part 573 report: 1,463 vehicles, all remedied by 19 November; the cause of the wrong build is from Rivian's public statements at the time, not the filing
- NHTSA recall 26V-316 (Hyundai, May 2026), Part 573 report; NHTSA recall 26V-400 (Hyundai, June 2026), Part 573 report, remedy type "Software, Software OTA"; and NHTSA recall 26V-047 (Hyundai, January 2026), instrument panel display software "updated over-the-air (OTA) or by a dealer"
- NHTSA recall 15V-461 (Chrysler/FCA US, July 2015), the Uconnect vulnerability recall covering the Jeep Cherokee among 1.4 million vehicles: "Chrysler will notify and mail affected owners a USB drive that includes a software update"
- Lumen Black Lotus Labs, "The Pumpkin Eclipse", 30 May 2024, updated 7 June 2024; the figures here follow the updated version
- Louis Casiano, Fox Business, "Hyundai recalls over 421,000 vehicles to fix software bug causing unexpected braking", 25 May 2026: https://www.foxbusiness.com/lifestyle/hyundai-recalls-over-421000-vehicles-fix-software-bug-causing-unexpected-braking
- Eclipse hawkBit: project page https://projects.eclipse.org/projects/iot.hawkbit, repository, release notes and shipped configuration defaults https://github.com/eclipse-hawkbit/hawkbit, original project proposal (September 2015)
- hawkBit capabilities named here are in the upstream source and API model: maintenance windows with cron schedule, duration and time zone (DdiDeployment, DdiMaintenanceWindowStatus), target types and assignment compatibility, auto-assignment rules, dynamic rollout groups, confirmation before installation, RSQL target filters
- Bosch IoT Rollouts documentation, docs.bosch-iot-rollouts.com, for the Sign & Encrypt extension, its separate licensing and its key management options
- hawkBit issue #1534, "Managing/Signing Artifacts?", January 2024: https://github.com/eclipse-hawkbit/hawkbit/issues/1534
- Zephyr carries hawkBit support in tree; Nordic ships it in the nRF Connect SDK
- RAUC security advisory GHSA-6hj7-q844-m2hx, CVE-2026-34155, 27 March 2026: https://github.com/rauc/rauc/security/advisories/GHSA-6hj7-q844-m2hx
- Regulation (EU) 2024/2847 (Cyber Resilience Act), Article 13(8), Article 14, Article 24, Article 71(2) and Annex I: https://eur-lex.europa.eu/eli/reg/2024/2847/oj
- UN Regulation No. 156, software update and software update management system: https://unece.org/transport/documents/2021/03/standards/un-regulation-no-156-software-update-and-software-update