Network device maintenance: firmware, configuration backups, change log

Why a network device goes out of date even while it keeps working: firmware updates, configuration backups, a change log, and the end of vendor support.

Topic
Networks
Author
ARLITECH expert team
Published

A switch has been forwarding traffic without a fault for six years. Its software has not changed in that time, none of the fixes released since then are on it, and nobody can say any longer who added which line to its configuration, or why.

That is why maintaining network devices is regular work even when nothing breaks. It covers updating the firmware, backing up the configuration and logging changes, tracking vendor support, and reviewing the rule base from time to time. None of this can be done without an accurate inventory, so the article ends with that.

Why a working network device goes out of date

Firmware reflects what was known on the day it was released. Some of its bugs are found only years later, by researchers, by the vendor's own team or by attackers. The vendor then releases a fixed version, but most enterprise network devices do not update themselves. Until someone installs the new version, the flaw stays open.

The risk is highest at the network edge. Firewalls, VPN gateways and remote access devices are visible from the internet, and a flaw exploited on one of them leads straight into the internal network. Flaws in firewalls, VPN gateways and routers show up regularly in the catalog CISA keeps of vulnerabilities exploited in the wild (Known Exploited Vulnerabilities Catalog).

A device goes out of date even if its own code is flawless, because the environment around it changes. A few typical examples:

  • Encryption protocols become obsolete. In 2021 the IETF deprecated TLS 1.0 and 1.1 in RFC 8996, yet the web management interface of old firmware often supports only these.
  • Management protocols are weak. Telnet and the first two versions of SNMP send credentials unencrypted, and they need to be replaced with SSH and SNMPv3.
  • The configuration drifts. Lines added for a troubleshooting session, a project or a partner often stay in after the reason is gone, because nobody takes them out.
  • Support runs out. After a while the vendor stops releasing fixes for the device. The fourth section covers this.

Firmware updates: release notes, scheduling, rollback

Revision 4 of NIST SP 800-40 (2022) calls patching preventive maintenance in its very title. This holds especially for network devices, because updating a switch or firewall almost always causes a short outage, so it has to be planned.

Release notes

Before a new version goes on, read its release notes. They tell you which bugs and vulnerabilities the version fixes and which known issues remain in it. They also give the upgrade path: from a very old version you can often reach the target version only through one or more intermediate versions. They list the hardware requirements, such as the memory and storage needed, and which features are removed or behave differently from now on. A changed default or a renamed command can cause an outage after the update just as a faulty version can.

The latest major version is not always the best choice. Many vendors mark a recommended or longer-supported release train separately. The vendor usually patches these for longer, and there is more field experience with them.

Scheduling

Routine updates go in a maintenance window, at a time agreed in advance. Order matters too: a less important device gets the new version first, then the rest. In redundant pairs, such as a high-availability firewall pair, the two members are updated one after the other, so one of them carries the traffic throughout.

Break from this rhythm if the fixed flaw is in the CISA catalog, or if the affected service, such as the VPN or the management interface, is reachable from the internet. In that case the update goes out of band, and if no fix exists yet, the affected service has to be restricted temporarily, following the vendor advisory.

Rollback

Before every update, decide how you will roll back if something goes wrong. You need the following:

  1. A fresh configuration backup taken right before the update.
  2. The new firmware from the vendor's official source, with a verified checksum or signature.
  3. The image file of the previous version. Many devices keep two boot partitions, so the old version can be brought back with a reboot.
  4. Console or other out-of-band access, because after a failed update the device often cannot be reached over the network.
  5. Knowing whether a downgrade is supported. If the new version converted the configuration format, you can go back to the old version only together with the old backup.

After the update, check that everything works: the interfaces are up, the routing adjacencies and VPN tunnels have come back, and there are no new errors in the log. The result, along with the version number, also goes into the change log. Staging updates for machines and applications is covered in the article on update rings, and the order of patching in the article on vulnerability management.

Configuration backups and the change log

The configuration of a switch or firewall holds years of work: VLANs, routes, rules, VPN settings, user accounts. If the device fails, the replacement returns to the original state only from the backup. Without a backup, the configuration has to be rebuilt from memory and old emails while part of the network is down.

What to back up and how

Many devices keep a separate running and startup configuration. A change first goes only into the running configuration, and if someone forgets to save it, it is gone after a reboot. The backup has to capture the state that is actually running, automatically, after every change or daily, to storage outside the device.

Keep the backups under version control, so any two states can be compared. The diff between two backups shows every change, and a change with no matching entry in the change log has to be investigated. NIST SP 800-128 (2011) describes, among other things, recording a baseline configuration, controlling changes and continuously monitoring for deviations.

The configuration file is sensitive data. It holds password hashes, VPN pre-shared keys, SNMP credentials and a complete map of the network, so the backup store needs the same protection as the device: encryption, tightly limited access and logging of that access. Some certificates, private keys and licenses are not part of the text configuration and have to be backed up separately.

A backup is worth something only if it can be restored. Every so often, try it on a spare device or in a test environment to confirm that the backup really produces a working device.

The change log

The change log answers the question of why the configuration is the way it is now. A well-kept entry contains the following:

FieldWhat it records
Time and deviceWhen the change was made and on which device.
Performed byWho made it, using a personal account rather than a shared admin account.
Reason and ticket numberWhich request or fault made it necessary, and where the request can be tracked.
ApprovalWho authorized it, if the change requires approval.
Exact changeWhich lines were added or removed, ideally as the diff between two backups.
RollbackHow it can be reverted if it causes trouble.
ExpiryFor a temporary rule or access, the date when it has to be removed.

If management access goes through central authentication and logging, the devices' own logs also record who logged in and which commands they ran. Comparing the two sources reveals undocumented changes. Where to collect these logs and for how long is described in the article on logging.

Vendor support and end of life

Every network device model has a lifecycle that the vendor publishes in advance. The milestones have different names from vendor to vendor, but their content is roughly the same:

  • End of sale: the model can no longer be ordered, but installed units are still supported.
  • End of software maintenance: no new features or general bug fixes, though security fixes still come.
  • End of security fixes: from this point on, newly discovered vulnerabilities get no fix.
  • End of support (or end of life): the vendor provides no fixes, troubleshooting or hardware replacement of any kind.

After the last date the device keeps working just as before, but every flaw discovered from then on stays open for good. CISA's list of practices to avoid (Bad Practices) singles out the use of unsupported software, especially on systems reachable from the internet. CIS Controls version 8 (2021), in Safeguard 12.1, asks that the software version and support status of network infrastructure be reviewed monthly or more often.

Vendor support also depends on the contract. With many vendors, downloading firmware, opening a case and replacing faulty hardware require a valid support contract. If the contract lapses unnoticed, the fix cannot be downloaded even when it is urgently needed. So the expiry date needs watching just like the lifecycle dates.

Start planning the replacement a good few years before end of support, because procurement, testing and migration take time. If a device cannot be replaced for the time being, its risk can be reduced. That means cutting the management interface off from the internet and the user networks, turning off unused services, placing the device in a tight segment and monitoring its traffic more closely. This situation is common in OT environments, and the article on unsupported systems covers it in detail.

Periodic rule base review

Updating the firmware and backing up the configuration say nothing about whether what the configuration contains is still correct. A rule base deteriorates on its own over the years: new rules get added, while old ones are rarely removed. So it has to be reviewed at a fixed interval and after every major network change.

The review applies to the other devices as well as the firewall. Some recurring questions:

  • Is there a firewall rule or access list that no traffic has passed through for months?
  • Who can reach the management interfaces, and from where? Only from a dedicated management network, or from any internal machine?
  • Are there still local accounts in the names of former employees or suppliers you no longer work with?
  • Are any default or easily guessed SNMP credentials still active?
  • Which VPN profiles and remote access entries belong to a site or partner that no longer exists?

The details of analyzing firewall rules, including how to spot overly broad and shadowed rules, are in the article on firewall rule base review.

Inventory and documentation

Every task so far rests on knowing which devices are on the network. If you do not know how many switches you have and which version runs on each, you cannot even tell whether a new vulnerability advisory affects you. A network device inventory should include at least the following:

DataWhat it is for
Vendor, model, serial numberSecurity advisories and support cases identify the device by these.
Location and roleSite, rack, and whether the device works at the edge, in the core or in the access layer.
Firmware versionThis decides whether a new vulnerability affects the device.
Management address and access methodWhere and over which protocol it is managed, and whether console access exists.
Lifecycle datesWhen security fixes and support end.
Support contractWhat it entitles you to, and when it expires.
Last backupWhen the last configuration backup was made, and whether it succeeded.
OwnerWho decides on changes to the device and its updates.

A spreadsheet maintained by hand goes stale quickly. SNMP polling and LLDP neighbor data help discover the devices and the links between them, and the backup system can record the version number with every backup. Tie the inventory to adding and retiring devices: whatever is removed comes out of it, and whatever is installed goes in before it is put into service.

The inventory needs documentation alongside it. This includes the network topology diagram, the IP addressing plan and the VLAN list, a description of the redundant links and the path for management access. Write down the update and rollback procedure step by step, so nobody has to rely on memory during a fault in the middle of the night. Admin credentials belong in a password manager, and the documentation should only point to where they are kept.

Quick self-check: pick your edge firewall and try to state, within five minutes, its firmware version, the date its security fixes end, the time of its last configuration backup, and who last changed it and why.

Related service: Network maintenance →

Contact

Ready to secure your critical systems?

Our experts will assess where your security stands and build a roadmap that fits your organization.

ARLITECH's office in Gödöllő

Visit us

You are welcome to visit our Gödöllő office in person.

Address
Centrum Office Building, 2100 Gödöllő, Dózsa György út 13, 3rd floor, office 309
Office hours
Mon–Fri, 9 a.m. to 5 p.m.
Get directions (opens in a new tab)