free-pmx-no-shred for Proxmox VE

UNRELEASED

Proxmox Cluster Filesystem holds the content of vital configurations in the mountpoint of /etc/pve. The backend is actually held in SQLite database residing on the OS disk under /var/lib/pve-cluster - this can be mounted over to RAM sparing most of the writes to hitting the actual block layer. The RAM hosted backend is flushed into the original location in regular intervals. Side effect of this setup is higher resiliency against database corruption as the flush operation is atomic.

Apart from this, configuration of rrdcached is also optimised for infrequent disk writes.

Quickstart

Download

wget -P /tmp https://free-pmx.pages.dev/tools/free-pmx-no-shred_0.1.0~20250509+TEST.deb

Install

apt install /tmp/free-pmx-no-shred_0.1.0~20250509+TEST.deb

Remove

apt purge free-pmx-no-shred

Audit

How to check what’s inside a debian package.

Glimpse at operation

The most useful command is plain no-shred:

free-pmx: NO-SHRED - Information summary

* Cluster filesystem backend database [pmxcfs]

Live:
Fri 2025-05-09 18:42:13 UTC lastmod 'config.db' - Size: 49152 B
Fri 2025-05-09 18:42:36 UTC lastmod 'config.db-wal' - Size: 4124152 B
Fri 2025-05-09 18:42:36 UTC last record - Version: 4361372 - Node: 2

Underlying:
Fri 2025-05-09 18:22:08 UTC lastmod 'config.db' - Size: 45056 B
Fri 2025-05-09 18:22:07 UTC last record - Version: 4358924 - Node: 1
20min behind: 2449 versions

Flush timer:
Fri 2025-05-09 19:22:07 UTC 39min left Fri 2025-05-09 18:22:07 UTC 20min ago no-shred-flush.timer no-shred-flush.service

* Time series data caching [rrdcached]

Process:
/usr/bin/rrdcached -B -f 2h -w 1h -b /var/lib/rrdcached/db/ -p /var/run/rrdcached.pid -l unix:/var/run/rrdcached.sock

Stats:
QueueLength: 0
UpdatesReceived: 1517
FlushesReceived: 0
UpdatesWritten: 0
DataSetsWritten: 0
TreeNodesNumber: 10
TreeDepth: 4
JournalBytes: 0
JournalRotate: 0

The Underlying section provides a good idea about the last flush time and how many ‘versions behind’ it is. Do note that there were many more actual writes than the ‘versions’ recorded made into RAM that the actual block layer was spared of - as can be explored in separate posts on how to briefly measure sector writes, or more in depth - how exactly does pmxcfs write onto disk.

The Flush timer information is particularly helpful to see when was the last and when is the next flush due. The history of flushes during current boot can also be conviently checked with no-shred log:

free-pmx: NO-SHRED - Log
May 09 18:17:08 pve3 systemd[1]: Starting no-shred.service - free-pmx: NO SHRED service...
May 09 18:17:08 pve3 no-shred[506]: Setting up flushing service timer ...
May 09 18:17:08 pve3 no-shred[506]: Creating ramfs for cluster filesystem backend database ...
May 09 18:17:08 pve3 no-shred[506]: Overriding rrdcached configuration ...
May 09 18:17:09 pve3 systemd[1]: Finished no-shred.service - free-pmx: NO SHRED service.
May 09 18:22:07 pve3 systemd[1]: Started no-shred-flush.service - free-pmx: NO SHRED flushing service.
May 09 18:22:08 pve3 no-shred[1683]: Flushing cluster filesystem backend database from ramfs ...
May 09 18:22:08 pve3 systemd[1]: no-shred-flush.service: Deactivated successfully.

Statistics of rrdcached are listed as well, despite out of scope here to delve into.

Manual flush

It is entirely possible to perform a manual pmxcfs backend database flush with: no-shred flush

Doing so will not be logged as it did not result from timed service run, but effect of which can be seen instantly seen in the no-shred summary run subsequently. During a flush - whether automated or manual - the database is atomically compacted onto the disk in a manner that cannot result in any (other than pre-existing) corruption.

As the actual directory written into is the very original one, there is no dependency on any tools or cleanup operation upon “uninstall” of the no-shred tool to have the host operational again. In fact, the resiliency against data corruption is increased due to this setup as regular SQLite database might e.g. end up corrupted during a power loss if it encountered it whilst performing a so-called checkpoint operation of its write-ahead log - this cannot happen when flushing the cache. See more details on this also in the linked manual page of the no-shred command.

Data since last flush

During e.g. a power-loss event, but also auto-reboots - due to how Proxmox watchdog operates, there will be a period since last flush when data were only held in RAM during such situation. On restart, the database state will be consistent, but equivalent to a node that had been turned off since the last flush. This is completely inconsequential to a node that is part of the cluster - nodes do synchronise their database state upon rejoining the cluster to the last one, essentially replaying the changes that had occurred since.

A special case would be either a sole node operation or a cluster where all nodes experienced such event all at once - the information lost would typically amount to e.g. state of HA services at the time of power failure, but this is not of a concern in respect to their recovery. Otherwise and also on non-HA system, configuration of e.g. newly created guests could be missing from within up to an hour back. If flushing this information is important at any stage, it is possible to manually flush as demonstrated above, including in own scripting.

Feedback

File an issue in the GitHub repository.

Disclaimer

THE TOOL IS A THIRD-PARTY PRODUCT AND IS PROVIDED ON AN “AS IS” BASIS AND WITHOUT WARRANTIES OF ANY KIND EITHER EXPRESS OR IMPLIED, STATUTORY OR OTHERWISE.