On 4/2/2026. Konnect introduced a change as part of our 'secure by default' initiatives for the upcoming 3.14 release, which caused expected_hash syncing issues for some customers' data planes.
For background, when a data plane connects to the Konnect control plane, the configuration is produced for that specific version of the Kong gateway. A configuration hash is calculated based off of the produced config, and stored as the expected config hash. When configuration changes occur, the control plane calculates the new configuration, broadcasts it to the connected data planes, and updates to the new expected hash.
The issue here was the preparation of the control plane for supporting SHA256 to store credential hashes. When this change was introduced, it resulted in a minor structural difference in the produced configuration, but no actual change to the configurations occurred. This led to the same configuration yielding a different hash before and after the deploy.
The way this could get a data plane to report 'out of sync' was the following:
- A data plane is connected and in sync with DP hash and Expected Config Hash `a111111a`
- The data plane is disconnected/reconnected, and since it is still in sync with DP hash and Expected Config Hash `a111111a`, a new config is not calculated
- The change to embed SHA256 support in the control plane is deployed
- A new data plane connects with no config loaded, and reports an empty config to the Control Plane
- The control plane generates the slightly altered config, and ends up with a new config hash, which updates the expected config hash to `b22222b`
- Since no configuration change occurred, the data planes already connected and on `1aaaaaa1` would not have been pushed a change event, and would still be running the same configuration as `b222222b` but the hash would not match, causing the DPs to appear out of sync
Note: Once *any* change to the control plane configuration occurred, a new config event would have been delivered, and all previous data planes would have been updated to the hash based on the new structure. Additionally, if an impacted data plane were to disconnect and reconnect, they also would have been brought into sync with the new hash. The issue was the system failing to consider the new hash as an 'event' since it did not change configured proxy behavior.
To address this in the short term, the Konnect engineering team has ensured structural changes like this which result in a config hash will trigger a configuration change event, even when no config has changed.
To address this in the long term, Konnect is moving to a config version tracking methodology that will not be impacted by structural, non-behavioral changes as the hash is.
Posted Apr 01, 2026 - 06:30 UTC