Downgrade Mesos

This document serves as a guide for users who wish to downgrade from an existing Mesos cluster to a previous version. This usually happens when rolling back from problematic upgrades. Mesos provides compatibility between any 1.x and 1.y versions of masters/agents as long as new features are not used. Since Mesos 1.8, we introduced a check for minimum capabilities on the master. If a backwards incompatible feature is used, a corresponding minimum capability entry will be persisted to the registry. If an old master (that does not possess the capability) tries to recover from the registry (e.g. when rolling back), an error message will be printed containing the missing capabilities. This document lists the detailed information regarding these minimum capabilities and remediation for downgrade errors.

List of Master Minimum Capabilities

CapabilityDescription
AGENT_DRAINING This capability is required when any agent is marked for draining or deactivated. These states were added in Mesos 1.9 and are triggered by using the DRAIN_AGENT or DEACTIVATE_AGENT operator APIs.
To remove this minimum capability requirement:
  1. Stop the master downgrade and return to the more recent version.
  2. Find all agents that are marked for draining or deactivated. This can be done by using the GET_AGENTS operator API and checking the deactivated boolean field of each agent. All draining agents will also be deactivated.
  3. Use the REACTIVATE_AGENT operator API for each deactivated agent.
QUOTA_V2 This capability is required when quota is configured in Mesos 1.9 or higher. When that happens, the newly configured quota will be persisted in the quota_configs field in the registry which requires this capability to decode.
To remove this minimum capability requirement:
  1. Stop the master downgrade and return to the more recent version.
  2. Use the /registrar(id)/registry endpoint to read the registry content and identify roles listed under the quota_configs field.
  3. Reset those roles' quota back to default (no guarantees and no limits). This will remove the roles from the quota_configs field. Once quota_configs becomes empty, the capability requirement will be removed.