健康检查¶
概览¶
Ceph 集群可能产生的健康消息是有限的——它们通通被定义为健康检查，都有唯一标识符。
The identifier is a terse pseudo-human-readable (i.e. like a variable name) string. It is intended to enable tools (such as UIs) to make sense of health checks, and present them in a way that reflects their meaning.
This page lists the health checks that are raised by the monitor and manager daemons. In addition to these, you may also see health checks that originate from MDS daemons (see CephFS 健康消息), and health checks that are defined by ceph-mgr python modules.
状态定义¶
监视器¶
MON_DOWN¶
One or more monitor daemons is currently down. The cluster requires a majority (more than 1/2) of the monitors in order to function. When one or more monitors are down, clients may have a harder time forming their initial connection to the cluster as they may need to try more addresses before they reach an operating monitor.
The down monitor daemon should generally be restarted as soon as possible to reduce the risk of a subsequen monitor failure leading to a service outage.
MON_CLOCK_SKEW¶
The clocks on the hosts running the ceph-mon monitor daemons are not
sufficiently well synchronized. This health alert is raised if the
cluster detects a clock skew greater than mon_clock_drift_allowed.
This is best resolved by synchronizing the clocks using a tool like
ntpd or chrony.
If it is impractical to keep the clocks closely synchronized, the
mon_clock_drift_allowed threshold can also be increased, but this
value must stay significantly below the mon_lease interval in
order for monitor cluster to function properly.
MON_MSGR2_NOT_ENABLED¶
The ms_bind_msgr2 option is enabled but one or more monitors is
not configured to bind to a v2 port in the cluster’s monmap. This
means that features specific to the msgr2 protocol (e.g., encryption)
are not available on some or all connections.
In most cases this can be corrected by issuing the command:
ceph mon enable-msgr2
That command will change any monitor configured for the old default port 6789 to continue to listen for v1 connections on 6789 and also listen for v2 connections on the new default 3300 port.
If a monitor is configured to listen for v1 connections on a non-standard port (not 6789), then the monmap will need to be modified manually.
管理器¶
MGR_DOWN¶
All manager daemons are currently down. The cluster should normally
have at least one running manager (ceph-mgr) daemon. If no
manager daemon is running, the cluster’s ability to monitor itself will
be compromised, and parts of the management API will become
unavailable (for example, the dashboard will not work, and most CLI
commands that report metrics or runtime state will block). However,
the cluster will still be able to perform all IO operations and
recover from failures.
The down manager daemon should generally be restarted as soon as
possible to ensure that the cluster can be monitored (e.g., so that
the ceph -s information is up to date, and/or metrics can be
scraped by Prometheus).
MGR_MODULE_DEPENDENCY¶
An enabled manager module is failing its dependency check. This health check should come with an explanatory message from the module about the problem.
For example, a module might report that a required package is not installed: install the required package and restart your manager daemons.
This health check is only applied to enabled modules. If a module is not enabled, you can see whether it is reporting dependency issues in the output of ceph module ls.
MGR_MODULE_ERROR¶
A manager module has experienced an unexpected error. Typically, this means an unhandled exception was raised from the module’s serve function. The human readable description of the error may be obscurely worded if the exception did not provide a useful description of itself.
This health check may indicate a bug: please open a Ceph bug report if you think you have encountered a bug.
If you believe the error is transient, you may restart your manager daemon(s), or use ceph mgr fail on the active daemon to prompt a failover to another daemon.
OSD¶
OSD_DOWN¶
至少有一个 OSD 被标记成了 down 状态，其 ceph-osd 守护进程可能已经停掉了、或者是对端 OSD 与此 OSD 之间的网络不通。常见起因有守护进程停止或崩溃、主机挂了、或者网络中断。
核实一下此主机是否健康、守护进程是否启动、网络是否正常。如果那个守护进程崩溃了，其守护进程日志文件（
/var/log/ceph/ceph-osd.* ）里会包含调试信息。
OSD_<crush type>_DOWN¶
(例如 OSD_HOST_DOWN, OSD_ROOT_DOWN)
某一个 CRUSH 子树里的所有 OSD 都被标记成 down 了，例如一台主机上的所有 OSD 。
OSD_ORPHAN¶
CRUSH 图分级结构里提到了这个 OSD ，但它并不存在。
CRUSH 图分级结构里的这个 OSD 可以用以下命令删除：
ceph osd crush rm osd.<id>
OSD_OUT_OF_ORDER_FULL¶
The utilization thresholds for backfillfull, nearfull, full, and/or failsafe_full are not ascending. In particular, we expect backfillfull < nearfull, nearfull < full, and full < failsafe_full.
The thresholds can be adjusted with:
ceph osd set-backfillfull-ratio <ratio>
ceph osd set-nearfull-ratio <ratio>
ceph osd set-full-ratio <ratio>
OSD_FULL¶
One or more OSDs has exceeded the full threshold and is preventing the cluster from servicing writes.
Utilization by pool can be checked with:
ceph df
The currently defined full ratio can be seen with:
ceph osd dump | grep full_ratio
A short-term workaround to restore write availability is to raise the full threshold by a small amount:
ceph osd set-full-ratio <ratio>
New storage should be added to the cluster by deploying more OSDs or existing data should be deleted in order to free up space.
OSD_BACKFILLFULL¶
One or more OSDs has exceeded the backfillfull threshold, which will prevent data from being allowed to rebalance to this device. This is an early warning that rebalancing may not be able to complete and that the cluster is approaching full.
Utilization by pool can be checked with:
ceph df
OSD_NEARFULL¶
One or more OSDs has exceeded the nearfull threshold. This is an early warning that the cluster is approaching full.
Utilization by pool can be checked with:
ceph df
OSDMAP_FLAGS¶
One or more cluster flags of interest has been set. These flags include:
full - the cluster is flagged as full and cannot service writes
pauserd, pausewr - paused reads or writes
noup - OSDs are not allowed to start
nodown - OSD failure reports are being ignored, such that the monitors will not mark OSDs down
noin - OSDs that were previously marked out will not be marked back in when they start
noout - down OSDs will not automatically be marked out after the configured interval
nobackfill, norecover, norebalance - recovery or data rebalancing is suspended
noscrub, nodeep_scrub - scrubbing is disabled
notieragent - cache tiering activity is suspended
With the exception of full, these flags can be set or cleared with:
ceph osd set <flag>
ceph osd unset <flag>
OSD_FLAGS¶
One or more OSDs or CRUSH {nodes,device classes} has a flag of interest set. These flags include:
noup: OSD is not allowed to start
nodown: failure reports for this OSD will be ignored
noin: if this OSD was previously marked out automatically after a failure, it will not be marked in when it stats
noout: if this OSD is down it will not automatically be marked out after the configured interval
这些标记可以这样批量设置和清除：
ceph osd set-group <flags> <who>
ceph osd unset-group <flags> <who>
例如：
ceph osd set-group noup,noout osd.0 osd.1
ceph osd unset-group noup,noout osd.0 osd.1
ceph osd set-group noup,noout host-foo
ceph osd unset-group noup,noout host-foo
ceph osd set-group noup,noout class-hdd
ceph osd unset-group noup,noout class-hdd
OLD_CRUSH_TUNABLES¶
CRUSH 图在使用很老的选项，应该更新它。还能使用（即，可连接此集群的最老客户端版本号）而不会触发此健康告警的最老可调选项由
mon_crush_min_required_version 配置选项决定。详情见
ref:crush-map-tunables 。
OLD_CRUSH_STRAW_CALC_VERSION¶
The CRUSH map is using an older, non-optimal method for calculating
intermediate weight values for straw buckets.
The CRUSH map should be updated to use the newer method
(straw_calc_version=1). See
可调选项 for more information.
CACHE_POOL_NO_HIT_SET¶
One or more cache pools is not configured with a hit set to track utilization, which will prevent the tiering agent from identifying cold objects to flush and evict from the cache.
Hit sets can be configured on the cache pool with:
ceph osd pool set <poolname> hit_set_type <type>
ceph osd pool set <poolname> hit_set_period <period-in-seconds>
ceph osd pool set <poolname> hit_set_count <number-of-hitsets>
ceph osd pool set <poolname> hit_set_fpp <target-false-positive-rate>
OSD_NO_SORTBITWISE¶
No pre-luminous v12.y.z OSDs are running but the sortbitwise flag has not
been set.
The sortbitwise flag must be set before luminous v12.y.z or newer
OSDs can start. You can safely set the flag with:
ceph osd set sortbitwise
POOL_FULL¶
One or more pools has reached its quota and is no longer allowing writes.
Pool quotas and utilization can be seen with:
ceph df detail
You can either raise the pool quota with:
ceph osd pool set-quota <poolname> max_objects <num-objects>
ceph osd pool set-quota <poolname> max_bytes <num-bytes>
or delete some existing data to reduce utilization.
设备健康¶
DEVICE_HEALTH¶
One or more devices is expected to fail soon, where the warning
threshold is controlled by the mgr/devicehealth/warn_threshold
config option.
This warning only applies to OSDs that are currently marked “in”, so
the expected response to this failure is to mark the device “out” so
that data is migrated off of the device, and then to remove the
hardware from the system. Note that the marking out is normally done
automatically if mgr/devicehealth/self_heal is enabled based on
the mgr/devicehealth/mark_out_threshold.
Device health can be checked with:
ceph device info <device-id>
Device life expectancy is set by a prediction model run by the mgr or an by external tool via the command:
ceph device set-life-expectancy <device-id> <from> <to>
You can change the stored life expectancy manually, but that usually doesn’t accomplish anything as whatever tool originally set it will probably set it again, and changing the stored value does not affect the actual health of the hardware device.
DEVICE_HEALTH_IN_USE¶
One or more devices is expected to fail soon and has been marked “out”
of the cluster based on mgr/devicehealth/mark_out_threshold, but it
is still participating in one more PGs. This may be because it was
only recently marked “out” and data is still migrating, or because data
cannot be migrated off for some reason (e.g., the cluster is nearly
full, or the CRUSH hierarchy is such that there isn’t another suitable
OSD to migrate the data too).
This message can be silenced by disabling the self heal behavior
(setting mgr/devicehealth/self_heal to false), by adjusting the
mgr/devicehealth/mark_out_threshold, or by addressing what is
preventing data from being migrated off of the ailing device.
DEVICE_HEALTH_TOOMANY¶
Too many devices is expected to fail soon and the
mgr/devicehealth/self_heal behavior is enabled, such that marking
out all of the ailing devices would exceed the clusters
mon_osd_min_in_ratio ratio that prevents too many OSDs from being
automatically marked “out”.
This generally indicates that too many devices in your cluster are expected to fail soon and you should take action to add newer (healthier) devices before too many devices fail and data is lost.
The health message can also be silenced by adjusting parameters like
mon_osd_min_in_ratio or mgr/devicehealth/mark_out_threshold,
but be warned that this will increase the likelihood of unrecoverable
data loss in the cluster.
数据健康（存储池和归置组们）¶
PG_AVAILABILITY¶
Data availability is reduced, meaning that the cluster is unable to service potential read or write requests for some data in the cluster. Specifically, one or more PGs is in a state that does not allow IO requests to be serviced. Problematic PG states include peering, stale, incomplete, and the lack of active (if those conditions do not clear quickly).
Detailed information about which PGs are affected is available from:
ceph health detail
In most cases the root cause is that one or more OSDs is currently
down; see the discussion for OSD_DOWN above.
The state of specific problematic PGs can be queried with:
ceph tell <pgid> query
PG_DEGRADED¶
Data redundancy is reduced for some data, meaning the cluster does not have the desired number of replicas for all data (for replicated pools) or erasure code fragments (for erasure coded pools). Specifically, one or more PGs:
has the degraded or undersized flag set, meaning there are not enough instances of that placement group in the cluster;
has not had the clean flag set for some time.
Detailed information about which PGs are affected is available from:
ceph health detail
In most cases the root cause is that one or more OSDs is currently
down; see the dicussion for OSD_DOWN above.
The state of specific problematic PGs can be queried with:
ceph tell <pgid> query
PG_DEGRADED_FULL¶
Data redundancy may be reduced or at risk for some data due to a lack of free space in the cluster. Specifically, one or more PGs has the backfill_toofull or recovery_toofull flag set, meaning that the cluster is unable to migrate or recover data because one or more OSDs is above the backfillfull threshold.
See the discussion for OSD_BACKFILLFULL or OSD_FULL above for steps to resolve this condition.
PG_BACKFILL_FULL¶
Data redundancy may be reduced or at risk for some data due to a lack of free space in the cluster. Specifically, one or more PGs has the backfill_toofull flag set, meaning that the cluster is unable to migrate or recover data because one or more OSDs is above the backfillfull threshold.
See the discussion for OSD_BACKFILLFULL above for steps to resolve this condition.
PG_DAMAGED¶
Data scrubbing has discovered some problems with data consistency in the cluster. Specifically, one or more PGs has the inconsistent or snaptrim_error flag is set, indicating an earlier scrub operation found a problem, or that the repair flag is set, meaning a repair for such an inconsistency is currently in progress.
详情见 修复 PG 不一致状态 。
LARGE_OMAP_OBJECTS¶
One or more pools contain large omap objects as determined by
osd_deep_scrub_large_omap_object_key_threshold (threshold for number of keys
to determine a large omap object) or
osd_deep_scrub_large_omap_object_value_sum_threshold (the threshold for
summed size (bytes) of all key values to determine a large omap object) or both.
More information on the object name, key count, and size in bytes can be found
by searching the cluster log for ‘Large omap object found’. Large omap objects
can be caused by RGW bucket index objects that do not have automatic resharding
enabled. Please see RGW Dynamic Bucket Index Resharding for more information on resharding.
The thresholds can be adjusted with:
ceph config set osd osd_deep_scrub_large_omap_object_key_threshold <keys>
ceph config set osd osd_deep_scrub_large_omap_object_value_sum_threshold <bytes>
CACHE_POOL_NEAR_FULL¶
A cache tier pool is nearly full. Full in this context is determined
by the target_max_bytes and target_max_objects properties on
the cache pool. Once the pool reaches the target threshold, write
requests to the pool may block while data is flushed and evicted
from the cache, a state that normally leads to very high latencies and
poor performance.
The cache pool target size can be adjusted with:
ceph osd pool set <cache-pool-name> target_max_bytes <bytes>
ceph osd pool set <cache-pool-name> target_max_objects <objects>
Normal cache flush and evict activity may also be throttled due to reduced availability or performance of the base tier, or overall cluster load.
TOO_FEW_PGS¶
The number of PGs in use in the cluster is below the configurable
threshold of mon_pg_warn_min_per_osd PGs per OSD. This can lead
to suboptimal distribution and balance of data across the OSDs in
the cluster, and similarly reduce overall performance.
This may be an expected condition if data pools have not yet been created.
The PG count for existing pools can be increased or new pools can be created. Please refer to 确定归置组数量 for more information.
POOL_TOO_FEW_PGS¶
One or more pools should probably have more PGs, based on the amount
of data that is currently stored in the pool. This can lead to
suboptimal distribution and balance of data across the OSDs in the
cluster, and similarly reduce overall performance. This warning is
generated if the pg_autoscale_mode property on the pool is set to
warn.
To disable the warning, you can disable auto-scaling of PGs for the pool entirely with:
ceph osd pool set <pool-name> pg_autoscale_mode off
To allow the cluster to automatically adjust the number of PGs,:
ceph osd pool set <pool-name> pg_autoscale_mode on
You can also manually set the number of PGs for the pool to the recommended amount with:
ceph osd pool set <pool-name> pg_num <new-pg-num>
TOO_MANY_PGS¶
The number of PGs in use in the cluster is above the configurable
threshold of mon_max_pg_per_osd PGs per OSD. If this threshold is
exceed the cluster will not allow new pools to be created, pool pg_num to
be increased, or pool replication to be increased (any of which would lead to
more PGs in the cluster). A large number of PGs can lead
to higher memory utilization for OSD daemons, slower peering after
cluster state changes (like OSD restarts, additions, or removals), and
higher load on the Manager and Monitor daemons.
The simplest way to mitigate the problem is to increase the number of OSDs in the cluster by adding more hardware. Note that the OSD count used for the purposes of this health check is the number of “in” OSDs, so marking “out” OSDs “in” (if there are any) can also help:
ceph osd in <osd id(s)>
Please refer to 确定归置组数量 for more information.
POOL_TOO_MANY_PGS¶
One or more pools should probably have more PGs, based on the amount
of data that is currently stored in the pool. This can lead to higher
memory utilization for OSD daemons, slower peering after cluster state
changes (like OSD restarts, additions, or removals), and higher load
on the Manager and Monitor daemons. This warning is generated if the
pg_autoscale_mode property on the pool is set to warn.
To disable the warning, you can disable auto-scaling of PGs for the pool entirely with:
ceph osd pool set <pool-name> pg_autoscale_mode off
To allow the cluster to automatically adjust the number of PGs,:
ceph osd pool set <pool-name> pg_autoscale_mode on
You can also manually set the number of PGs for the pool to the recommended amount with:
ceph osd pool set <pool-name> pg_num <new-pg-num>
POOL_TARGET_SIZE_RATIO_OVERCOMMITTED¶
One or more pools have a target_size_ratio property set to
estimate the expected size of the pool as a fraction of total storage,
but the value(s) exceed the total available storage (either by
themselves or in combination with other pools’ actual usage).
This is usually an indication that the target_size_ratio value for
the pool is too large and should be reduced or set to zero with:
ceph osd pool set <pool-name> target_size_ratio 0
For more information, see 配置期望的存储池尺寸.
POOL_TARGET_SIZE_BYTES_OVERCOMMITTED¶
One or more pools have a target_size_bytes property set to
estimate the expected size of the pool,
but the value(s) exceed the total available storage (either by
themselves or in combination with other pools’ actual usage).
This is usually an indication that the target_size_bytes value for
the pool is too large and should be reduced or set to zero with:
ceph osd pool set <pool-name> target_size_bytes 0
For more information, see 配置期望的存储池尺寸.
TOO_FEW_OSDS¶
The number of OSDs in the cluster is below the configurable
threshold of osd_pool_default_size.
SMALLER_PGP_NUM¶
One or more pools has a pgp_num value less than pg_num. This
is normally an indication that the PG count was increased without
also increasing the placement behavior.
This is sometimes done deliberately to separate out the split step
when the PG count is adjusted from the data migration that is needed
when pgp_num is changed.
This is normally resolved by setting pgp_num to match pg_num,
triggering the data migration, with:
ceph osd pool set <pool> pgp_num <pg-num-value>
MANY_OBJECTS_PER_PG¶
One or more pools has an average number of objects per PG that is
significantly higher than the overall cluster average. The specific
threshold is controlled by the mon_pg_warn_max_object_skew
configuration value.
This is usually an indication that the pool(s) containing most of the data in the cluster have too few PGs, and/or that other pools that do not contain as much data have too many PGs. See the discussion of TOO_MANY_PGS above.
The threshold can be raised to silence the health warning by adjusting
the mon_pg_warn_max_object_skew config option on the monitors.
POOL_APP_NOT_ENABLED¶
A pool exists that contains one or more objects but has not been tagged for use by a particular application.
Resolve this warning by labeling the pool for use by an application. For example, if the pool is used by RBD,:
rbd pool init <poolname>
If the pool is being used by a custom application ‘foo’, you can also label via the low-level command:
ceph osd pool application enable foo
For more information, see 关联存储池与应用程序.
POOL_FULL¶
One or more pools has reached (or is very close to reaching) its
quota. The threshold to trigger this error condition is controlled by
the mon_pool_quota_crit_threshold configuration option.
Pool quotas can be adjusted up or down (or removed) with:
ceph osd pool set-quota <pool> max_bytes <bytes>
ceph osd pool set-quota <pool> max_objects <objects>
Setting the quota value to 0 will disable the quota.
POOL_NEAR_FULL¶
One or more pools is approaching is quota. The threshold to trigger
this warning condition is controlled by the
mon_pool_quota_warn_threshold configuration option.
Pool quotas can be adjusted up or down (or removed) with:
ceph osd pool set-quota <pool> max_bytes <bytes>
ceph osd pool set-quota <pool> max_objects <objects>
Setting the quota value to 0 will disable the quota.
OBJECT_MISPLACED¶
One or more objects in the cluster is not stored on the node the cluster would like it to be stored on. This is an indication that data migration due to some recent cluster change has not yet completed.
Misplaced data is not a dangerous condition in and of itself; data consistency is never at risk, and old copies of objects are never removed until the desired number of new copies (in the desired locations) are present.
OBJECT_UNFOUND¶
One or more objects in the cluster cannot be found. Specifically, the OSDs know that a new or updated copy of an object should exist, but a copy of that version of the object has not been found on OSDs that are currently online.
Read or write requests to unfound objects will block.
Ideally, a down OSD can be brought back online that has the more recent copy of the unfound object. Candidate OSDs can be identified from the peering state for the PG(s) responsible for the unfound object:
ceph tell <pgid> query
If the latest copy of the object is not available, the cluster can be told to roll back to a previous version of the object. See 未找到的对象 for more information.
SLOW_OPS¶
One or more OSD requests is taking a long time to process. This can be an indication of extreme load, a slow storage device, or a software bug.
The request queue on the OSD(s) in question can be queried with the following command, executed from the OSD host:
ceph daemon osd.<id> ops
A summary of the slowest recent requests can be seen with:
ceph daemon osd.<id> dump_historic_ops
OSD 的位置可用此命令找到：
ceph osd find osd.<id>
PG_NOT_SCRUBBED¶
One or more PGs has not been scrubbed recently. PGs are normally
scrubbed every mon_scrub_interval seconds, and this warning
triggers when mon_warn_pg_not_scrubbed_ratio percentage of interval has elapsed
without a scrub since it was due.
PGs will not scrub if they are not flagged as clean, which may happen if they are misplaced or degraded (see PG_AVAILABILITY and PG_DEGRADED above).
You can manually initiate a scrub of a clean PG with:
ceph pg scrub <pgid>
PG_NOT_DEEP_SCRUBBED¶
One or more PGs has not been deep scrubbed recently. PGs are normally
scrubbed every osd_deep_mon_scrub_interval seconds, and this warning
triggers when mon_warn_pg_not_deep_scrubbed_ratio percentage of interval has elapsed
without a scrub since it was due.
PGs will not (deep) scrub if they are not flagged as clean, which may happen if they are misplaced or degraded (see PG_AVAILABILITY and PG_DEGRADED above).
You can manually initiate a scrub of a clean PG with:
ceph pg deep-scrub <pgid>
