主页 / user-255786

Erick Ramirez's questions

Martin Hope
Erick Ramirez
Asked: 2023-02-17 13:18:25 +0800 CST
  • 5

症状

添加到 Cassandra 集群的新节点无法与种子节点进行八卦。

另一个症状是当一个已重启的现有节点无法与集群中的其他节点进行 gossip 时，在启动期间报告以下错误：

ERROR [main] 2019-08-15 18:46:32,241 CassandraDaemon.java:749 - Exception encountered during startup
java.lang.RuntimeException: Unable to gossip with any peers
        at org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1435) ~[apache-cassandra-3.11.4.jar:3.11.4]
        at org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:566) ~[apache-cassandra-3.11.4.jar:3.11.4]
        at org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:823) ~[apache-cassandra-3.11.4.jar:3.11.4]
        at org.apache.cassandra.service.StorageService.initServer(StorageService.java:683) ~[apache-cassandra-3.11.4.jar:3.11.4]
        at org.apache.cassandra.service.StorageService.initServer(StorageService.java:632) ~[apache-cassandra-3.11.4.jar:3.11.4]
        at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:388) ~[apache-cassandra-3.11.4.jar:3.11.4]
        at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:620) ~[apache-cassandra-3.11.4.jar:3.11.4]
        at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:732) ~[apache-cassandra-3.11.4.jar:3.11.4]

在某些情况下，其他节点能够将受影响的节点视为可操作的，但有问题的节点无法与其他节点进行八卦。以下是 的示例输出nodetool gossipinfo：

/10.1.2.4
  generation:0
  heartbeat:0
/10.1.2.3
  generation:0
  heartbeat:0
/10.1.2.6
  generation:1444263348
  heartbeat:6232
  ...
  DC:DC1
  STATUS:NORMAL,-1041938454866204344
  ...
/10.1.2.5
  generation:0
  heartbeat:0

另一个症状是受影响的节点将另一个 DC 中的节点视为“关闭”（DN或关闭/正常），如以下示例nodetool status输出所示：

Datacenter: r1 
============== 
Status=Up/Down 
|/ State=Normal/Leaving/Joining/Moving 
-- Address Load Tokens Owns Host ID Rack 
DN 10.1.2.5 ? 256 9.0% 5279619a-550c-42b3-8150-61ad24f828f3 r1 
DN 10.1.2.3 ? 256 9.1% 5d1fa459-cdac-4658-b68d-c6e0933afcee r1 
DN 10.1.2.4 ? 256 10.5% a8f35c63-6a76-4e95-99f1-bef65d785366 r1 
Datacenter: DC1 
=============== 
Status=Up/Down 
|/ State=Normal/Leaving/Joining/Moving 
-- Address Load Tokens Owns Host ID Rack 
UN 10.1.2.6 18.9 GB 256 9.5% 36fdcf57-0274-43b8-a501-c0e475e3e30b RAC1
cassandra
Martin Hope
Erick Ramirez
Asked: 2023-02-17 10:58:43 +0800 CST
  • 5

大多数情况下运行的 Cassandra 集群有时会遇到服务中断，尤其是当一个（或多个）节点无法与集群中的其他节点进行八卦时。

症状

一个症状是节点无缘无故地随机上升和下降。system.log以下是节点的示例摘录：

INFO  [GossipTasks:1] 2016-04-29 02:47:32,559  Gossiper.java:1001 - InetAddress /10.1.2.3 is now DOWN
INFO  [GossipTasks:1] 2016-04-29 02:50:47,123  Gossiper.java:1001 - InetAddress /10.1.2.4 is now DOWN
INFO  [GossipTasks:1] 2016-04-29 02:54:59,640  Gossiper.java:1001 - InetAddress /10.1.2.5 is now DOWN
INFO  [SharedPool-Worker-2] 2016-04-29 03:01:23,828  Gossiper.java:987 - InetAddress /10.1.2.4 is now UP
INFO  [SharedPool-Worker-1] 2016-04-29 03:01:59,432  Gossiper.java:987 - InetAddress /10.1.2.5 is now UP
INFO  [SharedPool-Worker-7] 2016-04-29 03:02:01,839  Gossiper.java:987 - InetAddress /10.1.2.3 is now UP

nodetool status同样，根据运行命令的节点，不同的节点在输出中似乎已关闭，例如：

Datacenter: Cassandra
=====================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address   Load       Tokens  Owns    Host ID                               Rack
DN  10.1.2.3  8.97 GB    256     ?       a50dfef5-229d-4d15-89d9-971bec01094b  rack1
UN  10.1.2.5  8.9 GB     256     ?       a16b71a2-9b95-4669-a6bd-d7326bd279e2  rack1
DN  10.1.2.4  9.09 GB    256     ?       ac01b6f9-3cb9-47ff-83c6-0404836386eb  rack1
UN  10.1.2.6  10.65 GB   256     ?       9c0ef3a2-aad7-4d06-b015-f32ddccac750  rack1

什么可能导致这个问题？

cassandra
Martin Hope
Erick Ramirez
Asked: 2022-08-17 14:58:02 +0800 CST
  • 1

在某些情况下，操作员会在 Cassandra 日志中看到以下警告：

WARN ... Replica filtering protection has cached over 2000 rows during query \
  [SELECT ... FROM ... WHERE ...]. (See 'cached_replica_rows_warn_threshold' in cassandra.yaml.)

或者在最坏的情况下，一个错误：

ERROR ... Replica filtering protection has cached over 32000 rows during query \
  [SELECT ... FROM ... WHERE ...]. (See 'cached_replica_rows_fail_threshold' in cassandra.yaml.)

这些日志条目是什么意思？

cassandra