adinas's questions

Acho que as lápides não estão sendo excluídas em uma tabela do Cassandra 4.1.7.

gc_grace_seconds da tabela = 345600

Mesmo depois de executar nodetool garbagecollect(muito mais de 4 dias após a exclusão), se eu executar, nodetool tablestatsrecebo o seguinte:

O tamanho no disco também não diminui após executar o garbage collect.

Não tenho certeza se já corri no passado nodetool compact(o que acho que significa que preciso continuar usando?)

Pergunta paralela: como posso fazer com que o tablestats retorne informações mais recentes (em vez de 2024-12-12)

Obrigado

Temos um cluster de 2 nós (sei que 3 é o mínimo recomendado) que executou o Cassandra 3 e depois o 4 por anos sem incidentes.

Na semana passada, atualizamos ambos os nós para o Cassandra 5.

Desde então, os nós têm tido o que parecem ser problemas de fofoca e comunicação. Especialmente quando o esquema é atualizado (como ao criar ou excluir uma tabela).

Aqui está o log de depuração logo após fazer uma alteração de esquema (excluir uma tabela):

INFO  [Native-Transport-Requests-1] 2024-11-03 09:48:04,021 Keyspace.java:379 - Creating replication strategy r_internal params KeyspaceParams{durable_writes=true, replication=ReplicationParams{class=org.apache.cassandra.locator.SimpleStrategy, replication_factor=2}}
DEBUG [Native-Transport-Requests-1] 2024-11-03 09:48:04,064 MigrationCoordinator.java:430 - Reported schema a47919aa-79b3-362c-b023-d8ac89671885 at endpoint /10.10.2.27:7000
DEBUG [MigrationStage:1] 2024-11-03 09:48:04,070 MigrationCoordinator.java:749 - Pushing schema mutations: [Mutation(keyspace='system_schema', key='72617976656e5f696e7465726e616c', modifications=[
  [system_schema.columns] key=r_internal partition_deletion=deletedAt=-9223372036854775808, localDeletion=9223372036854775807 columns=[[] | []]
    Row[info=[ts=-9223372036854775808] del=deletedAt=1730627238890000, localDeletion=1730627238 ]: table_name=p1489_log_node_input_output, column_name=cdate |
    Row[info=[ts=-9223372036854775808] del=deletedAt=1730627238890000, localDeletion=1730627238 ]: table_name=p1489_log_node_input_output, column_name=device_id |
    Row[info=[ts=-9223372036854775808] del=deletedAt=1730627238890000, localDeletion=1730627238 ]: table_name=p1489_log_node_input_output, column_name=event_timestamp |
    Row[info=[ts=-9223372036854775808] del=deletedAt=1730627238890000, localDeletion=1730627238 ]: table_name=p1489_log_node_input_output, column_name=info |
    Row[info=[ts=-9223372036854775808] del=deletedAt=1730627238890000, localDeletion=1730627238 ]: table_name=p1489_log_node_input_output, column_name=input |
    Row[info=[ts=-9223372036854775808] del=deletedAt=1730627238890000, localDeletion=1730627238 ]: table_name=p1489_log_node_input_output, column_name=node_id |
    Row[info=[ts=-9223372036854775808] del=deletedAt=1730627238890000, localDeletion=1730627238 ]: table_name=p1489_log_node_input_output, column_name=output |
    Row[info=[ts=-9223372036854775808] del=deletedAt=1730627238890000, localDeletion=1730627238 ]: table_name=p1489_log_node_input_output, column_name=server |
    Row[info=[ts=-9223372036854775808] del=deletedAt=1730627238890000, localDeletion=1730627238 ]: table_name=p1489_log_node_input_output, column_name=unique_flow_process |
  [system_schema.indexes] key=r_internal partition_deletion=deletedAt=-9223372036854775808, localDeletion=9223372036854775807 columns=[[] | []]
    Row[info=[ts=-9223372036854775808] del=deletedAt=1730627238890000, localDeletion=1730627238 ]: table_name=p1489_log_node_input_output, index_name=p1489_log_node_input_output_unique_flow_process_idx |
  [system_schema.keyspaces] key=r_internal partition_deletion=deletedAt=-9223372036854775808, localDeletion=9223372036854775807 columns=[[] | [durable_writes replication]]
    Row[info=[ts=1730627238890000] ]: EMPTY | [durable_writes=true ts=1730627238890000], [replication=0000000200000005636c6173730000002b6f72672e6170616368652e63617373616e6472612e6c6f6361746f722e53696d706c655374726174656779000000127265706c69636174696f6e5f666163746f720000000132 ts=1730627238890000]
  [system_schema.tables] key=r_internal partition_deletion=deletedAt=-9223372036854775808, localDeletion=9223372036854775807 columns=[[] | []]
    Row[info=[ts=-9223372036854775808] del=deletedAt=1730627238890000, localDeletion=1730627238 ]: table_name=p1489_log_node_input_output |
])]
DEBUG [MigrationStage:1] 2024-11-03 09:48:04,071 MigrationCoordinator.java:757 - Pushing schema mutations to /10.10.2.28:7000: [Mutation(keyspace='system_schema', key='72617976656e5f696e7465726e616c', modifications=[
  [system_schema.columns] key=r_internal partition_deletion=deletedAt=-9223372036854775808, localDeletion=9223372036854775807 columns=[[] | []]
    Row[info=[ts=-9223372036854775808] del=deletedAt=1730627238890000, localDeletion=1730627238 ]: table_name=p1489_log_node_input_output, column_name=cdate |
    Row[info=[ts=-9223372036854775808] del=deletedAt=1730627238890000, localDeletion=1730627238 ]: table_name=p1489_log_node_input_output, column_name=device_id |
    Row[info=[ts=-9223372036854775808] del=deletedAt=1730627238890000, localDeletion=1730627238 ]: table_name=p1489_log_node_input_output, column_name=event_timestamp |
    Row[info=[ts=-9223372036854775808] del=deletedAt=1730627238890000, localDeletion=1730627238 ]: table_name=p1489_log_node_input_output, column_name=info |
    Row[info=[ts=-9223372036854775808] del=deletedAt=1730627238890000, localDeletion=1730627238 ]: table_name=p1489_log_node_input_output, column_name=input |
    Row[info=[ts=-9223372036854775808] del=deletedAt=1730627238890000, localDeletion=1730627238 ]: table_name=p1489_log_node_input_output, column_name=node_id |
    Row[info=[ts=-9223372036854775808] del=deletedAt=1730627238890000, localDeletion=1730627238 ]: table_name=p1489_log_node_input_output, column_name=output |
    Row[info=[ts=-9223372036854775808] del=deletedAt=1730627238890000, localDeletion=1730627238 ]: table_name=p1489_log_node_input_output, column_name=server |
    Row[info=[ts=-9223372036854775808] del=deletedAt=1730627238890000, localDeletion=1730627238 ]: table_name=p1489_log_node_input_output, column_name=unique_flow_process |
  [system_schema.indexes] key=r_internal partition_deletion=deletedAt=-9223372036854775808, localDeletion=9223372036854775807 columns=[[] | []]
    Row[info=[ts=-9223372036854775808] del=deletedAt=1730627238890000, localDeletion=1730627238 ]: table_name=p1489_log_node_input_output, index_name=p1489_log_node_input_output_unique_flow_process_idx |
  [system_schema.keyspaces] key=r_internal partition_deletion=deletedAt=-9223372036854775808, localDeletion=9223372036854775807 columns=[[] | [durable_writes replication]]
    Row[info=[ts=1730627238890000] ]: EMPTY | [durable_writes=true ts=1730627238890000], [replication=0000000200000005636c6173730000002b6f72672e6170616368652e63617373616e6472612e6c6f6361746f722e53696d706c655374726174656779000000127265706c69636174696f6e5f666163746f720000000132 ts=1730627238890000]
  [system_schema.tables] key=r_internal partition_deletion=deletedAt=-9223372036854775808, localDeletion=9223372036854775807 columns=[[] | []]
    Row[info=[ts=-9223372036854775808] del=deletedAt=1730627238890000, localDeletion=1730627238 ]: table_name=p1489_log_node_input_output |
])]
ERROR [GossipStage:1] 2024-11-03 09:48:04,187 JVMStabilityInspector.java:70 - Exception in thread Thread[GossipStage:1,5,GossipStage]
java.lang.RuntimeException: java.lang.AssertionError
        at org.apache.cassandra.net.InboundSink.accept(InboundSink.java:108)
        at org.apache.cassandra.net.InboundSink.accept(InboundSink.java:45)
        at org.apache.cassandra.net.InboundMessageHandler$ProcessMessage.run(InboundMessageHandler.java:430)
        at org.apache.cassandra.concurrent.ExecutionFailure$1.run(ExecutionFailure.java:133)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.AssertionError: null
        at org.apache.cassandra.db.TypeSizes.sizeof(TypeSizes.java:44)
        at org.apache.cassandra.gms.VersionedValue$VersionedValueSerializer.serializedSize(VersionedValue.java:365)
        at org.apache.cassandra.gms.VersionedValue$VersionedValueSerializer.serializedSize(VersionedValue.java:343)
        at org.apache.cassandra.gms.EndpointStateSerializer.serializedSize(EndpointState.java:344)
        at org.apache.cassandra.gms.EndpointStateSerializer.serializedSize(EndpointState.java:300)
        at org.apache.cassandra.gms.GossipDigestAckSerializer.serializedSize(GossipDigestAck.java:96)
        at org.apache.cassandra.gms.GossipDigestAckSerializer.serializedSize(GossipDigestAck.java:61)
        at org.apache.cassandra.net.Message$Serializer.payloadSize(Message.java:1088)
        at org.apache.cassandra.net.Message.payloadSize(Message.java:1131)
        at org.apache.cassandra.net.Message$Serializer.serializedSize(Message.java:769)
        at org.apache.cassandra.net.Message.serializedSize(Message.java:1111)
        at org.apache.cassandra.net.OutboundConnections.connectionTypeFor(OutboundConnections.java:215)
        at org.apache.cassandra.net.OutboundConnections.connectionFor(OutboundConnections.java:207)
        at org.apache.cassandra.net.OutboundConnections.enqueue(OutboundConnections.java:96)
        at org.apache.cassandra.net.MessagingService.doSend(MessagingService.java:473)
        at org.apache.cassandra.net.OutboundSink.accept(OutboundSink.java:70)
        at org.apache.cassandra.net.MessagingService.send(MessagingService.java:462)
        at org.apache.cassandra.net.MessagingService.send(MessagingService.java:437)
        at org.apache.cassandra.gms.GossipDigestSynVerbHandler.doVerb(GossipDigestSynVerbHandler.java:110)
        at org.apache.cassandra.net.InboundSink.lambda$new$0(InboundSink.java:78)
        at org.apache.cassandra.net.InboundSink.accept(InboundSink.java:97)
        ... 7 common frames omitted
DEBUG [GossipStage:1] 2024-11-03 09:48:04,187 HeapUtils.java:133 - Heap dump creation on uncaught exceptions is disabled.
DEBUG [GossipStage:1] 2024-11-03 09:48:04,187 HeapUtils.java:133 - Heap dump creation on uncaught exceptions is disabled.

A exceção Exception no thread Thread[GossipStage:1,5,GossipStage] continua se repetindo várias vezes.

Neste ponto, a execução do status do nodetool mostra que cada um pensa que o outro está inativo

Geralmente, preciso drenar e reiniciar ambos os nós (o que faço um após o outro) para que ambos se vejam e mostrem que seus esquemas estão sincronizados.

Aqui está a saída do nodetool gossipinfo (após reiniciar os nós)

/10.10.2.27
  generation:1730627829
  heartbeat:939
  STATUS:23:NORMAL,-1656170799351181049
  LOAD:898:2.029075780322E12
  SCHEMA:17:a47919aa-79b3-362c-b023-d8ac89671885
  DC:13:dc1
  RACK:15:rack1
  RELEASE_VERSION:6:5.0.2
  INTERNAL_IP:11:10.10.2.27
  RPC_ADDRESS:5:10.10.2.27
  NET_VERSION:2:12
  HOST_ID:3:8be5925c-0436-4bc7-bd4c-0fb5dd8fc9e5
  RPC_READY:35:true
  INTERNAL_ADDRESS_AND_PORT:9:10.10.2.27:7000
  NATIVE_ADDRESS_AND_PORT:4:10.10.2.27:9042
  STATUS_WITH_PORT:22:NORMAL,-1656170799351181049
  SSTABLE_VERSIONS:7:big-nb
  TOKENS:21:<hidden>
/10.10.2.28
  generation:1730628138
  heartbeat:619
  LOAD:586:2.215579324507E12
  SCHEMA:17:a47919aa-79b3-362c-b023-d8ac89671885
  DC:13:dc1
  RACK:15:rack1
  RELEASE_VERSION:6:5.0.2
  NET_VERSION:2:12
  HOST_ID:3:c9e8a4f8-c7b2-4cac-994a-e8de50c9f358
  RPC_READY:35:true
  INTERNAL_ADDRESS_AND_PORT:9:10.10.2.28:7000
  NATIVE_ADDRESS_AND_PORT:4:10.10.2.28:9042
  STATUS_WITH_PORT:22:NORMAL,-2024247660995288657
  SSTABLE_VERSIONS:7:big-nb
  TOKENS:21:<hidden>

E aqui do segundo nó

/10.10.2.27
  generation:1730627829
  heartbeat:906
  LOAD:898:2.029075780322E12
  SCHEMA:17:a47919aa-79b3-362c-b023-d8ac89671885
  DC:13:dc1
  RACK:15:rack1
  RELEASE_VERSION:6:5.0.2
  NET_VERSION:2:12
  HOST_ID:3:8be5925c-0436-4bc7-bd4c-0fb5dd8fc9e5
  RPC_READY:35:true
  INTERNAL_ADDRESS_AND_PORT:9:10.10.2.27:7000
  NATIVE_ADDRESS_AND_PORT:4:10.10.2.27:9042
  STATUS_WITH_PORT:22:NORMAL,-1656170799351181049
  SSTABLE_VERSIONS:7:big-nb
  TOKENS:21:<hidden>
/10.10.2.28
  generation:1730628138
  heartbeat:584
  STATUS:23:NORMAL,-2024247660995288657
  LOAD:524:2.215537773956E12
  SCHEMA:17:a47919aa-79b3-362c-b023-d8ac89671885
  DC:13:dc1
  RACK:15:rack1
  RELEASE_VERSION:6:5.0.2
  INTERNAL_IP:11:10.10.2.28
  RPC_ADDRESS:5:10.10.2.28
  NET_VERSION:2:12
  HOST_ID:3:c9e8a4f8-c7b2-4cac-994a-e8de50c9f358
  RPC_READY:35:true
  INTERNAL_ADDRESS_AND_PORT:9:10.10.2.28:7000
  NATIVE_ADDRESS_AND_PORT:4:10.10.2.28:9042
  STATUS_WITH_PORT:22:NORMAL,-2024247660995288657
  SSTABLE_VERSIONS:7:big-nb
  TOKENS:21:<hidden>

Quando eles estão tendo problemas, isso é o que o nodetool gossinfo mostra quando executado em 10.10.2.28:

/10.10.2.27
  generation:1730626480
  heartbeat:810
  LOAD:771:2.029304184276E12
  SCHEMA:17:9bcfb867-92ca-3144-92d7-da5fed3359d0
  DC:13:dc1
  RACK:15:rack1
  RELEASE_VERSION:6:5.0.2
  NET_VERSION:2:12
  HOST_ID:3:8be5925c-0436-4bc7-bd4c-0fb5dd8fc9e5
  RPC_READY:35:true
  INTERNAL_ADDRESS_AND_PORT:9:10.10.2.27:7000
  NATIVE_ADDRESS_AND_PORT:4:10.10.2.27:9042
  STATUS_WITH_PORT:22:NORMAL,-1656170799351181049
  SSTABLE_VERSIONS:7:big-nb
  TOKENS:21:<hidden>
/10.10.2.28
  generation:1730625184
  heartbeat:2523
  STATUS:23:NORMAL,-2024247660995288657
  LOAD:2513:2.214202366182E12
  SCHEMA:2243:a47919aa-79b3-362c-b023-d8ac89671885
  DC:13:dc1
  RACK:15:rack1
  RELEASE_VERSION:6:5.0.2
  INTERNAL_IP:11:10.10.2.28
  RPC_ADDRESS:5:10.10.2.28
  NET_VERSION:2:12
  HOST_ID:3:c9e8a4f8-c7b2-4cac-994a-e8de50c9f358
  RPC_READY:35:true
  INTERNAL_ADDRESS_AND_PORT:9:10.10.2.28:7000
  NATIVE_ADDRESS_AND_PORT:4:10.10.2.28:9042
  STATUS_WITH_PORT:22:NORMAL,-2024247660995288657
  SSTABLE_VERSIONS:7:big-nb
  INDEX_STATUS:2198:{"r_internal.p1643_log_node_input_output_unique_flow_process_idx":"BUILD_SUCCEEDED","r_internal.p1720_log_node_input_output_unique_flow_process_idx":"BUILD_SUCCEEDED","r_internal.p1479_log_node_input_output_unique_flow_process_idx":"BUILD_SUCCEEDED"...}
  TOKENS:21:<hidden>

Instalei o Cassandra na minha máquina Linux (Ubutnu).

Eu tinha o OpenJDK 11 instalado, depois instalei o JDK 17, configurei-o para o padrão e reiniciei o Cassandra.

Posso ver que a nova versão está sendo usada em /var/log/cassandra/system.log

Tudo está funcionando, mas não está claro para mim se usar 17 é considerado "experimental" ou não, ou se é preferível a 11?

Resumo: tenho um cluster de 2 nós e se um dos nós estiver inativo, não consigo fazer login. Eu recebo o erro.

Connection error: ('Unable to connect to any servers', {'192.168.1.104:9042': AuthenticationFailed('Failed to authenticate to 192.168.1.104:9042: Error from server: code=0100 [Bad credentials] message="Unable to perform authentication: Cannot achieve consistency level LOCAL_QUORUM"')})

• Estou usando um ROLE que criei (não um SUEPRUSER, então o problema de SUPERUSER sempre usar QUORUM não deve se aplicar aqui. Pelo que li deveria usar LOCAL_ONE mas não é!
• Eu configurei a replicação do keyspace system_auth para 2 propositalmente, pois isso é recomendado para não ter um único ponto de falha.

Aqui está uma captura de tela para mostrar que não estou usando o cassandra SUPERUSER e ainda estou recebendo o erro:

Gostaria que o login continuasse funcionando mesmo se um nó estivesse inativo.