Merge pull request #39199 from deimosfr/cassandra_exemple_fix_shutdown

Automatic merge from submit-queue (batch tested with PRs 39199, 37273, 29183, 39638, 40199)

fixing Cassandra shutdown example to avoid data corruption

Hi,

I was playing with Cassandra example stored in the Kubernetes project and I encountered issues on shutdown (not anytime). After checking it looks like the shutdown of a node is brutal and data corruption may occur during a flush on disk. To avoid that, I'm suggesting a hook to gracefully shutdown Cassandra before stopping the container.

Here are logs of corruption after a pod delete:
```
/10.2.76.4:[-8699848499000118463, -8567123670484406873, -8496767951391579058, -8426990834929543369, -7697118318683556771, -6942779781591907873, -6795880495022459877, -6496399078175245235, -5450122121479522544, -5002551029990001224, -4914532712178218138, -4884518674849288097, -3667338763252443465, -3316742521554936832, -2844544359955291760, -1291351295404368159, -794348397160283083, -705240847455001090, -652995206518489298, -284127251294286231, 173240967232234690, 616476682204879844, 826670457841382100, 1815369334084765465, 4431706613761077084, 4743606016174161647, 5637469692783959686, 5802957011124852712, 6759688243703331970, 7679657413128857702, 7713766696628426028, 9098158217036036188]

ERROR 16:23:06 Exception in thread Thread[CompactionExecutor:2,1,main]
org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted: /cassandra_data/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/mc-2-big-Data.db
	at org.apache.cassandra.io.sstable.format.big.BigTableScanner$KeyScanningIterator.computeNext(BigTableScanner.java:351) ~[apache-cassandra-3.9.jar:3.9]
	at org.apache.cassandra.io.sstable.format.big.BigTableScanner$KeyScanningIterator.computeNext(BigTableScanner.java:265) ~[apache-cassandra-3.9.jar:3.9]
	at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.9.jar:3.9]
	at org.apache.cassandra.io.sstable.format.big.BigTableScanner.hasNext(BigTableScanner.java:245) ~[apache-cassandra-3.9.jar:3.9]
	at org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:374) ~[apache-cassandra-3.9.jar:3.9]
	at org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:186) ~[apache-cassandra-3.9.jar:3.9]
	at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:155) ~[apache-cassandra-3.9.jar:3.9]
	at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.9.jar:3.9]
	at org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$2.hasNext(UnfilteredPartitionIterators.java:150) ~[apache-cassandra-3.9.jar:3.9]
	at org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:92) ~[apache-cassandra-3.9.jar:3.9]
	at org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:232) ~[apache-cassandra-3.9.jar:3.9]
	at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:184) ~[apache-cassandra-3.9.jar:3.9]
	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-3.9.jar:3.9]
	at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:82) ~[apache-cassandra-3.9.jar:3.9]
	at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60) ~[apache-cassandra-3.9.jar:3.9]
	at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:264) ~[apache-cassandra-3.9.jar:3.9]

```

It works well for me now and do not have data corruption anymore.
This commit is contained in:
Kubernetes Submit Queue 2017-01-23 00:30:11 -08:00 committed by GitHub
commit c08758e95d

View File

@ -34,6 +34,10 @@ spec:
capabilities:
add:
- IPC_LOCK
lifecycle:
preStop:
exec:
command: ["/bin/sh", "-c", "PID=$(pidof java) && kill $PID && while ps -p $PID > /dev/null; do sleep 1; done"]
env:
- name: MAX_HEAP_SIZE
value: 512M