产品环境中安装Cassandra 3.7

Chapter 1. Environment Prepare:

0. Environment description:

hardware: 7 commercial machine with 24 processors, 128 megabytes memory, 103G system disk + 3.6T*10 data disk
os: Red Hat 4.4.7-16 
ip: 192.168.1.11~17
cassandra: datastax-ddc-3.7.0
user: cassandra with sudo previlege.

1. Install JDK, recommend oracle JDK 8.

sudo  yum -y  install  jdk-8u101-linux-x64.rpm

I used a rpm package installment, it can save you time from configuration java_home and so on.

if you use openjdk, you will get a warn message as follows:

OpenJDK is not recommended. Please upgrade to the newest Oracle Java release

 

2.  Install jemalloc

sudo  yum -y  install  jemalloc.x86_64

if you miss the software, you will get a warn message as follows:

jemalloc shared library could not be preloaded to speed up memory allocations

3. Disable swap files and optimize system configuration

details, please see: https://docs.datastax.com/en/landing_page/doc/landing_page/recommendedSettingsLinux.html

Included in the file /etc/security/limits.conf:
# set for cassandra limits
coupang - memlock unlimited
coupang - nofile 100000
coupang - nproc 32768
coupang - as unlimited


For all installations, add the following line to /etc/sysctl.conf:
vm.max_map_count = 1048575


Disable swap files in the operations system
   sudo swapoff --all
To make this change permanent, remove all swap file entries from /etc/fstab.


Check the Java Hugepages setting
 echo never | sudo tee /sys/kernel/mm/transparent_hugepage/defrag

if not do this, then you will get  warn messages as follows:

Unable to lock JVM memory (ENOMEM). This can result in part of the JVM being swapped out, especially with mmapped I/O enabled. Increase RLIMIT_MEMLOCK or run Cassandra as root.

Cassandra server running in degraded mode. Is swap disabled? : false,  Address space adequate? : true,  nofile limit adequate? : true, nproc limit adequate? : true

4. Update OHC

update ohc from 0.4.3 to 0.4.5, including jars: 

# http://search.maven.org, search by ohc and download these, move them to $CASSANDRA_HOME/lib to replace ohc-core-0.4.3.jar & ohc-core-j8-0.4.3.jar
ohc-core-0.4.5.jar
ohc-core-j8-0.4.5.jar

 if not, you will get a warn message as follows:

Failed to load Java8 implementation ohc-core-j8 : java.lang.NoSuchMethodException: org.caffinitas.ohc.linked.UnsExt8.(java.lang.Class)

 

Chapter 2. Cassandra Configuration files

cassandra.yaml

this is the primary configuration for cassandra.

 for some confidential reason, this part can't be shared.

jvm.options

configuration for java virtual machine. the machine has 24 processors and 128 M memory, so we use 32G heap and  15 ParallelGCThreads.

###########################################################################
#                             jvm.options                                 #
#                                                                         #
# - all flags defined here will be used by cassandra to startup the JVM   #
# - one flag should be specified per line                                 #
# - lines that do not start with '-' will be ignored                      #
# - only static flags are accepted (no variables or parameters)           #
# - dynamic flags will be appended to these on cassandra-env              #
###########################################################################

######################
# STARTUP PARAMETERS #
######################

# Uncomment any of the following properties to enable specific startup parameters

# In a multi-instance deployment, multiple Cassandra instances will independently assume that all
# CPU processors are available to it. This setting allows you to specify a smaller set of processors
# and perhaps have affinity.
#-Dcassandra.available_processors=number_of_processors

# The directory location of the cassandra.yaml file.
#-Dcassandra.config=directory

# Sets the initial partitioner token for a node the first time the node is started.
#-Dcassandra.initial_token=token

# Set to false to start Cassandra on a node but not have the node join the cluster.
#-Dcassandra.join_ring=true|false

# Set to false to clear all gossip state for the node on restart. Use when you have changed node
# information in cassandra.yaml (such as listen_address).
#-Dcassandra.load_ring_state=true|false

# Enable pluggable metrics reporter. See Pluggable metrics reporting in Cassandra 2.0.2.
#-Dcassandra.metricsReporterConfigFile=file

# Set the port on which the CQL native transport listens for clients. (Default: 9042)
#-Dcassandra.native_transport_port=port

# Overrides the partitioner. (Default: org.apache.cassandra.dht.Murmur3Partitioner)
#-Dcassandra.partitioner=partitioner

# To replace a node that has died, restart a new node in its place specifying the address of the
# dead node. The new node must not have any data in its data directory, that is, it must be in the
# same state as before bootstrapping.
#-Dcassandra.replace_address=listen_address or broadcast_address of dead node

# Allow restoring specific tables from an archived commit log.
#-Dcassandra.replayList=table

# Allows overriding of the default RING_DELAY (1000ms), which is the amount of time a node waits
# before joining the ring.
#-Dcassandra.ring_delay_ms=ms

# Set the port for the Thrift RPC service, which is used for client connections. (Default: 9160)
#-Dcassandra.rpc_port=port

# Set the SSL port for encrypted communication. (Default: 7001)
#-Dcassandra.ssl_storage_port=port

# Enable or disable the native transport server. See start_native_transport in cassandra.yaml.
# cassandra.start_native_transport=true|false

# Enable or disable the Thrift RPC server. (Default: true)
#-Dcassandra.start_rpc=true/false

# Set the port for inter-node communication. (Default: 7000)
#-Dcassandra.storage_port=port

# Set the default location for the trigger JARs. (Default: conf/triggers)
#-Dcassandra.triggers_dir=directory

# For testing new compaction and compression strategies. It allows you to experiment with different
# strategies and benchmark write performance differences without affecting the production workload.
#-Dcassandra.write_survey=true

# To disable configuration via JMX of auth caches (such as those for credentials, permissions and
# roles). This will mean those config options can only be set (persistently) in cassandra.yaml
# and will require a restart for new values to take effect.
#-Dcassandra.disable_auth_caches_remote_configuration=true

########################
# GENERAL JVM SETTINGS #
########################

# enable assertions.  disabling this in production will give a modest
# performance benefit (around 5%).
-ea

# enable thread priorities, primarily so we can give periodic tasks
# a lower priority to avoid interfering with client workload
-XX:+UseThreadPriorities

# allows lowering thread priority without being root on linux - probably
# not necessary on Windows but doesn't harm anything.
# see http://tech.stolsvik.com/2010/01/linux-java-thread-priorities-workar
-XX:ThreadPriorityPolicy=42

# Enable heap-dump if there's an OOM
-XX:+HeapDumpOnOutOfMemoryError

# Per-thread stack size.
-Xss256k

# Larger interned string table, for gossip's benefit (CASSANDRA-6410)
-XX:StringTableSize=1000003

# Make sure all memory is faulted and zeroed on startup.
# This helps prevent soft faults in containers and makes
# transparent hugepage allocation more effective.
-XX:+AlwaysPreTouch

# Disable biased locking as it does not benefit Cassandra.
-XX:-UseBiasedLocking

# Enable thread-local allocation blocks and allow the JVM to automatically
# resize them at runtime.
-XX:+UseTLAB
-XX:+ResizeTLAB

# http://www.evanjones.ca/jvm-mmap-pause.html
-XX:+PerfDisableSharedMem

# Prefer binding to IPv4 network intefaces (when net.ipv6.bindv6only=1). See
# http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6342561 (short version:
# comment out this entry to enable IPv6 support).
-Djava.net.preferIPv4Stack=true

### Debug options

# uncomment to enable flight recorder
#-XX:+UnlockCommercialFeatures
#-XX:+FlightRecorder

# uncomment to have Cassandra JVM listen for remote debuggers/profilers on port 1414
#-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=1414

# uncomment to have Cassandra JVM log internal method compilation (developers only)
#-XX:+UnlockDiagnosticVMOptions
#-XX:+LogCompilation

#################
# HEAP SETTINGS #
#################

# Heap size is automatically calculated by cassandra-env based on this
# formula: max(min(1/2 ram, 1024MB), min(1/4 ram, 8GB))
# That is:
# - calculate 1/2 ram and cap to 1024MB
# - calculate 1/4 ram and cap to 8192MB
# - pick the max
#
# For production use you may wish to adjust this for your environment.
# If that's the case, uncomment the -Xmx and Xms options below to override the
# automatic calculation of JVM heap memory.
#
# It is recommended to set min (-Xms) and max (-Xmx) heap sizes to
# the same value to avoid stop-the-world GC pauses during resize, and
# so that we can lock the heap in memory on startup to prevent any
# of it from being swapped out.
-Xms32G
-Xmx32G

# Young generation size is automatically calculated by cassandra-env
# based on this formula: min(100 * num_cores, 1/4 * heap size)
#
# The main trade-off for the young generation is that the larger it
# is, the longer GC pause times will be. The shorter it is, the more
# expensive GC will be (usually).
#
# It is not recommended to set the young generation size if using the
# G1 GC, since that will override the target pause-time goal.
# More info: http://www.oracle.com/technetwork/articles/java/g1gc-1984535.html
#
# The example below assumes a modern 8-core+ machine for decent
# times. If in doubt, and if you do not particularly want to tweak, go
# 100 MB per physical CPU core.
#-Xmn800M

#################
#  GC SETTINGS  #
#################

### CMS Settings

#-XX:+UseParNewGC
#-XX:+UseConcMarkSweepGC
#-XX:+CMSParallelRemarkEnabled
#-XX:SurvivorRatio=8
#-XX:MaxTenuringThreshold=1
#-XX:CMSInitiatingOccupancyFraction=75
#-XX:+UseCMSInitiatingOccupancyOnly
#-XX:CMSWaitDuration=10000
#-XX:+CMSParallelInitialMarkEnabled
#-XX:+CMSEdenChunksRecordAlways
## some JVMs will fill up their heap when accessed via JMX, see CASSANDRA-6541
#-XX:+CMSClassUnloadingEnabled

### G1 Settings (experimental, comment previous section and uncomment section below to enable)

# Use the Hotspot garbage-first collector.
-XX:+UseG1GC

# Have the JVM do less remembered set work during STW, instead
# preferring concurrent GC. Reduces p99.9 latency.
-XX:G1RSetUpdatingPauseTimePercent=5

# Main G1GC tunable: lowering the pause target will lower throughput and vise versa.
## 200ms is the JVM default and lowest viable setting
## 1000ms increases throughput. Keep it smaller than the timeouts in cassandra.yaml.
-XX:MaxGCPauseMillis=350

## Optional G1 Settings

# Save CPU time on large (>= 16GB) heaps by delaying region scanning
# until the heap is 70% full. The default in Hotspot 8u40 is 40%.
-XX:InitiatingHeapOccupancyPercent=70

# For systems with > 8 cores, the default ParallelGCThreads is 5/8 the number of logical cores.
# Otherwise equal to the number of cores when 8 or less.
# Machines with > 10 cores should try setting these to <= full cores.
-XX:ParallelGCThreads=15
# By default, ConcGCThreads is 1/4 of ParallelGCThreads.
# Setting both to the same value can reduce STW durations.
-XX:ConcGCThreads=4

### GC logging options -- uncomment to enable

-XX:+PrintGCDetails
-XX:+PrintGCDateStamps
-XX:+PrintHeapAtGC
-XX:+PrintTenuringDistribution
-XX:+PrintGCApplicationStoppedTime
-XX:+PrintPromotionFailure
#-XX:PrintFLSStatistics=1
#-Xloggc:/var/log/cassandra/gc.log
-XX:+UseGCLogFileRotation
-XX:NumberOfGCLogFiles=10
-XX:GCLogFileSize=10M

 

cassandra-env.sh

configruation for JMX, open remote access with basic authentication.

please see: https://wiki.apache.org/cassandra/JmxSecurity

# set jmx ip
JVM_OPTS="$JVM_OPTS -Djava.rmi.server.hostname=192.168.1.11"

# disable local jmx
if [ "x$LOCAL_JMX" = "x" ]; then
    LOCAL_JMX=no
fi

## Basic file based authn & authz
JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.password.file=/etc/cassandra/jmxremote.password"
JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.access.file=/etc/cassandra/jmxremote.access"

 

/etc/cassandra/jmxremote.password

controlRole yourPassword

/etc/cassandra/jmxremote.access

controlRole readwrite \
create javax.management.monitor.*,javax.management.timer.* \
unregister

Chapter 3. Add Cassandra to service and auto-start

sudo  vim  /etc/init .d /cassandra
sudo  chmod  +x  /etc/init .d /cassandra
sudo  chkconfig --add cassandra
service cassandra start

 

/etc/init.d/cassandra 内容如下：

#!/bin/bash
# chkconfig: 2345 99 01
# description: Cassandra

.  /etc/rc .d /init .d /functions

CASSANDRA_HOME= /usr/cassandra3/datastax-ddc-3 .7.0
CASSANDRA_BIN=$CASSANDRA_HOME /bin/cassandra
CASSANDRA_NODETOOL=$CASSANDRA_HOME /bin/nodetool
CASSANDRA_LOG=$CASSANDRA_HOME /logs/cassandra .log
CASSANDRA_RUN=$CASSANDRA_HOME /run
CASSANDRA_PID=$CASSANDRA_RUN /cassandra .pid
CASSANDRA_LOCK=$CASSANDRA_RUN /cassandra .lock
JMX_AUTH= " -u controlRole -pw yourPassword "
PROGRAM= "cassandra"


# directory not exist, create it
if  [ ! -d $CASSANDRA_RUN ];  then
  echo  "Create directory: $CASSANDRA_RUN"
  mkdir  $CASSANDRA_RUN
fi

if  [ ! -f $CASSANDRA_BIN ];  then
  echo  "File not found: $CASSANDRA_BIN"
  exit  1
fi

RETVAL=0

start() {

  if  [ -f $CASSANDRA_PID ] && checkpid ` cat  $CASSANDRA_PID`;  then
    echo  "Cassandra is already running."
    exit  0
  fi

  echo  -n $ "Starting $PROGRAM ... "
  daemon $CASSANDRA_BIN -p $CASSANDRA_PID >> $CASSANDRA_LOG 2>&1
  usleep 500000
  RETVAL=$?
  if  [ $RETVAL - eq  0 ];  then
    touch  $CASSANDRA_LOCK
    echo_success
  else
    echo_failure
  fi

  echo
  return  $RETVAL
}

stop() {

  if  [ ! -f $CASSANDRA_PID ];  then
    echo  "Cassandra is already stopped."
    exit  0
  fi

  # disable gossip
  echo  $ "disable cassandra gossip ..."
  $CASSANDRA_NODETOOL -h 127.0.0.1 $JMX_AUTH disablegossip

  # disable thrift
  echo  $ "disable cassandra thrift ..."
  $CASSANDRA_NODETOOL -h 127.0.0.1 $JMX_AUTH disablethrift

  # write mmtable to sstable
  echo  $ "write mmtabe to sstabe ... (maybe the process cost some minutes, please wait it.)"
  $CASSANDRA_NODETOOL -h 127.0.0.1 $JMX_AUTH drain

  # kill cassandra
  echo  $ "Stopping $PROGRAM ... "
  if  kill  ` cat  $CASSANDRA_PID`;  then
    RETVAL=0
    rm  -f $CASSANDRA_LOCK
    echo_success
  else
    RETVAL=1
    echo_failure
  fi
  echo
  [ $RETVAL = 0 ]
}


status_fn() {
  if  [ -f $CASSANDRA_PID ] && checkpid ` cat  $CASSANDRA_PID`;  then
    echo  "Cassandra is running."
    exit  0
  else
    echo  "Cassandra is stopped."
    exit  1
  fi
}

case  "$1"  in

  start)
    start
    ;;

  stop)
    stop
    ;;

  status)
    status_fn
    ;;

  restart)
    stop
    start
    ;;
  *)
    echo  $ "Usage: $PROGRAM {start|stop|restart|status}"
    RETVAL=3
esac

exit  $RETVAL