Momentum 4 Maintenance
Table of Contents
- Directory Structure
- Log Rotations
- Database Maintenance
- Checking for Normal Momentum 4 Operation
- Alternative Vertica Admin Control
|/opt/msys/||Message Systems software|
|/opt/vertica/||HP Vertica Software|
|/var/db/||Database data (Cassandra, Postgres, Riak, and Vertica)|
|/var/ecconfigd/||Message Systems cluster repo|
|/var/eccluster/||Message Systems cluster data|
|/var/run/||Dynamic runtime data for running msys processes|
For a list of log files by process, see Log File Names and Locations. For more information on log files, see Momentum 4 Reference Manual - Logging Overview. Momentum provides the ec_rotate utility script you can use to rotate and compress logs that Momentum writes. We recommend you run this script daily from your system's crontab. For more information on ec_rotate, see Momentum 4 Reference Manual - Rotating Logs.
For a list of the log files that the ec_rotate utility script rotates, see Momentum 4 Reference Manual - ec_rotate.
Certain log files, such as those listed below, are not included in ec_rotate and do not need to be rotated daily. However, we do recommend that they are rotated periodically.
- /var/log/eccluster/* on manager/logaggregator
Set up rotations for these log files using the tool of your choice.For more information on rotating Vertica log files, see HP Vertica's documentation.
If logs are enabled, they should be monitored to ensure they are being properly consumed:
The tables in the mo_keyspace_default keyspace use the default compaction strategy SizeTieredCompactionStrategy. This means that by default four sstables of similar size are compacted into a new sstable. This pattern continues where compactions trigger on larger and larger sets of four tables. This strategy helps keep cpu/io usage down, but does not work well for tables that have many updates, deletes, or insertions with a TTL (this is because much of the mo_keyspace_default keyspace data is inserted with a small TTL). The two issues that can result are:
- Queries failing because of too many tombstones
- High disk space usage in /var/db/cassandra/data because the SizeTieredCompactionStrategy does not trigger often enough to evict tombstones
1. Verify the Cassandra cluster is entirely up and all nodes show a consistent view:
nodetool status2. Log onto the first Cassandra node, and flush the memtables for the mo_keyspace_default keyspace to sstables:
nodetool flush mo_keyspace_default3. Compact the sstables in the mo_keyspace_default keyspace:
nodetool compact mo_keyspace_default4. Monitor the status of compactions, till completion:
nodetool compactionstats5. Repeat steps 1 through 4 for the remaining nodes in the Cassandra cluster. If there is a system event or an abnormal operation, use the following commands to confirm that the required Momentum system processes are up and running correctly. Expected outcomes are listed in Expected Output for Normal Momentum 4 Operation.
ps -ef | grep webhooks; echo; echo ps -ef | grep ecelerity; echo; echo ps -ef | grep cassandra; echo; echo ps -ef | grep metrics-etl; echo; echo ps -ef | grep rabbitmq; echo; echo ps -ef | grep nginx; echo; echo ps -ef | grep vertica; echo; echo ps -ef | grep ecconfigd; echo; echo ps -ef | grep riak; echo; echoTo confirm that Cassandra is running on one node:
/opt/msys/3rdParty/cassandra/bin/nodetool infoTo confirm the status of the Cassandra cluster from one node:
/opt/msys/3rdParty/cassandra/bin/nodetool statusThere are occasions where the HP Vertica distributed database is in an uncooperative state and not responding to /etc/init.d level start and stop scripts. Although HP Vertica administration is beyond the scope of this document, there is a console command for checking the status of the database on all nodes and starting and stopping the database on all nodes. The Vertica Management Console may be installed separately and provides a GUI for viewing clusters. It can live independent of the Vertica nodes on its own hardware.
Additional information on HP Vertica Analytics Platform can be found in the HP Vertica Analytics Platform documentation.
If you have a question on system operation, please consult Message Systems Support.
1. To access the HP Vertica admin tool, run the following command:
su vertica_dba -c /opt/vertica/bin/admintoolsYou will see the following menu options:
Useful options are:
- 1. View Database Cluster State - displays the status of Vertica on each node. All nodes should be "Up".
- 4. Stop Database - stops the distributed database on all nodes. Use this to cleanly stop Vertica; for example, when one or more nodes are down or are having issues.
- 3. Start Database - starts the distributed database on all nodes. Use this to cleanly start Vertica. Run this command after running "Stop Database." This will give you a clean stop and restart of the database. You only need to run this command on one node in order to start the database on all nodes.
ps -ef | grep verticaYou should see the following output if Vertica is running correctly.
501 21792 1 0 16:20 ? 00:00:00 /opt/vertica/spread/sbin/spread -c /var/db/vertica/msys/v_msys_node0001_catalog/spread.conf 501 21794 1 10 16:20 ? 00:00:15 /opt/vertica/bin/vertica -D /var/db/vertica/msys/v_msys_node0001_catalog -C msys -n v_msys_node0001 -h 192.168.18.52 -p 5433 -P 4803 -Y ipv4 501 21801 21794 0 16:20 ? 00:00:00 /opt/vertica/bin/vertica-udx-zygote 12 10 21794 debug-log-off /var/db/vertica/msys/v_msys_node0001_catalog/UDxLogs root 22152 21525 0 16:22 pts/0 00:00:00 grep verticaIf you see something different or nothing at all, Vertica is either down or is stuck in an initialization state. If stuck in initialization state, use the admintool to "Stop Database" and then "Start Database."