Welcome, Guest Login

Support Center

Momentum 4 Maintenance

Last Updated: Feb 05, 2016 11:53AM EST

Operating Momentum
Momentum 4 Maintenance

Table of Contents

Directory Structure

At a summary level, the software to execute Momentum 4 is in /opt/, and the datafiles are in /var/. A list of directories and a brief description of each is shown below. For more information about the directories, see Directories and Mounts.
Directory Description
/opt/msys/ Message Systems software
/opt/vertica/ HP Vertica Software
/var/ Data files
/var/db/ Database data (Cassandra, Postgres, Riak, and Vertica)
/var/ecconfigd/ Message Systems cluster repo
/var/eccluster/ Message Systems cluster data
/var/log/ Log files
/var/run/ Dynamic runtime data for running msys processes
/var/spool/ecelerity/ Mail spool

For a list of log files by process, see Log File Names and Locations. For more information on log files, see Momentum 4 Reference Manual - Logging Overview.

Log Rotations

Momentum provides the ec_rotate utility script you can use to rotate and compress logs that Momentum writes. We recommend you run this script daily from your system's crontab.  For more information on ec_rotate, see Momentum 4 Reference Manual - Rotating Logs

For a list of the log files that the ec_rotate utility script rotates, see Momentum 4 Reference Manual - ec_rotate.
Certain log files, such as those listed below, are not included in ec_rotate and do not need to be rotated daily. However, we do recommend that they are rotated periodically.
  • /var/log/msys-nodejs/*.log
  • /var/log/msys-nginx/*.log
  • /var/log/msys-rabbitmq/*
  • /opt/vertica/log/*.log
  • /opt/vertica/log/**/*.log
  • /var/db/vertica/msys/*_catalog/vertica.log
  • /var/log/eccluster/* on manager/logaggregator

Set up rotations for these log files using the tool of your choice. 

For more information on rotating Vertica log files, see HP Vertica's documentation.

If logs are enabled, they should be monitored to ensure they are being properly consumed:

Database Maintenance

Compact Cassandra
The tables in the mo_keyspace_default keyspace use the default compaction strategy SizeTieredCompactionStrategy. This means that by default four sstables of similar size are compacted into a new sstable. This pattern continues where compactions trigger on larger and larger sets of four tables. This strategy helps keep cpu/io usage down, but does not work well for tables that have many updates, deletes, or insertions with a TTL (this is because much of the mo_keyspace_default keyspace data is inserted with a small TTL). The two issues that can result are:
  • Queries failing because of too many tombstones
  • High disk space usage in /var/db/cassandra/data because the SizeTieredCompactionStrategy does not trigger often enough to evict tombstones
When these issues occur, you must run manual compactions on the mo_keyspace_default. Note: This must be done one node at a time.

1.   Verify the Cassandra cluster is entirely up and all nodes show a consistent view:
nodetool status
2.   Log onto the first Cassandra node, and flush the memtables for the mo_keyspace_default keyspace to sstables:
nodetool flush mo_keyspace_default
3.   Compact the sstables in the mo_keyspace_default keyspace:
nodetool compact mo_keyspace_default
4.   Monitor the status of compactions, till completion:
nodetool compactionstats
5.   Repeat steps 1 through 4 for the remaining nodes in the Cassandra cluster.

Checking for Normal Momentum 4 Operation

If there is a system event or an abnormal operation, use the following commands to confirm that the required Momentum system processes are up and running correctly.  Expected outcomes are listed in Expected Output for Normal Momentum 4 Operation.
ps -ef | grep webhooks; echo; echo
ps -ef | grep ecelerity; echo; echo
ps -ef | grep cassandra; echo; echo
ps -ef | grep metrics-etl; echo; echo
ps -ef | grep rabbitmq; echo; echo
ps -ef | grep nginx; echo; echo
ps -ef | grep vertica; echo; echo
ps -ef | grep ecconfigd; echo; echo
ps -ef | grep riak; echo; echo
To confirm that Cassandra is running on one node:
/opt/msys/3rdParty/cassandra/bin/nodetool info
To confirm the status of the Cassandra cluster from one node: 
/opt/msys/3rdParty/cassandra/bin/nodetool status

Alternative Vertica Admin Control

There are occasions where the HP Vertica distributed database is in an uncooperative state and not responding to /etc/init.d level start and stop scripts. Although HP Vertica administration is beyond the scope of this document, there is a console command for checking the status of the database on all nodes and starting and stopping the database on all nodes. The Vertica Management Console may be installed separately and provides a GUI for viewing clusters. It can live independent of the Vertica nodes on its own hardware.
Additional information on HP Vertica Analytics Platform can be found in the HP Vertica Analytics Platform documentation.

If you have a question on system operation, please consult Message Systems Support.

1. To access the HP Vertica admin tool, run the following command:
su vertica_dba -c /opt/vertica/bin/admintools
You will see the following menu options:

Useful options are:
  • 1. View Database Cluster State - displays the status of Vertica on each node. All nodes should be "Up".
  • 4. Stop Database - stops the distributed database on all nodes. Use this to cleanly stop Vertica; for example, when one or more nodes are down or are having issues.
  • 3. Start Database - starts the distributed database on all nodes. Use this to cleanly start Vertica. Run this command after  running "Stop Database." This will give you a clean stop and restart of the database. You only need to run this command on one node in order to start the database on all nodes. 
2. To check status of Vertica on a node without using the admintool, issue the following command:
ps -ef | grep vertica
You should see the following output if Vertica is running correctly.  
501      21792     1  0 16:20 ?        00:00:00 /opt/vertica/spread/sbin/spread -c /var/db/vertica/msys/v_msys_node0001_catalog/spread.conf
501      21794     1 10 16:20 ?        00:00:15 /opt/vertica/bin/vertica -D /var/db/vertica/msys/v_msys_node0001_catalog -C msys -n v_msys_node0001 -h -p 5433 -P 4803 -Y ipv4
501      21801 21794  0 16:20 ?        00:00:00 /opt/vertica/bin/vertica-udx-zygote 12 10 21794 debug-log-off /var/db/vertica/msys/v_msys_node0001_catalog/UDxLogs
root     22152 21525  0 16:22 pts/0    00:00:00 grep vertica
If you see something different or nothing at all, Vertica is either down or is stuck in an initialization state.  If stuck in initialization state, use the admintool to "Stop Database" and then "Start Database." 
Previous Article Introduction Next Article

seconds ago
a minute ago
minutes ago
an hour ago
hours ago
a day ago
days ago
Invalid characters found