Managing Disk Space

There is no guaranteed system to prevent Mango from running out of disk space.  If a MangoES or server running Mango does run out of disk space, it will cause serious problems and the system will need to be manually resolved.

As part of the default MangoES setup, there is a data point that monitors the remaining disk space which has event detectors set up to trigger alarms at 2000MB and 1000MB remaining disk space.  It is recommended to set up associated event handlers for these point event detectors to email the system administrator so that it can be addressed before the disk is full.

In Mango, there are four primary things that can potentially fill the disk.

  1. Historical data stored in the MangoNoSQL Database under /opt/mango/databases/mangoTSDB

  2. Backup files stored in /opt/mango/backup

  3. Event messages stored in the Mango SQL (H2) database in /opt/mango/databases/mah2.h2.db

  4. Log files in /opt/mango/logs

Note: The file paths in this document are the exact paths used on a MangoES; depending on your installation, the "/opt/mango" part of the paths (often referred to as the 'MA_HOME' directory) could be different.

Historical Data Purging

Historical Data is the main thing that will use your available disk space.  The amount of disk space needed will be a function of how many data points are being logged, how often they are being logged and how long the logged data is stored before being purged.

By default Mango will keep all historical data and events for 12 months.  You can adjust the purge settings on the system setting page under the purge settings.  Individual purge setting can be set on each Data Source or Data Point to override the system purge setting.

As a an approximate value 200 million historical values consumes about 4 GB of Disk space.

If your Mango is operational you can adjust your purge settings lower and use the Purge Now button on the system setting page to have Mango free up disk space. 

To purge older data manually you can simply find and delete older shard files.

Note about the NoSQL database purging.  Data is stored in shared files in the /opt/mango/databases/mangoTSDB folder such as 685.data.rev The 685 is a hash of the date the shard started.  Each shard file contains about 24 days of data.  

On linux based systems you can use the following commands to find and could files with a certain name:

Change to the Mango Databases directory:

cd /opt/mango/databases/mangoTSDB

Run:

find . -name "686.data.rev" | wc -l

 This will tell you the total number of files for this data range in your database

You can create a Manifest files of the shards you want to delete:

find . -name '686.data.rev' > 686.manifest

Delete all the files in the Manifest file:

for f in $(cat 686.manifest); do rm $f; done

 

This is a useful script to do a full count on all the files in your mangoTSDB folder so you can find and delete the oldest shard files

directories="/usr/local/scada/databases/mangoTSDB"
for directory in ${directories} ; do
    echo ${directory}": " ;
    ( find ${directory} -type f -exec basename {} \;| sort | uniq -c) ;
done

To use the script follow these commands:

Save the script file: 

nano countDBFiles.sh

Paste the contents of the script above into the file

Ctr x and select to save the file. 

Make the file executable:

chmod u+x countDBFiles.sh

Now you can run

./countDBFiles.sh 

This can take some time so you might pipe the output of the script into a file with this command

./countDBFiles.sh > DBFileCount.txt

Backup Files

Backup files can be searched for and manually deleted in the /opt/mango/backup folder.

There are three types of backups performed by Mango which can be managed on the System Setting page.

  1. Mango NoSQL Backups under the "Mango NoSQL Configuration" Section. Too many backup files of the historical database can take up a lot of disk space.

  2. SQL Database Backup which makes a SQL dump of the default SQL database (H2 or MySQL)

  3. Configuration Backup which makes a .json file of the configuration, these files are generally fairly small.

Event Messages in Database

Event messages are stored in the SQL database (H2 or MySQL) in the "events" table.  Generally you will see on your events page when you have events but in some cases you may have your configuration set in a way that can create a lot of unnecessary events.  These events are usually created from a Data Source. As long as the Data Source is functioning property it's good practice to set your Data Source exception events to an "Ignore" level so they are not placed into the database.  It is important to note that a "None" level event is still logged into the database.

Event purging can be set on the system setting page under "Purge Settings" and default to 1 year

Screenshot 2019-01-04 15.45.08.png

You can use the SQL Console to count the numbers of events in your database.  See here for more information.

Log Files

Log files are stored in /opt/mango/logs and can be generated from several sources.  Some Data Sources and Publishers can generate log files for example.  The default system logs generated by Mango are configured to roll over so should never amount to any significant disk space usage however you may still want to check this folder for other large log files that you may have activated in the configuration.