Shane Rutter Blog

Category: Servers

  • NCDU – Find what’s using all your disk space on linux based system

    NCDU – Find what’s using all your disk space on linux based system

    NCDU is a text-based user interface, it allows you to visually see how and where disk space is being used via an easy-to-use interface. Simple run the command in the current directory to see a directory tree of folders & files and their space usage.

    I use this tool mainly for analysing why a server is running out of disk space, determine the reason why and if it’s simply expanded the disk space or is there unused files / massive logs which can be purged.

    Install

    Run the required command for your linux OS version.

    sudo apt install ncdu

    sudo yum install ndcu

    How to use

    Simple navigate to a directory you want to check and run the command:

    ncdu

    If you want to scan the whole system go to the root directory:

    cd /
    ncdu

    You can also specify a directory if you don’t want to change directory:

    ncdu /home

    Windows Server / PC

    For windows-based machines I use a bit of software called TreeSize, it does the same thing as NCDU, allowing you to see how your disk space is being used on a windows based OS.

    TreeSize on windows PC showing disk space usage
    TreeSize on windows PC showing disk space usage

     

    This is provided as it and may require changes to work for your needs.

     

  • How I deleted 530 million records from a large MySQL database table hitting nearly 200GB

    Came across the need to perform maintenance on a MySQL database which holds millions of records, we needed to remove all data which was 2+ years old which involved deleting around 530 million records from a very large table, hitting nearly 200GB.

    Problem

    Due to the size of the database and transaction logs / table locking, it wasn’t possible to do a bulk delete such as

    This caused the database to become very unresponsive and after hours of running eventually failed with table locking issues. This is because DELETE operations are very costly due to all the checking and transactional logging involved.

    I tried creating a second duplicate table and inserting the data we wanted to keep into this table, however this involved turning the system offline during this process, after some testing locally, was going to take roughly 90 hours to complete, this was not an option.

    Solution

    I created a MySQL stored procedure, which allowed me to split the DELETE query into chunks, stored procedures are not handled as a single transaction, this allowed each delete operation to complete and move onto the next chunk, slowly deleting the data from the active table whilst the table is still in use.

    As it was costly to query against the table or use LIMIT & OFFSET, which would take around 10 minutes to return results, it was not possible to use these as part of the procedure if we wanted this process to complete in a timely manner.

    As the ID where incremental, I was able to query the table until I found the largest ID I was willing to delete up to, in my case it was 530,000,000.

    The below stored procedure is what I ended up using to solve the issue.

    It is recommended a professional under takes these actions, please always backup existing data / files / configs.

  • cPanel – Sender x has an outgoing mail hold. Message will be reattempted later.

    A client was having issues sending emails and receiving a number of alerts regarding delayed emails. After looking investigating a number of outbound emails where queued with a failure reason of:

    The account in question had hit the sending limits configured within cPanel and there emails where being held.

    Pre-Checks

    It is important you pre-check and make sure the hold is not legitimate such as a spam attempt / hacked site, I checked the emails being sent and due to a influx in online orders there email usage per day has significantly increased.

    1st Attempt

    Try running the following commands under sudo / root, replaceing [accountname] with the cpanel account name of the user. Then try send an email or retry sending an email in the queue.

    2nd Attempt

    The above worked for a couple accounts, but for one this did not work. There is 3 files depending on the type of block. Have a look in the following files below and remove the accountname / email address if present.

    I found my user within “/etc/outgoing_mail_suspended_users” and once removed, everything went back to normal.

    It is recommended a professional under takes these actions, please always backup existing files / configs.

  • Ubuntu EC2 configure Amazon Time Sync Service for Amazon AWS

    Having problems keeping your server time in sync? or a project requiring accurate timing? If your using a Amazon EC2 instance you can swap to using Chrony and Amazon Time Sync using the below commands.

    (more…)

  • Linux: Killall command

    It is your responsibility to ensure safe usage of these commands, improper usage could cause damage to services and file systems. We will not be held responsible.

    There is a number way of killing processes on a Linux based OS, killall is one of these methods. It’s always good to be ready to kill an unwanted process or a rogue process which is eating resources.

    Usage

    Kill all instances of the “httpd” process

    Kill all proccesses running under the user “postfix”

    Kill a specific process but its full path

    Command not found?

    It does not always come pre-installed, it’s simple to install.

    Ubuntu / Debian

    RHEL – Red hat / Fedora / CentOS

    This package also contains the following useful programs:

    • fuser – identifies what processes are using files.
    • killall – kills a process by its name, similar to a pkill Unices.
    • pstree – Shows currently running processes in a tree format.
    • peekfd – Peek at file descriptors of running processes.