DirectAdmin since version 1.39 has function to detect brute force attack (Info) but DirectAdmin didn't block IP that brute force us. It just warn and annoy us by sending an email. DirectAdmin already provide how to automatic block IP when detect brute force for CentOS/Fedora here but they didn't provide for Debian so I will do for you

• First you have to install iptables

apt-get install iptables

• Then create script file at /usr/local/directadmin/scripts/custom/block_ip.sh

#!/bin/sh
 
curriptables()
{
        echo "<br><br><textarea cols=160 rows=60>";
        /sbin/iptables -nL
        echo "</textarea>";
}
 
### Make sure it's not already blocked
COUNT=`grep -c $ip /etc/network/iptables.save`;
if [ "$COUNT" -ne 0 ]; then
        echo "$ip already exists in iptables ($COUNT). Not blocking.";
        curriptables
        exit 2;
fi
 
echo "Adding $ip to iptables...<br>";
/sbin/iptables -I INPUT -s $ip -j DROP
/sbin/iptables-save > /etc/network/iptables.save
 
echo "<br><br>Result:";
curriptables
exit 0;

Then change permission to 700

chmod 700 /usr/local/directadmin/scripts/custom/block_ip.sh

This block_ip.sh script will add Block This IP button in DirectAdmin Administrator > Brute Force Monitor then click IP Info on IP that we want to block

• If you want iptables to be restored everytime when reboot. Put

post-up iptables-restore /etc/network/iptables.save

at the bottom of interface in /etc/network/interfaces

• If you want DirectAdmin to block ip automatically. Create script file /directadmin/scripts/custom/brute_force_notice_ip.sh

#!/bin/sh
SCRIPT=/usr/local/directadmin/scripts/custom/block_ip.sh
ip=$value $SCRIPT
exit $?;

Then change permission to 700

chmod 700 /usr/local/directadmin/scripts/custom/brute_force_notice_ip.sh

You can set the brute force time before that IP will be blocked by go to DirectAdmin Administrator > Administrator Settings > Notify Admins after an IP has and put the number you want

• If you want to unblock IP. You still have to manual put this command by yourself

iptables -D INPUT -s xxx.xxx.xxx.xxx -j DROP
iptables-save > /etc/network/iptables.save

Change xxx.xxx.xxx.xxx to IP that you want to unblock

Sources:
I wish to have a block_ip.sh so I can block IPs through DirectAdmin
How to block an IP in Linux