Welcome !

Bonjour à tous ,

Et bienvenue sur mon nouveau blog, j’y parle LINUX,
IT , RUNNING & HAMRADIO !

Bonne visite,
Bastien

F4EYQ – 14FDX897 / 14CRX004

 

Take control of your SHOREWALL FW.

Hi!

I purpose you into this article to configure and improve power of SHOREWALL via my script’s. I write it to control and block attack comming from the WEB and also i need some Firewall Log history into a MYSQL DB.

I’ve develop this little system with PHP/BASH/GEOIP libs, the script use FW database to log events and also create some DROP action for few days.

The system is very simple it based on two script’s witch run via CRON regulary.
=> First script is :  the log parsor and MYSQL sync.
=> Second script is : the main system witch can view DB, request some IP, bloc some IP and view statistics.
=> The second script can also run manualy ( the CRON mode is only for attack detection and auto blacklist )

Another script can be use for GEOIP database udpate.

Note here i’m using DEBIAN LINUX in last version with SHOREWALL and MARIADB and PHP7.

1) Setup SHOREWALL log and DB parsing script.

Here you need, your SHOREWALL setup, an empty MYSQL database.

Into SHOREWALL, install « ulog » daemon and configure SHOREWALL to log, here i use :
/var/log/firewallaccess.log

For ulog, simply add this bloc ( /etc/ulogd.conf ) :

[emu1]
file="/var/log/firewallaccess.log"
sync=1

For SHOREWALL here is :

LOGFILE=/var/log/firewallaccess.log
LOG_VERBOSITY=2
LOGTAGONLY=No
LOGFORMAT="Shorewall:%s:%s:"

Of course, your SHOREWALL policy file contain directive $LOG
here i want LOG for REJECT/DROP paquet from the Internet :

znet all DROP $LOG
all all REJECT $LOG

At the end, with $LOG option.

Now you can download and install CRX-FW script you need this ZIP file here : CRX-FW-0.1
Create also directory to unzip files :

mkdir -p /opt/crx/bin/fw/

The parsor script name is, do a chmod+x on it  :  log2mysql

Configure your database by :
Create a database for the FW, import crx_fw_events.sql  and crx_fw_events_actions.sql inside.

Configure your DB user and PASSWORD for this base via the file : config.php

<?php
$cfgbdd['host']='127.0.0.1';
$cfgbdd['user']='usrcrxfw';
$cfgbdd['pass']='*****';
$cfgbdd['bdd']='fwdatabase';
?>

From SHOREWALL part, now you must have the event log (ulog) with entrie like this format :

Jun 5 11:22:44 crx-hyp01 Shorewall:znet-zfw:DROP: IN=eth0 OUT= MAC=00:24:90:a3:32:18:ec:30:91:e0:ef:c0:08:00 SRC=31.184.194.109 DST=94.24.24.24 LEN=40 TOS=00 PREC=0x00 TTL=248 ID=18378 PROTO=TCP SPT=62586 DPT=8080 SEQ=1992689042 ACK=0 WINDOW=1024 SYN URGP=0 MARK=0

Wait few minutes, to have some entries into your SHOREWALL log, after that simply launch manualy the parsor like this :

All is ok here, we can configure now WHITELIST part, CRON part will be the last step.

2) Configure the whitelist for services and IP adress, and launch the engine !

Whitelist is simple, you have a file for :

Service : whitlist_services.php
IP adress : whitlist_ip.php

Start with IP adress and put your public IP.

The format must be :

<?php
$whitelist[]="86.23.0.1";
$whitelist[]="192.168.4.254";
$whitelist[]="10.0.0.2";
$whitelist[]="127.0.0.1";
$whitelist[]="10.0.0.1";
?>

For service :

<?php
$whitelistsvc=array();
$whitelistsvc[]='TCP_585';
$whitelistsvc[]='TCP_23';
?>

All is ok, now we can launch the main engine script   analyseAndAction  :

Without argument it look like this :

The program display the number of FW event’s in the last 30min.
Note that the ban period is 4 days, you can configure it with the config.php.


3) Configure the CRON

Here i’ve configure it like this ( you can ajust frequency depending to your SGBD configuration/capacity and server LOAD ).

#CRX-FIREWALL :
5 8 * * 3 [ `date +\%d` -le 7 ] && /opt/crx/bin/update_geo.bash
*/1 * * * * /opt/wraper.sh 5 "/opt/crx/bin/fw/log2mysql" > /dev/null 2> /dev/null
*/1 * * * * /opt/wraper.sh 15 "/opt/crx/bin/fw/analyseAndAction" > /dev/null 2> /dev/null
0 */1 * * * /opt/crx/bin/fw/analyseAndAction -d 'all' > /dev/null 2> /dev/null

Note the -d ‘all’  is update all event’s inside DB and decide to UBAN some of these if needed.

Wraper.sh is a simple script to launch every XX second some task witch not possible via classic CRON.

Here is the script :

#!/bin/bash
# Initialisation
if [ "$1" == "" ]; then
exit 1
fi
interval=$1
next=$(date +%M | awk -v interval=$interval '{ print int($0 * 60 % interval) }')
if [ $next -gt 0 ]; then
next=$(echo $next | awk -v interval=$interval '{print interval - $0}')
fi
for i in $(seq $next $interval 59)
do
(sleep $i; $2) &
done

4) Display info and also BAN/UBAN IP Adress :

Exemple i need top 5 country bloc by the FW :

Same for @IP ( -i « all » -l 5 ) :

You can also search by IP adress into events ( ./analyseAndAction -p 77.72.85.XX -l 5 ) :

To BAN/UBAN IP adress simply do : 

-m « @ip »      =>  DROP IP ( add event into MYSQL and call SHOREWALL DROP IP ).
-r « @IP »       =>  ALLOW IP ( remove event from MYSQL and call SHOREWALL ALLOW IP ).

Note that you can also view the FW DROP table content via :

# ./analyseAndAction -f "ip" | less

This command is the same as :

/sbin/shorewall show dynamic

That all for now,
Hope you find it usefull, all feedback is welcome !

 

 

Backup your Xen VM disk.

Hi!

Here i’m using a snapshot LVM method and compress over SSH on remote backup server.
With this method i’ve have disk image compress with gunzip, the disk image is ready for restauration
with simple dd pipe with gzip -d cmd.

I’ve develop a script for process automation here it’s:

#!/usr/bin/php
<?php
#Backup LV over SSH
#creation:24/11/2017
#update:24/05/2018
$conf_targets=array(
        'crx-fw01-disk',
        'crx-web01-disk',
        'crx-mail01-disk',
        'crx-vmtemplate01-disk',
        'crx-dxspiderham-disk',
        'crx-dxspider11m-disk',
        'crx-sche02-disk',
        'crx-web02-disk',
        'crx-sup01-disk'
);

$conf_vgname='vg__data_store';
$conf_ssh_uri='ssh crx-bck01.ext.crx.cloud -p9822';
$conf_pathremote='/data_crxbck01/backup_lvm_vg__data_store_from_crx-hyp01-5';

function lvmGetAllLv(){
        $lvminf=array();
        $out=`/sbin/lvdisplay`;
        $out=explode("\n",$out);
        $s=0;
        foreach($out as $k=>$v){
                if (preg_match("/Logical(.*)volume(.*)/",$v,$m)) {
                        $s=1;
                }
                if($s){
                        if(preg_match("/LV(.*)Name(.*)/",$v,$o)){
                                $lvkey=trim($o[2]);
                        }
                        if (preg_match("/LV(.*)Size(.*)/",$v,$m)) {
                                #var_dump($m);
                                $size=$m[2];
                                if(preg_match("/MiB/",$size,$n)){
                                        $size=(int)$size;
                                        $size=$size/1000;
                                        $size=ceil($size);
                                }else{ //GiB value:
                                        $size=str_replace(',','.',$size);
                                        $size=(float)$size;
                                        $size=ceil($size);
                                }
                                $lvminf[$lvkey]['size']=$size;
                        }

                        if (preg_match("/VG(.*)Name(.*)/",$v,$m)) {
                                $lvminf[$lvkey]['vg_name']=trim($m[2]);
                        }
                }
        }
        return $lvminf;
}
$curr_date=date('d-m-Y');
$all_lv=lvmGetAllLv();
foreach($all_lv as $lvname=>$lvinfo){
                if(in_array($lvname,$conf_targets)){
                        $opt_size=$lvinfo['size'].'g';
                        $opt_snap='snap';

                        $backup_file_name=$curr_date.'_'.$lvname.'_'.$opt_size.'.gz';

                        echo "\nBackup LV:$lvname";
                        echo "\nStart:".date('d-m-Y H:i:s')."\n";
                        $create_snapshot_cmd="/sbin/lvcreate --size $opt_size --snapshot --name $opt_snap /dev/$conf_vgname/$lvname";
                        echo $create_snapshot_cmd."\n";
                        $exec=`$create_snapshot_cmd`;
                        $copy_snap_to_rmt="/bin/dd if=/dev/$conf_vgname/snap bs=1k | /bin/gzip | $conf_ssh_uri /bin/dd of=$conf_pathremote/$backup_file_name bs=1k";
                        echo $copy_snap_to_rmt."\n";
                        $exec=`$copy_snap_to_rmt`;
                        $remove_snapshot_cmd="/sbin/lvremove -f /dev/$conf_vgname/snap";
                        echo $remove_snapshot_cmd."\n";
                        $exec=`$remove_snapshot_cmd`;
                        echo "\nEnd:".date('d-m-Y H:i:s')."\n\n";
                }
}

?>
backup_lv_to_ssh.php

So i run it into a screen, and i’m using this on the remote to control in real time :

watch "du -sh *"
control files

 

To restore img simply, re create a LV with the correct size, and import IMG into it :

lvcreate -L 40G -n myserver-disk vg_target
dd if=/path/to/file/backup/myserver-disk.gz | gzip -d | dd of=/dev/vg_target/myserver-disk
restore img into lv process

Hope it’ll help you to do your backup and export vm disk.

A+

 

 

Article: 26042018