• R/O
  • SSH
  • HTTPS

masspie: Zusammenfassung des Repository


Neueste Commits RSS

Rev. Zeit Autor Nachricht
r23 2020-03-08 16:18:55 elge openssldir=/etc/ssl
r22 2020-02-19 15:43:47 elge imporing the rest of shot1 incl enforce check
r21 2020-02-19 15:01:53 elge better iprev, count, and importing legacy san & valid
r20 2020-02-17 19:08:53 elge better intro and getting started guide writeup on initial...
r19 2020-02-17 19:05:38 elge re-factor validptr --> checkiprev
r18 2020-02-01 15:09:25 elge a copy of curl's concatenated certs in here and a few tun...
r17 2020-01-20 00:10:29 elge checking for 2nd-3rd-level domains, MXes and DANE
r16 2020-01-13 18:51:25 elge few improvements
r15 2020-01-12 00:18:56 elge adding cert validation
r14 2020-01-11 19:49:58 elge new checks described

README.md

MASSPIE

Mass-scanning port 25/tcp and eventually checking if STARTTLS is enforced and validates

REQUIREMENTS

Only Masscan and the host command are truly required, but a few tools may also become handy to handle the load.

Slackware

ls -lF /var/log/packages/{bind,traceroute,nmap,htop,iftop,nload}-*

Ubuntu

apt install dnsutils traceroute nmap htop iftop nload ntopng

INITIAL SCAN

Required as root & heavy network load

month=`date +%Y%m`
mkdir -p ~/mass/$month/
cd ~/mass/$month/
screen -S $month
cat /etc/masscan/exclude.conf
ls -alF
time masscan 0.0.0.0/0 -p25 --excludefile /etc/masscan/exclude.conf --rate=500000 -oG massp25.og

and check the network load e.g. with nload

At rate 250,000, it takes about 15 hours

At rate 500,000, network TX is about 210 Mbit/s and it takes about 2:15 hours

real    134m51.334s
user    17m6.783s
sys     28m2.972s

INSTALLATION

Just in case somebody manages to shell-inject code into our host probes, it's best to run all this as a user

useradd -m -s /bin/bash -g users mass

Move the scanned materials to its home directory. Got 11 to 13 millions?

grep -v ^# ~/mass/$month/massp25.og | wc -l
mv ~/mass/$month/ ~mass/
chown -R mass:users ~mass/$month/

and switch to it

su - mass

Then grab the Masspie scripts and start a GNU/Screen session in there

svn checkout https://svn.osdn.net/svnroot/masspie/
cd masspie/

month=`date +%Y%m`
cd ~/$month/
screen -S masspie

IPREV CHECK

As user and heavy CPU times

Stress some DNS services, possibly yours, or your ISP's. This can be split across multiple servers. Eventually switch around the order of nameservers into resolv.conf.

vi /etc/resolv.conf

cd ~/$month/
ls -lF massp25.og
#rm -rf splitted/
#rm -f splitted/*.ptr splitted/*.weird
~/masspie/checkiprev.bash
ls -lF massp25.og.ip.sort
ls -F splitted/ip[0-9][0-9][0-9]
tail splitted/ip099
tail -G splitted/ip099.ptr

and check the CPU load with htop

CUSTOM OPENSSL

Exit when EHLO/STARTTLS was not advertised

git clone git://git.openssl.org/openssl.git
cd openssl/
patch -p1 < ../openssl-shut.patch

removepkg openssl
#keep openssl-solibs as the whole system depends on it
mv /etc/ssl/ /etc/ssl.old/

./config --openssldir=/etc/ssl
#perl configdata.pm --dump
time make -j8 >/dev/null && echo BUILT
time make -j8 install >/dev/null && echo INSTALLED
ls -lF /usr/local/lib64/libssl.so
ls -lF /usr/local/lib64/libcrypto.so
ls -lF /usr/local/include/openssl/aes.h
#cat /etc/ld.so.conf
ldconfig
update-ca-certificates

VALID CN/SAN CHECK

for x in `seq -w 000 999`; do ./sslcheck.bash x$x.ptr & done; unset x
jobs

CERTIFICATE CHECK

Grab the latest concatenated Mozilla CA bundle

cd /etc/ssl/
curl -s --remote-name --time-cond - https://curl.haxx.se/ca/cacert.pem
sha256sum cacert.pem | tee -a cacert.pem.sha2

#2020/01/01
#adf770dfd574a0d6026bfaa270cb6879b063957177a991d453ff1d302c02081f  cacert.pem

Now make sure you're in position to validate certificates.

ehlo=YOUR-IPREV
mx=xc.nethence.com
echo Q | /usr/local/bin/openssl -4 s_client -showcerts -verify 5 -CAfile cacert.pem -starttls smtp -name $ehlo -servername $mx -connect $mx:25 -crlf > $mx.chain.crt
#-CApath /etc/ssl/certs
#-brief

Note -showcerts helps to get the intermediate certificate here.

/usr/local/bin/openssl crl2pkcs7 -nocrl -certfile $mx.chain.crt | openssl pkcs7 -print_certs -noout

/usr/local/bin/openssl verify -verbose -issuer_checks -verify_return_error -CAfile cacert.pem -untrusted $mx.chain.crt -no_alt_chains -ignore_critical $mx.chain.crt
#-CApath /etc/ssl/certs
#-crl_download -crl_check

Note -untrusted helps to define the intermediate certificate for chain validation.

Then finally proceed with mass validation.

    for x in `seq -w 000 999`; do ./sslvalid.bash x$x.ptr.validcn & done; unset x
jobs

PTR DOMAINS

We further need to look at MX records to query correct TLSA records, and to get MX records, we need zone names. We will not obtain all of public zones, but here's an attempt to get some using the PTRs we've collected.

./checkdomains.bash

MXes & DANE

Now we can query MX against a few domains and eventually seek for DANE-enabled SMTP hosts (including PKIX-TA/EE).

./dane.bash

ADDITIONAL NOTES

The second-level domain public suffixes in file SLDs were obtained as such

wget https://raw.githubusercontent.com/gavingmiller/second-level-domains/master/SLDs.csv
cut -f2 -d, SLDs.csv | sed 's/^\.//' > SLDs
dos2unix SLDs
Show on old repository browser