This is active banning of IPs that are brute forcing login attempts
to SSH, versus the passive banning of 10,000 abusive IPs from the
abuseipdb.com blacklist. For now I am banning IPs that fail to log
in successfully more than twelve times in a one-hour period, but
these settings might change, and I can override them at the group
and host level if needed.
Currently this works for CentOS 7, Ubuntu 16.04, and Ubuntu 18.04,
with minor differences in the systemd configuration due to older
versions on some distributions.
You can see the status of the jail like this:
# fail2ban-client status sshd
Status for the jail: sshd
|- Filter
| |- Currently failed: 0
| |- Total failed: 0
| `- Journal matches: _SYSTEMD_UNIT=sshd.service + _COMM=sshd
`- Actions
|- Currently banned: 1
|- Total banned: 1
`- Banned IP list: 106.13.112.20
You can unban IPs like this:
# fail2ban-client set sshd unbanip 106.13.112.20
Seems to work around an issue when firewalld is using the nftables
backend with iptables 1.8.2 on Debian 10. Alternatively I could go
back to using the iptables backend... hmm.
See: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=914694
This causes problems every time I re-run the Ansible tag because the
nightly apt security uses a different sources.list and the indexes
are then missing buster-backports. I could add a cache update to the
task, but actually I think the original bug I was trying to solve is
finally fixed, and I'm going to switch to nftables anyways.
This comes from the AbuseIPDB with a confidence level of 95%. I use
the following command to download and sort the IPs:
$ curl -G https://api.abuseipdb.com/api/v2/blacklist -d confidenceMinimum=95 -H "Key: $ABUSEIPDB_API_KEY" -H "Accept: text/plain" | sort > /tmp/ips.txt
Then I add the XML formatting to the file and run it through tidy.
This comes from the AbuseIPDB with a confidence level of 95%. I use
the following command to download and sort the IPs:
$ curl -G https://api.abuseipdb.com/api/v2/blacklist -d confidenceMinimum=95 -H "Key: $ABUSEIPDB_API_KEY" -H "Accept: text/plain" | sort > /tmp/ips.txt
Then I add the XML formatting to the file and run it through tidy.
This comes from the AbuseIPDB with a confidence level of 95%. I use
the following command to download and sort the IPs:
$ curl -G https://api.abuseipdb.com/api/v2/blacklist -d confidenceMinimum=95 -H "Key: $ABUSEIPDB_API_KEY" -H "Accept: text/plain" | sort > /tmp/ips.txt
Then I add the XML formatting to the file and run it through tidy.
Now that I'm blocking ~10,000 malicious IPs from AbuseIPDB I feel
more comfortable using a more relaxed rate limit for SSH. A limit
of 12 per minute is about one every five seconds.
I updated the list with a few dozen more hosts that we brute forcing
SSH but failed to even negotiate a connection because they are using
old ciphers. I will still block them because they attempted 100+ co-
nnections.
This uses the ipsets feature of the Linux kernel to create lists of
IPs (though could be MACs, IP:port, etc) that we can block via the
existing firewalld zone we are already using. In my testing it works
on CentOS 7, Ubuntu 16.04, and Ubuntu 18.04.
The list of abusive IPs currently comes from HPC's systemd journal,
where I filtered for hosts that had attempted and failed to log in
over 100 times. The list is formatted with tidy, for example:
$ tidy -xml -iq -m -w 0 roles/common/files/abusers-ipv4.xml
See: https://firewalld.org/2015/12/ipset-support
If a user has RSA, ECDSA, and ED25519 private keys present on their
system then the ssh client will offer all of these to the server
and they may not get a chance to try password auth before it fails.
There is a bug in iptables 1.8.2 in Debian 10 "Buster" that causes
firewalld to fail when restoring rules. The bug has been fixed in
iptables 1.8.3, which is currently in buster-backports.
See: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=914694
For some reason the key ID I had here was wrong. According to the
Tarsnap website the key ID is 0x6D97F5A4CA38CF33.
ee: https://www.tarsnap.com/pkg-deb.html
Debian 10 comes with Python 2 and Python 3 (at least from the ISO),
so we should prefer the Python 3 version of pycurl. We'll see whet-
her cloud providers like Linode and Digital Ocean ship with Python
3 or not in their default image.
We can register changes when adding repositories and keys and then
update the apt package index conditionally. This should make it be
more consistent between initial host setup and subsequent re-runs.
Ansible errors on adding the tarsnap signing key because it is not
valid (expired a month ago). I contacted Colin Percival about this
on Twitter but he did not seem worried for some reason.
I had previously been removing some packages for security reasons,
then removing others because they were annoying, and yet *others*
because they were annoying on newer Ubuntus only. It is easier to
just unify these tasks and remove them all in one go.
On older Ubuntus where some packages don't exist the task will just
succeed because the package is absent anyways.
The default systemd journal configuration on CentOS 7 and Ubuntu
16.04 does not keep journal logs for multiple boots. This limits
the usefulness of the journal entirely (for example, try to see
sshd logs from even two or three months ago!).
Changing the storage to "persistent" makes systemd keep the logs
on disk in /var/log/journal for up to 2% of the partition size.
The default in later OpenSSH is 6, which seems too high. If you can't
get your password correct after 3 tries then I think you need help.
Eventually I'd like an easy way to enable blocking of repeated login
attempts at the firewall level. I think it's possible in firewalld.
After reörganizing for dynamic includes these tags will never be reached
because the children of dynamic includes do not inherit tags from their
parents as they did with static imports.
Use dynamic includes instead of static imports when you are running
tasks conditionally or using variable interpolation. The down side
is that you need to then tag the parent task as well as all child
tasks, as tags only apply to children of statically imported tasks.
Basically, when using conditionals or variables in your tasks you should
use include_tasks instead of import_tasks. The down side is that you now
need to tag all included tasks individually or with a block, unlike when
using static imports (tags are applied to all imported child tasks).
I would actually like to reduce this task to a single one that uses the
host's ansible_distribution variable, but Ansible 2.5.1 currently gives
the following error: ansible_distribution is undefined.
Vanilla Ubuntu (and Debian actually) defaults to using the official
mirror for security updates rather than country or regional mirrors.
Also, for what it's worth, Ubuntu mirrors didn't always sync these
security archives. I'd prefer to stay closer to vanilla Ubuntu but
also it kinda makes sense to get security updates from the official
source than a mirror (in case of delay or errors).
We stopped being able to do dynamic includes from the playbooks around
Ansible 2.4.0.0 if I recall correctly. Instead we can create a task to
include the variables and make it always run by using the special tag.
For now the Debian and Ubuntu vars files are the same, but I will keep
them separate so that it is more flexible in the future.
Ansible 2.4 and 2.5 are moving away from specialized loop functions
and the old syntax will eventually be deprecated and removed. I did
not change the with_fileglob loops because I'm not sure about their
syntax yet.
See: https://docs.ansible.com/ansible/latest/user_guide/playbooks_loops.html
You should only use the "shell" module when you need shell functions
like flow control and redirects. Also, the "command" module is safer
because it is not affected by the user's environment.
Something seems to have happened as of Ansible 2.4.0.0 where this no
longer works. I suspect it is related to the major changes to static
and dynamic imports that landed around this same time.
In practice this achieves the same function, but without the "magic"
ability to use one task for different operating systems.
Something seems to have happened as of Ansible 2.4.0.0 where this no
longer works. I suspect it is related to the major changes to static
and dynamic imports that landed around this same time.
We make sure that this tasks always runs by using the special tag of
the same name.
Ansible 2.4 changes the way includes work. Now you have to use "import"
for playbooks and tasks that are static, and "include" for those that
are dynamic (ie, those that use variables, loops, etc).
See: http://docs.ansible.com/ansible/devel/playbooks_reuse_includes.html
I'm surprised this worked all these years actually. Since Ansible
version 1.6 it has been possible to use the version_compare filter
instead of doing math logic on strings.
See: https://docs.ansible.com/ansible/playbooks_tests.html
This reverts commit 201165cff6.
Turns out this actually breaks initial deployments, because the
cache gets updated in the first task, then you add sources for
nginx and mariadb, but it doesn't update the indexes because the
cache is < 3600 seconds old, so you end up getting the distro's
versions of nginx and mariadb.
I have added cache_valid_time=3600 for the first task in each
tag that could be possibly running apt-related commands. For ex,
the "nginx" tag is also in the "packages" tag, but sometimes you
run the nginx tag by itself (perhaps repeatadely), so you'd want
to limit the update unless the cache was 1 hour old
I had removed them from Debian 8 and Ubuntu 14.04 configs last year
when the NSA's Suite B crypto guidelines dropped 128-bit algorithms
but those changes didn't make it to my new Ubuntu 16.04 config.
It is probably overkill and paranoid, but this server is mine, so I
can make those decisions (and I only connect from modern clients).
For idempotence we need to run all apt-related tasks, like editing
source files, adding keys, installing packages, etc, when running
the 'packages' tag.
Automatically uses the best mirror for your location, see:
http://httpredir.debian.org/demo.html
Should be much better than any hardcoded default for most hosts.
Signed-off-by: Alan Orth <alan.orth@gmail.com>
Note: I've only tested this on a Debian container, and you can't
set these sysctls on containers (the host controls them). To make
matters worse, there is no fact to make ansible skip this on hosts
that are running in containers. For now I will just skip it on
hosts that are "virtualization" servers... even though we actually
do have KVM running on Debian on real hardware. *sigh*
Signed-off-by: Alan Orth <alan.orth@gmail.com>
Needed in Ubuntu 15.04 where iptables-persistent is going away. I
have added translations of the current IPv4 and IPv6 iptables rules.
Signed-off-by: Alan Orth <alan.orth@gmail.com>
- Don't run the static files as templates
- Use a separate playbook for related tasks
- Use a template for security.sources.list
Signed-off-by: Alan Orth <alan.orth@gmail.com>
New hosts often fail due to not having an apt_mirror, because there
isn't one defined for their group and their host_vars haven't over-
ridden it.
We want new hosts to deploy successfully, so let's just use a default
apt_mirror if there isn't one defined. Rather have a slow mirror than
a failed deployment. And in any case, Linode can download from KENET's
mirror at 10MB/sec. ;)
Signed-off-by: Alan Orth <alan.orth@gmail.com>
