It seems that it is no longer recommended/supported to install the
standalone certbot-auto client on Ubuntu 20.04. We apparently need
to use the one provided by Ubuntu, which luckily includes a service
and timer to renew the certs.
Older Ubuntus originally didn't use a persistent journal, which was
somewhat of a surprise when looking at logs after a few months. Now
this does not seem to be an issue since Ubuntu 18.04. As for CentOS
I do not use that distro here so I don't need to care.
The nftables backend should be more performant and flexible. I had
been planning to use it on Ubuntu 18.04 and Debian 10 as well, but
there were issues with the specific versions used in those distros.
See: https://firewalld.org/2018/07/nftables-backend
We never used this simple firewall utility and in at least one case
a user on the server tried to use it and messed up the rules I had
set via firewalld.
This comes from the AbuseIPDB with a confidence level of 95%. I use
the following command to download and sort the IPs:
$ curl -G https://api.abuseipdb.com/api/v2/blacklist -d \
confidenceMinimum=95 -H "Key: $ABUSEIPDB_API_KEY" \
-H "Accept: text/plain" | sort | sed -e '/:/w /tmp/ipv6.txt' \
-e '/:/d' > /tmp/ipv4.txt
I manually add the XML formatting to each file and run them through
tidy:
$ tidy -xml -utf8 -m -iq -w 0 roles/common/files/abusers-ipv4.xml
$ tidy -xml -utf8 -m -iq -w 0 roles/common/files/abusers-ipv6.xml
This config stopped working a few years ago probably when hosts started
getting IPv6 and we need to allow ::1. Make sure we're only listening
on local loopback interfaces also.
This comes from the AbuseIPDB with a confidence level of 95%. I use
the following command to download and sort the IPs:
$ curl -G https://api.abuseipdb.com/api/v2/blacklist -d confidenceMinimum=95 -H "Key: $ABUSEIPDB_API_KEY" -H "Accept: text/plain" | sort | sed -e '/:/w /tmp/ipv6.txt' -e '/:/d' > /tmp/ipv4.txt
I manually add the XML formatting to each file and run them through
tidy:
$ tidy -xml -utf8 -m -iq -w 0 roles/common/files/abusers-ipv4.xml
$ tidy -xml -utf8 -m -iq -w 0 roles/common/files/abusers-ipv6.xml
ansible-lint told me not to test equality with booleans using literal
"True" and "False", but it Ansible complains if I use "is True" also.
It seems that I need to adjust the syntax slightly.
This comes from the AbuseIPDB with a confidence level of 95%. I use
the following command to download and sort the IPs:
$ curl -G https://api.abuseipdb.com/api/v2/blacklist -d confidenceMinimum=95 -H "Key: $ABUSEIPDB_API_KEY" -H "Accept: text/plain" | sort | sed -e '/:/w /tmp/ipv6.txt' -e '/:/d' > /tmp/ipv4.txt
I manually add the XML formatting to each file and run them through
tidy.
This comes from the AbuseIPDB with a confidence level of 95%. I use
the following command to download and sort the IPs:
$ curl -G https://api.abuseipdb.com/api/v2/blacklist -d confidenceMinimum=95 -H "Key: $ABUSEIPDB_API_KEY" -H "Accept: text/plain" | sort | sed -e '/:/w /tmp/ipv6.txt' -e '/:/d' > /tmp/ipv4.txt
I manually add the XML formatting to each file and run them through
tidy.
For some reason the nftables set support in firewalld doesn't seem
to be working. I see that sets (aka ipsets in nftables lingo) are
created, but they are empty. For now I will just leave these tasks
as they are to revert the behavior on current hosts (should do no
change on new installed, as the regexp won't match).
This comes from the AbuseIPDB with a confidence level of 95%. I use
the following command to download and sort the IPs:
$ curl -G https://api.abuseipdb.com/api/v2/blacklist -d confidenceMinimum=95 -H "Key: $ABUSEIPDB_API_KEY" -H "Accept: text/plain" | sort > /tmp/ips.txt
I manually remove the IPv6 addresses and save them to a different
filr, then I add the XML formatting to files and run them through
tidy.
This is active banning of IPs that are brute forcing login attempts
to SSH, versus the passive banning of 10,000 abusive IPs from the
abuseipdb.com blacklist. For now I am banning IPs that fail to log
in successfully more than twelve times in a one-hour period, but
these settings might change, and I can override them at the group
and host level if needed.
Currently this works for CentOS 7, Ubuntu 16.04, and Ubuntu 18.04,
with minor differences in the systemd configuration due to older
versions on some distributions.
You can see the status of the jail like this:
# fail2ban-client status sshd
Status for the jail: sshd
|- Filter
| |- Currently failed: 0
| |- Total failed: 0
| `- Journal matches: _SYSTEMD_UNIT=sshd.service + _COMM=sshd
`- Actions
|- Currently banned: 1
|- Total banned: 1
`- Banned IP list: 106.13.112.20
You can unban IPs like this:
# fail2ban-client set sshd unbanip 106.13.112.20
Seems to work around an issue when firewalld is using the nftables
backend with iptables 1.8.2 on Debian 10. Alternatively I could go
back to using the iptables backend... hmm.
See: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=914694
This causes problems every time I re-run the Ansible tag because the
nightly apt security uses a different sources.list and the indexes
are then missing buster-backports. I could add a cache update to the
task, but actually I think the original bug I was trying to solve is
finally fixed, and I'm going to switch to nftables anyways.
This comes from the AbuseIPDB with a confidence level of 95%. I use
the following command to download and sort the IPs:
$ curl -G https://api.abuseipdb.com/api/v2/blacklist -d confidenceMinimum=95 -H "Key: $ABUSEIPDB_API_KEY" -H "Accept: text/plain" | sort > /tmp/ips.txt
Then I add the XML formatting to the file and run it through tidy.
This comes from the AbuseIPDB with a confidence level of 95%. I use
the following command to download and sort the IPs:
$ curl -G https://api.abuseipdb.com/api/v2/blacklist -d confidenceMinimum=95 -H "Key: $ABUSEIPDB_API_KEY" -H "Accept: text/plain" | sort > /tmp/ips.txt
Then I add the XML formatting to the file and run it through tidy.
This comes from the AbuseIPDB with a confidence level of 95%. I use
the following command to download and sort the IPs:
$ curl -G https://api.abuseipdb.com/api/v2/blacklist -d confidenceMinimum=95 -H "Key: $ABUSEIPDB_API_KEY" -H "Accept: text/plain" | sort > /tmp/ips.txt
Then I add the XML formatting to the file and run it through tidy.
Now that I'm blocking ~10,000 malicious IPs from AbuseIPDB I feel
more comfortable using a more relaxed rate limit for SSH. A limit
of 12 per minute is about one every five seconds.
I updated the list with a few dozen more hosts that we brute forcing
SSH but failed to even negotiate a connection because they are using
old ciphers. I will still block them because they attempted 100+ co-
nnections.
This uses the ipsets feature of the Linux kernel to create lists of
IPs (though could be MACs, IP:port, etc) that we can block via the
existing firewalld zone we are already using. In my testing it works
on CentOS 7, Ubuntu 16.04, and Ubuntu 18.04.
The list of abusive IPs currently comes from HPC's systemd journal,
where I filtered for hosts that had attempted and failed to log in
over 100 times. The list is formatted with tidy, for example:
$ tidy -xml -iq -m -w 0 roles/common/files/abusers-ipv4.xml
See: https://firewalld.org/2015/12/ipset-support
If a user has RSA, ECDSA, and ED25519 private keys present on their
system then the ssh client will offer all of these to the server
and they may not get a chance to try password auth before it fails.
There is a bug in iptables 1.8.2 in Debian 10 "Buster" that causes
firewalld to fail when restoring rules. The bug has been fixed in
iptables 1.8.3, which is currently in buster-backports.
See: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=914694
For some reason the key ID I had here was wrong. According to the
Tarsnap website the key ID is 0x6D97F5A4CA38CF33.
ee: https://www.tarsnap.com/pkg-deb.html
Debian 10 comes with Python 2 and Python 3 (at least from the ISO),
so we should prefer the Python 3 version of pycurl. We'll see whet-
her cloud providers like Linode and Digital Ocean ship with Python
3 or not in their default image.
We can register changes when adding repositories and keys and then
update the apt package index conditionally. This should make it be
more consistent between initial host setup and subsequent re-runs.
Ansible errors on adding the tarsnap signing key because it is not
valid (expired a month ago). I contacted Colin Percival about this
on Twitter but he did not seem worried for some reason.
I had previously been removing some packages for security reasons,
then removing others because they were annoying, and yet *others*
because they were annoying on newer Ubuntus only. It is easier to
just unify these tasks and remove them all in one go.
On older Ubuntus where some packages don't exist the task will just
succeed because the package is absent anyways.
The default systemd journal configuration on CentOS 7 and Ubuntu
16.04 does not keep journal logs for multiple boots. This limits
the usefulness of the journal entirely (for example, try to see
sshd logs from even two or three months ago!).
Changing the storage to "persistent" makes systemd keep the logs
on disk in /var/log/journal for up to 2% of the partition size.
The default in later OpenSSH is 6, which seems too high. If you can't
get your password correct after 3 tries then I think you need help.
Eventually I'd like an easy way to enable blocking of repeated login
attempts at the firewall level. I think it's possible in firewalld.
Avoids the following error in apt:
Skipping acquire of configured file 'nginx/binary-i386/Packages' as repository 'https://nginx.org/packages/ubuntu bionic InRelease' doesn't support architecture 'i386'
No need to give Google even more data or free advertising by using
this as the default! In practice I always use the DNS servers from
the VPS provider anyways.
Instead of looping over a list of items to install, we can actually
just give a list directly to the apt module. This allows the module
to install all packages in one transaction, which is faster as well
as slightly safer for some dependency resolution scenarios.
This tag is no longer reachable after switching to the new dynamic
includes in Ansible 2.4 and 2.5. Anyways, I've been questioning my
decision to add the "packages" tag to any task that uses the apt
module.
Instead of looping over a list of items to install, we can actually
just give a list directly to the apt module. This allows the module
to install all packages in one transaction, which is faster as well
as slightly safer for some dependency resolution scenarios.
Because of the shift from static imports to dynamic includes these
tags will never be reached unless they have their own task that is
tagged at the top-level (dynamic includes don't pass their tags to
their children).
After reörganizing for dynamic includes these tags will never be reached
because the children of dynamic includes do not inherit tags from their
parents as they did with static imports.
As of Ansible 2.4 and 2.5 the behavior for importing tasks has changed
to introduce the notion of static imports and dynamic includes. If the
tasks doing the import is using variable interpolation or conditionals
then the task should be dynamic. This results in quicker playbook runs
due to less importing of unneccessary tasks.
One side effect of this is that child tasks of dynamic includes do not
inherit their parents' tags so you must tag them explicitly or a block.
Also, I had to move the letsencrypt tasks to the main task file so the
tags were available (due to dynamic tasks not inheriting tags).
As of Ansible 2.4 and 2.5 the behavior for importing tasks has changed
to introduce the notion of static imports and dynamic includes. If the
tasks doing the import is using variable interpolation or conditionals
then the task should be dynamic. This results in quicker playbook runs
due to less importing of unneccessary tasks.
One side effect of this is that child tasks of dynamic includes do not
inherit their parents' tags so you must tag them explicitly or a block.
Use dynamic includes instead of static imports when you are running
tasks conditionally or using variable interpolation. The down side
is that you need to then tag the parent task as well as all child
tasks, as tags only apply to children of statically imported tasks.
Instead of iterating over fifteen packages with a loop that does fifteen
separate apt transactions, it is better to give the apt module a list so
it can install them all in one transaction. This is both quicker and te-
chnically more safe for dependency resolution.
Instead of iterating over fifteen packages with a loop that does fifteen
separate apt transactions, it is better to give the apt module a list so
it can install them all in one transaction. This is both quicker and te-
chnically more safe for dependency resolution.
Use dynamic includes instead of static imports when you are running
tasks conditionally or using variable interpolation. The down side
is that you need to then tag the parent task as well as all child
tasks, as tags only apply to children of statically imported tasks.
Basically, when using conditionals or variables in your tasks you should
use include_tasks instead of import_tasks. The down side is that you now
need to tag all included tasks individually or with a block, unlike when
using static imports (tags are applied to all imported child tasks).
I would actually like to reduce this task to a single one that uses the
host's ansible_distribution variable, but Ansible 2.5.1 currently gives
the following error: ansible_distribution is undefined.
Vanilla Ubuntu (and Debian actually) defaults to using the official
mirror for security updates rather than country or regional mirrors.
Also, for what it's worth, Ubuntu mirrors didn't always sync these
security archives. I'd prefer to stay closer to vanilla Ubuntu but
also it kinda makes sense to get security updates from the official
source than a mirror (in case of delay or errors).
We stopped being able to do dynamic includes from the playbooks around
Ansible 2.4.0.0 if I recall correctly. Instead we can create a task to
include the variables and make it always run by using the special tag.
For now the Debian and Ubuntu vars files are the same, but I will keep
them separate so that it is more flexible in the future.
These tasks are conditional and mutually exclusive due to the "when"
clause. Using import_tasks means that these are imported before the
playbook execution and then skipped during evaluation of the test.
It makes sense in this case to use include_tasks so that the tasks
are only imported during playbook execution if the condition is met.
Ansible 2.4 and 2.5 are moving away from specialized loop functions
and the old syntax will eventually be deprecated and removed. I did
not change the with_fileglob loops because I'm not sure about their
syntax yet.
See: https://docs.ansible.com/ansible/latest/user_guide/playbooks_loops.html
I have zero idea if we have IE6 clients any more, but according to the
H5BP community IE6 actually did support gzip and only represents 0.1%
of Internet traffic in 2015 (!) anyways.
See: https://github.com/h5bp/server-configs-nginx/issues/125
You should only use the "shell" module when you need shell functions
like flow control and redirects. Also, the "command" module is safer
because it is not affected by the user's environment.
Something seems to have happened as of Ansible 2.4.0.0 where this no
longer works. I suspect it is related to the major changes to static
and dynamic imports that landed around this same time.
In practice this achieves the same function, but without the "magic"
ability to use one task for different operating systems.
Something seems to have happened as of Ansible 2.4.0.0 where this no
longer works. I suspect it is related to the major changes to static
and dynamic imports that landed around this same time.
We make sure that this tasks always runs by using the special tag of
the same name.
[ANSIBLE0006] systemctl used in place of systemd module
Also, move the functionality of the changed check to the systemd task,
because it has the ability to simply daemon-reload itself now.
I can't remember right now why I needed to use Debian's MariaDB build
but now I just want to use upstream's latest stable. Debian's version
is 10.1 and upstream has moved on to 10.2.
