This comes from the AbuseIPDB with a confidence level of 95%. I use
the following command to download and sort the IPs:
$ curl -G https://api.abuseipdb.com/api/v2/blacklist -d confidenceMinimum=95 -H "Key: $ABUSEIPDB_API_KEY" -H "Accept: text/plain" | sort > /tmp/ips.txt
Then I add the XML formatting to the file and run it through tidy.
This comes from the AbuseIPDB with a confidence level of 95%. I use
the following command to download and sort the IPs:
$ curl -G https://api.abuseipdb.com/api/v2/blacklist -d confidenceMinimum=95 -H "Key: $ABUSEIPDB_API_KEY" -H "Accept: text/plain" | sort > /tmp/ips.txt
Then I add the XML formatting to the file and run it through tidy.
Now that I'm blocking ~10,000 malicious IPs from AbuseIPDB I feel
more comfortable using a more relaxed rate limit for SSH. A limit
of 12 per minute is about one every five seconds.
I updated the list with a few dozen more hosts that we brute forcing
SSH but failed to even negotiate a connection because they are using
old ciphers. I will still block them because they attempted 100+ co-
nnections.
This uses the ipsets feature of the Linux kernel to create lists of
IPs (though could be MACs, IP:port, etc) that we can block via the
existing firewalld zone we are already using. In my testing it works
on CentOS 7, Ubuntu 16.04, and Ubuntu 18.04.
The list of abusive IPs currently comes from HPC's systemd journal,
where I filtered for hosts that had attempted and failed to log in
over 100 times. The list is formatted with tidy, for example:
$ tidy -xml -iq -m -w 0 roles/common/files/abusers-ipv4.xml
See: https://firewalld.org/2015/12/ipset-support
If a user has RSA, ECDSA, and ED25519 private keys present on their
system then the ssh client will offer all of these to the server
and they may not get a chance to try password auth before it fails.
There is a bug in iptables 1.8.2 in Debian 10 "Buster" that causes
firewalld to fail when restoring rules. The bug has been fixed in
iptables 1.8.3, which is currently in buster-backports.
See: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=914694
For some reason the key ID I had here was wrong. According to the
Tarsnap website the key ID is 0x6D97F5A4CA38CF33.
ee: https://www.tarsnap.com/pkg-deb.html
Debian 10 comes with Python 2 and Python 3 (at least from the ISO),
so we should prefer the Python 3 version of pycurl. We'll see whet-
her cloud providers like Linode and Digital Ocean ship with Python
3 or not in their default image.
We can register changes when adding repositories and keys and then
update the apt package index conditionally. This should make it be
more consistent between initial host setup and subsequent re-runs.
Ansible errors on adding the tarsnap signing key because it is not
valid (expired a month ago). I contacted Colin Percival about this
on Twitter but he did not seem worried for some reason.
I had previously been removing some packages for security reasons,
then removing others because they were annoying, and yet *others*
because they were annoying on newer Ubuntus only. It is easier to
just unify these tasks and remove them all in one go.
On older Ubuntus where some packages don't exist the task will just
succeed because the package is absent anyways.
The default systemd journal configuration on CentOS 7 and Ubuntu
16.04 does not keep journal logs for multiple boots. This limits
the usefulness of the journal entirely (for example, try to see
sshd logs from even two or three months ago!).
Changing the storage to "persistent" makes systemd keep the logs
on disk in /var/log/journal for up to 2% of the partition size.
The default in later OpenSSH is 6, which seems too high. If you can't
get your password correct after 3 tries then I think you need help.
Eventually I'd like an easy way to enable blocking of repeated login
attempts at the firewall level. I think it's possible in firewalld.
Avoids the following error in apt:
Skipping acquire of configured file 'nginx/binary-i386/Packages' as repository 'https://nginx.org/packages/ubuntu bionic InRelease' doesn't support architecture 'i386'
No need to give Google even more data or free advertising by using
this as the default! In practice I always use the DNS servers from
the VPS provider anyways.
Instead of looping over a list of items to install, we can actually
just give a list directly to the apt module. This allows the module
to install all packages in one transaction, which is faster as well
as slightly safer for some dependency resolution scenarios.
This tag is no longer reachable after switching to the new dynamic
includes in Ansible 2.4 and 2.5. Anyways, I've been questioning my
decision to add the "packages" tag to any task that uses the apt
module.
Instead of looping over a list of items to install, we can actually
just give a list directly to the apt module. This allows the module
to install all packages in one transaction, which is faster as well
as slightly safer for some dependency resolution scenarios.
Because of the shift from static imports to dynamic includes these
tags will never be reached unless they have their own task that is
tagged at the top-level (dynamic includes don't pass their tags to
their children).
After reörganizing for dynamic includes these tags will never be reached
because the children of dynamic includes do not inherit tags from their
parents as they did with static imports.
As of Ansible 2.4 and 2.5 the behavior for importing tasks has changed
to introduce the notion of static imports and dynamic includes. If the
tasks doing the import is using variable interpolation or conditionals
then the task should be dynamic. This results in quicker playbook runs
due to less importing of unneccessary tasks.
One side effect of this is that child tasks of dynamic includes do not
inherit their parents' tags so you must tag them explicitly or a block.
Also, I had to move the letsencrypt tasks to the main task file so the
tags were available (due to dynamic tasks not inheriting tags).
As of Ansible 2.4 and 2.5 the behavior for importing tasks has changed
to introduce the notion of static imports and dynamic includes. If the
tasks doing the import is using variable interpolation or conditionals
then the task should be dynamic. This results in quicker playbook runs
due to less importing of unneccessary tasks.
One side effect of this is that child tasks of dynamic includes do not
inherit their parents' tags so you must tag them explicitly or a block.
Use dynamic includes instead of static imports when you are running
tasks conditionally or using variable interpolation. The down side
is that you need to then tag the parent task as well as all child
tasks, as tags only apply to children of statically imported tasks.
Instead of iterating over fifteen packages with a loop that does fifteen
separate apt transactions, it is better to give the apt module a list so
it can install them all in one transaction. This is both quicker and te-
chnically more safe for dependency resolution.
Instead of iterating over fifteen packages with a loop that does fifteen
separate apt transactions, it is better to give the apt module a list so
it can install them all in one transaction. This is both quicker and te-
chnically more safe for dependency resolution.
Use dynamic includes instead of static imports when you are running
tasks conditionally or using variable interpolation. The down side
is that you need to then tag the parent task as well as all child
tasks, as tags only apply to children of statically imported tasks.
Basically, when using conditionals or variables in your tasks you should
use include_tasks instead of import_tasks. The down side is that you now
need to tag all included tasks individually or with a block, unlike when
using static imports (tags are applied to all imported child tasks).
I would actually like to reduce this task to a single one that uses the
host's ansible_distribution variable, but Ansible 2.5.1 currently gives
the following error: ansible_distribution is undefined.
Vanilla Ubuntu (and Debian actually) defaults to using the official
mirror for security updates rather than country or regional mirrors.
Also, for what it's worth, Ubuntu mirrors didn't always sync these
security archives. I'd prefer to stay closer to vanilla Ubuntu but
also it kinda makes sense to get security updates from the official
source than a mirror (in case of delay or errors).
We stopped being able to do dynamic includes from the playbooks around
Ansible 2.4.0.0 if I recall correctly. Instead we can create a task to
include the variables and make it always run by using the special tag.
For now the Debian and Ubuntu vars files are the same, but I will keep
them separate so that it is more flexible in the future.
These tasks are conditional and mutually exclusive due to the "when"
clause. Using import_tasks means that these are imported before the
playbook execution and then skipped during evaluation of the test.
It makes sense in this case to use include_tasks so that the tasks
are only imported during playbook execution if the condition is met.
Ansible 2.4 and 2.5 are moving away from specialized loop functions
and the old syntax will eventually be deprecated and removed. I did
not change the with_fileglob loops because I'm not sure about their
syntax yet.
See: https://docs.ansible.com/ansible/latest/user_guide/playbooks_loops.html
I have zero idea if we have IE6 clients any more, but according to the
H5BP community IE6 actually did support gzip and only represents 0.1%
of Internet traffic in 2015 (!) anyways.
See: https://github.com/h5bp/server-configs-nginx/issues/125
You should only use the "shell" module when you need shell functions
like flow control and redirects. Also, the "command" module is safer
because it is not affected by the user's environment.
Something seems to have happened as of Ansible 2.4.0.0 where this no
longer works. I suspect it is related to the major changes to static
and dynamic imports that landed around this same time.
In practice this achieves the same function, but without the "magic"
ability to use one task for different operating systems.
Something seems to have happened as of Ansible 2.4.0.0 where this no
longer works. I suspect it is related to the major changes to static
and dynamic imports that landed around this same time.
We make sure that this tasks always runs by using the special tag of
the same name.
[ANSIBLE0006] systemctl used in place of systemd module
Also, move the functionality of the changed check to the systemd task,
because it has the ability to simply daemon-reload itself now.
I can't remember right now why I needed to use Debian's MariaDB build
but now I just want to use upstream's latest stable. Debian's version
is 10.1 and upstream has moved on to 10.2.
Ansible 2.4 changes the way includes work. Now you have to use "import"
for playbooks and tasks that are static, and "include" for those that
are dynamic (ie, those that use variables, loops, etc).
See: http://docs.ansible.com/ansible/devel/playbooks_reuse_includes.html
Using www-data was a temporary measure while I was waiting for the
official nginx.org packages to be released for Debian 9 and we had
to use Debian's own nginx package.
Sync with latest packaged version from Debian 9. Effectively this
only updates comments and disabled options, but having less changes
to look at during an apt upgrade is nice and reduces the chance of
adding errors.
There are currently no nginx.org builds for Debian 9, so we need to
use the package from Debian's repository. This package provides a
www-data user and group instead of an nginx one.
We can revert some of this after Debian 9 is released and official
builds come from nginx.org (though it might be useful to keep the
main nginx.conf as a template).
Debian 9 is still in beta and doesn't have nginx.org builds yet, so
we need to use the nginx package in Debian's repositories, and that
required a bit of a different configuration.
After official nginx.org builds are released we can revert this.
I'm surprised this worked all these years actually. Since Ansible
version 1.6 it has been possible to use the version_compare filter
instead of doing math logic on strings.
See: https://docs.ansible.com/ansible/playbooks_tests.html
I realized the other day that due to complex logic in the location
blocks, various WordPress static files like images and stylesheets
didn't get the HTTP Strict Transport Security header set. We need
to include it on each level where we are setting headers, because
nginx overwrites headers if you set them again in a child block.
There was some knowledge floating around that 860 bytes was the
optimal size, I think it was from an Akamai engineer or something,
but the HTML 5 Boilerplate server configs use 256 bytes, and I
actually have HTML content that is less than 860 bytes, so I guess
I could benefit from compressing it. gzip compression is costly
for the compression side, but very quick for the client, so this
is a good thing.
See: https://github.com/h5bp/server-configs-nginx/blob/master/nginx.conf
The variable name is misleading as this really does is install the
certbot client and its dependencies, and we generally want this to
always happen. If a host doesn't want it, they can override it in
their host vars.
Perhaps I should rename this variable to "bootstrap_letsencrypt" or
something so it is more accurate.
The `ConditionFileIsExecutable` goes in the [Unit] section! This
fixes the error:
systemd[1]: [/etc/systemd/system/renew-letsencrypt.service:6] Unknown lvalue 'ConditionFileIsExecutable' in section 'Service'
Only vhosts running WordPress, etc need PHP. Make sure to set the
appopriate variables for each vhost in the host's vars, ie:
nginx_vhosts:
- domain_name: example.com
has_wordpress: True
- domain_name: example.net
needs_php: True
You can set either of them, but not both—needing WordPress implies
needing PHP.
Used to indicate if a vhost needs PHP configuration or not, like
for a static site. Set in the hosts's nginx_vhosts block. Defaults
to "False" if unset.
This reverts commit 201165cff6.
Turns out this actually breaks initial deployments, because the
cache gets updated in the first task, then you add sources for
nginx and mariadb, but it doesn't update the indexes because the
cache is < 3600 seconds old, so you end up getting the distro's
versions of nginx and mariadb.
When you give Ansible the key id it will check if the key exists
before trying to download and add it. I got the long fingerprint
from `sudo apt-key finger`.
When I originally deployed Ubuntu 16.04 there were no public xenial
MariaDB builds yet, so I used wily (15.10). Now there are official
xenial builds so we can use them instead.
I have added cache_valid_time=3600 for the first task in each
tag that could be possibly running apt-related commands. For ex,
the "nginx" tag is also in the "packages" tag, but sometimes you
run the nginx tag by itself (perhaps repeatadely), so you'd want
to limit the update unless the cache was 1 hour old
I never modify file in the git repo, but the WordPress updater does
updates from the web (for example TwentySixteen theme), and this
always causes the task to fail when I switch WordPress versions.
This reverts commit a38d822fad.
The docs definitely recommend twice a day. From a note on certbot's
installation page:
> if you're setting up a cron or systemd job, we recommend running
> it twice per day (it won't do anything until your certificates
> are due for renewal or revoked, but running it regularly would
> give your site a chance of staying online in case a Let's
> Encrypt-initiated revocation happened for some reason). Please
> select a random minute within the hour for your renewal tasks.
See: https://certbot.eff.org/#ubuntuxenial-nginx
I had removed them from Debian 8 and Ubuntu 14.04 configs last year
when the NSA's Suite B crypto guidelines dropped 128-bit algorithms
but those changes didn't make it to my new Ubuntu 16.04 config.
It is probably overkill and paranoid, but this server is mine, so I
can make those decisions (and I only connect from modern clients).
For idempotence we need to run all apt-related tasks, like editing
source files, adding keys, installing packages, etc, when running
the 'packages' tag.
Take an opinionated stance on HTTPS and assume that hosts are using
HTTPS for all vhosts. This can either be via custom TLS cert/key
pairs defined in the host's variables (could even be self-signed
certificates on dev boxes) or via Let's Encrypt.
Hosts can specify use_letsencrypt: 'yes' in their host_vars. For
now this assumes that the certificates already exist (ie, you have
to manually run Let's Encrypt first to register/create the certs).
Alan Orth