diff options
author | Jakob Odersky <jakob@odersky.com> | 2018-12-04 21:31:01 -0800 |
---|---|---|
committer | Jakob Odersky <jakob@odersky.com> | 2018-12-04 21:39:07 -0800 |
commit | 9588e9366d3455f203e5482a41f712777595bb13 (patch) | |
tree | 272aeababb1b68f477301d67198a82c80d044c01 | |
parent | db27247dd7d7209ab93419eb33d2ecb21e74c1ec (diff) | |
download | infra-9588e9366d3455f203e5482a41f712777595bb13.tar.gz infra-9588e9366d3455f203e5482a41f712777595bb13.tar.bz2 infra-9588e9366d3455f203e5482a41f712777595bb13.zip |
Simplify terraform and provisioning scripts. Move away from config packages.
47 files changed, 331 insertions, 749 deletions
diff --git a/Makefile b/Makefile deleted file mode 100644 index e52ddd7..0000000 --- a/Makefile +++ /dev/null @@ -1,5 +0,0 @@ -all: - $(MAKE) -C packages all - (cd terraform && bash -c "terraform apply -var-file=<(pass infra/terraform)") - -.PHONY: all @@ -1,74 +1,3 @@ ## Quasi-Immutable Infrastructure for Small-Scale Deployments -This project combines [terraform](https://www.terraform.io) with -[debian configuration packages](https://wiki.debian.org/Packaging) to -manage infrastructure automatically. - -It has a very opinionated structure, ideal for managing a small amount -of servers. - -It is used to manage the crashbox.io services. - -## Overview - -Management of infrastructure revolves around two central concepts: - -1. Provisioning of infrastructure, such as virtual private - servers and DNS entries, with terraform. - -2. Configuration of servers with debian packages. Custom debian - packages integrate easily into the debian ecosystem and provide a - robust way of managing files. - -These two concepts are brought together by *roles* which aggregate DNS -entries and packages. - -**In a nutshell, all infrastructure is configured by assigning sets of -roles to servers. A role will apply a debian configuration package to -a server and create a CNAME to the server's A record.** - -For example, assigning the `ip` role to server `server.crashbox.io` will: - -1. Create the server and A record if it doesn't already exist. -2. Install the package `crashbox-ip-config` on the server. -3. Create a DNS CNAME, aliasing `ip.crashbox.io` to `server.crashbox.io`. - -In the given example, the ip-config package will ensure a webserver is -installed and configure it to serve an ip address echo website. - -## Structure - -- Provisioning scripts are in `terraform/`. - -- Configuration package sources are in `packages/`. - -## Running - -### Bootstrap - -Before infrastructure configuration can be automated, a couple of -bootstrapping steps need to be performed manually: - -0. Create accounts for the various providers specified in the - terraform configuration. - -1. Provision a storage space for the terraform state file. - -2. Install dependencies for this project: - - pass - - terraform - See notes in `packages/` for additional dependencies. - -### Apply - -Run `make` to apply configuration. - -## Note about immutability - -This project uses debian packages for stronger consistency guarantees -when removing packages. Nevertheless, it is recommended to completely -reprovision a server if a role is removed. (It isn't necessary to -reprovision if a role is added.) - -Keeping in mind that the goal of this project is to automate -deployments, regular reprovisions are encouraged. +Used to manage the crashbox.io services. diff --git a/packages/.dockerignore b/packages/.dockerignore deleted file mode 100644 index 1c44263..0000000 --- a/packages/.dockerignore +++ /dev/null @@ -1,3 +0,0 @@ -target/build/ -target/base.img -target/snapshot.qcow2 diff --git a/packages/Dockerfile b/packages/Dockerfile deleted file mode 100644 index c09ce4a..0000000 --- a/packages/Dockerfile +++ /dev/null @@ -1,25 +0,0 @@ -# Bare image for testing config packages. -# -# This image is configured to use apt-cacher-ng running on the host -# machine, and hence is designed to not require an internet connection -# to build and run (assuming the base image is cached). -# -# To enable access from a running container to the host's -# apt-cacher-ng, the following firewall rule is required: -# -# iptables -A INPUT -i docker0 -p tcp --dport 3142 -j ACCEPT - -FROM crashbox-config-base:latest - -RUN echo 'Acquire::http::Proxy "http://172.17.0.1:3142";' \ - > /etc/apt/apt.conf.d/00aptproxy - -COPY target/archive/ /usr/local/share/archive/ -RUN echo "deb [trusted=yes] file:/usr/local/share/archive ./" \ - > /etc/apt/sources.list.d/local-archive.list -RUN apt-get update \ - -o Dir::Etc::sourcelist="sources.list.d/local-archive.list" \ - -o Dir::Etc::sourceparts="-" \ - -o APT::Get::List-Cleanup="0" - -CMD ["bash"] diff --git a/packages/Dockerfile.base b/packages/Dockerfile.base deleted file mode 100644 index a188b76..0000000 --- a/packages/Dockerfile.base +++ /dev/null @@ -1,11 +0,0 @@ -# Base image for the bare image for testing config packages (see Dockerfile). -# Note that building this image requires an internet connection. - -FROM debian:buster - -RUN echo 'deb http://httpredir.debian.org/debian buster main' > /etc/apt/sources.list -RUN echo 'Acquire::ForceIPv4 "true";' > /etc/apt/apt.conf.d/99force-ipv4 -RUN apt-get update -RUN apt-get install curl -y - -CMD ["bash"] diff --git a/packages/README.md b/packages/README.md deleted file mode 100644 index 6f407e9..0000000 --- a/packages/README.md +++ /dev/null @@ -1,54 +0,0 @@ -# Config Packages - -This directory contains sources and scripts for building and testing -Debian config packages for common server roles, tailored to the -conventions used for managing crashbox.io. - -All packages are built from a single [debian source -package](https://wiki.debian.org/Packaging/SourcePackage) in -`crashbox-config`. - -Although this source package can be built in the regular way (e.g. by -cd'ing into debian and running `debuild`), a makefile is also provided -for cleaner builds and testing the packages. Some notable makefile -targets are: - -- `all`: builds a package archive (that may be used as a "deb" - source for `apt`) - -- `run`: starts a virtual machine with the package archive configured - as an additional "deb" source. Config packages may be tested by - running `apt update` and `apt install <package>`. - -## Dependencies - -Config packages declare their own dependencies, however the following -packages are required to get started: - -```bash -apt install \ - apt-cacher-ng \ - build-essential \ - devscripts \ - lintian \ - qemu-kvm \ - vmdebootstrap -``` -## Example - -1. `make run`: builds packages, bundles tem in an archive and starts a - virtual machine. Note that running this first time may take a while - a require and require significatnt bandwidth, since the base image - of the virtual machine needs to be bootstrapped. - -2. login with `root` (no password) - -3. `apt update`: update apt's list of available packages - -4. `apt install crashbox-ip-config`: configure the vm to become an - "ip-echo" service. This will install and configure dependent - packages too, such as `crashbox-nginx-config` and - `crshbox-base-config`. - -5. back on the host, visit `https://ip.localhost:10443` to confirm the - service os running diff --git a/packages/crashbox-config/debian/changelog b/packages/crashbox-config/debian/changelog deleted file mode 100644 index 4991b8d..0000000 --- a/packages/crashbox-config/debian/changelog +++ /dev/null @@ -1,5 +0,0 @@ -crashbox-config (1) unstable; urgency=medium - - * Initial Release. - - -- Jakob Odersky <infra@crashbox.io> Tue, 28 Aug 2018 21:47:21 -0700 diff --git a/packages/crashbox-config/debian/compat b/packages/crashbox-config/debian/compat deleted file mode 100644 index b4de394..0000000 --- a/packages/crashbox-config/debian/compat +++ /dev/null @@ -1 +0,0 @@ -11 diff --git a/packages/crashbox-config/debian/control b/packages/crashbox-config/debian/control deleted file mode 100644 index aacca52..0000000 --- a/packages/crashbox-config/debian/control +++ /dev/null @@ -1,38 +0,0 @@ -Source: crashbox-config -Section: admin -Priority: optional -Maintainer: Jakob Odersky <infra@crashbox.io> -Build-Depends: debhelper (>= 11) -Standards-Version: 4.1.3 - -Package: crashbox-base-config -Architecture: any -Depends: ${shlibs:Depends}, ${misc:Depends}, apt-listchanges, ca-certificates, curl, jq, openssl, rsync, ufw, unattended-upgrades, wget, sudo -Provides: ${diverted-files} -Conflicts: ${diverted-files} -Description: configuration for base system - Adds local customizations to the base system configuration. - -Package: crashbox-nginx-config -Architecture: any -Depends: ${shlibs:Depends}, ${misc:Depends}, ssl-cert, nginx, crashbox-base-config -Provides: ${diverted-files} -Conflicts: ${diverted-files} -Description: local nginx configuration - Adds local customizations to nginx config - -Package: crashbox-ip-config -Architecture: any -Depends: ${shlibs:Depends}, ${misc:Depends}, crashbox-nginx-config -Provides: ${diverted-files} -Conflicts: ${diverted-files} -Description: what-is-my-ip website - Adds an nginx site that echoes back a remote IP address - -Package: crashbox-git-config -Architecture: any -Depends: ${shlibs:Depends}, ${misc:Depends}, crashbox-nginx-config, cgit, python3-pygments, python3-markdown, git-core, fcgiwrap, adduser -Provides: ${diverted-files} -Conflicts: ${diverted-files} -Description: cgit web interface - Adds an nginx site that serves a CGit instance
\ No newline at end of file diff --git a/packages/crashbox-config/debian/copyright b/packages/crashbox-config/debian/copyright deleted file mode 100644 index ac7fbf4..0000000 --- a/packages/crashbox-config/debian/copyright +++ /dev/null @@ -1,27 +0,0 @@ -Format: https://www.debian.org/doc/packaging-manuals/copyright-format/1.0/ -Upstream-Name: crashbox-config - -Files: * -Copyright: 2018 Jakob Odersky <jakob@odersky.com> -License: GPL-3.0+ - -Files: debian/* -Copyright: 2018 Jakob Odersky <jakob@odersky.com> -License: GPL-3.0+ - -License: GPL-3.0+ - This program is free software: you can redistribute it and/or modify - it under the terms of the GNU General Public License as published by - the Free Software Foundation, either version 3 of the License, or - (at your option) any later version. - . - This package is distributed in the hope that it will be useful, - but WITHOUT ANY WARRANTY; without even the implied warranty of - MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - GNU General Public License for more details. - . - You should have received a copy of the GNU General Public License - along with this program. If not, see <https://www.gnu.org/licenses/>. - . - On Debian systems, the complete text of the GNU General - Public License version 3 can be found in "/usr/share/common-licenses/GPL-3".
\ No newline at end of file diff --git a/packages/crashbox-config/debian/crashbox-base-config.install b/packages/crashbox-config/debian/crashbox-base-config.install deleted file mode 100644 index ef80655..0000000 --- a/packages/crashbox-config/debian/crashbox-base-config.install +++ /dev/null @@ -1 +0,0 @@ -base/20auto-upgrades etc/apt/apt.conf.d/ diff --git a/packages/crashbox-config/debian/crashbox-base-config.postinst b/packages/crashbox-config/debian/crashbox-base-config.postinst deleted file mode 100644 index b48f01f..0000000 --- a/packages/crashbox-config/debian/crashbox-base-config.postinst +++ /dev/null @@ -1,42 +0,0 @@ -#!/bin/sh -# postinst script for crashbox-base-config -# -# see: dh_installdeb(1) - -set -e - -# summary of how this script can be called: -# * <postinst> `configure' <most-recently-configured-version> -# * <old-postinst> `abort-upgrade' <new version> -# * <conflictor's-postinst> `abort-remove' `in-favour' <package> -# <new-version> -# * <postinst> `abort-remove' -# * <deconfigured's-postinst> `abort-deconfigure' `in-favour' -# <failed-install-package> <version> `removing' -# <conflicting-package> <version> -# for details, see https://www.debian.org/doc/debian-policy/ or -# the debian-policy package - - -case "$1" in - configure) - ufw allow 22/tcp || true - ufw default deny || true - ufw --force enable || true - ;; - - abort-upgrade|abort-remove|abort-deconfigure) - ;; - - *) - echo "postinst called with unknown argument \`$1'" >&2 - exit 1 - ;; -esac - -# dh_installdeb will replace this with shell code automatically -# generated by other debhelper scripts. - -#DEBHELPER# - -exit 0 diff --git a/packages/crashbox-config/debian/crashbox-git-config.cron.d b/packages/crashbox-config/debian/crashbox-git-config.cron.d deleted file mode 100644 index d9cadfd..0000000 --- a/packages/crashbox-config/debian/crashbox-git-config.cron.d +++ /dev/null @@ -1 +0,0 @@ -0 0 * * * git /usr/bin/gh-mirror-all
\ No newline at end of file diff --git a/packages/crashbox-config/debian/crashbox-git-config.install b/packages/crashbox-config/debian/crashbox-git-config.install deleted file mode 100644 index a7d3e36..0000000 --- a/packages/crashbox-config/debian/crashbox-git-config.install +++ /dev/null @@ -1,3 +0,0 @@ -git/etc/* etc -git/usr/* usr -git/var/* var diff --git a/packages/crashbox-config/debian/crashbox-git-config.postinst b/packages/crashbox-config/debian/crashbox-git-config.postinst deleted file mode 100644 index 774869e..0000000 --- a/packages/crashbox-config/debian/crashbox-git-config.postinst +++ /dev/null @@ -1,45 +0,0 @@ -#!/bin/sh -# -# see: dh_installdeb(1) - -set -e - -# summary of how this script can be called: -# * <postinst> `configure' <most-recently-configured-version> -# * <old-postinst> `abort-upgrade' <new version> -# * <conflictor's-postinst> `abort-remove' `in-favour' <package> -# <new-version> -# * <postinst> `abort-remove' -# * <deconfigured's-postinst> `abort-deconfigure' `in-favour' -# <failed-install-package> <version> `removing' -# <conflicting-package> <version> -# for details, see https://www.debian.org/doc/debian-policy/ or -# the debian-policy package - - -case "$1" in - configure) - adduser --group --system --home /var/lib/git git - mkdir -p /srv/git - chown -R git:git /srv/git - mkdir -p /var/lib/git/www/ - ln -s /usr/share/cgit/cgit.css /var/lib/git/www/cgit.css - ln -s /usr/share/cgit/robots.txt /var/lib/git/www/robots.txt - deb-systemd-invoke restart nginx - ;; - - abort-upgrade|abort-remove|abort-deconfigure) - ;; - - *) - echo "postinst called with unknown argument \`$1'" >&2 - exit 1 - ;; -esac - -# dh_installdeb will replace this with shell code automatically -# generated by other debhelper scripts. - -#DEBHELPER# - -exit 0 diff --git a/packages/crashbox-config/debian/crashbox-ip-config.install b/packages/crashbox-config/debian/crashbox-ip-config.install deleted file mode 100644 index 2646928..0000000 --- a/packages/crashbox-config/debian/crashbox-ip-config.install +++ /dev/null @@ -1 +0,0 @@ -ip/ip.conf etc/nginx/sites-enabled/ diff --git a/packages/crashbox-config/debian/crashbox-ip-config.postinst b/packages/crashbox-config/debian/crashbox-ip-config.postinst deleted file mode 100644 index 90e58d6..0000000 --- a/packages/crashbox-config/debian/crashbox-ip-config.postinst +++ /dev/null @@ -1,40 +0,0 @@ -#!/bin/sh -# postinst script for crashbox-ip-config -# -# see: dh_installdeb(1) - -set -e - -# summary of how this script can be called: -# * <postinst> `configure' <most-recently-configured-version> -# * <old-postinst> `abort-upgrade' <new version> -# * <conflictor's-postinst> `abort-remove' `in-favour' <package> -# <new-version> -# * <postinst> `abort-remove' -# * <deconfigured's-postinst> `abort-deconfigure' `in-favour' -# <failed-install-package> <version> `removing' -# <conflicting-package> <version> -# for details, see https://www.debian.org/doc/debian-policy/ or -# the debian-policy package - - -case "$1" in - configure) - deb-systemd-invoke restart nginx - ;; - - abort-upgrade|abort-remove|abort-deconfigure) - ;; - - *) - echo "postinst called with unknown argument \`$1'" >&2 - exit 1 - ;; -esac - -# dh_installdeb will replace this with shell code automatically -# generated by other debhelper scripts. - -#DEBHELPER# - -exit 0 diff --git a/packages/crashbox-config/debian/crashbox-nginx-config.install b/packages/crashbox-config/debian/crashbox-nginx-config.install deleted file mode 100644 index f2ed0d3..0000000 --- a/packages/crashbox-config/debian/crashbox-nginx-config.install +++ /dev/null @@ -1 +0,0 @@ -nginx/etc/* etc diff --git a/packages/crashbox-config/debian/crashbox-nginx-config.postinst b/packages/crashbox-config/debian/crashbox-nginx-config.postinst deleted file mode 100644 index 7a22244..0000000 --- a/packages/crashbox-config/debian/crashbox-nginx-config.postinst +++ /dev/null @@ -1,54 +0,0 @@ -#!/bin/sh -# postinst script for crashbox-nginx-config -# -# see: dh_installdeb(1) - -set -e - -# summary of how this script can be called: -# * <postinst> `configure' <most-recently-configured-version> -# * <old-postinst> `abort-upgrade' <new version> -# * <conflictor's-postinst> `abort-remove' `in-favour' <package> -# <new-version> -# * <postinst> `abort-remove' -# * <deconfigured's-postinst> `abort-deconfigure' `in-favour' -# <failed-install-package> <version> `removing' -# <conflicting-package> <version> -# for details, see https://www.debian.org/doc/debian-policy/ or -# the debian-policy package - - -case "$1" in - configure) - ln -f -s /etc/nginx/sites-available/default.conf /etc/nginx/sites-enabled/default - usermod --append --groups ssl-cert www-data - ufw allow 80/tcp - ufw allow 443/tcp - - if [ ! -r /etc/ssl/private/server.key.pem ] \ - || [ ! -r /etc/ssl/server.cert.pem ] \ - || [ ! -r /etc/ssl/issuer.cert.pem ]; then - ln -f -s /etc/ssl/private/ssl-cert-snakeoil.key /etc/ssl/private/server.key.pem - ln -f -s /etc/ssl/certs/ssl-cert-snakeoil.pem /etc/ssl/server.cert.pem - ln -f -s /etc/ssl/certs/ssl-cert-snakeoil.pem /etc/ssl/issuer.cert.pem - echo "WARNING: no certificates found, falling back to snakeoil certificates!" >&2 - fi - - deb-systemd-invoke restart nginx - ;; - - abort-upgrade|abort-remove|abort-deconfigure) - ;; - - *) - echo "postinst called with unknown argument \`$1'" >&2 - exit 1 - ;; -esac - -# dh_installdeb will replace this with shell code automatically -# generated by other debhelper scripts. - -#DEBHELPER# - -exit 0 diff --git a/packages/crashbox-config/debian/rules b/packages/crashbox-config/debian/rules deleted file mode 100755 index 9946432..0000000 --- a/packages/crashbox-config/debian/rules +++ /dev/null @@ -1,18 +0,0 @@ -#!/usr/bin/make -f -# See debhelper(7) (uncomment to enable) -# output every command that modifies files on the build system. -#export DH_VERBOSE = 1 - - -# see FEATURE AREAS in dpkg-buildflags(1) -#export DEB_BUILD_MAINT_OPTIONS = hardening=+all - -# see ENVIRONMENT in dpkg-buildflags(1) -# package maintainers to append CFLAGS -#export DEB_CFLAGS_MAINT_APPEND = -Wall -pedantic -# package maintainers to append LDFLAGS -#export DEB_LDFLAGS_MAINT_APPEND = -Wl,--as-needed - - -%: - dh $@ diff --git a/packages/crashbox-config/debian/source/format b/packages/crashbox-config/debian/source/format deleted file mode 100644 index 89ae9db..0000000 --- a/packages/crashbox-config/debian/source/format +++ /dev/null @@ -1 +0,0 @@ -3.0 (native) diff --git a/terraform/deploy b/terraform/deploy new file mode 100755 index 0000000..a7fdf57 --- /dev/null +++ b/terraform/deploy @@ -0,0 +1,3 @@ +#!/bin/bash + +terraform apply -var-file=<(pass infra/terraform) diff --git a/terraform/main.tf b/terraform/main.tf index b25598e..e29cf6a 100755..100644 --- a/terraform/main.tf +++ b/terraform/main.tf @@ -16,36 +16,128 @@ provider "cloudflare" { } provider "acme" { - #server_url = "https://acme-staging-v02.api.letsencrypt.org/directory" server_url = "https://acme-v02.api.letsencrypt.org/directory" } -################################################################################ +############################################################ resource "hcloud_ssh_key" "root" { name = "root" - public_key = "${file("~/.ssh/id_rsa.pub")}" + public_key = "${file("root-ssh-key")}" } -module "vps" { - source = "./stdvps" - location = "nbg1" - ssh_key_name = "${hcloud_ssh_key.root.name}" +resource "tls_private_key" "private_key" { + algorithm = "RSA" +} + +resource "acme_registration" "registration" { + account_key_pem = "${tls_private_key.private_key.private_key_pem}" + email_address = "jakob@odersky.com" +} + +resource "acme_certificate" "certificate" { + account_key_pem = "${acme_registration.registration.account_key_pem}" + common_name = "crashbox.io" + + subject_alternative_names = [ + "ip.crashbox.io", + "git.crashbox.io", + ] + + dns_challenge { + provider = "cloudflare" + + config { + CLOUDFLARE_EMAIL = "jakob@odersky.com" + CLOUDFLARE_API_KEY = "${var.secret_cloudflare_token}" + } + } +} + +resource "cloudflare_record" "record_caa" { + domain = "crashbox.io" + name = "crashbox.io" + + data = { + flags = "0" + tag = "issue" + value = "letsencrypt.org" + } + + type = "CAA" +} + +resource "random_id" "peter" { + prefix = "peter-" + byte_length = 2 +} + +resource "hcloud_server" "peter" { + name = "${random_id.peter.hex}" + image = "debian-9" + server_type = "cx11" + location = "nbg1" + ssh_keys = ["${hcloud_ssh_key.root.name}"] + + provisioner "file" { + content = "${acme_certificate.certificate.private_key_pem}" + destination = "/etc/ssl/private/server.key.pem" + } + + provisioner "file" { + content = "${acme_certificate.certificate.certificate_pem}" + destination = "/etc/ssl/server.cert.pem" + } + + provisioner "file" { + content = "${acme_certificate.certificate.issuer_pem}" + destination = "/etc/ssl/issuer.cert.pem" + } + + provisioner "file" { + source = "./provision" + destination = "/usr/local/share/" + } + + provisioner "remote-exec" { + inline = [ + "chmod +x /usr/local/share/provision/provision", + "/usr/local/share/provision/provision --force", + ] + } +} + +module "peter_mount_volume" { + source = "./mount_volume" volume_name = "master" + host = "${hcloud_server.peter.ipv4_address}" + server_id = "${hcloud_server.peter.id}" +} + +resource "cloudflare_record" "peter_a" { + domain = "crashbox.io" + name = "${hcloud_server.peter.name}" + value = "${hcloud_server.peter.ipv4_address}" + type = "A" } -module "roles" { - source = "./role" - secret_cloudflare_token = "${var.secret_cloudflare_token}" - host = "${module.vps.fqdn}" - id = "${module.vps.id}" - roles = ["ip", "git"] +resource "cloudflare_record" "peter_aaaa" { + domain = "crashbox.io" + name = "${hcloud_server.peter.name}" + value = "${hcloud_server.peter.ipv6_address}1" + type = "AAAA" } -output "vps_address" { - value = "${module.vps.fqdn}" +resource "cloudflare_record" "record_ip" { + domain = "crashbox.io" + name = "ip" + value = "${cloudflare_record.peter_a.hostname}" + type = "CNAME" } -output "vps_roles" { - value = "${join(" ", module.roles.roles)}" +resource "cloudflare_record" "record_git" { + domain = "crashbox.io" + name = "git" + value = "${cloudflare_record.peter_a.hostname}" + type = "CNAME" } diff --git a/terraform/mount_volume/main.tf b/terraform/mount_volume/main.tf new file mode 100644 index 0000000..aed5324 --- /dev/null +++ b/terraform/mount_volume/main.tf @@ -0,0 +1,104 @@ +variable "volume_name" { + type = "string" +} + +variable "host" { + type = "string" +} + +variable "server_id" { + type = "string" +} + +# volumes contain persistent storage and thus need to be initialized +# manually +data "hcloud_volume" "master" { + name = "${var.volume_name}" +} + +resource "hcloud_volume_attachment" "master_attachment" { + volume_id = "${data.hcloud_volume.master.id}" + server_id = "${var.server_id}" +} + +resource "null_resource" "volume_mount" { + triggers = { + attachement_id = "${hcloud_volume_attachment.master_attachment.id}" + } + + connection { + host = "${var.host}" + } + + provisioner "remote-exec" { + inline = ["mkdir -p /mnt/storage"] + } + + provisioner "file" { + content = <<EOF +[Unit] +Description=Mount /mnt/storage directory + +[Mount] +What=${data.hcloud_volume.master.linux_device} +Where=/mnt/storage +Type=ext4 +Options=defaults + +[Install] +WantedBy=multi-user.target +EOF + + destination = "/etc/systemd/system/mnt-storage.mount" + } + + provisioner "file" { + content = <<EOF +[Unit] +Description=Mount /srv to persistent volume storage +After=mnt-storage.mount +BindsTo=mnt-storage.mount + +[Mount] +What=/mnt/storage/srv +Where=/srv +Type=ext4 +Options=bind + +[Install] +WantedBy=multi-user.target +EOF + + destination = "/etc/systemd/system/srv.mount" + } + + provisioner "file" { + content = <<EOF +[Unit] +Description=Mount /home to persistent volume storage +After=mnt-storage.mount +BindsTo=mnt-storage.mount + +[Mount] +What=/mnt/storage/home +Where=/home +Type=ext4 +Options=bind + +[Install] +WantedBy=multi-user.target +EOF + + destination = "/etc/systemd/system/home.mount" + } + + provisioner "remote-exec" { + inline = [ + "systemctl daemon-reload", + "systemctl enable mnt-storage.mount", + "systemctl enable srv.mount", + "systemctl enable home.mount", + "systemctl reboot", + ] + } +} diff --git a/terraform/provision/provision b/terraform/provision/provision new file mode 100755 index 0000000..3a32a63 --- /dev/null +++ b/terraform/provision/provision @@ -0,0 +1,81 @@ +#!/bin/bash + +panic() { + echo "$1" >&2 + echo "Aborting." + exit 1 +} + +[[ $1 == "--force" ]] || panic "Must be run with --force" +[[ $(id --user) -eq 0 ]] || panic "This script must be run as root." + +log() { + echo "provision: $1" >&2 +} + +log "install and configure most essential packages" +apt-get update --quiet=2 +apt-get install --yes --quiet=2 ufw +ufw allow 22/tcp +ufw default deny +ufw --force enable + +log "install service packages" +apt-get install --yes --quiet=2 \ + adduser \ + apt-listchanges \ + ca-certificates \ + cgit \ + curl \ + fcgiwrap \ + git-core \ + jq \ + nginx \ + openssl \ + python3-markdown \ + python3-pygments \ + rsync \ + ssl-cert \ + sudo \ + ufw \ + unattended-upgrades \ + wget + +log "copy package configurations" +rsync -r /usr/local/share/provision/rootfs/ / + +log "ensure certificate bundle exists" +# the ceritifcate bundle should be provisioned by terraform, however +# for testing purposes (such as in a vm) this copies the default +# "snakeoil" test certificates to the appropriate locations if they do +# not already exist +if [[ ! -r /etc/ssl/private/server.key.pem ]] \ + || [[ ! -r /etc/ssl/server.cert.pem ]] \ + || [[ ! -r /etc/ssl/issuer.cert.pem ]]; then + ln -f -s /etc/ssl/private/ssl-cert-snakeoil.key /etc/ssl/private/server.key.pem + ln -f -s /etc/ssl/certs/ssl-cert-snakeoil.pem /etc/ssl/server.cert.pem + ln -f -s /etc/ssl/certs/ssl-cert-snakeoil.pem /etc/ssl/issuer.cert.pem + log "WARNING: no certificates found, falling back to snakeoil certificates!" +fi + +log "configure nginx" +rm -r /etc/nginx/sites-enabled/default +usermod --append --groups ssl-cert www-data +ufw allow 80/tcp +ufw allow 443/tcp + +log "configure git" +adduser --group --system --home /var/lib/git git +mkdir -p /srv/git +chown -R git:git /srv/git +mkdir -p /var/lib/git/www/ +ln -s /usr/share/cgit/cgit.css /var/lib/git/www/cgit.css +ln -s /usr/share/cgit/robots.txt /var/lib/git/www/robots.txt + +log "configure shell accounts" +adduser --uid 1000 --disabled-password --gecos "" jodersky + +log "restart services" +systemctl restart nginx + +log "configuration complete!" diff --git a/packages/crashbox-config/base/20auto-upgrades b/terraform/provision/rootfs/etc/apt/apt.conf.d/20auto-upgrades index 8d6d7c8..8d6d7c8 100644 --- a/packages/crashbox-config/base/20auto-upgrades +++ b/terraform/provision/rootfs/etc/apt/apt.conf.d/20auto-upgrades diff --git a/packages/crashbox-config/git/etc/cgitrc.d/crashbox b/terraform/provision/rootfs/etc/cgitrc.d/crashbox index fdafab6..fdafab6 100644 --- a/packages/crashbox-config/git/etc/cgitrc.d/crashbox +++ b/terraform/provision/rootfs/etc/cgitrc.d/crashbox diff --git a/packages/crashbox-config/git/etc/gh-mirror b/terraform/provision/rootfs/etc/gh-mirror index 4fc987b..4fc987b 100644 --- a/packages/crashbox-config/git/etc/gh-mirror +++ b/terraform/provision/rootfs/etc/gh-mirror diff --git a/packages/crashbox-config/nginx/etc/nginx/conf.d/ssl.conf b/terraform/provision/rootfs/etc/nginx/conf.d/ssl.conf index bb96ec7..bb96ec7 100644 --- a/packages/crashbox-config/nginx/etc/nginx/conf.d/ssl.conf +++ b/terraform/provision/rootfs/etc/nginx/conf.d/ssl.conf diff --git a/packages/crashbox-config/nginx/etc/nginx/sites-available/default.conf b/terraform/provision/rootfs/etc/nginx/sites-enabled/default.conf index e10725d..e10725d 100644 --- a/packages/crashbox-config/nginx/etc/nginx/sites-available/default.conf +++ b/terraform/provision/rootfs/etc/nginx/sites-enabled/default.conf diff --git a/packages/crashbox-config/git/etc/nginx/sites-enabled/git.conf b/terraform/provision/rootfs/etc/nginx/sites-enabled/git.conf index 7210dbc..7210dbc 100644 --- a/packages/crashbox-config/git/etc/nginx/sites-enabled/git.conf +++ b/terraform/provision/rootfs/etc/nginx/sites-enabled/git.conf diff --git a/packages/crashbox-config/ip/ip.conf b/terraform/provision/rootfs/etc/nginx/sites-enabled/ip.conf index 2f3ab1e..2f3ab1e 100644 --- a/packages/crashbox-config/ip/ip.conf +++ b/terraform/provision/rootfs/etc/nginx/sites-enabled/ip.conf diff --git a/packages/crashbox-config/git/usr/bin/gh-mirror b/terraform/provision/rootfs/usr/bin/gh-mirror index 54985cb..54985cb 100755 --- a/packages/crashbox-config/git/usr/bin/gh-mirror +++ b/terraform/provision/rootfs/usr/bin/gh-mirror diff --git a/packages/crashbox-config/git/usr/bin/gh-mirror-all b/terraform/provision/rootfs/usr/bin/gh-mirror-all index fa9054f..fa9054f 100755 --- a/packages/crashbox-config/git/usr/bin/gh-mirror-all +++ b/terraform/provision/rootfs/usr/bin/gh-mirror-all diff --git a/packages/crashbox-config/git/var/lib/git/www/about.md b/terraform/provision/rootfs/var/lib/git/www/about.md index 55e68fa..55e68fa 100644 --- a/packages/crashbox-config/git/var/lib/git/www/about.md +++ b/terraform/provision/rootfs/var/lib/git/www/about.md diff --git a/packages/crashbox-config/git/var/lib/git/www/crashbox.svg b/terraform/provision/rootfs/var/lib/git/www/crashbox.svg index 87ff69c..87ff69c 100644 --- a/packages/crashbox-config/git/var/lib/git/www/crashbox.svg +++ b/terraform/provision/rootfs/var/lib/git/www/crashbox.svg diff --git a/packages/crashbox-config/git/var/lib/git/www/instagram.png b/terraform/provision/rootfs/var/lib/git/www/instagram.png Binary files differindex dcaff14..dcaff14 100644 --- a/packages/crashbox-config/git/var/lib/git/www/instagram.png +++ b/terraform/provision/rootfs/var/lib/git/www/instagram.png diff --git a/terraform/role/README.md b/terraform/role/README.md deleted file mode 100644 index 11e2e21..0000000 --- a/terraform/role/README.md +++ /dev/null @@ -1,4 +0,0 @@ -# Role-based configuration for standalone hosts. - -Applying a role to a host will install corresponding config packages -and create a role CNAME record to the host. diff --git a/terraform/role/main.tf b/terraform/role/main.tf deleted file mode 100644 index e85fd3b..0000000 --- a/terraform/role/main.tf +++ /dev/null @@ -1,91 +0,0 @@ -variable "host" { - type = "string" -} - -variable "id" { - type = "string" -} - -variable "roles" { - type = "list" -} - -variable "secret_cloudflare_token" { - type = "string" -} - -resource "tls_private_key" "private_key" { - algorithm = "RSA" -} - -resource "acme_registration" "reg" { - account_key_pem = "${tls_private_key.private_key.private_key_pem}" - email_address = "jakob@odersky.com" -} - -resource "acme_certificate" "certificate" { - account_key_pem = "${acme_registration.reg.account_key_pem}" - common_name = "${var.host}" - subject_alternative_names = "${formatlist("%s.crashbox.io", var.roles)}" - - dns_challenge { - provider = "cloudflare" - - config { - CLOUDFLARE_EMAIL = "jakob@odersky.com" - CLOUDFLARE_API_KEY = "${var.secret_cloudflare_token}" - } - } -} - -resource "cloudflare_record" "role_cname" { - count = "${length(var.roles)}" - - domain = "crashbox.io" - name = "${element(var.roles, count.index)}" - value = "${var.host}" - type = "CNAME" -} - -resource "null_resource" "role_config" { - triggers = { - host_id = "${var.id}" - config_packages = "${join(" ", sort(formatlist("crashbox-%s-config", var.roles)))}" - } - - connection { - host = "${var.host}" - } - - provisioner "file" { - content = "${acme_certificate.certificate.certificate_pem}" - destination = "/etc/ssl/server.cert.pem" - } - - provisioner "file" { - content = "${acme_certificate.certificate.issuer_pem}" - destination = "/etc/ssl/issuer.cert.pem" - } - - provisioner "file" { - content = "${acme_certificate.certificate.private_key_pem}" - destination = "/etc/ssl/private/server.key.pem" - } - - provisioner "file" { - source = "${path.root}/../packages/target/archive" - destination = "/usr/local/share/" - } - - provisioner "remote-exec" { - inline = [ - "echo deb [trusted=yes] file:/usr/local/share/archive ./ > /etc/apt/sources.list.d/local-archive.list", - "apt update --quiet=2", - "apt install --quiet=2 --yes ${null_resource.role_config.triggers.config_packages}", - ] - } -} - -output "roles" { - value = "${var.roles}" -} diff --git a/terraform/root-ssh-key b/terraform/root-ssh-key new file mode 100644 index 0000000..11a7a78 --- /dev/null +++ b/terraform/root-ssh-key @@ -0,0 +1 @@ +ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDG5x/YyeCRcIV9TsdyahDpSKQkn/78967VVFvMNjBxDj1/NekINjTgNep7QUxiT16wKH2HFc9ahRGHZ/vK71JDHGZUic+DgvC8lHTTv/Y6aBtstZ9uEv9lz1OOgUDz2sfxLfajCr+40LbgBv8CAJSLBkB0PD5CypUcOtjaD07E6xQD4lMTL1/wsgbLcx4tIQRdQjUziT5d0Zlx9geV/cR4Z3YMNILKMk6vsh4vf3P7qebXv9XOjXE091EIgkfbawYmaB6zcBXoWYDdjPQ+YybqgLFTIjM2OiBmOI+eQFSFlALhP6HUjgbjOub+DnZDGWhH8SlCft6gVMK1lD+Sgud+57R4pXMHIPrIyOJcBF4cWX8biTPg4iLk01sF8hE/5J8fEArs48IhAFj91WW9O/l5torwB5P2g1lM9JxIyqZWSAS63oV4G+8CkgP4TdLYT/LA08F+a40CRateP5WvRGlk1WosMaN+IVHRYd8iFtJBT78hnPyFx3d9UoTCCFskigAueedNzw545nZOJJ4oXbw4WvlTkS/bPDBrj62XRPyrlFPimXLrfGKuY/N7f2wiWbV1Al2EkNurXVUg9sekXEd5cSYyghI14Qp5CyN+D2xkqwNKxC+WL0svIF1wZu39KZaQ2tOFy4ZYae0vut0LjFhJejo+TILLCSeyWf1vVPFrPw== cardno:000605258808 diff --git a/terraform/stdvps/README.md b/terraform/stdvps/README.md deleted file mode 100644 index 316641c..0000000 --- a/terraform/stdvps/README.md +++ /dev/null @@ -1,2 +0,0 @@ -# stdvps -A basic cloud virtual machine. diff --git a/terraform/stdvps/main.tf b/terraform/stdvps/main.tf deleted file mode 100644 index 2e94f1d..0000000 --- a/terraform/stdvps/main.tf +++ /dev/null @@ -1,148 +0,0 @@ -variable "ssh_key_name" { - type = "string" -} - -variable "location" { - type = "string" -} - -variable "volume_name" { - type = "string" - default = "" -} - -resource "random_id" "server" { - prefix = "peter-" - byte_length = 2 -} - -resource "hcloud_server" "server" { - name = "${random_id.server.hex}.crashbox.io" - image = "debian-9" - server_type = "cx11" - location = "${var.location}" - ssh_keys = ["${var.ssh_key_name}"] -} - -resource "cloudflare_record" "record_a" { - domain = "crashbox.io" - name = "${hcloud_server.server.name}" - value = "${hcloud_server.server.ipv4_address}" - type = "A" -} - -resource "cloudflare_record" "record_aaaa" { - domain = "crashbox.io" - name = "${hcloud_server.server.name}" - value = "${hcloud_server.server.ipv6_address}1" - type = "AAAA" -} - -resource "hcloud_volume" "master" { - count = "${var.volume_name == "" ? 0 : 1}" - name = "${var.volume_name}" - size = 50 - server_id = "${hcloud_server.server.id}" -} - -# volumes contain persistent storage and thus need to be initialized manually -resource "null_resource" "volume_mount" { - count = "${var.volume_name == "" ? 0 : 1}" - - triggers = { - server_id = "${hcloud_server.server.id}" - volume_id = "${hcloud_volume.master.id}" - } - - connection { - host = "${hcloud_server.server.ipv4_address}" - } - - provisioner "remote-exec" { - inline = ["mkdir -p /mnt/storage"] - } - - provisioner "file" { - content = <<EOF -[Unit] -Description=Mount /mnt/storage directory - -[Mount] -What=${hcloud_volume.master.linux_device} -Where=/mnt/storage -Type=ext4 -Options=defaults - -[Install] -WantedBy=multi-user.target -EOF - destination = "/etc/systemd/system/mnt-storage.mount" - } - - provisioner "file" { - content = <<EOF -[Unit] -Description=Mount /srv to persistent volume storage -After=mnt-storage.mount -BindsTo=mnt-storage.mount - -[Mount] -What=/mnt/storage/srv -Where=/srv -Type=ext4 -Options=bind - -[Install] -WantedBy=multi-user.target -EOF - destination = "/etc/systemd/system/srv.mount" - } - - provisioner "file" { - content = <<EOF -[Unit] -Description=Mount /home to persistent volume storage -After=mnt-storage.mount -BindsTo=mnt-storage.mount - -[Mount] -What=/mnt/storage/home -Where=/home -Type=ext4 -Options=bind - -[Install] -WantedBy=multi-user.target -EOF - destination = "/etc/systemd/system/home.mount" - } - - provisioner "remote-exec" { - inline = [ - "systemctl daemon-reload", - "systemctl enable --now mnt-storage.mount", - "systemctl enable --now srv.mount", - "systemctl enable --now home.mount" - ] - } -} - -output "ipv4" { - value = "${hcloud_server.server.ipv4_address}" -} - -output "ipv6" { - value = "${hcloud_server.server.ipv6_address}" -} - -output "fqdn" { - value = "${cloudflare_record.record_aaaa.hostname}" -} - -output "id" { - value = "${hcloud_server.server.id}" -} - -output "name" { - value = "${hcloud_server.server.name}" -} diff --git a/packages/.gitignore b/terraform/test/.gitignore index 2f7896d..2f7896d 100644 --- a/packages/.gitignore +++ b/terraform/test/.gitignore diff --git a/packages/Makefile b/terraform/test/Makefile index 01805c7..7f484e0 100644 --- a/packages/Makefile +++ b/terraform/test/Makefile @@ -1,22 +1,4 @@ -src-packages=crashbox-config - -all: target/archive - -target/build/%: % - mkdir --parent target/build - cp -rf $< $@ - (cd $@ && debuild -us -uc) - -target/build/Packages.gz: $(addprefix target/build/,$(src-packages)) - mkdir --parent target/build - (cd target/build && dpkg-scanpackages . | gzip -9c > Packages.gz) - -target/archive: target/build/Packages.gz - mkdir --parent $@ - cp \ - --target-directory $@ \ - $(wildcard target/build/*.deb) \ - $< +all: run # Create a base Debian image. # @@ -48,7 +30,7 @@ target/snapshot.qcow2: target/base.img -b $(notdir $<) $@ # Start a VM for testing config packages -run: target/snapshot.qcow2 target/archive +run: target/snapshot.qcow2 qemu-system-x86_64 \ -enable-kvm \ -machine q35,accel=kvm,kernel-irqchip=split \ @@ -62,7 +44,7 @@ run: target/snapshot.qcow2 target/archive id=net0 \ -device e1000,netdev=net0 \ -virtfs local,\ - path=target/archive,\ + path=../provision,\ mount_tag=host0,\ security_model=mapped,\ id=host0 \ @@ -71,16 +53,7 @@ run: target/snapshot.qcow2 target/archive -monitor none \ -serial stdio -docker-base: - docker build --tag crashbox-config-base:latest -f Dockerfile.base . - -docker: $(archive) - docker build --tag crashbox-config . - docker run --tty --interactive crashbox-config - clean: - rm -rf target/archive - rm -rf target/build rm -rf target/snapshot.qcow2 dist-clean: diff --git a/terraform/test/README.md b/terraform/test/README.md new file mode 100644 index 0000000..f418b36 --- /dev/null +++ b/terraform/test/README.md @@ -0,0 +1,26 @@ +# Test Utilities for Provisioning Scripts + +## Dependencies + +```bash +apt install \ + apt-cacher-ng \ + build-essential \ + qemu-kvm \ + vmdebootstrap +``` +## Example + +1. `make run`: starts a virtual machine and mounts provisioning + scripts. Note that running this the first time will build a base + virtual machine image, requiring significatnt time and + bandwidth. Any changes applied to the filesystem from within a + running VM will be contained in a copy-on-write snapshot image. + +2. login with `root` (no password) + +3. `/usr/local/share/provision/provision --force` to run provisioning + scripts + +4. back on the host, visit `https://ip.localhost:10443` to confirm the + service is running diff --git a/packages/vm/customize.sh b/terraform/test/vm/customize.sh index e8f67b6..8d18de0 100755 --- a/packages/vm/customize.sh +++ b/terraform/test/vm/customize.sh @@ -10,15 +10,12 @@ rootdir="$1" # avoid messing with host system, in case this script is run by accident [[ -n $rootdir ]] || abort "root directory is not set" -# set up local apt archive -mkdir -p $rootdir/usr/local/share/archive -echo "deb [trusted=yes] file:/usr/local/share/archive ./" > $rootdir/etc/apt/sources.list.d/local.list - -# mount local apt archive from host on startup +mkdir -p $rootdir/usr/local/share/provision +# mount local provision script directory from host on startup echo 9p >> $rootdir/etc/initramfs-tools/modules echo 9pnet >> $rootdir/etc/initramfs-tools/modules echo 9pnet_virtio >> $rootdir/etc/initramfs-tools/modules -echo "host0 /usr/local/share/archive 9p trans=virtio,version=9p2000.L 0 0" >> $rootdir/etc/fstab +echo "host0 /usr/local/share/provision 9p trans=virtio,version=9p2000.L 0 0" >> $rootdir/etc/fstab # boot immediately sed --in-place 's/GRUB_TIMEOUT=[0-9]\+/GRUB_TIMEOUT=0/g' $rootdir/etc/default/grub diff --git a/terraform/tf b/terraform/tf deleted file mode 100755 index 888e681..0000000 --- a/terraform/tf +++ /dev/null @@ -1,3 +0,0 @@ -#!/bin/bash - -exec terraform "$@" -var-file=<(pass infra/terraform)
\ No newline at end of file |