docs: Move the project documentation from GitHub Wiki

Having it in the same directory as the source code makes it simplier to
keep in sync with the source code itself.

.build.yml

@@ -13,7 +13,7 @@ tasks:
       cd maddy
       go test ./... -cover -race
   - build-man-pages: |
-      cd maddy/man
+      cd maddy/docs/man
       scdoc < maddy.1.scd > /dev/null
       scdoc < maddy-auth.5.scd > /dev/null
       scdoc < maddy-config.5.scd > /dev/null

docs/README.md

@@ -0,0 +1 @@
+../README.md

docs/get.sh.md

@@ -0,0 +1,31 @@
+# get.sh script
+
+get.sh script does the following:
+
+- Makes sure you have a supported version of Go toolchain, if this is not the
+  case - it downloads one.
+- Downloads and compiles maddy executables.
+- Installs maddy executables to /usr/local/bin.
+- Installs the dist/ directory contents
+- Installs the man/ directory contents
+- Install the default configuration.
+- Creates maddy user and group.
+
+It is Linux-specific, users of other systems will have to use Manual
+installation.
+
+## Environmental variables
+
+Users can be set following environmental variables to control the exact
+behavior of the get.sh script.
+
+|  Variable       |  Default value        |  Description |
+| --------------- | --------------------- | --- |
+| GOVERSION       | 1.13.4                | Go toolchain version to download if the system toolchain is not compatible. |
+| MADDYVERSION    | master                | maddy version to download & install. |
+| DESTDIR         |                       | Interpret all paths as relative to this directory during installation. |
+| PREFIX          | /usr/local            | Installation prefix. |
+| SYSTEMDUNITS    | $PREFIX/lib/systemd   | Directory to install systemd units to. |
+| CONFDIR         | /etc/maddy            | Path to write configuration files to. |
+| FAIL2BANDIR     | /etc/fail2ban         | Path to install fail2ban configs to. |
+| SUDO            | sudo                  | Executable to call to elevate privileges during installation. |

docs/internals/quirks.md

@@ -0,0 +1,30 @@
+# Implementation quirks
+
+This page documents unusual behavior of the maddy protocols implementations.
+Some of these problems break standards, some don't but still can hurt
+interoperability.
+
+## SMTP
+
+- `for` field is never included in the `Received` header field.
+
+  This is allowed by [RFC 2821].
+
+## IMAP
+
+### `sql`
+
+- `\Recent` flag is not implemented and it always set.
+
+  This _does not_ break [RFC 3501]. Clients relying on it will work (much) less
+  efficiently.
+
+- Sequence numbers don't stay consistent between SELECT/CHECK commands.
+
+  This _does not_ break [RFC 3501] which is unclear about synchronization
+  issues, however it deviates from behavior implemented by most servers. This
+  can lead to operations applied to the wrong messages if sequence numbers are
+  used by multiple clients connected at the same time.
+
+[RFC 2821]: https://tools.ietf.org/html/rfc2821
+[RFC 3501]: https://tools.ietf.org/html/rfc3501

docs/internals/sqlite.md

@@ -0,0 +1,49 @@
+# maddy & SQLite
+
+SQLite is a perfect choice for small deployments because no additional
+configuration is required to get started. It is recommended for cases when you
+have less than 10 mail accounts.
+
+**Note: SQLite requires DB-wide locking for writing, it means that multiple
+messages can't be accepted in parallel. This is not the case for server-based
+RDBMS where maddy can accept multiple messages in parallel even for a single
+mailbox.**
+
+## WAL mode
+
+maddy forces WAL journal mode for SQLite. This makes things reasonably fast and
+reduces locking contention which may be important for a typical mail server.
+
+maddy uses increased WAL autocheckpoint interval. This means that while
+maintaining a high write throughput, maddy will have to stop for a bit (0.5-1
+second) every time 78 MiB is written to the DB (with default configuration it
+is 15 MiB).
+
+Note that when moving the database around you need to move WAL journal (`-wal`)
+and shared memory (`-shm`) files as well, otherwise some changes to the DB will
+be lost.
+
+## Query planner statistics
+
+maddy updates query planner statistics on shutdown and every 5 hours. It
+provides query planner with information to access the database in more
+efficient way because go-imap-sql schema does use a few so called "low-quality
+indexes".
+
+## Auto-vacuum
+
+maddy turns on SQLite auto-vacuum feature. This means that database file size
+will shrink when data is removed (compared to default when it remains unused).
+
+## Manual vacuuming
+
+Auto-vacuuming can lead to database fragmentation and thus reduce the read
+performance.  To do manual vacuum operation to repack and defragment database
+file, install the SQLite3 console utility and run the following commands:
+```
+sqlite3 -cmd 'vacuum' database_file_path_here.db
+sqlite3 -cmd 'pragma wal_checkpoint(truncate)' database_file_path_here.db
+```
+
+It will take some time to complete, you can close the utility when the
+`sqlite>` prompt appears.

docs/man/.gitignore

@@ -0,0 +1 @@
+_generated_*.md

docs/man/README.md

@@ -0,0 +1,16 @@
+maddy manual pages
+-------------------
+
+The reference documentation is maintained in the scdoc format and is compiled
+into a set of Unix man pages viewable using the standard `man` utility.
+
+See https://git.sr.ht/~sircmpwn/scdoc for information about the tool used to
+build pages.
+It can be used as follows:
+```
+scdoc < maddy-filters.5.scd > maddy-filters.5
+man ./maddy-filters.5
+```
+
+get.sh script in the repo root compiles and installs man pages if the scdoc
+utility is installed in the system.

docs/man/maddy-tls.5.scd

@@ -32,6 +32,7 @@ performance
 List of supported cipher suites, in preference order. Not used with TLS 1.3.
 
 Valid values:
+
 - RSA-WITH-RC4128-SHA
 - RSA-WITH-3DES-EDE-CBC-SHA
 - RSA-WITH-AES128-CBC-SHA

docs/man/maddy.1.scd

@@ -1,7 +1,6 @@
 maddy(1) "maddy mail server" "maddy reference documentation"
 
 ; TITLE Introduction
-; NO_TOC
 
 # Name
 

docs/man/prepare_md.py

@@ -0,0 +1,57 @@
+#!/usr/bin/python3
+
+"""
+This script does all necessary pre-processing to convert scdoc format into
+Markdown.
+
+Usage:
+    prepare_md.py < in > out
+    prepare_md.py file1 file2 file3
+        Converts into _generated_file1.md, etc.
+"""
+
+import sys
+import re
+
+anchor_escape = str.maketrans(r' #()./\+-_', '__________')
+
+def prepare(r, w):
+    new_lines = list()
+    title = str()
+    previous_h1_anchor = ''
+
+    inside_literal = False
+
+    for line in r:
+        if not inside_literal:
+            if line.startswith('; TITLE ') and title == '':
+                title = line[8:]
+            if line[0] == ';':
+                continue
+            # turn *page*(1) into [**page(1)**](../_generated_page.1)
+            line = re.sub(r'\*(.+?)\*\(([0-9])\)', r'[*\1(\2)*](../_generated_\1.\2)', line)
+            # *aaa* => **aaa**
+            line = re.sub(r'\*(.+?)\*', r'**\1**', line)
+            # remove ++ from line endings
+            line = re.sub(r'\+\+$', '<br>', line)
+            # turn whatever looks like a link into one
+            line = re.sub(r'(https://[^ \)\(\\]+[a-z0-9_\-])', r'[\1](\1)', line)
+            # escape underscores inside words
+            line = re.sub(r'([^ ])_([^ ])', r'\1\\_\2', line)
+
+        if line.startswith('```'):
+            inside_literal = not inside_literal
+
+        new_lines.append(line)
+
+    if title != '':
+        print('#', title, file=w)
+
+    print(''.join(new_lines[1:]), file=w)
+
+if len(sys.argv) == 1:
+    prepare(sys.stdin, sys.stdout)
+else:
+    for f in sys.argv[1:]:
+        new_name = '_generated_' + f[:-4] + '.md'
+        prepare(open(f, 'r'), open(new_name, 'w'))

docs/tutorials/manual-installation.md

@@ -0,0 +1,76 @@
+# Manual installation & configuration
+
+## Dependencies
+
+- [Go](https://golang.org) toolchain (1.13 or newer)
+
+  If your distribution ships an outdated Go version, you can use
+  following commands to get a newer version:
+  ```
+  go get golang.org/dl/go1.13
+  go1.13 download
+  ```
+  Then use `go1.13` instead of `go` in commands below.
+
+- C compiler (**optional**, set CGO_ENABLED env. variable to 0 to disable)
+
+  Required for SQLite3-based storage (default configuration) and PAM
+  authentication.
+
+## Building
+
+First, make sure Go Modules support is enabled:
+```
+export GO111MODULE=on
+```
+
+There are two binaries to install, server itself and DB management
+utility. Use the following command to install them:
+```
+go get github.com/foxcpp/maddy/cmd/{maddy,maddyctl}@master
+```
+
+Executables will be placed in the $GOPATH/bin directory (defaults to
+$HOME/go/bin).
+
+## Configuration
+
+*Note*: explaination below is short and assumes that you already have
+basic ideas about how email works.
+
+1. Install maddy and maddyctl (see above)
+2. Copy maddy.conf from this repo to /etc/maddy/maddy.conf
+3. Create /run/maddy and /var/lib/maddy, make sure they are writable
+   for the maddy user. Though, you don't have to use system directories,
+   see `maddy -help`.
+4. Open maddy.conf with ~~vim~~your favorite editor and change
+   the following:
+- `tls ...`
+  Change to paths to TLS certificate and key.
+- `$(hostname)`
+  Server identifier. Put your domain here if you have only one server.
+- `$(primary_domain)`
+  Put the "main" domain you are handling messages for here.
+5. Run the executable.
+6. On first start-up server will generate a RSA-2048 keypair for DKIM and tell
+   you where file with DNS record text is placed. You need to add it to your
+   zone to make signing work.
+7. Create user accounts you need using `maddyctl`:
+   ```
+   maddyctl users create foxcpp@example.org
+   ```
+
+Congratulations, now you have your working mail server.
+IMAP endpoint is on port 993 with TLS enforced ("implicit TLS").
+SMTP endpoint is on port 465 with TLS enforced ("implicit TLS").
+
+### systemd unit
+
+You can use the systemd unit file from the [dist/](dist) directory in
+this repo. It will automatically set-up user account and directories.
+Additionally, it will apply strict sandbox to maddy to ensure additional
+security.
+
+You need a relatively new systemd version (235+) both of these things to work
+properly. Otherwise, you still have to manually create a user account and the
+state + runtime directories with read-write permissions for the maddy user.

docs/tutorials/setting-up.md

@@ -0,0 +1,177 @@
+# Installation & initial configuration
+
+This is the practical guide on how to set up a mail server using maddy for
+personal use. It omits most of the technical details for brevity and just gives
+you the minimal list of things you need to be aware of and what to do to make
+stuff work.
+
+For purposes of clarity, these values are used in this tutorial as examples,
+wherever you see them, you need to replace them with your actual values:
+
+- Domain: example.org
+- IPv4 address: 10.2.3.4
+- IPv6 address: 2001:beef::1
+
+## Getting a server
+
+Where to get a server to run maddy on is out of the scope of this article. Any
+VPS (virtual private server) will work fine for small configurations. However,
+there are a few things to keep in mind:
+
+- Make sure your provider does not block SMTP traffic (25 TCP port). Most VPS
+  providers don't do it, but some "cloud" providers (such as Google Cloud) do
+  it, so you can't host your mail there.
+- ...
+
+## Installing maddy
+
+Since there are currently no pre-compiled binaries for maddy, we are going to
+build it from the source. Nothing scary, this is relatively easy to do with Go.
+
+One dependency you need to figure out is the C compiler, it is needed for
+SQLite3 support which is used in the default configuration. On Debian-based
+distributions, this should be enough:
+```
+# apt-get install build-essential
+```
+
+get.sh script will do the rest for you:
+
+```
+$ curl 'https://foxcpp.dev/maddy/get.sh' | bash
+```
+
+It will leave `maddy-setup` directory lying around, you might want to keep it
+so you don't have to redownload and recompile everything on update.
+
+*Note:* If you can't / don't use this script for some reason, instructions for
+manual installation can be found
+[here](../manual-installation)
+
+## TLS certificates
+
+One thing that can't be automagically configured is TLS certs. If you already
+have them somewhere - use them, open /etc/maddy/maddy.conf and put the right
+paths in. You need to make sure maddy can read them while running as
+unprivileged user (maddy never runs as root, even during start-up), one way to
+do so is to use ACLs (replace with your actual paths):
+```
+$ sudo setfacl -R -m u:maddy:rX /etc/ssl/example.org.crt /etc/ssl/example.org.key
+```
+
+### Let's Encrypt and certbot
+
+If you use certbot to manage your certificates, you can simply symlink
+/etc/maddy/certs into /etc/letsencrypt/live. maddy will pick the right
+certificate depending on the domain you specified during installation.
+
+You still need to make keys readable for maddy, though:
+```
+$ sudo setfacl -R -m u:maddy:rX /etc/letsencrypt/{live,archive}
+```
+
+Additionally, it is a good idea to automatically restart
+maddy on certificate renewal.
+Put that into /etc/letsencrypt/renewal-hooks/post/restart:
+```shell
+#!/bin/bash
+systemctl restart maddy
+```
+And make it executable:
+```
+$ sudo chmod +x /etc/letsencrypt/renewal-hooks/post/restart
+```
+
+## First run
+
+```
+systemctl start maddy
+```
+
+Well, it should be running now, except that it is useless because we haven't
+configured DNS records.
+
+## DNS records
+
+How it is configured depends on your DNS provider (or server, if you run your
+own). Here is how your DNS zone should look like:
+```
+; Basic domain->IP records, you probably already have them.
+example.org.   A     10.2.3.4
+example.org.   AAAA  2001:beef::1
+
+; It says that "server example.org is handling messages for example.org".
+example.org.   MX    10 example.org.
+
+; Use SPF to say that the servers in "MX" above are allowed to send email
+; for this domain, and nobody else.
+example.org.   TXT   "v=spf1 mx -all"
+
+; Opt-in into DMARC with permissive policy and request reports about broken
+; messages.
+_dmarc.example.org.   TXT    "v=DMARC1; p=none; ruf=postmaster@example.org"
+```
+
+And the last one, DKIM key, is a bit tricky. maddy generated a key for you on
+the first start-up. You can find it in
+/var/lib/maddy/dkim_keys/example.org_default.dns. You need to put it in a TXT
+record for `default._domainkey.example.org` domain, like that:
+```
+default._domainkey.example.org    TXT   "v=DKIM1; k=ed25519; p=nAcUUozPlhc4VPhp7hZl+owES7j7OlEv0laaDEDBAqg="
+```
+
+## postmaster and other user accounts
+
+A mail server is useless without mailboxes, right? Unlike software like postfix
+and dovecot, maddy uses "virtual users" by default, meaning it does not care or
+know about system users.
+
+Here is the command to create virtual 'postmaster' account, it will prompt you
+for a password:
+```
+$ maddyctl users create postmaster@example.org
+```
+
+Note that account names include the domain. When authenticating in the mail
+client, full address should be specified as a username as well.
+
+Btw, it is a good idea to learn what else maddyctl can do. Given the
+non-standard structure of messages storage, maddyctl is the only way to
+comfortably inspect it.
+
+## Optional: Install and use fail2ban
+
+The email world is full of annoying bots just like Web (these annoying scanners
+looking for PhpMyAdmin on your blog). fail2ban can help you get rid of them by
+temporary blocking offending IPs.
+
+1. Install the fail2ban package using your distribution package manager
+For Debian-based distributions:
+```
+apt-get install fail2ban
+```
+
+2. get.sh already installed necessary jail configuration files, but you have to
+   enable them. Open /etc/fail2ban/jail.d/common.local (create one if needed)
+   and add the following lines:
+```
+[maddy-auth]
+enabled = true
+
+[maddy-dictonary-attack]
+enabled = true
+```
+
+Now start or restart the fail2ban daemon:
+```
+systemctl restart fail2ban
+```
+
+Keep in mind that the maddy jail configuration uses a different much longer
+bantime value. This means users will get IP-banned for quite a long time (4
+days) after 5 failed login attempts. You might want to change that to a smaller
+period by editing /etc/fail2ban/jail.d/common.local:
+```
+[maddy-auth]
+bantime = 1h
+```

man/prepare_md.py

@@ -1,93 +0,0 @@
-#!/usr/bin/python3
-
-"""
-This script does all necessary pre-processing to convert scdoc format into
-Markdown.
-
-Also, it generates Table of Contents for files with more than one
-section.
-If '; NO_TOC' line is present in file, ToC will not be generated.
-
-And finally, '; TITLE Whatever' is converted into YAML front matter with 'title'
-value.
-
-Usage:
-    prepare_md.py < in > out
-    prepare_md.py file1 file2 file3
-        Converts into _generated_file1.md, etc.
-"""
-
-import sys
-import re
-
-anchor_escape = str.maketrans(r' #()./\+-_', '__________')
-
-def toc_entry(line, previous_h1_anchor):
-    level = line.count('#')
-    indent = '  ' * (level-1)
-    name = line[line.rfind('#')+1:].strip().replace('_', r'\_')
-    anchor = name.translate(anchor_escape).lower()
-    if level == 2:
-        anchor = previous_h1_anchor + '.' + anchor
-    return '{indent}- [{name}](#{anchor})'.format(**locals()), anchor
-
-def prepare(r, w):
-    new_lines = list()
-    h1_count = 0
-    no_toc = False
-    table_of_contents = list()
-    title = str()
-    previous_h1_anchor = ''
-
-    inside_literal = False
-
-    for line in r:
-        if line.startswith('#') and not inside_literal:
-            entry, anchor = toc_entry(line, previous_h1_anchor)
-            table_of_contents.append(entry)
-            new_lines.append('<a id="{}">\n\n'.format(anchor))
-            if not line.startswith('##'):
-                h1_count += 1
-                previous_h1_anchor = anchor
-            line = '#' + line
-
-        if not inside_literal:
-            if line.startswith('; NO_TOC'):
-                no_toc = True
-            if line.startswith('; TITLE '):
-                title = line[8:].strip()
-            if line[0] == ';':
-                continue
-            # turn *page*(1) into [**page(1)**](_generated_page.1.md)
-            line = re.sub(r'\*(.+?)\*\(([0-9])\)', r'[*\1(\2)*](_generated_\1.\2.md)', line)
-            # *aaa* => **aaa**
-            line = re.sub(r'\*(.+?)\*', r'**\1**', line)
-            # remove ++ from line endings
-            line = re.sub(r'\+\+$', '<br>', line)
-            # turn whatever looks like a link into one
-            line = re.sub(r'(https://[^ \)\(\\]+[a-z0-9_\-])', r'[\1](\1)', line)
-            # escape underscores inside words
-            line = re.sub(r'([^ ])_([^ ])', r'\1\\_\2', line)
-
-        if line.startswith('```'):
-            inside_literal = not inside_literal
-
-        new_lines.append(line)
-
-    if title != '':
-        print('---', file=w)
-        print('title:', title, file=w)
-        print('---', file=w)
-
-    if h1_count > 1 and not no_toc:
-        print('## Table of contents\n', file=w)
-        print('\n'.join(table_of_contents), file=w)
-
-    print(''.join(new_lines[1:]), file=w)
-
-if len(sys.argv) == 1:
-    prepare(sys.stdin, sys.stdout)
-else:
-    for f in sys.argv[1:]:
-        new_name = '_generated_' + f[:-4] + '.md'
-        prepare(open(f, 'r'), open(new_name, 'w'))