There is abstraction 'updates pipe' defined for future use with
configuration involving IMAP data replication (e.g. multiple nodes with
maddy instances + PostgreSQL replicas + S3 bucket for messages).
However, for the case of local SQLite3 DB, limited UDS-based
implementation is provided. It solves the problem of maddyctl not being
able to tell the server about modifications it makes. Alternative to
this approach would be to have server actually perform operations and
maddyctl being a dumb API client, but this requires a lot more complex
IPC interface and will not work when the server is down.
The SMTPUTF8 support is not required from the remote server unless it is
needed to transmit the SMTP envelope. The implementation assumes that we
will not accept a message without the SMTPUTF8 flag set with non-ASCII
addresses.
Newly added functions to address and dns packages implement Unicode and
IDNA2008-aware "clean" and "equal" operations.
The equality check for local-parts of addresses is intentionally defined
to be case-insensitive, this is consistent with other implementations
and ensures safer behavior for the situation when the final MTA handles
local-part in a case-insensitive way.
The equality check for the domain part diverges from the strict
definition provided by IDNA2008, converting both values into U-labels
form instead of A-labels if they are in different forms in the first
place. However, it is believed to yield the same results as the
comparsion strictly as defined by IDNA2008 as long as NFC normalization
is applied afterwards (the 'symmetry rule' also defined by IDNA2008
will hold then). The actual IDNA2008 equivalence is more problematic to
implement for cases where NFC normalization requirement ("MUST" from RFC
5890) is broken by the client. As we all know, clients are terrible at
following complex standards and Unicode is hard. Remember the
robustness principle: "Be conservative in what you send, be liberal in
what you accept".
Additionally, U-labels form is planned to be used internally in maddy in
all places converting the A-labels form only where necessary for
interoperability. This form is more useful in logs (A-labels look like a
gibberish for the end-user) and also is practically considered to be the
"true" representation of IDNs and should be prefferred in environments
where legacy compatibility is not a concern.
References:
- https://unicode.org/reports/tr15/ (UAX#15)
- https://tools.ietf.org/html/rfc5890 (IDNA2008)
- https://blog.golang.org/normalization
This commit does not enable support for SMTPUTF8 (we are not conformant
to RFC 6531 yet, #165) nor for REQUIRETLS (more changes are needed to
implement it correctly, #123).
It just fixes code to account for backward-incompatible changes and adds
a new field to MsgMetadata to store MAIL FROM arguments in.
since the endpoint/smtp initialization is rather complex, tests use
New+Init instead of creating the Endpoint object directly like it is
done for target/queue and target/remote. This requires a couple of
tricks to create a valid configuration tree for it. This is the reason
dummy module was moved into 'module' package, this wasy it can be
registered and then referenced by the Init code when reading the
configuration.
A small problem was found and fixed in go-mockdns along the way.
We merely do an rDNS lookup on the IP address and use the returned
domain name to do everything. It seems to be the only one way to
verify certificates while using IP literals, since most servers don't
have a certificate valid for its IP, only for domain.
Most legitimate mail servers have forward-confirmed rDNS records, so it
should work fine.
applyResults modifies the header while FetchRecord (running in parallel
calls extractDomains that reads it.
Additionally, another race condition was caused by go-mockdns not
copying the slice before returning, that was addressed upstream:
* ed42e5b Copy slice before returning it from Lookup*
Minimal mock server is implemented using go-smtp in testutils package.
This means some problems can be hidden by the go-smtp processing, but
this believed to not be a significant problems.
Also I have to export some of the dns and mtasts packages internals
to make them mockable.
Several hooks had to be added to the module object itself:
- net.Dial replacement (Target.dialer)
- mtasts.Cache.Get replacement (Target.mtastsGet)
In meanwhile I learned that message body must end with CRLF and go-smtp
implicitly adds it if it is not present, so I adjusted CheckTestMessage
and DoTestDelivery routines to include it in the body.
* dkim: Add signer method to return only value of the DKIM-Signature field
* dmarc: Separate Parse from Lookup
* dkim: Always close done channel in Sign
* dkim: Don't pass hash function to Sign for ed25519
* dkim: Add tests for ed25519 signing
This support is based on github.com/foxcpp/go-msgauth fork until
emerison/go-msgauth#13 gets merged.
Further extensions are required to make sure only messages we can
actually "take responsibility for" are signed.
RSA-2048 is used as a default algorithm when generating new keys.
RSA-4096 can cause trouble with UDP-only DNS due to responses being
bigger than 512 octets. RSA-1024 is too weak and explicitly
disallowed in maddy for new keys. It could be possible to use Ed25519
but support is not widely deployed yet (according to warning in rspamd
docs dated 2019-09). Users concerned about security of RSA-2048 can
switch to RSA-4096 or Ed25519, keeping relevant problems in mind.
Ed25519 key format uses PKCS#8, this seems to be different from other
implementations that just dump key material into a file without any
wrapping. Interoperability is not considered to encourage key
rotation when migration, which is a good thing to do anyway.
There is no option to use "body limit", since it is dangerous
and go-msgauth/dkim does not support it for signing.
The default set of signed header fields is the list used by rspamd.
Most "core" fields are oversigned to provide strict integrity.
"Conditional oversigning" similar to rspamd is not implemented, though
it may be useful, further research is required.
Multi-tentant configuration with DKIM and DMARC is much more verbose,
configuration example is added to config.d/multitentant-dkim.conf to
explain how to make it work.
Mailing lists break DKIM signatures by modifing Subject and other header
fields. They are supposed to either include their own DKIM signature
and/or ARC (RFC 8617) seal.
Now it is a wrapper around miekg/dns library that reports
whether AD flag is set in the response. It does not perform
any verification on its own. This is not going to be implemented due
to complexity of code required to make it work reasonably fast.
1. There is only one version for maddy and imapsql-ctl utility.
This prevents confusion about compatibility.
2. Modified imapsql-ctl understands maddy config format, this allows
it to read needed values from it without the need for lengthy commmand
line arguments.
Closes#148.
- fsstore is now required, this is reflected by documentation updates.
Upstream commits:
* 8ee5c96 Fix handling of "null" compression
* d6bc61c Add support for zstd compression
* aa76135 Implement support for LZ4 compression support
* cde9a24 Update VersionStr constant
* 2008a7b Remove schema upgrade operations for older versions
* b6668d0 Remove section about Internal/External BLOBs from README
* e91826c cmd/imapsql-ctl: Remove support for reading driver/DSN from text file
* 0dca68f Rename imapsql.Store -> imapsql.FSStore
* 4071c69 Remove support for handling messages in table rows
* 4c8996e Remove most of the code for storing messages in table rows
* b682ac1 Skip header in openBody if needHeader = false
* 2a263c3 Require fsstore to be used
* 45b437d Remove CI job for testing with MariaDB
* 8d30346 Add 'go 1.12' to go.mod
* 1679626 Fix wrong order of arguments for setInboxId query
* da0b12c Remove dangling External Store key when message is not added to DB
* c706691 Use Bytes() method on Reader passed to Delivery.Body
* 3d2b5b0 Fix User.inboxId = 0 for newly created users
* 45cd989 Remove schema upgrade restriction
Use of context.Context for cancellation turned out to be impractical.
Additionally, most maddy configuration will not make use of
configuration where checks can be cancelled.
E.g. it is important to run all checks to generate complete
Authentication-Results header.
Additionally, due to scoring system use (will be used by default
after DMRAC support introduction), we can't rely on a subset of checks
to reject the message.
Now exterrors.IsTemporary should be used to check whether error is a
temporary one.
This implementation relies on Go 1.13 error inspection primitives and
thus increases minimal Go version required by maddy accordingly.
Go 1.13 and newer always adds that string, so it can be annoying when
doing unrelated changes.
Note that it is go 1.12, not go 1.12. Later would be pointless since
we don't use anything from Go 1.13. Some distros ship older versions
of Go so it is actually bad to require it to be newer than needed.
In general, the checks interface with added scoring and quarantining
support was not convenient to use enough. Also it was problematic
to add support for Authentication-Results header field generation.
Per-sender and per-recipient checks were not applied to body.
This is fixed now.
Checks inspecting the message header was able to see header
modifications done by other checks. This could lead to unwanted
side-effects and so now checks can't modify the header directly
and instead can only prepend fields to it by returning them.
Additionally, it allows checks to return values for
Authentication-Results field. Each server handling the message should
add only one field, so it is not possible to implement it using header
prepending.
MsgMetadata.CheckScore is removed, now it is managed internally by
dispatcher code and not exposed where it is not needed.
MsgMetadata.Quarantine is no longer set directly by checks code. Future
refactoring may be remove it altogether as it is discouraged to have
mutable flags in MsgMetadata.
On top of that, tests are added for all new code.
