Several comments were removed since they are not worth the trouble.
A few minor issues were addressed.
Most of remaining comments got corresponding GitHub issues assigned.
This makes server more aggressive in retrying delivery. The total
attempts time is approximately the same (around 5 days).
Notably, the increased amount of attempts is needed to handle large
recipient lists where the remote server sometimes rejects subset of them
with "Too Many Recipients" code.
See #149.
While was not strictly needed before, it is necessary in the presence of
limits and other internal target errors that should cause the delivery
to get rescheduled without increasing retries counter.
Set of flow restrictions is represented as a "limits" module instance
that can be either created inline via "limits" directive in some modules
(including "remote" target and "smtp" endpoint) or defined globally and
referenced in configuration of modules mentioned above.
This permits a variety of use cases, including shared and separate
counters for various endpoints and also "modules group" style sharing
described in #195.
Previous approach consisted of multiple independent options with unknown
interaction between each other and not offering enough flexibility for
local policy configuration.
Additionally, it was not possible to implement downgrade protection
mentioned in #178 because it was not clear what is "downgrade" since
options were not related in any linear order, this commit makes it
explicit via the "security levels" system:
MX: DNSSEC > MTA-STS > Nothing
TLS: Authenticated+Encrypted > Encrypted > Plaintext
Note DNSSEC and MTA-STS being different levels, they provide different
security guarantees. Keeping them together under "authenticated" level
would not provide enough granularity for levels-based downgrade
protection and local policies.
'common_domain' MX authentication option is removed. It was offering no
real protection and therefore is was problematic to use together with
planned downgrade protection.
All security level errors are marked as temporary to force requeueing
and allow local admin to troubleshoot them without losing messages.
'remote' tests are changed to use testTarget function to initialize
tested module instance, since security levels mapping requires some
pre-initialization.
Support for IP literals in address domain-part is disabled because it
is incompatible with the new verification logic and was broken anyway
(#176).
TLS without authentication is still better than no TLS at all.
To save latency in transactions with a misconfigured recipient server
that cannot use TLS at all but still advertises STARTTLS support,
downgrade to non-authenticated TLS is attempted only on verification
errors (x509.UnknownAuthorityError or x509.HostnameError) and malformed
certificate errors (x509.ConstraintViolationError and
x509.CertificateInvalidError). In all other cases 'remote' module
fallbacks to plaintext directly.
While rearranging code to support this, some additional changes were
made to allow simplier implementation of security levels idea from #178.
See https://tools.ietf.org/html/rfc7435.
See #178.
As revealed by latency tracing using runtime/trace, MTA-STS cache miss
essentially doubles the connection time for outbound delivery. This is
mostly because MTA-STS lookup have to estabilish a TCP+TLS connection to
obtain the policy text (shame on Google for pushing that terribly
misdesigned protocol, but, well, it is better than nothing so we adopt
it).
Additionally, there is a number of additional DNS lookups needed (e.g.
TLSA record for DANE). This commit rearranges connection code so it is
possible to run all "additional" queries in parallel with the connection
estabilishment. However, this changes the behavior of TLS requirement
checks (including MTA-STS). The connection to the candidate MX is
already estabilished and STARTTLS is always attempted if it is
available. Only after that the policy check is done, using the result of
TLS handshake attempt (if any). If for whatever reason, the candidate MX
cannot be used, the connection is then closed. This might bring
additional overhead in case of configuration errors on the recipient
side, but it is believed to not be a major problem since this should not
happen often.
runtime/trace together with 'go tool trace' provides extremely powerful
tooling for performance (latency) analysis. Since maddy prides itself on
being "optimized for concurrency", it is a good idea to actually live up
to this promise.
Closes#144. No need to reinvent the wheel. The original issue
proposed a solution to use in production to detect "performance
anomalies", it is possible to use runtime/trace in production too, but
the corresponding flag to enable profiler endpoint is hidden behind the
'debugflags' build tag at the moment.
For SMTP code, the basic latency information can be obtained from
regular logs since they include timestamps with millisecond granularity.
After the issue is apparent, it is possible to deploy the server
executable compiled with tracing support and obtain more information
... Also add missing context.Context arguments to smtpconn.C.
It is useful to define background tasks lifetimes more precisely,
especially involving timeouts and other cancellation methods.
On top of that, several tracing facilities are context-based (e.g.
runtime/trace), so it is possible to use them now.
Allow to override DNS resolver address via the -debug.dnsoverride flag
and SMTP port via -debug.smtpport.
All flags are not available unless maddy is built using the 'debugflags'
tag.
The intention is to keep to repo root clean while the list of packages
is slowly growing.
Additionally, a bunch of small (~30 LoC) files in the repo root is
merged into a single maddy.go file, for the same reason.
Most of the internal code is moved into the internal/ directory. Go
toolchain will make it impossible to import these packages from external
applications.
Some packages are renamed and moved into the pkg/ directory in the root.
According to https://github.com/golang-standards/project-layout this is
the de-facto standard to place "library code that's ok to use by
external applications" in.
To clearly define the purpose of top-level directories, README.md files
are added to each.
