The intention is to keep to repo root clean while the list of packages
is slowly growing.
Additionally, a bunch of small (~30 LoC) files in the repo root is
merged into a single maddy.go file, for the same reason.
Most of the internal code is moved into the internal/ directory. Go
toolchain will make it impossible to import these packages from external
applications.
Some packages are renamed and moved into the pkg/ directory in the root.
According to https://github.com/golang-standards/project-layout this is
the de-facto standard to place "library code that's ok to use by
external applications" in.
To clearly define the purpose of top-level directories, README.md files
are added to each.
It was used only in queue tests and so was moved there. This fixes
-tests.debuglog command line flag being present in the maddy executable
as a side effect of testutils package import.
This allows to define session - message relation in a more clear way and
avoid certain hacks in the endpoint/smtp implementation.
The new ConnState structure is shared between all messages received over
a single connection and is initialized only once what saves resources.
I am trying to split the MsgMetadata structure since it is getting
rather big.
Strictly speaking, BodyLength is the property of the body, not message
as a whole. Additionally, keeping it separate creates the possibility of
being inconsistent with the real body size i.e. in case of any changes.
If this is needed, 'bounce' pipeline should use 'deliver_to queue ...'.
For cases where this is not needed, it introduces a lot of overhead
and adds unnecessary and potentially misleading spam to the log.
While at it, simplify the TimeWheel code to be callback-based rather
than a channel-based. This allows to get rid of second goroutine
(dispatch) to make shutdown logic simplier.
Change the order of operations to close the TimeWheel before waiting for
goroutines to stop. This prevents starting of new goroutines.
Make operations on closed TimeWheel no-op. Otherwise it will cause panics
when the TimeWheel is already closed. No-op behavior is fine, we are
about to discard TimeWheel state anyway and the TimeWheel caller already
saved meta-data needed to restore it on the next start-up to the disk.
Use correct enhanced codes in Status fields.
Use null return-path in MAIL FROM, as required by RFC 5321.
Don't generate DSNs for messages with null return-path.
Add tests.
Mostly to clear the noise in the output of golangci-lint to make
actually useful warnings more visible.
check/dkim: Silence textproto.WriteHeader errcheck warning
config/lexer: Remove unused isNewLine function
config/lexer: Remove "ineffectual assignment" to quoted
future: Don't drop context cancel function.
log: Remove unnecessary conversion in marshalOrderedJSON
module.go: Preallocate mods slice
Actually useful fixes:
log: Handle errors from marshalOrderedJSON
target/queue: Log the error of os.Rename in discardBroken
msgpipeline: Check delivery.Abort error in tests
This includes: queue dispatch, check runner and some checks (only SPF
atm) doing various stuff in parallel to the main delivery flow.
Queue panic handler drops the message that caused the panic from queue
to avoid further errors. It is assumed that most panics are caused due
to corruption of per-message state, so it is possible to reasonably
recover from it by stopping attempts to handle the problematic message.
NDN is not generated in this case, since it could cause interaction with
the state of a possibly corrupted message. This could change in the
future, as per RFC 5321 the server "MUST NOT lose the message for frivolous
reasons".
Additionally, when panic causes SMTP-time rejection, the server returns
a 4xx code, essentially deferring the message instead of dropping it.
That's the safety net for cases where panic is caused by a transient
problem (e.g. a race condition).
Previous error reporting code was inconsistent in terms of what is
logged, when and by whom. This caused several problems such as: logs
missing important error context, duplicated error messages, too verbose
messages, etc.
Instead of logging all generated errors, module should log
only errors it 'handles' somehow and does not simply pass it to the
caller. This removes duplication, however, also it removes context
information. To fix this, exterrors package was extended to provide
utilities for error wrapping. These utilities provide ability to
'attach' arbitrary key-value ('fields') pairs to any error object while
preserving the original value (using to Go 1.13 error handling
primitives).
In additional to solving problems described above this commit makes logs
machine-readable, creating the possibility for automated analysis.
Three new functions were added to the Logger object, providing
loosely-typed structured logging. However, they do not completely
replace plain logging and are used only where they are useful (to allow
automated analysis of message processing logs).
So, basically, instead of being logged god knows where and when,
all context information is attached to the error object and then it is
passed up until it is handled somewhere, at this point it is logged
together with all context information and then discarded.
Some fixed warnings are not actual problems. I just want output
golangci-run to be smaller. Otherwise it is problematic to see
actual problems found by linters due to amount of noise.
testutils, ctl/maddyctl: Remove quite a lot of unused entities.
msgpipeline: Fix typo in log message.
msgpipeline: Add missing 'case 0' for 'reject' directive parsing
msgpipeline: Preallocate slice in parseChecksGroup
Check errors in some places in tests.
smtp_upstream: Move DATA command use into Commit
target/queue: Avoid copying sync.Mutex
Not a problem since it is not used after copying, but I just want
govet to shut up.
New name more precisely describes what it is doing now. It was initally
meant to be more generic and usable for other purposes, but I don't
think we will need that flexibility.
Log successfull delivery to each recipient.
Log permanent failures separately for each recipients. It makes it
easier to process log in scripts.
Add list of recipients that will be retried to the "will retry"
message.
Most previously existing ways to specify Config/State/Libexec
directories are removed. Now they can be set only using
command line flags. Generally, having multiple ways to set these
values can be confusing and also believed to be unnecessary.
All relative paths in the configuration are now interpreted
relatively to the State directory.
Additionally, new value is added: Runtime directory. It will used
in the future for temporary files.
All directories that should be writable will be created on startup
if possible and checked for writebility.
It makes sense to allow having default of bool flag to true and
allow writing 'name no' to disable it. This allows getting rid of
"dont", "no" etc in names or certain directives which are meant to be
enabled by default. IMO 'do_x no' looks better than 'dont_do_x'
Due to .gitignore, target/queue directory got ignored and was not
commited.
This commit makes problematic ignore filter more strict and adds
accidentally removed directory.
