revert back to ubuntu-latest for https://github.com/Kozea/Radicale/issues/1594

Fix 1515

.github/workflows/generate-documentation.yml

@@ -6,7 +6,7 @@ on:
 
 jobs:
   generate:
-    runs-on: ubuntu-22.04
+    runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v4
         with:

CHANGELOG.md

@@ -1,5 +1,13 @@
 # Changelog
 
+## 3.5.1.dev
+
+* Fix: auth/htpasswd related to detection and use of bcrypt
+* Add: option [auth] ldap_ignore_attribute_create_modify_timestamp for support of Authentik LDAP server
+* Extend: [storage] hook supports now placeholder for "cwd" and "path" (and catches unsupported placeholders)
+* Fix: location of lock file for in case of dedicated cache folder is activated
+* Extend: log and create base folders if not existing during startup
+
 ## 3.5.0
 
 * Add: option [auth] type oauth2 by code migration from https://gitlab.mim-libre.fr/alphabet/radicale_oauth/-/blob/dev/oauth2/

DOCUMENTATION.md

@@ -20,26 +20,19 @@ Radicale is a small but powerful CalDAV (calendars, to-do lists) and CardDAV
 
 #### Installation
 
-Radicale is really easy to install (for testing purposes) and works out-of-the-box.
+Check
 
-```bash
-# Run as normal user
-python3 -m pip install --user --upgrade https://github.com/Kozea/Radicale/archive/master.tar.gz
-python3 -m radicale --logging-level info --storage-filesystem-folder=~/.var/lib/radicale/collections
-```
+* [Tutorials](#tutorials)
+* [Documentation](#documentation-1)
+* [Wiki on GitHub](https://github.com/Kozea/Radicale/wiki)
+* [Disussions on GitHub](https://github.com/Kozea/Radicale/discussions)
+* [Open and already Closed Issues on GitHub](https://github.com/Kozea/Radicale/issues?q=is%3Aissue)
 
-When the server is launched, open <http://localhost:5232> in your browser!
-You can login with any username and password.
-
-Want more? Check the [tutorials](#tutorials) and the
-[documentation](#documentation-1).
-
-Instead of downloading from PyPI look for packages provided by used [distribution](#linux-distribution-packages), they contain also startup scripts to run daemonized.
+Hint: instead of downloading from PyPI look for packages provided by used [distribution](#linux-distribution-packages), they contain also startup scripts to run daemonized.
 
 #### What's New?
 
-Read the
-[changelog on GitHub.](https://github.com/Kozea/Radicale/blob/master/CHANGELOG.md)
+Read the [Changelog on GitHub](https://github.com/Kozea/Radicale/blob/master/CHANGELOG.md).
 
 ## Tutorials
 
@@ -59,27 +52,56 @@ Follow one of the chapters below depending on your operating system.
 First, make sure that **python** 3.9 or later and **pip** are installed. On most distributions it should be
 enough to install the package ``python3-pip``.
 
-Then open a console and type:
+##### as normal user
+
+Recommended only for testing - open a console and type:
 
 ```bash
 # Run the following command to only install for the current user
-#  data is also stored for the current user only
 python3 -m pip install --user --upgrade https://github.com/Kozea/Radicale/archive/master.tar.gz
+```
+
+If _install_ is not working and instead `error: externally-managed-environment` is displayed, create and activate a virtual environment in advance
+
+```bash
+python3 -m venv ~/venv
+source ~/venv/bin/activate
+```
+
+and try to install with
+
+```bash
+python3 -m pip install --upgrade https://github.com/Kozea/Radicale/archive/master.tar.gz
+```
+
+Start the service manually, data is stored only for the current user
+
+```bash
+# Start, data is stored for the current user only
 python3 -m radicale --storage-filesystem-folder=~/.var/lib/radicale/collections
 ```
 
-Alternative one can install as root or system user
+##### as system user (or as root)
+
+Alternative one can install and run as system user or as root (not recommended)
 
 ```bash
-# Run the following command as root
-# or non-root system user (can require --user in case of dependencies are not available system-wide)
-#  requires existence of and write permissions to /var/lib/radicale/collections
+# Run the following command as root (not required)
+# or non-root system user (can require --user in case of dependencies are not available system-wide and/or virtual environment)
 python3 -m pip install --upgrade https://github.com/Kozea/Radicale/archive/master.tar.gz
-python3 -m radicale --storage-filesystem-folder=/var/lib/radicale/collections
 ```
 
+Start the service manually, data is stored in a system folder
+
+```bash
+# Start, data is stored in a system folder (requires write permissions to /var/lib/radicale/collections)
+python3 -m radicale --storage-filesystem-folder=/var/lib/radicale/collections --auth-type none
+```
+
+##### common
+
 Victory! Open <http://localhost:5232> in your browser!
-You can log in with any username and password.
+You can log in with any username and password (no authentication is required as long as not proper configured - INSECURE).
 
 #### Windows
 
@@ -93,11 +115,11 @@ Launch a command prompt and type:
 
 ```powershell
 python -m pip install --upgrade https://github.com/Kozea/Radicale/archive/master.tar.gz
-python -m radicale --storage-filesystem-folder=~/radicale/collections
+python -m radicale --storage-filesystem-folder=~/radicale/collections --auth-type none
 ```
 
 Victory! Open <http://localhost:5232> in your browser!
-You can log in with any username and password.
+You can log in with any username and password (no authentication is required as long as not proper configured - INSECURE).
 
 ### Basic Configuration
 
@@ -831,27 +853,31 @@ Available backends:
 `remote_user`
 : Takes the username from the `REMOTE_USER` environment variable and disables
   HTTP authentication. This can be used to provide the username from a WSGI
-  server.
+  server which authenticated the client upfront. Required to validate, otherwise
+  client can supply the header itself which is unconditionally trusted then.
 
 `http_x_remote_user`
 : Takes the username from the `X-Remote-User` HTTP header and disables HTTP
   authentication. This can be used to provide the username from a reverse
-  proxy.
+  proxy which authenticated the client upfront. Required to validate, otherwise
+  client can supply the header itself which is unconditionally trusted then.
 
 `ldap` _(>= 3.3.0)_
-: Use a LDAP or AD server to authenticate users.
+: Use a LDAP or AD server to authenticate users by relaying credentials from client and handle result.
 
 `dovecot` _(>= 3.3.1)_
-: Use a Dovecot server to authenticate users.
+: Use a Dovecot server to authenticate users by relaying credentials from client and handle result.
 
 `imap` _(>= 3.4.1)_
-: Use an IMAP server to authenticate users.
+: Use an IMAP server to authenticate users by relaying credentials from client and handle result.
 
 `oauth2` _(>= 3.5.0)_
-: Use an OAuth2 server to authenticate users.
+: Use an OAuth2 server to authenticate users by relaying credentials from client and handle result.
+  Oauth2 authentication (SSO) directly on client is not supported. Use herefore `http_x_remote_user`
+  in combination with SSO support in reverse proxy (e.g. Apache+mod_auth_openidc).
 
 `pam` _(>= 3.5.0)_
-: Use local PAM to authenticate users.
+: Use local PAM to authenticate users by relaying credentials from client and handle result..
 
 Default: `none` _(< 3.5.0)_ `denyall` _(>= 3.5.0)_
 
@@ -1038,6 +1064,16 @@ The path to the CA file in pem format which is used to certificate the server ce
 
 Default:
 
+##### ldap_ignore_attribute_create_modify_timestamp
+
+_(>= 3.5.1)_
+
+Add modifyTimestamp and createTimestamp to the exclusion list of internal ldap3 client
+so that these schema attributes are not checked. This is needed at least for Authentik
+LDAP server as not providing these both attributes.
+
+Default: false
+
 ##### dovecot_connection_type = AF_UNIX
 
 _(>= 3.4.1)_
@@ -1302,7 +1338,9 @@ Command that is run after changes to storage. Take a look at the
 Default:
 
 Supported placeholders:
- - `%(user)`: logged-in user
+ - `%(user)s`: logged-in user
+ - `%(cwd)s`: current working directory _(>= 3.5.1)_
+ - `%(path)s`: full path of item _(>= 3.5.1)_
 
 Command will be executed with base directory defined in `filesystem_folder` (see above)
 

config

@@ -74,6 +74,9 @@
 ## Expiration time of caching failed logins in seconds
 #cache_failed_logins_expiry = 90
 
+# Ignore modifyTimestamp and createTimestamp attributes. Required e.g. for Authentik LDAP server
+#ldap_ignore_attribute_create_modify_timestamp = false
+
 # URI to the LDAP server
 #ldap_uri = ldap://localhost
 
@@ -220,10 +223,13 @@
 
 # Command that is run after changes to storage, default is emtpy
 #  Supported placeholders:
-#   %(user): logged-in user
+#   %(user)s: logged-in user
+#   %(cwd)s : current working directory
+#   %(path)s: full path of item
 #  Command will be executed with base directory defined in filesystem_folder
 #  For "git" check DOCUMENTATION.md for bootstrap instructions
-# Example: git add -A && (git diff --cached --quiet || git commit -m "Changes by \"%(user)s\"")
+# Example(test): echo \"user=%(user)s path=%(path)s cwd=%(cwd)s\"
+# Example(git): git add -A && (git diff --cached --quiet || git commit -m "Changes by \"%(user)s\"")
 #hook =
 
 # Create predefined user collections

pyproject.toml

@@ -3,7 +3,7 @@ name = "Radicale"
 # When the version is updated, a new section in the CHANGELOG.md file must be
 # added too.
 readme = "README.md"
-version = "3.5.0"
+version = "3.5.1.dev"
 authors = [{name = "Guillaume Ayoub", email = "guillaume.ayoub@kozea.fr"}, {name = "Unrud", email = "unrud@outlook.com"}, {name = "Peter Bieringer", email = "pb@bieringer.de"}]
 license = {text = "GNU GPL v3"}
 description = "CalDAV and CardDAV Server"
@@ -33,6 +33,7 @@ dependencies = [
     "passlib",
     "vobject>=0.9.6",
     "pika>=1.1.0",
+    "requests",
 ]
 
 

radicale/app/put.py

@@ -165,7 +165,7 @@ class ApplicationPartPut(ApplicationBase):
              bool(rights.intersect(access.permissions, "Ww")),
              bool(rights.intersect(access.parent_permissions, "w")))
 
-        with self._storage.acquire_lock("w", user):
+        with self._storage.acquire_lock("w", user, path=path):
             item = next(iter(self._storage.discover(path)), None)
             parent_item = next(iter(
                 self._storage.discover(access.parent_path)), None)

radicale/app/report.py

@@ -2,7 +2,11 @@
 # Copyright © 2008 Nicolas Kandel
 # Copyright © 2008 Pascal Halter
 # Copyright © 2008-2017 Guillaume Ayoub
-# Copyright © 2017-2018 Unrud <unrud@outlook.com>
+# Copyright © 2017-2021 Unrud <unrud@outlook.com>
+# Copyright © 2024-2024 Pieter Hijma <pieterhijma@users.noreply.github.com>
+# Copyright © 2024-2024 Ray <ray@react0r.com>
+# Copyright © 2024-2024 Georgiy <metallerok@gmail.com>
+# Copyright © 2024-2025 Peter Bieringer <pb@bieringer.de>
 #
 # This library is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -171,7 +175,11 @@ def xml_report(base_prefix: str, path: str, xml_request: Optional[ET.Element],
                        xmlutils.make_human_tag(root.tag), path)
         return client.FORBIDDEN, xmlutils.webdav_error("D:supported-report")
 
-    props: Union[ET.Element, List] = root.find(xmlutils.make_clark("D:prop")) or []
+    props: Union[ET.Element, List]
+    if root.find(xmlutils.make_clark("D:prop")) is not None:
+        props = root.find(xmlutils.make_clark("D:prop")) # type: ignore[assignment]
+    else:
+        props = []
 
     hreferences: Iterable[str]
     if root.tag in (

radicale/auth/htpasswd.py

@@ -113,6 +113,7 @@ class Auth(auth.BaseAuth):
                         "The htpasswd encryption method 'bcrypt' or 'autodetect' requires "
                         "the bcrypt module (entries found: %d)." % self._htpasswd_bcrypt_use) from e
             else:
+                self._has_bcrypt = True
                 if self._encryption == "autodetect":
                     if self._htpasswd_bcrypt_use == 0:
                         logger.info("auth htpasswd encryption is 'radicale.auth.htpasswd_encryption.%s' and bycrypt module found, but currently not required", self._encryption)
@@ -122,8 +123,8 @@ class Auth(auth.BaseAuth):
                 self._verify = functools.partial(self._bcrypt, bcrypt)
             else:
                 self._verify = self._autodetect
-                self._verify_bcrypt = functools.partial(self._bcrypt, bcrypt)
-            self._has_bcrypt = True
+                if self._htpasswd_bcrypt_use:
+                    self._verify_bcrypt = functools.partial(self._bcrypt, bcrypt)
         else:
             raise RuntimeError("The htpasswd encryption method %r is not "
                                "supported." % self._encryption)

radicale/auth/ldap.py

@@ -63,6 +63,12 @@ class Auth(auth.BaseAuth):
                 self.ldap = ldap
             except ImportError as e:
                 raise RuntimeError("LDAP authentication requires the ldap3 module") from e
+
+        self._ldap_ignore_attribute_create_modify_timestamp = configuration.get("auth", "ldap_ignore_attribute_create_modify_timestamp")
+        if self._ldap_ignore_attribute_create_modify_timestamp:
+            self.ldap3.utils.config._ATTRIBUTES_EXCLUDED_FROM_CHECK.extend(['createTimestamp', 'modifyTimestamp'])
+            logger.info("auth.ldap_ignore_attribute_create_modify_timestamp applied")
+
         self._ldap_uri = configuration.get("auth", "ldap_uri")
         self._ldap_base = configuration.get("auth", "ldap_base")
         self._ldap_reader_dn = configuration.get("auth", "ldap_reader_dn")
@@ -242,7 +248,7 @@ class Auth(auth.BaseAuth):
                 logger.debug("_login3 LDAP groups of user: %s", ",".join(self._ldap_groups))
             if self._ldap_user_attr:
                 if user_entry['attributes'][self._ldap_user_attr]:
-                    login = user_entry['attributes'][self._ldap_user_attr][0]
+                    login = user_entry['attributes'][self._ldap_user_attr]
                     logger.debug(f"_login3 user set to: '{login}'")
             conn.unbind()
             logger.debug(f"_login3 {login} successfully authenticated")

radicale/config.py

@@ -259,6 +259,10 @@ DEFAULT_CONFIG_SCHEMA: types.CONFIG_SCHEMA = OrderedDict([
             "value": "1",
             "help": "incorrect authentication delay",
             "type": positive_float}),
+        ("ldap_ignore_attribute_create_modify_timestamp", {
+            "value": "false",
+            "help": "Ignore modifyTimestamp and createTimestamp attributes. Need if Authentik LDAP server is used.",
+            "type": bool}),
         ("ldap_uri", {
             "value": "ldap://localhost",
             "help": "URI to the ldap server",

radicale/storage/multifilesystem/__init__.py

@@ -147,8 +147,13 @@ class Storage(
     def __init__(self, configuration: config.Configuration) -> None:
         super().__init__(configuration)
         logger.info("Storage location: %r", self._filesystem_folder)
-        self._makedirs_synced(self._filesystem_folder)
+        if not os.path.exists(self._filesystem_folder):
+            logger.warning("Storage location: %r not existing, create now", self._filesystem_folder)
+            self._makedirs_synced(self._filesystem_folder)
         logger.info("Storage location subfolder: %r", self._get_collection_root_folder())
+        if not os.path.exists(self._get_collection_root_folder()):
+            logger.warning("Storage location subfolder: %r not existing, create now", self._get_collection_root_folder())
+            self._makedirs_synced(self._get_collection_root_folder())
         logger.info("Storage cache subfolder usage for 'item': %s", self._use_cache_subfolder_for_item)
         logger.info("Storage cache subfolder usage for 'history': %s", self._use_cache_subfolder_for_history)
         logger.info("Storage cache subfolder usage for 'sync-token': %s", self._use_cache_subfolder_for_synctoken)
@@ -170,7 +175,9 @@ class Storage(
         logger.debug("Storage cache action logging: %s", self._debug_cache_actions)
         if self._use_cache_subfolder_for_item is True or self._use_cache_subfolder_for_history is True or self._use_cache_subfolder_for_synctoken is True:
             logger.info("Storage cache subfolder: %r", self._get_collection_cache_folder())
-            self._makedirs_synced(self._get_collection_cache_folder())
+            if not os.path.exists(self._get_collection_cache_folder()):
+                logger.warning("Storage cache subfolder: %r not existing, create now", self._get_collection_cache_folder())
+                self._makedirs_synced(self._get_collection_cache_folder())
         if sys.platform != "win32":
             if not self._folder_umask:
                 # retrieve current umask by setting a dummy umask

radicale/storage/multifilesystem/lock.py

@@ -1,7 +1,8 @@
 # This file is part of Radicale - CalDAV and CardDAV server
 # Copyright © 2014 Jean-Marc Martins
 # Copyright © 2012-2017 Guillaume Ayoub
-# Copyright © 2017-2019 Unrud <unrud@outlook.com>
+# Copyright © 2017-2022 Unrud <unrud@outlook.com>
+# Copyright © 2023-2025 Peter Bieringer <pb@bieringer.de>
 #
 # This library is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -37,10 +38,11 @@ class CollectionPartLock(CollectionBase):
         if self._storage._lock.locked == "w":
             yield
             return
-        cache_folder = os.path.join(self._filesystem_path, ".Radicale.cache")
+        cache_folder = self._storage._get_collection_cache_subfolder(self._filesystem_path, ".Radicale.cache", ns)
         self._storage._makedirs_synced(cache_folder)
         lock_path = os.path.join(cache_folder,
                                  ".Radicale.lock" + (".%s" % ns if ns else ""))
+        logger.debug("Lock file (CollectionPartLock): %r" % lock_path)
         lock = pathutils.RwLock(lock_path)
         with lock.acquire("w"):
             yield
@@ -54,11 +56,12 @@ class StoragePartLock(StorageBase):
     def __init__(self, configuration: config.Configuration) -> None:
         super().__init__(configuration)
         lock_path = os.path.join(self._filesystem_folder, ".Radicale.lock")
+        logger.debug("Lock file (StoragePartLock): %r" % lock_path)
         self._lock = pathutils.RwLock(lock_path)
         self._hook = configuration.get("storage", "hook")
 
     @types.contextmanager
-    def acquire_lock(self, mode: str, user: str = "") -> Iterator[None]:
+    def acquire_lock(self, mode: str, user: str = "", *args, **kwargs) -> Iterator[None]:
         with self._lock.acquire(mode):
             yield
             # execute hook
@@ -73,8 +76,17 @@ class StoragePartLock(StorageBase):
                 else:
                     # Process group is also used to identify child processes
                     preexec_fn = os.setpgrp
-                command = self._hook % {
-                    "user": shlex.quote(user or "Anonymous")}
+                # optional argument
+                path = kwargs.get('path', "")
+                try:
+                    command = self._hook % {
+                        "path": shlex.quote(self._get_collection_root_folder() + path),
+                        "cwd": shlex.quote(self._filesystem_folder),
+                        "user": shlex.quote(user or "Anonymous")}
+                except KeyError as e:
+                    logger.error("Storage hook contains not supported placeholder %s (skip execution of: %r)" % (e, self._hook))
+                    return
+
                 logger.debug("Executing storage hook: '%s'" % command)
                 try:
                     p = subprocess.Popen(

setup.py.legacy

@@ -20,7 +20,7 @@ from setuptools import find_packages, setup
 
 # When the version is updated, a new section in the CHANGELOG.md file must be
 # added too.
-VERSION = "3.5.0"
+VERSION = "3.5.1.dev"
 
 with open("README.md", encoding="utf-8") as f:
     long_description = f.read()
@@ -38,6 +38,7 @@ web_files = ["web/internal_data/css/icon.png",
 
 install_requires = ["defusedxml", "passlib", "vobject>=0.9.6",
                     "pika>=1.1.0",
+                    "requests",
                     ]
 bcrypt_requires = ["bcrypt"]
 ldap_requires = ["ldap3"]