mirrors/prosody
1
0
Fork 0
mirror of https://github.com/bjc/prosody.git synced 2025-04-03 05:07:42 +03:00
Code Issues Projects Releases Packages Wiki Activity Actions
13812 commits 12 branches 100 tags 40 MiB
Commit graph

132 commits

Author SHA1 Message Date
Jonas Schäfer
b9f73bc724 util.datamanager: fix duplicated word in log message 2024-11-16 09:20:29 +01:00
Kim Alvefur
934bd2a65e util.datamanager: Fix missing pack format when reading first index entry 
Thanks MattJ
 2023-11-30 18:09:53 +01:00
Kim Alvefur
b97c4ffc16 util.datamanager: Load first item into index earlier 
Should get rid of fseek() call
 2023-11-27 00:14:56 +01:00
Kim Alvefur
94f9295042 util.datamanager: Clean up list index files on purge (i.e. user deletion) 2023-11-02 17:35:10 +01:00
Kim Alvefur
8cf6d8d959 util.datamanager: Always reset index after list shift 
Shifting the index does not work reliably yet, better to rebuild it from
scratch. Since there is minimal parsing involved in that, it should be
more efficient anyway.
 2023-07-22 14:02:01 +02:00
Kim Alvefur
e6d0a3e639 util.datamanager: Add way to close indexed list store 2023-07-21 18:30:06 +02:00
Kim Alvefur
cd5b2664be util.datamanager: Close file handle when done using it 
It gets closed eventually but at high load they could potentially
lead to reaching FD limits faster.
 2023-07-21 18:28:54 +02:00
Kim Alvefur
1312963f73 util.datamanager: Disable blockwise removal 
In desperate need of tests
 2023-07-21 18:28:16 +02:00
Kim Alvefur
8699c98ad8 util.datamanager: Disable block alignment 
Until we have more test coverage. Somehow the index becomes incorrect
after inserting padding, unclear why.
 2023-07-21 18:03:22 +02:00
Kim Alvefur
33986e97b7 util.datamanager: Pad list writes to avoid crossing block boundaries 
By padding items so that they do not cross block boundaries, it becomes
eaiser to delete whole blocks with fallocate() without cutting items
in half, improving efficiency of such operations.

Since list stores are used for message archives, where the most common
deletion operation would be of the oldest entires, at the top of the
file. With this, all blocks that contain items to be removed could be
deleted without needing to read, delete and write out the whole file.
 2023-06-07 00:39:30 +02:00
Kim Alvefur
3346561d43 util.datamanager: Efficiently remove whole blocks to shift lists 
Using the new pposix.remove_blocks() it should be very performant to
delete whole sections of a file, given a supporting file system.
 2023-07-12 11:45:12 +02:00
Kim Alvefur
50ae2083ca util.datamanager: Add way to efficiently remove first items in a list 
Copying data without parsing it should be more performant than parsing
it serializing back.
 2023-07-12 11:42:41 +02:00
Kim Alvefur
deecaafbfb util.datamanager: Fix indexing first item if not at the very start 
If the first item does not start at position 0 then the index function
produces a phantom first entry covering position zero until where the
real first item starts. When using the index, this would make it either
appear as the first item was missing or cause an off-by-one issue with
remaining items.
 2023-07-10 17:19:05 +02:00
Kim Alvefur
5ee4b4e8a6 util.datamanager: Reduce log level of left over debug messages to debug (thanks Trung) 
These were mostly 'warn' to make them stand out from the debug noise
 2023-07-12 10:24:28 +02:00
Kim Alvefur
96bc5fd439 util.datamanager: Fix missing separator in log line 2023-07-10 17:52:52 +02:00
Kim Alvefur
2d69c8c4ea util.datamanager: Halve size of list index 
Instead of storing (start, length) tuples, store the offset to the end
of items and derive length using the previous entry.
 2021-05-14 05:49:35 +02:00
Kim Alvefur
74ea750bc4 util.datamanager: Add O(1) list indexing with on-disk index 
Index file contains offsets and lengths of each item() which allows
seeking directly to each item and reading it without parsing the entire
file.

Also allows tricks like binary search, assuming items have some defined
order.

We take advantage of the 1-based indexing in tables to store a magic
header in the 0 position, so that table index 1 ends up at file index 1.
 2021-05-11 02:09:56 +02:00
Kim Alvefur
43531740f9 util: Prefix module imports with prosody namespace 2023-03-17 16:23:16 +01:00
Kim Alvefur
c9e223e4f7 spelling: non-existing mistakes (thanks timeless) 2022-03-07 00:35:29 +01:00
Kim Alvefur
43351d2b54 Spelling: Fix various spelling mistakes (thanks timeless) 
Words, sometimes I wonder how they even work

Maybe I missed something.
 2022-03-07 00:13:56 +01:00
Kim Alvefur
ac2d84a47b util.datamanager: Support iterating over any file extension 
The 'typ' argument to all other functions is the actual file extension,
but not here for some reason.

May need this for iterating over the .bin files created by
mod_http_file_share in the future.
 2021-02-07 19:23:33 +01:00
Kim Alvefur
2f9a03eba0 util.datamanager: Fix not passing variables as format string to logger (thanks semgrep) 
Passing error messages as format string could lead to errors, although
not a problem anymore since util.format sanitizes this.
 2021-01-16 15:49:46 +01:00
Kim Alvefur
9a83f0e20b Merge 0.11->trunk 2020-02-24 23:33:53 +01:00
Kim Alvefur
d8fda82e40 util.datamanager: Fix iterating over "users" (thanks marc0s) 
The 'store' path componend used to be unescaped until 756a2a00e7e7 added
escaping to address issues with characters like '/' used in PEP, but
with a special case for '_' which was already in common use in 'store'
path components.

Missed adding this escaping here.
 2020-02-24 23:16:08 +01:00
Kim Alvefur
0bbfb60aae util.datamanager: Ignore unused 'errno' variable [luacheck] 2019-12-23 21:01:55 +01:00
Kim Alvefur
43b814a83b vairious: Add annotation when an empty environment is set [luacheck] 2018-02-28 20:06:26 +01:00
Kim Alvefur
2572082717 util.datamanager: Encode the 'store' path component, preserving underscores 2017-10-08 18:34:57 +02:00
Kim Alvefur
33be2cd463 util.datamanager: Use already known index instead of measuring length each iteration 2017-04-17 03:26:21 +02:00
Kim Alvefur
0f578fb9f3 util.datamanager: Log where the error came from 2017-03-28 17:31:55 +02:00
Kim Alvefur
5c4a8d13e2 util.datamanager: Rearrange locals 2017-03-28 17:31:24 +02:00
Kim Alvefur
7b4ad0e694 util.datamanager: Use pposix.atomic_append 2017-02-28 11:33:43 +01:00
Kim Alvefur
2e06e8e75d util.datamanager: Remove check for file non-existence since ENOENT tells us that already 2017-03-22 12:02:52 +01:00
Kim Alvefur
d496a96705 util.datamanager: Split long line [luacheck] 2017-03-22 12:00:21 +01:00
Kim Alvefur
cb0953029e util.datamanager: Expose atomic_store as store_raw 2017-03-01 17:37:29 +01:00
Kim Alvefur
c3aae40b19 util.datamanager: Use atomic store function when writing the first entry of a list 2017-03-01 17:15:56 +01:00
Kim Alvefur
f0c676e8c6 util.datamanager: Import value of ENOENT from util.pposix (if available) 2017-03-01 17:04:48 +01:00
Kim Alvefur
fc38b8c4f4 util.datamanager: Ignore ENOENT (no such file) when loading data 2017-03-01 16:44:59 +01:00
Kim Alvefur
f65c017ee1 Fix various spelling mistakes [codespell] 2019-05-03 20:54:24 +02:00
Kim Alvefur
f36fed39e8 util.datamanager: Shorter pattern 2016-11-20 12:06:36 +01:00
Kim Alvefur
f0bd3ec403 util.datamanager: Remove extraneous decoding of + to space, which we never never encode to 2016-11-20 12:05:43 +01:00
Kim Alvefur
e3ab638262 util.datamanager: Use the 'base' argument to tonumber() to indicate hexadecimal 2016-11-20 12:03:24 +01:00
Kim Alvefur
d56bd9dbdb util.datamanager: Add annotations to ignore name clashes [luacheck] 2016-09-24 18:31:16 +02:00
Kim Alvefur
c331fe1594 util.datamanager: Rename variables to avoid name clashes [luacheck] 2016-09-24 18:31:00 +02:00
Kim Alvefur
baba98852d util.datamanager: Import tostring and type (fix global access) 2016-05-22 14:38:07 +02:00
Kim Alvefur
afe389d870 util.datamanager: Explicit handling of each error condition (see #632) 2016-02-27 15:29:56 +01:00
Kim Alvefur
d17785e91f Merge 0.9->0.10 2016-02-25 22:37:41 +01:00
Kim Alvefur
fc2ad03510 util.datamanager: Unreference file handle after closing it to prevent trying to close it again (fixes #632) 2016-02-25 22:36:42 +01:00
Kim Alvefur
30b54f4ad7 util.datamanager: Add append to public api 2015-12-11 20:31:55 +01:00
Kim Alvefur
7816dfc838 util.datamanager: Return extra location info 2015-12-11 20:31:19 +01:00
Kim Alvefur
206330cea1 util.datamanager: Make sure only strings are passed as data to append() 2015-12-11 20:30:39 +01:00