If you prefer hosting without Docker, see Self-Hosting instead.
+Docker Engine and Docker Compose are required.
+Note that built images are not provided via Docker Hub. +If you can't or don't want to build them on your server +and don't want to setup a CI/CD system, +let us know, +we'll consider setting up a GitHub Actions workflow.
+git clone https://github.com/txtdot/txtdot.git
+cd txtdot
+docker compose build
+docker compose up -d
+txtdot can be configured either with environment variables
+or with the .env file in the working directory which has higher priority.
+For sample config, see .env.example.
Default: 0.0.0.0
Host where HTTP server should listen for connections.
+Set it to 127.0.0.1 if your txtdot instance is behind reverse proxy,
+0.0.0.0 otherwise.
Default: 8080
Port where HTTP server should listen for connections.
+Default: false
Set it to true only if your txtdot instance runs behind reverse proxy.
+Needed for processing X-Forwarded headers.
Basically, you just need to set the domain, TLS certificates, +Host and X-Forwarded headers (so txtdot could know the hostname) +and pass all requests to txtdot.
+server {
+ listen 443 ssl http2;
+ listen [::]:443 ssl http2;
+
+ # Replace the domain
+ server_name txt.dc09.ru;
+
+ ssl_certificate ...pem;
+ ssl_certificate_key ...key;
+ # More options here:
+ # https://ssl-config.mozilla.org/#server=nginx&config=modern
+
+ location / {
+ # Replace 8080 port if needed
+ proxy_pass http://127.0.0.1:8080;
+
+ proxy_set_header Host $host;
+ proxy_set_header X-Real-IP $remote_addr;
+ proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+ proxy_set_header X-Forwarded-Proto $scheme;
+ }
+}
+On the official instance, TLS is configured in the main nginx config, +so we omit these options below.
+Nginx serves static files faster than NodeJS, let's configure it:
+server {
+ ...
+
+ location /static/ {
+ alias /home/txtdot/src/dist/static/;
+ }
+}
+What about rate-limiting? We don't want the hackers to overload our proxy.
+The config below rate-limits to 2 requests per second, +allows to put up to 4 requests into the queue, +sets the maximum size for zone to 10 megabytes. +See the Nginx blog post for detailed explanation.
+limit_req_zone $binary_remote_addr zone=txtdotapi:10m rate=2r/s;
+
+server {
+ ...
+ location / {
+ limit_req zone=txtdotapi burst=4;
+ ...
+ }
+ ...
+}
+Let's put all together. +Here's our sample config:
+limit_req_zone $binary_remote_addr zone=txtdotapi:10m rate=2r/s;
+
+server {
+ listen 443 ssl http2;
+ listen [::]:443 ssl http2;
+
+ server_name txt.dc09.ru;
+
+ location / {
+ limit_req zone=txtdotapi burst=4;
+ proxy_pass http://127.0.0.1:8080;
+
+ proxy_set_header Host $host;
+ proxy_set_header X-Real-IP $remote_addr;
+ proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+ proxy_set_header X-Forwarded-Proto $scheme;
+ }
+
+ location /static/ {
+ alias /home/txtdot/src/dist/static/;
+ }
+}
+Coming soon. +If you are familiar with Apache httpd and want to help, +write a config here (a small explanation as above also would be great) +and open a pull request.
+ + + + + + +txtdot is a proxy that requests the page by the given URL, extracts only useful data including text, links, pictures and tables, and returns it as an HTML page with a minimalistic design optimized for text reading.
txtdot increases the loading speed and reduces client's bandwidth usage since no unnecessary code and no scripts are transferred. Also, you won't see any advertisement (unless it's a static picture that is hard to detect as ads). There are no trackers too.
"},{"location":"#how-to-use-it","title":"How to use it","text":"txtdot is an open source software, so everyone can host it on his own server. The official instance is txt.dc09.ru, the list of all instances is here.
On the main page, there's a handy form where you can specify a URL, choose an engine and a format for parsed data. On the /get page, \"Home\" button returns you to /, \"Original page\" opens the entered URL in the same window without txtdot proxy.
The latest docs for API endpoints can be found here. For handy JSON API, use /api/parse returning an engine result object (see below). For pure HTML response, use /api/raw-html. Note that both API and browser endpoints on txt.dc09.ru are ratelimited to 2 requests per second.
This project exists thanks to great Mozilla's Readability.js library. The initial idea was to process HTML with it on the server so the client does not need to download and execute heavy JS, doesn't need to use an adblock.
Readability performs its work very well in most cases. But not always. For example, check any StackOverflow page or Google search results.
So artegoser wrote the basis of the code keeping in mind that we'll extend txtdot with other engines. For now, engines are functions taking a URL as a parameter, returning an object that contains extracted HTML and plain text, page title and language. The object is rendered with ejs template (or, in /api/parse, just sent as JSON).
If an ?engine= parameter wasn't passed, but txtdot found that a specific engine is assigned to the requested domain, for example, \"stackoverflow.com\": stackoverflow, it uses that engine to process the URL. Otherwise, the page is parsed with the engine assigned to * (it's Readability).
Install Node and NPM:
# Debian, Ubuntu\nsudo apt install nodejs npm\n# CentOS\nsudo yum install nodejs\n# Arch\nsudo pacman -S nodejs npm\n# Alpine\ndoas apk add nodejs npm\nCreate a user for txtdot, log in:
# Not Alpine (coreutils)\nsudo useradd -r -m -s /sbin/nologin -U txtdot\nsudo -u txtdot -i\n\n# Alpine (busybox)\ndoas addgroup -S txtdot\ndoas adduser -h /home/txtdot -s /sbin/nologin -G txtdot -S -D txtdot\ndoas -u txtdot bash\nClone the repo:
git clone https://github.com/txtdot/txtdot.git src\nInstall packages, compile TS:
cd src\nnpm install\nnpm run build\nManually start the server to check if it works (Ctrl+C to exit):
npm run start\nLog out from txtdot account: exit
Either using systemd unit file:
wget https://github.com/TxtDot/txtdot/blob/main/txtdot.service\nsudo chown root:root txtdot.service\nsudo chmod 755 txtdot.service\nsudo mv txtdot.service /etc/systemd/system/\nsudo systemctl daemon-reload\nsudo systemctl enable txtdot\nsudo systemctl start txtdot\nOr using OpenRC script:
wget -O txtdot https://github.com/TxtDot/txtdot/blob/main/txtdot.init\ndoas chown root:root txtdot\ndoas chmod 755 txtdot\ndoas mv txtdot /etc/init.d/\ndoas rc-update add txtdot\ndoas rc-service txtdot start\nOr using crontab:
sudo crontab -u txtdot -e\n# The command will open an editor\n# Add this line to the end of the file:\n@reboot sleep 10 && cd /home/txtdot/src && npm run start\n# Save the file and exit\nDocker Engine and Docker Compose are required.
Note that built images are not provided via Docker Hub. If you can't or don't want to build them on your server and don't want to setup a CI/CD system, let us know, we'll consider setting up a GitHub Actions workflow.
git clone https://github.com/txtdot/txtdot.git\ncd txtdot\ndocker compose build\ndocker compose up -d\ntxtdot is a proxy that requests the page by the given URL, extracts only useful data including text, links, pictures and tables, and returns it as an HTML page with a minimalistic design optimized for text reading.
txtdot increases the loading speed and reduces client's bandwidth usage since no unnecessary code and no scripts are transferred. Also, you won't see any advertisement (unless it's a static picture that is hard to detect as ads). There are no trackers too.
"},{"location":"#how-to-use-it","title":"How to use it","text":"txtdot is an open source software, so everyone can host it on his own server. The official instance is txt.dc09.ru, the list of all instances is here.
On the main page, there's a handy form where you can specify a URL, choose an engine and a format for parsed data. On the /get page, \"Home\" button returns you to /, \"Original page\" opens the entered URL in the same window without txtdot proxy.
The latest docs for API endpoints can be found here. For handy JSON API, use /api/parse returning an engine result object (see below). For pure HTML response, use /api/raw-html. Note that both API and browser endpoints on txt.dc09.ru are ratelimited to 2 requests per second.
This project exists thanks to great Mozilla's Readability.js library. The initial idea was to process HTML with it on the server so the client does not need to download and execute heavy JS, doesn't need to use an adblock.
Readability performs its work very well in most cases. But not always. For example, check any StackOverflow page or Google search results.
So artegoser wrote the basis of the code keeping in mind that we'll extend txtdot with other engines. For now, engines are functions taking a URL as a parameter, returning an object that contains extracted HTML and plain text, page title and language. The object is rendered with ejs template (or, in /api/parse, just sent as JSON).
If an ?engine= parameter wasn't passed, but txtdot found that a specific engine is assigned to the requested domain, for example, \"stackoverflow.com\": stackoverflow, it uses that engine to process the URL. Otherwise, the page is parsed with the engine assigned to * (it's Readability).
If you prefer hosting without Docker, see Self-Hosting instead.
Docker Engine and Docker Compose are required.
Note that built images are not provided via Docker Hub. If you can't or don't want to build them on your server and don't want to setup a CI/CD system, let us know, we'll consider setting up a GitHub Actions workflow.
git clone https://github.com/txtdot/txtdot.git\ncd txtdot\ndocker compose build\ndocker compose up -d\ntxtdot can be configured either with environment variables or with the .env file in the working directory which has higher priority. For sample config, see .env.example.
Default: 0.0.0.0
Host where HTTP server should listen for connections. Set it to 127.0.0.1 if your txtdot instance is behind reverse proxy, 0.0.0.0 otherwise.
Default: 8080
Port where HTTP server should listen for connections.
"},{"location":"env/#reverse_proxy","title":"REVERSE_PROXY","text":"Default: false
Set it to true only if your txtdot instance runs behind reverse proxy. Needed for processing X-Forwarded headers.
Basically, you just need to set the domain, TLS certificates, Host and X-Forwarded headers (so txtdot could know the hostname) and pass all requests to txtdot.
server {\n listen 443 ssl http2;\n listen [::]:443 ssl http2;\n\n # Replace the domain\n server_name txt.dc09.ru;\n\n ssl_certificate ...pem;\n ssl_certificate_key ...key;\n # More options here:\n # https://ssl-config.mozilla.org/#server=nginx&config=modern\n\n location / {\n # Replace 8080 port if needed\n proxy_pass http://127.0.0.1:8080;\n\n proxy_set_header Host $host;\n proxy_set_header X-Real-IP $remote_addr;\n proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;\n proxy_set_header X-Forwarded-Proto $scheme;\n }\n}\nOn the official instance, TLS is configured in the main nginx config, so we omit these options below.
Nginx serves static files faster than NodeJS, let's configure it:
server {\n ...\n\n location /static/ {\n alias /home/txtdot/src/dist/static/;\n }\n}\nWhat about rate-limiting? We don't want the hackers to overload our proxy.
The config below rate-limits to 2 requests per second, allows to put up to 4 requests into the queue, sets the maximum size for zone to 10 megabytes. See the Nginx blog post for detailed explanation.
limit_req_zone $binary_remote_addr zone=txtdotapi:10m rate=2r/s;\n\nserver {\n ...\n location / {\n limit_req zone=txtdotapi burst=4;\n ...\n }\n ...\n}\nLet's put all together. Here's our sample config:
limit_req_zone $binary_remote_addr zone=txtdotapi:10m rate=2r/s;\n\nserver {\n listen 443 ssl http2;\n listen [::]:443 ssl http2;\n\n server_name txt.dc09.ru;\n\n location / {\n limit_req zone=txtdotapi burst=4;\n proxy_pass http://127.0.0.1:8080;\n\n proxy_set_header Host $host;\n proxy_set_header X-Real-IP $remote_addr;\n proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;\n proxy_set_header X-Forwarded-Proto $scheme;\n }\n\n location /static/ {\n alias /home/txtdot/src/dist/static/;\n }\n}\nComing soon. If you are familiar with Apache httpd and want to help, write a config here (a small explanation as above also would be great) and open a pull request.
"},{"location":"selfhost/","title":"Self-Hosting","text":"If you prefer hosting with Docker, see Docker instead.
"},{"location":"selfhost/#install-nodejs-and-npm","title":"Install nodejs and npm","text":"For Debian, Ubuntu: packages in the repository are so old, consider installing them with NodeSource. Minimal required version is NodeJS 18.
Other distros:
# CentOS\nsudo yum install nodejs\n# Arch\nsudo pacman -S nodejs npm\n# Alpine\ndoas apk add nodejs npm\nAlmost all distros except Alpine:
sudo useradd -r -m -s /sbin/nologin -U txtdot\nsudo -u txtdot bash\nAlpine Linux with busybox and doas:
doas addgroup -S txtdot\ndoas adduser -h /home/txtdot -s /sbin/nologin -G txtdot -S -D txtdot\ndoas -u txtdot bash\nClone the git repository, cd into it:
git clone https://github.com/txtdot/txtdot.git src\ncd src\nCopy and modify the sample config file (see the Configuring section):
cp .env.example .env\nnano .env\nInstall packages, compile TS:
npm install\nnpm run build\nManually start the server to check if it works (Ctrl+C to exit):
npm run start\nLog out from the txtdot account:
exit\nEither using systemd unit file:
wget https://raw.githubusercontent.com/TxtDot/txtdot/main/config/txtdot.service\nsudo chown root:root txtdot.service\nsudo chmod 644 txtdot.service\nsudo mv txtdot.service /etc/systemd/system/\nsudo systemctl daemon-reload\nsudo systemctl enable txtdot\nsudo systemctl start txtdot\nOr using OpenRC script:
wget -O txtdot https://raw.githubusercontent.com/TxtDot/txtdot/main/config/txtdot.init\ndoas chown root:root txtdot\ndoas chmod 755 txtdot\ndoas mv txtdot /etc/init.d/\ndoas rc-update add txtdot\ndoas rc-service txtdot start\nOr using crontab:
sudo crontab -u txtdot -e\n# The command will open an editor\n# Add this line to the end of the file:\n@reboot sleep 10 && cd /home/txtdot/src && npm run start\n# Save the file and exit\nInstall Node and NPM:
-# Debian, Ubuntu
-sudo apt install nodejs npm
-# CentOS
+If you prefer hosting with Docker, see Docker instead.
+Install nodejs and npm
+For Debian, Ubuntu: packages in the repository are so old,
+consider installing them with NodeSource.
+Minimal required version is NodeJS 18.
+Other distros:
+# CentOS
sudo yum install nodejs
# Arch
sudo pacman -S nodejs npm
# Alpine
doas apk add nodejs npm
-Create a user for txtdot, log in:
-# Not Alpine (coreutils)
-sudo useradd -r -m -s /sbin/nologin -U txtdot
-sudo -u txtdot -i
-
-# Alpine (busybox)
-doas addgroup -S txtdot
+Create a user for txtdot
+Almost all distros except Alpine:
+sudo useradd -r -m -s /sbin/nologin -U txtdot
+sudo -u txtdot bash
+
+Alpine Linux with busybox and doas:
+doas addgroup -S txtdot
doas adduser -h /home/txtdot -s /sbin/nologin -G txtdot -S -D txtdot
doas -u txtdot bash
-Clone the repo:
+Build, config and launch
+Clone the git repository, cd into it:
git clone https://github.com/txtdot/txtdot.git src
+cd src
+
+Copy and modify the sample config file (see the Configuring section):
+cp .env.example .env
+nano .env
Install packages, compile TS:
-cd src
-npm install
+npm install
npm run build
Manually start the server to check if it works (Ctrl+C to exit):
npm run start
-Log out from txtdot account: exit
-Add txtdot to autostart
+Log out from the txtdot account:
+exit
+
+Add txtdot to autostart
Either using systemd unit file:
-wget https://github.com/TxtDot/txtdot/blob/main/txtdot.service
+wget https://raw.githubusercontent.com/TxtDot/txtdot/main/config/txtdot.service
sudo chown root:root txtdot.service
-sudo chmod 755 txtdot.service
+sudo chmod 644 txtdot.service
sudo mv txtdot.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable txtdot
sudo systemctl start txtdot
Or using OpenRC script:
-wget -O txtdot https://github.com/TxtDot/txtdot/blob/main/txtdot.init
+wget -O txtdot https://raw.githubusercontent.com/TxtDot/txtdot/main/config/txtdot.init
doas chown root:root txtdot
doas chmod 755 txtdot
doas mv txtdot /etc/init.d/
@@ -477,18 +547,6 @@ doas rc-service txtdot start
@reboot sleep 10 && cd /home/txtdot/src && npm run start
# Save the file and exit
-With Docker
-Docker Engine and Docker Compose are required.
-Note that built images are not provided via Docker Hub.
-If you can't or don't want to build them on your server
-and don't want to setup a CI/CD system,
-let us know,
-we'll consider setting up a GitHub Actions workflow.
-git clone https://github.com/txtdot/txtdot.git
-cd txtdot
-docker compose build
-docker compose up -d
-
diff --git a/sitemap.xml.gz b/sitemap.xml.gz
index d6e5a0d3d0ccc699cf5ae55a03d67cc62a2cd4e6..c80ac85dc11823902b10a326ccb0c4069a8dedbd 100644
GIT binary patch
delta 13
Ucmb=gXP58h;5c^o<3#of03t*MWB>pF
delta 13
Ucmb=gXP58h;5cRbVIq4403SF6#{d8T