documentation/search/search_index.json

{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"Getting Started","text":""},{"location":"#what-is-this","title":"What is this","text":"<p>txtdot is a proxy that requests the page by the given URL, extracts only useful data including text, links, pictures and tables, and returns it as an HTML page with a minimalistic design optimized for text reading.</p> <p>txtdot increases the loading speed and reduces client's bandwidth usage since no unnecessary code and no scripts are transferred. Also, you won't see any advertisement (unless it's a static picture that is hard to detect as ads). There are no trackers too.</p>"},{"location":"#how-to-use-it","title":"How to use it","text":"<p>txtdot is an open source software, so everyone can host it on his own server. The official instance is txt.dc09.ru, the list of all instances is here.</p> <p>On the main page, there's a handy form where you can specify a URL, choose an engine and a format for parsed data. On the <code>/get</code> page, \"Home\" button returns you to <code>/</code>, \"Original page\" opens the entered URL in the same window without txtdot proxy.</p> <p>The latest docs for API endpoints can be found here. For handy JSON API, use <code>/api/parse</code> returning an engine result object (see below). For pure HTML response, use <code>/api/raw-html</code>. Note that both API and browser endpoints on txt.dc09.ru are ratelimited to 2 requests per second.</p>"},{"location":"#how-it-works","title":"How it works","text":"<p>This project exists thanks to great Mozilla's Readability.js library. The initial idea was to process HTML with it on the server so the client does not need to download and execute heavy JS, doesn't need to use an adblock.</p> <p>Readability performs its work very well in most cases.</p> <p>If an <code>?engine=</code> parameter wasn't passed, but txtdot found that a specific engine is assigned to the requested domain, for example, <code>\"stackoverflow.com\": engines.StackOverflow</code>, it uses that engine to process the URL. Otherwise, the page is parsed with the engine assigned to <code>*</code> (it's Readability).</p>"},{"location":"#plugins","title":"Plugins","text":"<p>Readability is good, but now always, so artegoser wrote the basis of the code keeping in mind that we'll extend txtdot with other engines. Back then, it was functions taking a URL as a parameter, returning an object that contains extracted HTML and plain text, page title and language. The object is rendered with ejs template (or, in <code>/api/parse</code>, just sent as JSON).</p> <p>But after a while it became unwieldy and we decided to create a monorepo. We created classes Engines, Middlewares that handle the necessary parts. Now you can create such functions for different domains, and routes. Also we added support for JSX for simplifying the code of plugins.</p>"},{"location":"#engines","title":"Engines","text":"<p>Creation of engines is easy.</p> <pre><code>import { Engine, Route } from \"@txtdot/sdk\";\n\nconst Readability = new Engine(\n  \"Readability\", // Name of the engine\n  \"Engine for parsing content with Readability\", // Description\n  [\"*\"] // Domains that use this engine\n);\n\nReadability.route(\"*path\", async (input, ro: Route&lt;{ path: string }&gt;) =&gt; {\n  // ...\n\n  // If any of the parameters except content is empty, txtdot will try to extract it from the page automatically\n  return {\n    content: parsed.content,\n    title: parsed.title,\n    lang: parsed.lang,\n  };\n});\n</code></pre>"},{"location":"#middlewares","title":"Middlewares","text":"<p>Creation of middlewares similar to engines.</p> <pre><code>import { Middleware } from \"@txtdot/sdk\";\n\nconst Highlight = new Middleware(\n  \"Highlight\",\n  \"Highlights code with highlight.js only when needed\",\n  [\"*\"]\n);\n\nHighlight.use(async (input, ro, out) =&gt; {\n  if (out.content.indexOf(\"&lt;code\") !== -1)\n    return {\n      ...out,\n      content: &lt;Highlighter content={out.content} /&gt;,\n    };\n\n  return out;\n});\n</code></pre>"},{"location":"docker/","title":"Docker","text":"<p>If you prefer hosting without Docker, see Self-Hosting instead.</p> <p>Download docker-compose.yml and txtdot configs, edit them and then start the container:</p> <pre><code>wget https://raw.githubusercontent.com/TxtDot/txtdot/main/docker-compose.yml\nwget -O .env https://raw.githubusercontent.com/TxtDot/txtdot/main/.env.example\nnano .env\ndocker compose up -d\n</code></pre> <p>Alternatively, you can configure txtdot with the <code>environment</code> section of docker-compose config (don't forget to remove .env and <code>volumes</code>).</p>"},{"location":"env/","title":"Configuring","text":"<p>txtdot can be configured either with environment variables or with the <code>.env</code> file in the working directory which has higher priority. For sample config, see <code>.env.example</code>.</p>"},{"location":"env/#server-settings","title":"Server Settings","text":""},{"location":"env/#host","title":"HOST","text":"<p>Default: <code>0.0.0.0</code></p> <p>Host where HTTP server should listen for connections. Set it to <code>127.0.0.1</code> if your txtdot instance is behind reverse proxy, <code>0.0.0.0</code> otherwise.</p>"},{"location":"env/#port","title":"PORT","text":"<p>Default: <code>8080</code></p> <p>Port where HTTP server should listen for connections.</p>"},{"location":"env/#timeout","title":"Timeout","text":"<p>Default: <code>0</code></p> <p>Max response time in milliseconds. If it's reached, the request is aborted. If set to <code>0</code>, the timeout is disabled.</p>"},{"location":"env/#reverse_proxy","title":"REVERSE_PROXY","text":"<p>Default: <code>false</code></p> <p>Set it to <code>true</code> only if your txtdot instance runs behind reverse proxy. Needed for processing X-Forwarded headers.</p>"},{"location":"env/#proxy","title":"Proxy","text":""},{"location":"env/#proxy_res","title":"PROXY_RES","text":"<p>Default: <code>true</code></p> <p>Whether to allow proxying images, video, audio and everything else through your txtdot instance.</p>"},{"location":"env/#img_compress","title":"IMG_COMPRESS","text":"<p>Default: <code>true</code></p> <p>Whether to compress images through your txtdot instance.</p>"},{"location":"env/#documentation","title":"Documentation","text":""},{"location":"env/#swagger","title":"SWAGGER","text":"<p>Default: <code>false</code></p> <p>Whether to add <code>/doc</code> route for Swagger API docs.</p>"},{"location":"env/#third-party","title":"Third-party","text":""},{"location":"env/#searx_url","title":"SEARX_URL","text":"<p>SearXNG base URL, if set, txtdot will use it for searching and add search form to the page with /search route.</p>"},{"location":"env/#webder_url","title":"WEBDER_URL","text":"<p>Webder base URL, if set, txtdot will use it for rendering web pages.</p>"},{"location":"reverse/","title":"Reverse Proxy","text":""},{"location":"reverse/#nginx","title":"Nginx","text":"<p>Basically, you just need to set the domain, TLS certificates, Host and X-Forwarded headers (so txtdot could know the hostname) and pass all requests to txtdot.</p> <pre><code>server {\n    listen 443 ssl http2;\n    listen [::]:443 ssl http2;\n\n    # Replace the domain\n    server_name txt.dc09.ru;\n\n    ssl_certificate ...pem;\n    ssl_certificate_key ...key;\n    # More options here:\n    # https://ssl-config.mozilla.org/#server=nginx&amp;config=modern\n\n    location / {\n        # Replace 8080 port if needed\n        proxy_pass http://127.0.0.1:8080;\n\n        proxy_set_header Host $host;\n        proxy_set_header X-Real-IP $remote_addr;\n        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;\n        proxy_set_header X-Forwarded-Proto $scheme;\n    }\n}\n</code></pre> <p>On the official instance, TLS is configured in the main nginx config, so we omit these options below.</p> <p>Nginx serves static files faster than NodeJS, let's configure it:</p> <pre><code>server {\n    ...\n\n    location /static/ {\n        alias /home/txtdot/src/dist/static/;\n    }\n}\n</code></pre> <p>What about rate-limiting? We don't want the hackers to overload our proxy.</p> <p>The config below rate-limits to 2 requests per second, allows to put up to 4 requests into the queue, sets the maximum size for zone to 10 megabytes. See the Nginx blog post for detailed explanation.</p> <pre><code>limit_req_zone $binary_remote_addr zone=txtdotapi:10m rate=2r/s;\n\nserver {\n    ...\n    location / {\n        limit_req zone=txtdotapi burst=4;\n        ...\n    }\n    ...\n}\n</code></pre> <p>Let's put all together. Here's our sample config:</p> <pre><code>limit_req_zone $binary_remote_addr zone=txtdotapi:10m rate=2r/s;\n\nserver {\n    listen 443 ssl http2;\n    listen [::]:443 ssl http2;\n\n    server_name txt.dc09.ru;\n\n    location / {\n        limit_req zone=txtdotapi burst=4;\n        proxy_pass http://127.0.0.1:8080;\n\n        proxy_set_header Host $host;\n        proxy_set_header X-Real-IP $remote_addr;\n        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;\n        proxy_set_header X-Forwarded-Proto $scheme;\n    }\n\n    location /static/ {\n        alias /home/txtdot/src/dist/static/;\n    }\n}\n</code></pre>"},{"location":"reverse/#apache","title":"Apache","text":"<p>Coming soon. If you are familiar with Apache httpd and want to help, write a config here (a small explanation as above also would be great) and open a pull request.</p>"},{"location":"selfhost/","title":"Self-Hosting","text":"<p>If you prefer hosting with Docker, see Docker instead.</p>"},{"location":"selfhost/#install-nodejs-and-npm","title":"Install nodejs and npm","text":"<p>For Debian, Ubuntu: packages in the repository are so old, consider installing them with NodeSource. Minimal required version is NodeJS 18.</p> <p>Other distros:</p> <pre><code># CentOS\nsudo yum install nodejs\n# Arch\nsudo pacman -S nodejs npm\n# Alpine\ndoas apk add nodejs npm\n</code></pre>"},{"location":"selfhost/#create-a-user-for-txtdot","title":"Create a user for txtdot","text":"<p>Almost all distros except Alpine:</p> <pre><code>sudo useradd -r -m -s /sbin/nologin -U txtdot\nsudo -u txtdot bash\n</code></pre> <p>Alpine Linux with busybox and doas:</p> <pre><code>doas addgroup -S txtdot\ndoas adduser -h /home/txtdot -s /sbin/nologin -G txtdot -S -D txtdot\ndoas -u txtdot bash\n</code></pre>"},{"location":"selfhost/#build-config-and-launch","title":"Build, config and launch","text":"<p>Clone the git repository, cd into it:</p> <pre><code>git clone https://github.com/txtdot/txtdot.git src\ncd src\n</code></pre> <p>Copy and modify the sample config file (see the Configuring section):</p> <pre><code>cp .env.example .env\nnano .env\n</code></pre> <p>Install packages, compile TS:</p> <pre><code>npm install\nnpm run build\n</code></pre> <p>Manually start the server to check if it works (Ctrl+C to exit):</p> <pre><code>npm run start\n</code></pre> <p>Log out from the txtdot account:</p> <pre><code>exit\n</code></pre>"},{"location":"selfhost/#add-txtdot-to-autostart","title":"Add txtdot to autostart","text":"<p>Either using systemd unit file:</p> <pre><code>wget https://raw.githubusercontent.com/TxtDot/txtdot/main/config/txtdot.service\nsudo chown root:root txtdot.service\nsudo chmod 644 txtdot.service\nsudo mv txtdot.service /etc/systemd/system/\nsudo systemctl daemon-reload\nsudo systemctl enable txtdot\nsudo systemctl start txtdot\n</code></pre> <p>Or using OpenRC script:</p> <pre><code>wget -O txtdot https://raw.githubusercontent.com/TxtDot/txtdot/main/config/txtdot.init\ndoas chown root:root txtdot\ndoas chmod 755 txtdot\ndoas mv txtdot /etc/init.d/\ndoas rc-update add txtdot\ndoas rc-service txtdot start\n</code></pre> <p>Or using crontab:</p> <pre><code>sudo crontab -u txtdot -e\n# The command will open an editor\n# Add this line to the end of the file:\n@reboot sleep 10 &amp;&amp; cd /home/txtdot/src &amp;&amp; npm run start\n# Save the file and exit\n</code></pre>"}]}