Advanced usage

Running in Docker

webchanges can be run in a Docker container. Please see https://github.com/yubiuser/webchanges-docker for one such implementation.

Making POST requests

The POST HTTP request method is used to submit form-encoded data to the specified resource (server). In webchanges, simply supply your data in the data directive. The method will be automatically changed to POST and, if no Content-type header is supplied, it will be set to application/x-www-form-urlencoded.

If the data needs to be sent in JSON, add the directive data_as_json to the job. If no Content-type header is supplied, it will be set to application/json.

See examples here.

Selecting items from a JSON dictionary

If you are watching JSON-encoded dictionary data but are only interested in the data contained in certain key, you can use the jq filter (Linux/macOS only, ASCII only) to extract it, or write a cross-platform Python command like the one below:

url: https://example.com/api_data.json
user_visible_url: https://example.com
filters:
  - execute: "python3 -c \"import sys, json; print(json.load(sys.stdin)['data'])\""

Escaping of the Python is a bit complex due to being inside a double quoted shell string inside a double quoted YAML string. For example, " code becomes \\\" and \n becomes \\n – and so on. The example below provides seemingly complex escaping and also informs the downstream html reporter that the extracted data is in Markdown:

url: https://example.com/api_data.json
user_visible_url: https://example.com
filters:
  - execute: "python3 -c \"import sys, json; d = json.load(sys.stdin); [print(f\\\"[{v['Title']}]\\n({v['DownloadUrl']})\\\") for v in d['value']]\""
is_markdown: true

Alternatively, you could run a script like this

url: https://example.com/api_data.json
user_visible_url: https://example.com
filters:
  - execute: python3 ~/.config/webchanges/parse.py
is_markdown: true

With the script file ~/.config/webchanges/parse.py containing the following:

# ~/.config/webchanges/parse.py
import json
import sys

data = json.load(sys.stdin)
for v in d['value']:
    print(f"[{v['Title']}]\n({v['DownloadUrl']})")

More advanced programmers can write their own Class and hook it into webchanges.

Selecting HTML elements with wildcards

Some pages appends/generates random characters to the end of the class name, which change every time it’s loaded. For example: contentWrap–qVat7asG contentWrap–wSlxapCk contentWrap–JV0HGsqD etc.

element-by-class does not support this, but XPATH does:

filters:
  - xpath: //div[contains(@class, 'contentWrap-')]
  - html2text

Alternatively, especially if you want to do more custom filtering, you can write an external Python script that uses e.g. Beautiful Soup and call it:

filters:
  - execute: python3 ~/.config/webchanges/content_wrap.py
  - html2text

With the script file ~/.config/webchanges/content_wrap.py containing the following:

# ~/.config/webchanges/content_wrap.py
import os
import re
import sys

from bs4 import BeautifulSoup

data = sys.stdin.read()
soup = BeautifulSoup(data, 'lxml')

# search for "div" elements with the according class
for element in soup.find_all('div', {'class' : re.compile(r'contentWrap-*')}):
    print(element)

More advanced programmers can write their own Class and hook it into webchanges.

.onion (Tor) top level domain name

.onion is a special-use top level domain name designating an anonymous onion service reachable only via the Tor network. As sites with URLs in the .onion pseudo-TLD are not accessible via public DNS and TCP, you need to run a Tor service as a SOCKS5 proxy service and use it to proxy these websites through it, as per this example:

name: A .onion website (unencrypted http)
url: http://www.example.onion
proxy: socks5h://localhost:9050
---
name: Another .onion website
url: https://www.example2.onion
proxy: socks5h://localhost:9050

Note the “h” in socks5h//, which tells the underlying urllib3 library to resolve the hostname using the SOCKS5 server (see here).

Setting up Tor is out of scope for this document, but in Windows install the Windows Expert Bundle from here and execute tor --service install as an Administrator; in Linux the installation of the tor package usually is sufficient to create a SOCKS5 proxy service, otherwise run with tor --options RunAsDaemon 1. Some useful options may be HardwareAccel 1 CircuitPadding 0 ConnectionPadding 0 ClientUseIPv6 1 FascistFirewall 1 (check documentation).

Alternatively (Linux/macOS only), instead of proxying those sites you can use the torsocks (fka torify) tool from the tor package to to make every Internet communication go through the Tor network. Just run webchanges within the torsocks wrapper:

torsocks webchanges

Passing diff output to a custom script

In some situations, it might be useful to run a script with the diff as input when changes were detected (e.g. to start an update or process something). This can be done by combining diff_filter with the shellpipe filter, which can run any custom script.

The output of the custom script will then be the diff result as reported by webchanges, so if it outputs any status, the CHANGED notification that webchanges does will contain the output of the custom script, not the original diff. This can even have a “normal” filter attached to only watch links (the css: a part of the filter definitions):

url: https://example.org/downloadlist.html
filters:
  - css: a
diff_filters:
  - execute: /usr/local/bin/process_new_links.sh

If running on Linux/macOS, please read about file permission restrictions in the filter’s explanation here.

Using word-based differ for Markdown (pandiff)

You can also specify an external diff-style tool (a tool that takes two filenames (old, new) as parameter and returns the difference of the files on its standard output). For example, to get Markdown differences you can use PanDiff:

url: https://example.com/
differ:
  name: command
  command: pandiff

or:

url: https://example.com/
differ:
  name: command
  command: pandiff --to=HTML
  is_html: true

In order for this to work, pandiff needs to be installed separately (see PanDiff.

Creating a separate notification for each change

Each type of reports (Text, HTML or Markdown) have an optional sub-directive separate, which when set to true will cause webchanges to send a report for each job separately instead of a single combined report with all jobs.

These sub-directives are set in the configuration.

Selecting recipients by individual job

Currently, configuring reporters on a per-job basis is not supported. All jobs share the same reporter configuration.

To manage different reporting needs you can use multiple job files each with its separate configuration (--config) or, if using email, setting separate reports (separate: true in config.yaml) and use email filtering to forward the reports as needed.

Using environment variables

The !env tag inserts the value of environment variables anywhere within a string value of the jobs or configuration file. Tag the value with !env and reference each environment variable as ${VAR}, or as ${VAR:-default} to fall back to default when VAR is not set. If a referenced variable is not set and no default is given, webchanges exits with an error reporting the variable’s name and the line it is referenced in.

name: !env 'Support for i386 on NVIDIA ${NVIDIA_SERIES:-570} series in Ubuntu'
url: !env 'http://archive.ubuntu.com/ubuntu/pool/multiverse/n/nvidia-graphics-drivers-${NVIDIA_SERIES:-570}/#1'

This allows a value shared by multiple jobs to be defined in a single place (the environment). It also allows the value to be computed by another program before webchanges is invoked, e.g. in a wrapper script:

export NVIDIA_SERIES=$(curl -s https://www.example.com/versions | grep -oP '^\d+' | head -1)
webchanges

The tag is also useful for keeping secrets out of the YAML files. In a job:

url: https://api.example.com/data
headers:
  Authorization: !env 'Bearer ${API_TOKEN}'

Or in the configuration file:

report:
  telegram:
    bot_token: !env '${TELEGRAM_BOT_TOKEN}'

Please note:

  • The substituted value is always a string, so the tag cannot be used for boolean or numeric directives such as use_browser.

  • If the variable is used inside url or command, changing its value changes the job’s location and therefore its identity: webchanges will treat it as a new job and start its snapshot history afresh.

  • Command line arguments that rewrite the jobs file (--add, --delete and --change-location) save the substituted values, losing the !env tags; --edit is safe, as it operates on the raw text.

  • If your editor uses the YAML language server for validation, add !env to its custom tags setting (e.g. in Visual Studio Code, add "yaml.customTags": ["!env scalar"] to settings.json) to avoid an unresolved tag warning.

Added in version 3.38.

Authenticated requests

Set the Authorization header to provide credentials that authenticate a url job with a server, allowing access to a protected resource. Some of the most popular authentication schemes are Basic, Digest and NTLM. For more information, see here.

Using persistent browser storage (for e.g. authentication)

Some sites may use a combination of cookies and/or their functional equivalent of storing data in ‘Local Storage’ to authenticate or initialize their state and will not display the content you want unless you first authenticate (or accept cookies or whatever). In these circumstances, you can use webchanges with use_browser: true directive and its user_data_dir sub-directive to instruct it to use a pre-existing user directory, which you can pre-initialize beforehand. Specifically:

  1. Create an empty directory somewhere (e.g. mkdir ~/chrome_user_data_webchanges);

  2. Run a Google Chrome browser with the --user-data-dir switch pointing to this directory (e.g. chrome.exe --user-data-dir=~/chrome_user_data_webchanges);

  3. Browse to the site that you’re interested in tracking and log in or do whatever is needed for it to save the authentication data in local storage;

  4. Exit the browser.

You can now run a webchanges job defined like this:

url: https://example.org/usedatadir.html
use_browser: true
user_data_dir: ~/chrome_user_data_webchanges

Attaching to an already-running browser via CDP

Instead of having webchanges launch a new browser for every run, you can attach Playwright to a Chromium browser you have already started yourself, using the Chrome DevTools Protocol (CDP). This is useful when:

  • You want to reuse a long-lived, already-authenticated session (cookies and local storage stay in the running browser between runs).

  • The browser is running on a different machine or inside a container that exposes a remote debugging port.

  • You want to keep a single browser process across many jobs to amortize startup cost.

To use it:

  1. Launch Chrome (or another Chromium build) with the --remote-debugging-port switch, pointing at a user data directory you control:

    chrome.exe --remote-debugging-port=9222 --user-data-dir=~/chrome_user_data_webchanges

  2. In another browser tab (or with curl), open http://127.0.0.1:9222/json/version and copy the value of webSocketDebuggerUrl — this is the endpoint Playwright will connect to.

  3. Configure the job with connect_over_cdp:

    url: https://example.org/needs-existing-session.html
use_browser: true
connect_over_cdp: ws://127.0.0.1:9222/devtools/browser/<id-from-step-2>

    Alternatively, connect_over_cdp: true attaches to the default endpoint ws://127.0.0.1:58489/devtools/browser.

All jobs that use connect_over_cdp are run on a single dedicated worker thread, so each unique CDP URL only requires a manual browser authorization once per run, regardless of how many jobs reference it. The connected browser is then cached for the lifetime of the webchanges process, so consecutive runs (e.g. --test cycles) reuse the same connection. To disable caching (open and close a new CDP connection for every job), set the environment variable WEBCHANGES_BROWSER_CDP_CACHE=0.

Overriding the content encoding

(rare) For web pages with missing or incorrect 'Content-type' HTTP header or whose encoding cannot be correctly guessed by the chardet library our default HTTP client uses, it may be useful to explicitly specify an encoding from Python’s Standard Encodings list like this:

url: https://example.com/
encoding: utf-8

Monitoring the HTTP response status code

To monitor the HTTP response status code of a resource and be notified when it changes, use an external command like curl to extract it. Here’s a job example:

command: curl --silent --output /dev/null --write-out '%{response_code}' https://example.com
name: Example.com response status code
note: Requires curl

Creating job urls based on keywords

webchanges does not support arrays and loops to generate jobs (e.g. to check different pricing of a set of products on a set of shots). The best way to do this is to use some template language outside of webchanges and let it generate the urls.yaml file from that template.

Add bullet points in reports for clarity

You can improve the readability of line-by-line data by adding bullet points using the technique below. Notice the two spaces before the asterisk (*): this ensure it works in Markdown, and therfore also after the `html2text` filter, which outputs Markdown. When using an HTML reporter, these bullet points will be converted to proper <ul><li> tags, and most browsers will display each line with an indented bullet point (●), providing clear visual separation between items.

filters:
  - html2text #  example only: can be used with any or no previous filters!
  - re.sub
      pattern: (?m)^
      repl: '  * '

Speeding up browser jobs by blocking elements

Warning

This Pyppeteer feature is not (yet?) implemented by Playwright, and therefore the block_elements directive is ignored (does nothing) for the time being.

If you’re running a browser job (use_browser: true) and not interested in all elements of a website, you can skip downloading the ones that you don’t care, paying attention that some elements may be required for the correct rendering of the website (always test!). Typical elements to skip include stylesheet, font, image, media, and other, and they can be specified like this on a job-by-job basis:

name: This is a Javascript site
note: It's just a test
url: https://www.example.com
use_browser: true
block_elements:
  - stylesheet
  - font
  - image
  - media
  - other

or like this in the config file for all use_browser: true jobs:

job_defaults:
  browser:
    block_elements:
      - stylesheet
      - font
      - image
      - media
      - other

The full list of supported resources is the following (from here):

  • document

  • stylesheet

  • image

  • media

  • font

  • script

  • texttrack

  • xhr

  • fetch

  • eventsource

  • websocket

  • manifest

  • other