Wikimedia-contrib is a collection of user scripts and Toolforge tools intended for users of Wikimedia Foundation wikis.

Contents

• For users
  • Tools
  • User scripts
• First-time setup on Toolforge
  • Tool accounts
  • User scripts
  • Legacy redirects
• Deploy an update
• Server maintenance
  • Background jobs
  • Web logs
  • Deferred requests

For users

Tools

Toolforge is part of the Wikimedia Cloud infrastructure hosted by the Wikimedia Foundation for community-developed tools and bots. These tools provide analysis and data to support wiki editors and functionaries.

• Account Eligibility analyzes a user account to determine whether it's eligible to vote in the specified event.
• Category Analysis analyzes edits to pages in the category tree rooted at the specified category (or pages rooted at a prefix). This is primarily intended for test project analysis by the Wikimedia Foundation language committee.
• Crosswiki Activity measures a user's latest edit, bureaucrat, or sysop activity across all wikis.
• Global Groups shows a live review of extra permissions assigned to global groups on Wikimedia Foundation wikis.
• Global User Search searches and filters global users on Wikimedia wikis.
• Magic Redirect redirects to an arbitrary URL with tokens based on user and wiki filled in. This is primarily intended for Wikimedia templates (see example).
• Stalktoy shows global details about a user across all Wikimedia wikis. You can provide an account name (like Pathoschild), an IPv4 address (like 127.0.0.1), an IPv6 address (like 2001:db8:1234::), or a CIDR block (like 212.75.0.1/16 or 2600:3C00::/48).
• Stewardry estimates which users in a group are available based on their last edit or action.
• Synchbot synchronises user pages across Wikimedia projects in every language. This allows users to create user pages on every wiki, or to have global JavaScript and CSS. (Due to the potential for misuse, this bot is not open-source.)
• User Pages shows a user's pages on all wikis (or finds wikis where they don't have user pages).

User scripts

These user scripts extend the wiki interface seen by a user, and they're sometimes available to all users as gadgets (particularly TemplateScript). See Gadget kitchen for an introduction to user scripts & gadgets.

• ForceLTR enforces left-to-right layout and editing on right-to-left wikis. This resolves editing glitches in many browsers when one's preferred language is left-to-right, and corrects display when the interface language is not right-to-left.
• StewardScript extends the user interface for Wikimedia stewards' convenience. It extends the sidebar (with links to steward pages), Special:Block (with links to stalktoy and Special:CentralAuth if preloaded with a target), Special:CentralAuth (with links to external tools, one-click status selection, a preselected template reason, and convenient links in the 'local accounts' list), global renaming and Special:UserRights (with template summaries).
• TemplateScript adds a menu of configurable templates and scripts to the sidebar. It automatically handles templates for various forms (from editing to protection), edit summaries, auto-submission, and filtering which templates are shown based on namespace, form, or arbitrary conditions. Templates can be inserted at the cursor position or at a preconfigured position, and scripts can be invoked when a sidebar link is activated. TemplateScript is also used as a framework for other scripts, and includes a fully-featured regex editor.
• UseJS imports JavaScript for the current page when the URL contains a parameter like &usejs=MediaWiki:Common.js. It only accepts scripts in the protected MediaWiki: namespace.

First-time setup on Toolforge

This section covers setting up the Toolforge accounts which host the tools from scratch.

Tool accounts

Each tool has its own account, with a subdomain matching its folder name. For example, Stalktoy is at stalktoy.toolforge.org.

To set up a tool:

1. Connect to Toolforge via SSH.
2. Run this script (editing the # configure section as needed):

   # configure
   become stalktoy
   toolName=stalktoy
   
   # add required folders
   mkdir --parents bin cache logs public_html
   
   # add tool files
   git clone https://github.com/Pathoschild/Wikimedia-contrib.git git/wikimedia-contrib
   ln --symbolic git/wikimedia-contrib/tool-labs/.lighttpd.conf .lighttpd.conf
   ln --symbolic git/wikimedia-contrib/tool-labs/backend public_html/backend --relative
   ln --symbolic git/wikimedia-contrib/tool-labs/content public_html/content --relative
   ln --symbolic git/wikimedia-contrib/tool-labs/$toolName public_html/tool --relative
   cp /usr/bin/kubectl bin/kubectl # scheduled jobs (jobs.yaml) can only access home folder
   
   # launch server
   toolforge webservice php8.2 start --cpu 2 --mem 2Gi
   
   # start scheduled jobs (e.g. log rotation)
   toolforge jobs load ~/git/wikimedia-contrib/tool-labs/_scheduledJobs/jobs.yaml

User scripts

The user scripts are deployed to the shared meta account, and made available through the Toolforge static CDN (via https://tools-static.wmflabs.org/meta/scripts/*.js).

To set up meta:

1. Connect to Toolforge via SSH.
2. Run this script:

   # switch to the project
   become meta
   
   # add required folders
   mkdir --parents bin logs
   
   ## add tool files
   git clone https://github.com/Pathoschild/Wikimedia-contrib.git git/wikimedia-contrib
   ln --symbolic git/wikimedia-contrib/tool-labs/.lighttpd.conf .lighttpd.conf
   cp /usr/bin/kubectl bin/kubectl # scheduled jobs (jobs.yaml) can only access home folder
   
   # set up script CDN
   mkdir --parents www/static
   ln --symbolic git/wikimedia-contrib/user-scripts www/static/scripts --relative
   
   ## launch server
   toolforge webservice php8.2 start --cpu 2 --mem 2Gi
   
   ## start scheduled jobs (e.g. log rotation)
   toolforge jobs load ~/git/wikimedia-contrib/tool-labs/_scheduledJobs/jobs.yaml

Legacy redirects

The tools were previously hosted in three shared tool accounts (meta, meta2, and meta3). These still exist to redirect requests to the new per-tool accounts.

To set up meta, see user scripts above.

To set up meta2 or meta3:

1. Connect to Toolforge via SSH.
2. Run this script (editing the # configure section as needed):

   # switch to the project
   become meta2
   
   ## set up tool files
   git clone https://github.com/Pathoschild/Wikimedia-contrib.git git/wikimedia-contrib
   mkdir --parents bin logs public_html
   ln --symbolic git/wikimedia-contrib/tool-labs/.lighttpd.conf .lighttpd.conf
   cp /usr/bin/kubectl bin/kubectl # scheduled jobs (jobs.yaml) can only access home folder
   
   ## launch server
   toolforge webservice php8.2 start --cpu 2 --mem 2Gi
   
   ## start scheduled jobs (e.g. log rotation)
   toolforge jobs load ~/git/wikimedia-contrib/tool-labs/_scheduledJobs/jobs.yaml

Deploy an update

To update one tool account:

1. Connect to Toolforge via SSH.
2. Run this script (editing the # configure section as needed):

   # configure
   become stalktoy
   
   # update tool
   git -C git/wikimedia-contrib pull --ff-only --no-stat
   cp --update /usr/bin/kubectl bin/kubectl
   
   # update scheduled jobs
   toolforge jobs flush
   toolforge jobs load ~/git/wikimedia-contrib/tool-labs/_scheduledJobs/jobs.yaml
   
   # (optional) restart service to bypass caching, or if .lighttpd.conf changed
   webservice restart

To update every tool account at once:

1. Connect to Toolforge via SSH.
2. Run this script from your main account (not a tool account) which has access to all the tools:

   for toolName in accounteligibility catanalysis crossactivity globalgroups gusersearch magicredirect stalktoy stewardry userpages meta meta2 meta3 meta-dev; do
       echo "=============== $toolName ==============="
       become "$toolName" bash -s <<'EOF'
           set -o errexit -o nounset
   
           # update tool
           git -C "$HOME/git/wikimedia-contrib" pull --ff-only --no-stat
           cp --update /usr/bin/kubectl "$HOME/bin/kubectl"
   
           # update scheduled jobs
           toolforge jobs flush
           toolforge jobs load "$HOME/git/wikimedia-contrib/tool-labs/_scheduledJobs/jobs.yaml"
   
           # (optional) restart service to bypass caching, or if .lighttpd.conf changed
           webservice restart
   EOF
       [ $? -eq 0 ] || echo "FAILED: $toolName"
       echo
   done

Server maintenance

Background jobs

Each tool account runs three background jobs, configured in tool-labs/_scheduledJobs/jobs.yaml. The deploy steps delete any background job that isn't defined in that file.

task	schedule	description
rotate-logs	daily¹	Move each log file into a dated and compressed ~/logs/old backup, and delete backups older than a week.
track-server-load	continuous	Maintains a server-load.json file with Lighttpd connection metrics, used for request deferral.
report-errors	daily¹	Read error.log files covering the preceding 24 hours, and send an email notification to tool maintainers for any errors found.

¹ Toolforge runs each daily job at a random time of day. The random time can be different for each job (even within one tool), but it's fixed for a given tool + job name across deploys.

Web logs

These logs are created automatically on each tool account:

• ~/error.log logs Lighttpd errors (enabled by default).
• ~/logs/access.log logs each incoming request via Lighttpd (configured via ~/.lighttpd.conf).
• ~/logs/job-*.log + ~/logs/job-*.err logs output from scheduled jobs.

The log files are rotated into ~/logs/old daily.

Deferred requests

Each tool supports 'deferring' a request, which means that the user must submit the form to confirm before the results are shown.

This happens in two cases:

1. The URL has a defer=1 query argument.

   The tools add the argument when outputting HTML links to expensive requests, and JavaScript removes it when a user interacts with the link element. Since bots usually read URLs directly from the HTML and don't submit forms, that prevents bots from triggering a cascade of expensive queries.

2. The tool account is overloaded (i.e. too many requests are getting queued), usually due to an excessive spike in bot traffic.

   In that case, some requests get deferred automatically to prevent a death spiral. The deferred portion is proportional to the queue size.