Performance and load on the Nextcloud server

Where the work happens

The app holds the plan and the record of a migration; it moves no files itself. Reading the sources, migrating the files and their versions and writing the reports is done by the SMART Migration server, on a Windows machine in your network, which writes into Nextcloud Files like any other client. What the Nextcloud server does for the app is store the job definitions and the results, and answer the calls of the SMART Migration server and of the people using the app.

Short calls only

Every call SMART Migration makes is answered at once. No request is held open on the Nextcloud server while SMART Migration waits for work.

One poll per open app

An open app asks Nextcloud one thing at a time: every 5 seconds while a job runs, every 30 seconds while nothing does, and not at all while its browser tab is hidden.

Paged on the server

Every list is filtered, sorted and paged on the server, every figure on the Dashboard is counted by the database, and a tree is built only as far as it is open.

The calls SMART Migration makes

SMART Migration comes to the app; the app never calls out to it. Its calls are few, short, and answered at once.

Call	How often
Asking for the next job	About every 10 seconds while it has a free slot and no job is waiting, and at once again after it was given one. A job set to Ready therefore starts within about 10 seconds.
The heartbeat	Every 30 to 60 seconds, so the Live mark knows the server is running.
Progress, results and the session log	While a job runs; the session log in batches of up to 1,000 messages.
File issues	In batches of up to 1,000 per call.

Nextcloud's own rate limit answers more than 120 requests for a job in a minute from one account with too many requests, so even a client that asks nonstop cannot load the server.

The app open in a browser

An open app polls its own Nextcloud with one small request, and the Live mark, the tables, the run notifications and the Dashboard all follow its answers. While a job runs or is about to start, it asks every 5 seconds; while nothing does, every 30 seconds; while the browser tab is hidden, not at all. A table is read again only when what it shows has changed, so SMART Migration checking in without changing anything costs the open tables nothing. Refresh, at the top right, reads the open tab again at once, one request like the table's own.

The Dashboard's migration figures are counted by the database and kept until a run starts or finishes: file issues recorded while jobs run do not have them counted again.

Sizes it is built and measured for

The app is sized and measured for 100 groups, 50 tags, 25,000 jobs, 75,000 Run History rows, 200,000 file issues and four jobs running at once, on a small server with 2 CPU cores and PHP's memory_limit at 512 MB. At those sizes a tab's list answers in about a second or less, and the poll of an open app in a fraction of a second.

PHP memory per request

Each request of the app takes its PHP memory for a moment and gives it back once it is answered. Measured at the sizes above, with Nextcloud's own share of about 40 MB included:

Request	PHP memory
The poll of an open app, and the Dashboard	About 45 MB
Run History: the tree, the list, the filters and Expand all	45 to 75 MB
Run History: the summary card over a group	About 45 MB
File Issues	85 to 95 MB
Jobs	About 140 MB

Nextcloud writes a warning to its log for every request that uses more than half of memory_limit — 256 MB at 512 MB — so each request of the app stays well clear of it.

Run History keeps growing

Run History is the one list that only grows: its runs are kept until someone deletes them or a job's Retention schedule does, and every recurring job adds more. So its views never hold all the runs in memory. They read them a slice at a time, keep a few numbers per run, and read in full only the runs on screen and the groups that are open. A view of 75,000 runs needs 45 to 75 MB of PHP memory, and a view of 225,000 runs 50 to 115 MB: each further run adds a few hundred bytes at most. The summary card over a group reads that group's runs only.

Plan PHP for the people, not for the app

The app needs no PHP workers of its own: SMART Migration's calls are short, and none is held open. Size PHP-FPM for the people using Nextcloud, as Nextcloud's own tuning advice describes. The largest request, the Jobs tab with 25,000 jobs, needs about 140 MB of PHP memory for a moment, so keep memory_limit at 512 MB, as Nextcloud recommends.

Background jobs

Two background jobs run in Nextcloud's own cron, and neither calls out:

Every 5 minutes, the app passwords made for destination users are revoked once their runs are over — read from a table of a few rows.
Every hour, the jobs' retention schedules are applied: the runs past their job's Retention schedule are deleted, for the jobs that have one.

Databases

The app runs on the databases Nextcloud supports — MariaDB and MySQL, PostgreSQL and SQLite — and is installed and checked on each of them automatically whenever the app changes. Its tables carry the indexes its lists and polls read.

See also