FAQ: Production and performance in the real world

Every group of questions: Nextcloud FAQ

Performance expectations: the real world

A migration never runs in an ideal world. Networks drop, Microsoft 365 throttles, servers reboot for updates, permissions are missing and people keep working in the source. Plan for that, not for the speed of a quiet test.

What a migration involves, and what sets its speed
Progress for jobs without a Discovery run
Size the servers for a large migration

What a migration involves, and what sets its speed

These questions are explained in full in What a migration involves, and what sets its speed

What performance can I expect in the real world?

Less than the speed a quiet pilot run shows, and less than the network's speed. Every file takes many steps on both sides, and over weeks of work there are always network dropouts, throttling in business hours, server reboots and updates, missing permissions, files in use and expired credentials. Measure the speed in a pilot run through ordinary working days, read the Dashboard's forecast as the best case, and plan the calendar with a margin.

Why does a migration take longer than copying the files?

Every file takes the same steps on both sides: it is found in the source, looked up and compared in the destination, read, written and given its dates, versions and metadata. For many small files those steps take far longer than the bytes.

Why is the throughput so much lower than the network's speed?

It is the volume moved per hour of a job's whole time — preparing, comparing with the destination, versions, metadata, throttling and the calls to both sides included — not the bare transfer speed.

How does the Dashboard work out the time left?

The files left, times the average time per file, divided by the jobs running at once. The time per file is each migration's time from its start in the log to its latest log row, divided by the files it moved, so the preparing is shared out over the files.

What does Finished about on the Dashboard mean?

A guess from now: the files left at the rate the running jobs reach now, added up. It reacts at once to a slow source or throttling, and with nothing running there is no guess.

Why does the Dashboard show a time left and a throughput when nothing is running?

Both come from the migrations that have already run: the time left is the files left at their average time per file, one job at a time, and the throughput their average per job. While nothing runs, the lines say so — at the past pace and past migrations — and only Finished about waits for a running job.

What interruptions should I plan for?

Network dropouts, Microsoft 365 throttling, server reboots and updates, missing permissions, files in use, expired certificates or passwords, a full disk, and the days decisions and fixes take. Each costs the time until someone runs the job again.

Does the time left on the Dashboard include interruptions?

Only those met inside the runs, such as throttling waits. Hours when no job ran — a server down, a job waiting for a fix — are not in it: read it as the best case, and plan a margin.

Progress for jobs without a Discovery run

These questions are explained in full in Progress for jobs without a Discovery run

Why can a job without a Discovery run not be measured precisely?

Nothing has counted its files, versions or volume. The app never reads the source itself, so until a run has counted it, there is nothing exact to measure the job against.

How are jobs without a Discovery run counted in the progress?

A job that has finished a Migration counts with what that run moved; every other one as the average job — the average of every job that has figures. Each bar says how much is estimated, and the estimate improves as jobs finish. While no job has figures at all, the bars show what has been moved so far over full, dotted bars, with no total to measure against.

How far can the estimate be off?

A long way for a single job: an average says nothing about whether a job is a small OneDrive or a site with a million files. A few estimated jobs among many measured ones hardly matter; many of them make the whole progress a rough guide.

Why did the progress bars jump when a job finished?

The job had no Discovery run and was counted at an estimate. When it finished, its real figures replaced the estimate, and the totals moved up or down.

How do I get exact progress figures?

Run the jobs as Discovery first, then switch them to Migration: their Discovery figures carry over, and they are measured precisely from their first migration.

See also

Nextcloud FAQ — every group of questions

Size the servers for a large migration

These questions are explained in full in Size the servers for a large migration

How many jobs and files can the app handle?

Tens of thousands of jobs with their Run History, and at least 100 million files. The app keeps one row per run with its counts, not a row per file, and its tables are filtered and paged on the server; it is measured with 25,000 jobs, 75,000 runs and 200,000 file issues on a server with 2 virtual CPUs, where no request needs more than about 140 MB of PHP memory. Run History can grow well past that: a view of 225,000 runs needs about 115 MB at most.

What computer does Power BI need for a large migration?

Power BI Desktop holds the whole file inventory in memory, and a refresh needs up to twice that. For tens of millions of files, a computer with 64–96 GB of memory is not uncommon. Microsoft's minimum of 2 to 4 GB only runs the program.

How large must the PostgreSQL reporting database server be?

At tens of millions of files, a server of its own: fast SSD or NVMe storage with several hundred GB, 32–64 GB of memory and 8 cores or more as a starting point, and a fast network to the SMART Migration server and to the computers that refresh the report.

How must the Nextcloud server be prepared for a large migration?

Every file is a request of its own to Nextcloud, so size it for the files per hour four jobs write at once: enough PHP workers, Redis for file locking, a fast database and storage, and limits that allow the largest files. Keep PHP's memory_limit at 512 MB: the app's largest request needs about 140 MB of it. Put the SMART Migration server's address on the brute-force allow list, so a failed sign-in cannot throttle it.

Can the Power BI service hold the report for a large migration?

Only on a capacity large enough for it. With Power BI Pro a semantic model holds at most 1 GB; on a Premium or Fabric capacity it can grow to the capacity's memory — 25 GB on an F64 — and a refresh can need twice the model's size. Power BI Desktop, which is free, needs no capacity at all.