- What the app handles
- The Nextcloud server
- Throttling: 429 Too Many Requests
- Where the file-level data is kept
- The PostgreSQL server
- The Power BI computer
- The Power BI service
- Size it before the large waves
What the app handles
Section titled “What the app handles”The app keeps one row per job and one row per run, each with the run's counts — files, versions, volume, issues — not a row per file. A run that moved a million files is one row in Run History, so the number of files makes no difference to the app: 100 million files and more are counted without it growing.
Its tables are filtered, sorted and paged on the server, and the Dashboard's figures are counted by the database, so every screen stays quick however large the migration grows. It is measured at this size on a small server, with 2 virtual CPUs and 512 MB for PHP:
| What | Measured with |
|---|---|
| Jobs | 25,000, in 100 groups and with 50 tags |
| Run History rows | 75,000 — and 225,000 for the PHP memory a view needs |
| File issues | 200,000 and more |
| Files counted by the runs | At least 100 million — only their counts are in the app |
| Jobs running at once | 4, the most the SMART Migration server runs |
| PHP memory a request needs | About 140 MB at most, for the Jobs tab; 45 to 95 MB for the other tabs. See PHP memory per request |
The app handles any number of jobs, but a single job should not be too large: a library of hundreds of thousands or millions of files is split into jobs by folder, or it is slow and may not finish. See Sources
The Nextcloud server the files are written to is sized for the files themselves, as the next section explains.
The Nextcloud server
Section titled “The Nextcloud server”SMART Migration writes every file straight into Nextcloud Files over WebDAV, signed in with an app password. The app holds only the jobs and their results, so the files never pass through it. Every file, and every version of it, is a request of its own, so the Nextcloud server is sized for the number of files per hour that four jobs write at once, not only for the volume.
| What | Why it matters |
|---|---|
| PHP workers | Every upload holds a PHP worker while it is written. Too few, and the requests queue: the migration slows down, and so do Nextcloud's pages for everyone else |
| Redis | Nextcloud locks every file it writes. With Redis for file locking and caching, the locks stay out of the database |
| The database | Every file is a row in Nextcloud's file cache, written as the file arrives. A fast database on SSD keeps up with many small files |
| Storage | Fast disks with room for the whole migration, versions included, and quotas on the destination accounts and Team Folders that leave room for it |
| Size limits | PHP's limits, and those of any proxy in front of Nextcloud, on the size and duration of a request must allow the largest files |
| PHP memory | Keep memory_limit at 512 MB, as Nextcloud recommends. The app's largest request, the Jobs tab, needs about 140 MB of it, and Run History reads its runs a slice at a time, so its views stay small as it grows. See Performance and load on the Nextcloud server |
| Work on every upload | Antivirus scanning, server-side encryption and File access control rules add time to every file. Test the speed with them switched on, as they will be in the migration |
Throttling: 429 Too Many Requests
Section titled “Throttling: 429 Too Many Requests”Nextcloud does not limit how many files a signed-in account writes. Its brute-force protection guards against password guessing instead. Every failed sign-in from an address slows down the next requests from that address, the good ones too. After too many, Nextcloud refuses them with 429 Too Many Requests, for hours. Since the SMART Migration server signs in from one address, this stops every job, and the app shows Offline, because the server can no longer reach the app's API either.
A failed sign-in during a migration usually means a password that no longer works: an app password revoked under Settings → Security, a changed password for the service account, or the app password the app issued for a run into a user's files, used after the run was over. Here is how to prevent it, and how to clear it:
Fix the password first
Give the SMART Migration server a working app password. While it keeps sending one that fails, Nextcloud keeps throttling it.
Put the server's address on the allow list
In Nextcloud's Administration settings → Security, add the SMART Migration server's address to the Brute-force allow list and tick Bypass rate limiting for allowed IPs. An address on the list can try passwords without being throttled, so add only that one fixed address.
Clear an address already throttled
occ security:bruteforce:attempts <address>shows how far it has gone, andocc security:bruteforce:reset <address>clears it at once.
Behind a proxy, every client shares one address
When Nextcloud sits behind a reverse proxy without trusted_proxies set in its configuration, it sees every client with the proxy's address. A wrong password typed by any user then throttles everyone, the SMART Migration server included. Set trusted_proxies, so Nextcloud sees each client's own address.
A 429 can also come from in front of Nextcloud: a reverse proxy, a firewall or a hosting provider's protection that limits requests per address. Let the SMART Migration server's address through there too. A 503 Service Unavailable means Nextcloud is overloaded or in maintenance mode. See What a migration involves, and what sets its speed
Where the file-level data is kept
Section titled “Where the file-level data is kept”Every file a Discovery or migration run handles is recorded in the PostgreSQL reporting database: where it is, its size, versions, dates, labels, metadata and what happened to it. That is the data SMART BI.pbix reads, and the data that grows with the migration.
It grows with every run, not only with every file
A file discovered, then migrated in a test and once more for real, is recorded by each of those runs. Recurring catch-up runs add to it too. Plan the reporting database for the runs the project will make, not only for the files in the source.
The PostgreSQL server
Section titled “The PostgreSQL server”At tens of millions of files, the reporting database is a server of its own, not a corner of the SMART Migration server. Every run writes to it while the migration goes on, and a refresh in Power BI reads the whole file inventory from it in one go.
| Starting point at 100 million files | Why | |
|---|---|---|
| Storage | Fast SSD or NVMe, with several hundred GB for the data, its indexes and room to grow | The file rows hold long paths and metadata; slow disks make every refresh and every run's writes wait |
| Memory | 32–64 GB | So the indexes and the data read most stay in memory |
| Processor | 8 cores or more | Runs write while refreshes read |
| Network | A fast link to the SMART Migration server and to the computers that refresh the report | A refresh moves the whole inventory across it |
These are starting points: the real need depends on the metadata the sources carry and on how many runs each file has. A hosted PostgreSQL service works as well, sized the same way. Back the database up as any production database: it is the migration's record.
The Power BI computer
Section titled “The Power BI computer”Power BI Desktop imports the report's data into the computer's memory: while SMART BI.pbix is open, the whole file inventory it holds is in RAM, and a refresh needs up to twice that while the new copy is built beside the old one. Long text columns — paths and file names — compress least and take most of it.
Microsoft's minimum is for the program, not for this data
Microsoft asks for 2 GB of memory, 4 GB recommended, to run Power BI Desktop. That opens the program; it does not hold a file inventory of tens of millions of rows. For a migration of that size, a computer with 64–96 GB of memory is not uncommon.
| Files in the report | Memory, as a rough starting point |
|---|---|
| A few million | 16–32 GB |
| Tens of millions | 64 GB |
| 100 million and more | 64–96 GB, or more |
- 64-bit Windows and the 64-bit Power BI Desktop, a fast processor with many cores, and an SSD.
- Refresh where the memory is: one well-equipped computer, or a virtual machine sized for it, refreshes the report and uploads it to Nextcloud; everyone else opens the uploaded file. Opening needs less than refreshing.
- Keep the report to what you need: leave out columns no page uses, or filter the data to the waves or sites in question. See Adapt the SMART BI report to your needs
The Power BI service
Section titled “The Power BI service”Published to the Power BI service, the report's semantic model has the same appetite, and the licence sets its ceiling. With Power BI Pro, a semantic model can hold at most 1 GB. On a Premium or Fabric capacity, with the large semantic model storage format, a model can grow up to the capacity's memory:
| Fabric capacity | Most memory for one semantic model |
|---|---|
| F32 | 10 GB |
| F64 | 25 GB |
| F128 | 50 GB |
| F256 | 100 GB |
A refresh can need twice the model's size, so a model near half the capacity's memory may fail to refresh. Microsoft's limits change: check them in its documentation before choosing a capacity. Nothing in SMART Migration needs the service; Power BI Desktop is free for everyone. See Power BI for first-time users
Size it before the large waves
Section titled “Size it before the large waves”The figures above are a guide. The size that counts is your own, and a pilot shows it:
Discover a representative part
Run Discovery on a few sites, OneDrives and shares that look like the rest — not only the smallest.
Refresh the report and measure
Refresh
SMART BI.pbixin Power BI Desktop and note the memory it takes in Windows Task Manager, the file's size and the reporting database's size.Scale to the whole migration
Multiply by the files the whole migration holds, with the runs each file will have, and size the PostgreSQL server and the Power BI computer before the large waves start.
See also