Incremental runs: only the files changed since the last run

• Two kinds of incremental
• Incremental Discovery
• Switch it on
• How From Date moves
• Start over with every file
• For DORA: selected libraries, not a backup
• Keep the Run History of an incremental job

Two kinds of incremental

Incremental is said of two things in SMART Migration — what a run reads, and what it writes — and they work together:

Kind	What it does	Where it is set	By default
Incremental run	Each run reads only the files created or modified since the last successful run	Incremental (only files changed since the last successful run), under From Date — Source step, Advanced Settings	Off: every run reads the whole source
Incremental copy	Each run writes only the files that are new, or newer in the source, and leaves the rest	Rules for adding and overwriting files, Add New Overwrite Existing By Newer — Destination step, Advanced Settings	On, for every migration

• Without the incremental run, every run — Discovery included — reads the whole source again, even when the incremental copy then writes only a few files.
• The incremental run speeds up the incremental copy. The copy checks every file a run reads against the destination — is it there, is the source copy newer? A run that reads only the files that changed checks only those, instead of every file in the library: far fewer calls to Nextcloud, and a much shorter run.
• With both, a run reads only what changed and writes only what is new or newer: the shortest run there is.
• A Discovery job writes nothing, so for it only the incremental run matters.

Read more: Rules for adding and overwriting files · Advanced options for discovery and migration

Incremental Discovery

A Discovery job can be incremental too, and a source that is discovered again and again should be.

The first run reads everything

It records every file of the source in the BI data, as any Discovery does.

Every run after it reads the changes

Only the files created or modified since the last successful run: new files, new versions, changed metadata.

The report follows the documents

Each file is in the Power BI report once, so a file found again is brought up to date, not listed twice. Each run's report marks its files New or Modified in its StatusInBI column.

A daily incremental Discovery keeps the BI data current enough to act on, at a fraction of the cost of reading everything every night — the base for a copy of exactly the critical files, picked by a query on that data.

Read more: Discovery · Pick files with a Power BI or SQL query · Business intelligence with Power BI

Switch it on

The setting is on the Source step of the job dialog, under From Date, in Advanced Settings.

1. Open the job and show Advanced Settings

   Open the Discovery, Test Migration or Migration job and switch on Advanced Settings at the top of the dialog.

2. Tick Incremental

   On the Source step, under From Date: Incremental (only files changed since the last successful run). Leave From Date empty for a first run that reads everything, or set the date the first run should start from.

3. Make it recur

   Set Recurrence to Daily, Weekly or Monthly and the Scheduled Date to the time of the first run — at night, for example.

4. Save and set it to Ready

   SMART Migration runs it at its scheduled date, and From Date moves on after every run that ends with no error or warning.

Read more: Scheduling, recurrence and priority · Choose which files a job takes

How From Date moves

When a run ends with no error and no warning in its log, the app sets the job's From Date to the moment that run began — the time of the log's first row, Session start. A run with an error or a warning leaves From Date as it was, so its files are read again.

Run	From Date it starts from	What it reads
The first run	Empty, or the date you set	Every file, or every file from that date on
After a run with no error or warning	When that run began	The files created or modified since
After a run with an error or a warning	Unchanged	The same files again, and what changed since
After Reset From Date	Empty	Every file again — the runs after it follow on from there

• The begin time, not the end. A file changed while a run was scanning is read again by the next run rather than missed.
• A warning counts. A run whose log holds a warning — a file it could not read, for example — does not move the date, so the files behind the warning are read again next time.
• Only forward. A late report from an older run never moves the date back.
• Each run records its window. The From Date a run started from is kept with the run in Run History, so you can see later which changes each run covered.
• Visible on the run. When a run has moved the date, its Message in Run History ends with a line saying so: From Date moved to and the new date.
• Visible on the job. From Date in the job dialog shows where the next run starts, and the job's summary card says Incremental run: Only files changed since the last successful run.

Start over with every file

Reset From Date, under From Date in the job dialog, clears the date when you save: the next run reads every file, and the runs after it follow on from there.

A full scan now and then

An incremental run reads what was added or modified since the last one. To bring the BI data fully in line with the source — files deleted or moved there included — reset From Date now and then, monthly for example, so the next run is a full scan.

For DORA: selected libraries, not a backup

Incremental runs are what make a resilience copy practical: each night reads only what changed, so the whole cycle stays short enough to repeat every night.

Why this matters for DORA

A DORA setup keeps an independent, regularly refreshed copy of the documents a critical business function depends on. With incremental runs, each refresh reads only what changed since the last good copy: runs stay short enough to repeat every day, the load and throttling on SharePoint stay low, and Run History shows for every refresh exactly which window of changes it covered and how it ended — the evidence an auditor asks for.

A DORA copy of selected libraries, not a backup

Microsoft 365 throttles heavy and frequent reads, so no recurring job can keep a whole tenant in sync. Keep the recurring scope to fewer than 20 libraries of under 50,000 files each as a rule of thumb, and never more than about 50 libraries in all. This setup is a resilience copy of selected libraries, to reach in an emergency. It is not a backup of SharePoint Online, SharePoint Server or file shares to Nextcloud: backing up a whole tenant or file server is a job for a backup product built for it.

When the emergency is over, the work done on the copy goes back to the source with a restore run: see Restore: files back from Nextcloud to SharePoint Online or a file share

Read more: Digital Resilience Operation (DORA) — its section Choosing what to protect · Pick files with a Power BI or SQL query · Estimate volume and duration

Keep the Run History of an incremental job

Each run of an incremental job counted only the files that changed since the one before, so no newer run replaces an older one.

Keep these runs

Every run's figures reach the Power BI report through its Run History row. Deleting runs of an incremental job — by hand, or by the job's Retention schedule — takes their data out of the report, and the report no longer adds up to what the source or the copy holds. Keep these runs and leave the job's Retention schedule at None; archive old runs instead to put them out of everyday view.

Important: the Dashboard's figures change only once SMART Migration has purged the runs. Refresh the report when the Power BI heading on the Dashboard shows a green Up to date instead of a red mark. See Run History

See also

• Choose which files a job takes
• Rules for adding and overwriting files
• Scheduling, recurrence and priority
• Pick files with a Power BI or SQL query