Skip to content
Contact us

GA download and change log

SharePointFixedNumeric values having >6 decimals were returned in scientific format that would make any numeric calculation fail and also fail in SQL commands
SharePointFixedManaged Metadata: When a Term does not exist in the destination Term Store and the Allow any user name or Managed Metadata tag - Only M365 is selected then the Term is allowed to pass through. The issue was that the Term was lower case converted before applied to the destination.
SystemFixedThe Application Icon failed to show one some computers.

FileManagement

FileManagementNewSupport for Nextcloud v32 and newer.
FileManagementEnhancement[File Type] and [File Owner (AD)] is now supported in source filters
FileManagementEnhancement[bytesizeallversions] and [versioncount] is now calculated i Test mode and when using Proxy. This is essential to do BI analysis prior to migration
FileManagementEnhancementNew columns for Document Library [File Size All Versions] and [Version Count]
FileManagementDocumentationUsing wildcards in Filters
FileManagementEnhancementOffload & Migration Support for managed metadata tags that does not resolve against the current TermStore. This enables you to migrate Document Library files having old or foreign terms assigned that do not exist in the destination TermStore
FileManagementEnhancementMS Migration API Updated to the latest .NET 8 version and re-implemented using the new Microsoft recommendations to eliminate the use of Azure Queues.
FileManagementEnhancementFile_VersionCount added to the view report_migrated_files. This column shows how many versions were processed for the file.
FileManagementEnhancementFile_SizeAllVersions added to the view report_migrated_files. This column shows the size of the entire file item including all version herstory.
FileManagementJob LogThe status messages now includes the total number of version processed and the total volume processed. This gives a much better indication of speed and the actual volume transferred
FileManagementEnhancementThe option to only update metadata now supports incremental so only files that has an older timestamp are processed. NoFilesOnlyUpdateMetadataByNewer Corresponding override value added.
FileManagementEnhancement5 new tables and centralized Database Server connection for BI. This enables you to update the same database tables from multiple SMART Migration servers. You enable the BI table updates from the options tab in the section Business intelligence - smart_bi_file - smart_bi_filetype - smart_bi_metadata_issue - smart_bi_session - smart_bi_step
FileManagementEnhancementAWS S3 & Azure Blob Storage: Support for Group-By.This allows you to retrieve anonymous blobs and enrich them with any metadata — not just file names and folder structures, but also Managed Metadata and other custom properties — sourced from external systems such as SQL Server, JSON, or CSV files, before directing everything into SharePoint. This approach can even accommodate version history if required.
FileManagementEnhancement[file_accessed] and [file_checkedoutto] added to the view Report_MigratedFiles
FileManagementEnhancementProxy: “MOVE” now supports deletion of proxy items when the file the proxy points to is moved. Also empty proxy folders are deleted. Use [<Override-CopyMove>] = Move DeleteProxy
FileManagementEnhancementAmazon S3 Connection: The AWS S3 Bucket Name can now be 100 characters long. (Extended from 40)
FileManagementFixed2 jobs validating the same user at the exact same time make the job fail
FileManagementFixedA job creating folders and no files will fail when one of the folders to create was created manually or by another job in the time-lap between preparing and executing the fodder creation
FileManagementFixedWhen a File Management name exceeded 40 characters a fatal database error happened.
FileManagementFixedJob tab: When clicking the button Cancel Pending Files a confirmation is now shown with a detail explanation. Previously it just did it without notice.
FileManagementFixedSharePoint: new SharePoint columns did not show in dropdownlist for destination assignment until the Step was reopened
FileManagementFixedSharePoint: Sometimes files could remain “Pending” even the files were successfully migrated and never change status to Created etc.
FileManagementFixedSharePoint Filter: Server-Side filter tab did not always show.
FileManagementFixedFile Connection: Copy to new connection did not work for AWS S3 and Azure BLOB.
FileManagementFixedFile Connection: Control to set upload file size limit was invisible.
FileManagementFixedSharePoint: On very rare occasions, a bug in the Microsoft backend API causes the document content to override the specified content-type. Because the API does not return this information, it is impossible to detect when it occurs. SMART Migration has reissued the content‑type assignment to work around this backend issue.
DataETL
DataETLEnhancementSharePoint destination. Support for automatic and dynamic column mapping (AutoMapAllColumns). This enables you to copy content between 2 custom lists having one or more identical column names.
DataETLEnhancementFileShare inventory: Skip! keyword support in filter
DataETLEnhancementDatabase: Skip! keyword support for the IN SQL where clause keyword when using a Variable.
DataETLEnhancementSQL Server Source/Destination support for column names like From, To, Select that are SQL Server reserved words.
DataETLEnhancementSharePoint (Options Tab): A new option lets you limit how many items are returned from SharePoint. This is useful when a SharePoint list supplies parameters for multiple parallel jobs, reducing load during retrieval. To avoid multiple jobs using the same parameters, use the DataETL CRUD option “Update Items. If unverified then exit Group Step.”
DataETLEnhancementSharePoint On-Prem: Support for [FileSizeAllVersions] and [VersionCount].
Mapping ListEnhancementNew Save button.
DataETLEnhancementSharePoint: New columns for Document Library [File Size All versions] and [version Count] The values are ONLY retrieved when “Include version history” in NOT selected
DataETLEnhancementSharePoint: Sub Sites Hierarchy (VirtualSubSites) now supports Group enabled site collections (like Teams)
DataETLEnhancementSharePoint: Support for Dynamic URL in the virtual lists: - Sub Sites Hierarchy (VirtualSubSites) - Tenant groups (VirtualGroupList) - Tenant Users (VirtualUerList)
DataETLFixedFileShare Discovery: IN filter did not work for more than 1 value
Data ETLFixedSQL Server Bulk mode: A warning was issued when bulk mode cannot initialize and a fallback to conventional SQL is used. This is now changed to a message.
DataETLFixedSQL Server: Bulk mode did not engage when the target table was created externally with certain parameters that forces SQL Server to encapsulate database object names in [ ] for example [dbo].[mytable]
DataETLFixedSharePoint Online: [FileSize] was incorrect for versions when retrieving version history for files in a document library.
Installation
InstallationEnhancementThe default database server for the repository database is now SQL Server Express 2025.
InstallationEnhancementIt is now optional to install “SQL Server Express”. So, if you plan to use a remote SQL Server you no longer need to install the Express version first.
InstallationEnhancementIf SQL Server Express 2022 is already installed then the installation of SQL Server Express 2025 is skipped
InstallationEnhancementThe “Show changelog” is now un-checked.
InstallationEnhancementThe SQL Server 2025 services are now pre configured to start immediately. The Service configuration takes effect after reboot. (Previously “Delayed” – that was sometime annoying after a reboot to wait several minutes for it to start )
SQL Server
SQL ServerEnhancementUser/Password authentication: Password max length is now 250 characters.
SQL ServerEnhancementTransaction retry to accommodate known Azure SQL transient fault – the database is briefly unavailable during load balancing, failover, or maintenance.
SQL ServerEnhancementRepository SQL Server Express: Support for the new limit of 50GB database size in the free SQL Server Express 2025.
SQL ServerEnhancementRepository: SQL Server Express: in the beginning of each job session the used/available space is reported.
SQL ServerFixedUser/Password authentication: Password can now contain ' (single apostrophe).
SQL ServerFixedSQL Server deadlocks
SharePoint
SharePointNew3 New columns for Sensitivity labels (Read Only). More info
SharePointEnhancementDocumentation revised Create M365 App Registration
SharePointEnhancementMigration workload management to ensure the Azure Migration queues do get overloaded
SharePointEnhancementNew setting on the account tab Lower the max URL length (for testing) Setting this to true will effectively change the conditions for when SharePoint’s “too long path or filename” detection is triggered, generating warnings much earlier than SharePoint’s default limits. Use this setting only for testing or understanding what happens when a file is migrated to SharePoint and has a path or filename that exceeds SharePoint’s restrictions. The test values are set to: - 30 characters for maximum path length. - 20 characters for maximum filename length.
SharePointFeature removalThe below SharePoint Online (M365) legacy authentication methods has been removed as they are no longer officially supported by Microsoft: Note that WebLogin is supported only for list items, and it is strongly recommended not to use it unless you specifically need legacy remote trigger support for list items. File uploads are not supported by WebLogin, and rate limit management is unavailable, so large data sets may be throttled. Removed: - App ID/Secret (non AppReg) - User name/Password
SharePointFixedIn rare cases the ‘CT:’ identifier for content-type was stripped from the library display value in the FileMangement and DataETL library dropdown. Happened when the site was provisioned from a DANISH template.
SharePointFixedWhen any metadata — including the filename — was processed using a transformation, and the beginning of the value before the first space could be interpreted as a date, that portion was automatically formatted using the local date format. This resulted in corrupted data. For example, a filename could be unintentionally converted into a date format, which in turn caused validation errors because the file effectively received a new name.
SharePointFixedError when calculating the version size when it exceeded 2 GB.
SharePointFixedBoolean column having empty string as value was updated with ‘False’. Now any value the Boolean column may have is transferred unmodified
Functions
FunctionEnhancementNew function: CallStep( 'Step name or ID' ) Returns Ok, Warning, Error or a system error if the step is not found.
FunctionEnhancementNew function: SPConvertToDirectFolderURL( 'Browser URL' ) Returns the direct url without encoding and aspx references.
FunctionEnhancementNew function: ReplaceFirst Replaces the first occurrences of a string with another string. (Non case sensitive)
FunctionEnhancementNew function for the Match filter function: WildCardToMatch( ‘. wildcard spec’ )
FunctionEnhancementNew function for SQL: WildCardToSQLLike( '*.* wildcard spec' )
FunctionEnhancementNew function for SQL: DateTimeToNumber( 'DateTime' or 'Date' )
FunctionEnhancementNew function for SQL: GetSQLSelect() Returns the latest SQL SELECT statement executed in DataETL or For-Each
FunctionEnhancementNew function FolderExist('my folder') to test if a folder exists or not
Miscellaneous
SystemEnhancementRemove .Net 6.0 dependency
Step UIEnhancementSave All toolbar button in top right corner is now much faster
AccountEnhancementNew button on the Account/Repository tab: Truncate BI tables Use this to reset the data in the BI tables and free up storage in the database.
DashboardEnhancementNew status for how many MS Migration API packages are waiting for Microsoft.
DashboardEnhancementRedesigned toolbar for easier session filtering and querying of files associated with the selected session(s), either within the Query ETL module or via a new DataETL or For-Each Step..
DashboardEnhancementNew column [End]. Shows when the job stopped. The tooltip calculates the job duration.
DashboardEnhancementNew column [SessionLog_id]. Shows the id for the job session. Use this value in various reports.
DashboardEnhancementNew Select files processed in job session to generate query criteria based on the session selected.
Job ServerEnhancementWindows Service: The max length for account and password individually is now 500 characters.
RepositoryEnhancementRepository database connection can now authenticate with username/password
PowerShellFixedDid not work: Set-FlowVariable -Name "[<SessionLog-CustomInfo>]" -Value $MyVariable Set-FlowVariable -Name "[<SessionLog-Name>]" -Value $MyVariable
Mapping ListFixedPaste values from clipboard did not trigger save unless a value was subsequently changed
File ManagementFixedSharePoint destination using Group-By could fail to migrate specific combinations of filenames with % as part of the name or path. - Example: 2019.02.10_% détention.xlsx
File ManagementFixedSharePoint: Unicode characters in user names could in some cases show up encoded when using Allow any user name (Only M365)
Amazon S3 ConnectionEnhancementA specific Bucket can now be configured, eliminating the need to call the SDK to list available buckets. As a result, no permission to list buckets is required.
Connection hyperlinkBehaviorWhen clicked, now works consistent for all connection types and opens the connection browse.
TransformationFixedClicking the documentation help button could crash the program
FileMangementFixedMoving files would fail and work incorrectly when choosing also to delete the root folder if empty.
FileMangementFixedA file in a FileShare having a corrupted date Created or Modified would crash the job
MappingFixedCopying a mapping list could make the name longer than 50 characters thereby generating an error
DashboardFixedRemoved Select files processed in job session. It is a 7.0 feature and was accidently visible in v6.63.x
DataETLFixedFileShare Inventory failed in some cases when including permissions in the scan.
GeneralEnhancementAt start up “Check for updates” can now be disabled (Do not show again)
DataETLFixedDatabase source: selecting/deselecting a column via the column check-box did not react to single-click. Only double-clicking or clinking the column name worked.
FileManagementFixedMigrate file to SharePoint. When a file name or a folder name contained and encoded illegal character like My%5CFolder would fail to migrate because %5C translates into \ when decoded. The name containing the encoding will now be migrated untouched.
FileManagementFixedMigrate file to SharePoint. Under certain conditions an incremental migration would fail if the path contained illegal characters encoded in the file or folder names like the following My%5CFolder. The error would be an Azure exception stating that the file is already there.
Job logFixedOn error and the setting for On error was Continue and log a warning then the variable [<SessionLog-Result>] returned Warning instead of Error.
TeamsFixedWhen an error happens the job was set to ‘Warning’ instead of ‘Error’. For example when encountering an invalid URL.
FileManagementEnhancementSharePoint Online: Handling of multiple concurrent jobs that simultaneously upload multiple files to the same destination folder with identical file names.
DataETLEnhancementSQL insert performance. On SQL Server x15 faster and on AzureSQL x50 faster
SystemEnhancementPowerBuilder Runtime upgraded to 2025 build 3683
SystemEnhancementRemoved dependency on Aspose (3rd party .Net extension).
UIChangeModule for SharePoint, Azure, AWS S3 has been renamed to “File Storage” and have a new icon.
ModuleTypeDescription
TeamsFixedValidation: - SharePoint To SharePoint did not work when referencing subfolder in the source - When a fatal error happens for example an invalid URL specified in the the job then the job did not get notified and was “hanging” as running. - Job Date, Job result and Job message was not updated.
FileManagementFixedIf SharePoint source was selected without choosing a library then the Filter button was disabled if switching to FileShare source.
Value MappingFixedWhen renaming a Mapping list in dialog-window mode (via double click) the new name was not reflected in the main Mapping list UI.
SQL ServerFixedAdded prefix N' for strings in dynamically generated SQL to ensure proper handling of Unicode independent of selected database codepage
FileManagementFixedWhen using % (percent) in folder name or file name the migration could in some cases fail
SystemFixedEnhanced Process monitoring to prevent false “Job session was terminated unexpectedly”
MappingFixedThe name of a mapping list could exceed the 50 allowed characters when typing the name.
SystemFixedCall tree system error when clicking a Step.
SharePointFixedMultiple jobs (+10) could trigger a deadlock database error when updating file status.
SharePointFixedIllegal non visual control characters (ascii 0 – 31, 127 – 159) in file name, folder name and metadata is now removed prior to migration.
SharePointFixedHEX encoded Illegal non visual control characters (ascii 0 – 31, 127 – 159) in file name, folder name is now removed prior to migration.
DataETLFixedFileShare scanning: When an exception was thrown by the file system due to lack of permissions or file system corruption the error message logged did not include the filename and folder name
TeamsEnhancementSite provisioning: The following 2 keywords have been added to the list of allowed PowerShell commands if and Add-AzADGroupMember
Value MappingEnhancementFaster UI in the Mapping list module.
SystemEnhancementFaster SMART Migration program start and close.
FileManagementConfigurationNumber of download threads reduced from 10 to 3 when downloading form SharePoint to FileShare. This is to reduce overhead.
ModuleTypeDescription
FileManagementFixedThe character 0×04 (decimal 4) within metadata to a file caused the file to fail migration
FileManagementFixedSharePoint source filter UI did not show CAML server-side filters.
FileManagementFixedA false warning could be issued about a non existing Variable named: - [<Override-PopulateBItables>]
InstallerEnhancementAutomated .net installation
ModuleTypeDescription
DataETLNewUsing the special list Sub Sites Hierarchy you can now make your own custom provisioning features. - [SchemaXML] , [Column Display Name] , [Column Name]. New columns added. Please run “Refresh Connection”. - Columns associated with each List/Library is reported along with column names and SchemaXML specific to each column. - SchemaXML is reported on the List/Library level
FunctionNewXMLEscape('string') to encode a XML string so that all characters can be processed by a XML parser.
FunctionNewWriteFile('File Name', 'Data') write data to the end of an existing file or create a new file.
FunctionNewMappingListModified('Mapping List Name') determine if a given mapping list has been modified dynamically during job execution.
AccountNewThe size of the repository database is now reported on the Repository tab. Also, if running on SQL Server Express, a warning is shown if the used space exceeds 8GB, since the maximum allowed size is 10GB. This serves as an early warning that you are running out of database space and need to purge old logs.
Session LogNewIf running on SQL Server Express the database size is monitored and a warning is issued if the used space exceeds 8GB, since the maximum allowed size is 10GB. This serves as an early warning that you are running out of database space and need to purge old logs.
OffloadEnhancedReporting and error handling in the Restore process of offloaded files is now more consistent and messages are more readable and relevant.
SharePointEnhancedTermStore: Enhanced handling of root-level anchor tags.
SharePointEnhancedTermStore: Enhanced program logging when Diagnostics mode is enabled.
FileManagementFixedThe column [File owner (AD)] Is now returning blank if a files owner property points to an AD user that no longer exists in the AD instead of reporting an error.
FileManagementFixedWhen running many jobs concurrently targeting the same folder structure Azure is reporting an invalid error. This case is now detected and ignored.
DataETLFixedExcel source could in some cases not be configured.
InstallerFixedIssue when upgrading older repositories. Could sometimes throw a SQL error.
SharePointFixedTermStore: Unable to read an anchor tag in the third level.
SharePointFixedTermStore: If there are multiple languages and the default language has been changed then the term hierarchy display and was or incorrect. Also, terms could not be resolved at runtime.
ModuleDescription
UIThe high DPI for crisp and clear UI (Account tab) can now select - <Disabled> (The standard windows option) - DPI unaware, scale using the Windows GDI. Better than <Disabled> - DPI aware, Display crisp and clear UI. The best option but is not supported by all display adapters. This option is the default.
AccountWhen license is expired it is more clearly stated.
InstallerThe executable files (StepsUI.exe, StepsJob.exe, StepsServer.exe) now have the version number embedded. When viewing the .exe file properties, you can see the version number.
FileManagement (Issue fixed)Unable to perform “metadata update only”.
Version 6.60GA – February 11, 2025
DebuggingStop processing and go into “Preview mode” with variable inspection. Right-click on Step in call tree
SQL ServerUpdate support for Azure SQL. Now ODBC 18.4.1 or newer is used to leverage encrypted connection.
SQL ServerText columns are now created with the data type nvarchar
SQL ServerBoth SQL Server version 2019 and 2022 are now supported. You can download the installation SQL Server .BAK file here if you want to install on a remote server
InstallerThe latest MS ODBC 18.x is now a requirement and is automatically installed.
Installer.Net 8.0 is now a requirement and is automatically installed.
FileManagementOffload SharePoint online documents to Azure Blob Storage, Amazon S3 Bucket, FileShare
FileManagementMove option in FileManagement now supports confirmation of successful migration before source file is deleted.
FileManagementDelete support in File Management
DataETLAzure Blob Storage & Amazon S3 Bucket as input to DataETL and FileManagement (analysis and data delivery in M&A instead of USB-stick)
DataETLNew view DeleteFiles
DataETLThe “MigratedFiles” view has 2 extra columns [DetsinationModifiedBy] [DestinationOverwrittenModifiedBy]
DataETLFileShare discovery: support for relative path in folder name split.
PowerShellDynamic connection support for PowerShell LoadSharePointConnectionForPowerShell(URL)
UISupport for high DPI (Account tab)
UIDataETL & FileManagement “Preview mode” is now locked as top window to make it easier to access
UIUI in SMART Migration when using the Teams template: Standard/Advanced Mode
Teams UI
  • Simplified UI in Teams - Simplified install and management in SMART Migration - Support for Offload to AWS S3, Azure Blob Storage, FileShare - PowerApp for Restore of offloaded content (open source) - Site Provisioning - “Move File” option in Migration (Direct with Lift&Shift and via Proxy) - Lift&Shift without proxies - Consolidated Content Reporting - Job import from CSV - Delete support (direct and via Proxy) - Validation (also for migrations done by other products like MS SPMT - OneDrive support using UPN (source e-mail and destination e-mail) - Archive option to move completed jobs to Archive library
General100+ minor optimizations
Version 5.21GA – April 22, 2024
SharePointThe column [Property Bag] is now supported for retrieval.
FileManagementEnhancement to Pending files gives quicker detection of when the files are in fact completed by the associated Azure Migration Job.
FileManagementSharePoint >> FileShare with version history now includes all metadata columns in the associated csv file.
TeamsSupport for empty folders in fie Discovery
Version 5.14March 21, 2024
File ManagementNew options for alternative modified detection
PowerShellSharePoint Online: Support for Connect-PnPOnline using Entra App
Date ETLFile Share Discovery: Support for unlimited File Share size.
Date ETLFile Share Discovery: performance enhancement when scanning FileShares >10.000 files. The destination is populated in parallel. So while the scanning takes place chunks of 10.000 file names are applied to the destination in parallel.
SQL ServerAll templates are now using the UTF8 codepage. Latin1_General_100_CI_AS_SC_UTF8
Version 5.10March 12, 2024
TeamsSMART Migration for Teams: Multiple Teams channels are now processed automatically (Enterprise+ license required)
TeamsSMART Migration for Teams: Enhanced layout in SMART Migration Job
TeamsSMART Migration for Teams: Enhanced Color coding in views
TeamsSMART Migration for Teams: Enhanced issue tracking with direct link to associated Dashboard item from an issue item.
TeamsSMART Migration for Teams: Discovery performance enhancement x5. Now creates up to 80.000 FileProxy items per hour.
FileManagementSharePoint Online: 30-150% performance enhancement
FileManagementSharePoint Online: Folders are now created on demand along with the files.
FileManagementFileShare source: The column [File Type] now has the value <folder> when the item being processed is a folder.
Version 5.0February 16, 2024
TeamsSMART Migration for Teams initial release
SharePointZERO THROTTELING implemented by using the Rate-Limit headers. This effectively eliminates 429 Exceptions
UINumerous enhancement to the UI including new module to do interactive query and ETL without having to create a dedicated Step
InstallerAll prerequisites are now installed automatically such as .NET 6.0 etc.
RepositorySQL Server is now the repository
RepositoryNew ability to mange SQL Server : create new databases etc. (Account tab)
Job ServerAbility to reconnect to SQL Server if it has been temporarily unavailable.
FileManagementAbility to auto-shorten too long path before upload to SharePoint
FileManagementNew property [<IgnoreOnlyNewer>] to force a file overwrite. To activate set the property to 1
FileManagementAbility to update metadata on folders (Options tab)
SharePoint[Approval-status] and [Approval comments] are now migrated as is (if mapped). Previously all files would get the status Approved.
Functions
  • GetInstalledOfficeVersion() - GetJobSessionRateLimitdelay()
ETLDatabase query sow supports the Skip! keyword
Version 4.28December 29, 2023
DashboardSearch for [SessionLog-Id]
DataETLSharePoint Online: Emoji support. Use Windows key and . (dot) to insert emoji
SharePointSupport for: [Folder Child Count] & [Item Child Count]
FileManagementSharePoint: Upload is now returning the GUID for the uploaded file and apply it to the Report_MigratedFiles view. Notice the GUID can be used in SharePoint search for find the file within a site collection. It takes up to 24 hours before the GUID is searchable within SharePoint.
FileManagementValidation including complex data types and handling of identical values accomplished by multiple edits of multi-value values.
FileManagementSharePoint Online: Folder color support
TransformationNew function: ExtractMetadataColumn() String data,String column, String Default value
SharePointSession log: New format for Managed Metadata that show the assigned tag much more clearly and can be used directly in assignments from the New function: ExtractMetadataColumn()
Version 4.01September 16, 2023
DataETLSharePoint: Selecting a different list or library is now keeping settings.
FileManagementSharePoint: Selecting a different library is now keeping settings.
FileManagementSharePoint: Upload is now returning the GUID for the uploaded file and apply it to the Report_MigratedFiles view. Notice the GUID can be used in SharePoint search for find the file within a site collection. It takes up to 24 hours before the GUID is searchable within SharePoint.
DataETLYou can now validate that files have been successfully migrated.
TransformationNew function: ExtractMetadataColumn() String data,String column, String Default value
SharePointSession log: New format for Managed Metadata that show the assigned tag much more clearly and can be used directly in assignments from the New function: ExtractMetadataColumn()
DashboardSearch for [SessionLog-Id]
DataETLSharePoint Online: Emoji support. Use Windows key and . (dot) to insert emoji
Version 3.99August 17, 2023
Session LogSharePoint: New format [Display name<Internal name>]
Duplicates detectionNew Function FileCheckSum that allows you to calculate the file check sum (or hash key) that you can use for file duplicates detection
RepositoryLatest Microsoft MS SQL Server support OLEDB 19.3 May 18, 2023 C:\Windows\SysWOW64\msoledbsql19.dll
VariablesNew Variable SessionLog-Name. Sets the session name visible in the “Job session drop down menu”
SharePointAutomatic time zone detection. The site no longer has to be in the same time zone as the local machine.
FileManagementNew metadata assignment: [<Ignore «Only newer» condition>] When True a file that already exist on the destination is overwritten with the matching source file regardless of it having a newer or identical timestamp compared to the source file. Use this option to force remigration of specific files. Valid values: Unassigned/True/False
File ManagementNew option: [x] Transfer files having data validation warning(s) Files are transferred despite any warnings like: Wrong Content Type Wrong username Invalid managed metadata Invalid dates etc. Please note that the files having data validation issues are inconsistent and will not be considered successfully migrated. (Default it is [x]Checked) IMPORTANT: if you [ ]Uncheck this option you get the benefit of never uploading files that have inconsistent metadata. This makes it significantly easier to fix your data issues on the specific files and then run an incremental job.
Version 3.94July 17, 2023
UIReengineered Step-Tab system with close buttons on the Step-Tabs etc.
UIReengineered dropdown menus and many other minor UI enhancements
DataETLSharePoint: Support for creating folders also in Document Libraries.
FileManagementOverride Variables
Version 3.80May 15, 2023
DataETLLoad into SharePoint now support Create Folder for Custom Lists. To create a folder set the column [Item Type] = 1 (The value 1 is the SharePoint identifier for a folder.) The folder name you can assign to the column [Title] or include it in the folder path that you assign to the columns [Path]. The path columns supports both / (forward slash) and \ (back slash) for hierarchy folder level separation. Features include: - Create folder structures in a Custom List from any other SharePoint Custom List or Library. - Create folders dynamically based on any variable or input data set including FileShare scans. - Support for Content Type for each folder Item. (assign [Content Type ID] with eighter the Content Type Label or GUID.
RepositoryThe SharePoint connections and Database connections now remember the latest used connation and setup when you close and restart the UI.
RepositoryThe SharePont connections Data Tab has 3 new button to refresh the result set.
RepositoryThe Database connections Data Tab has new buttons to refresh the result set, export to excel and Notepad.
File ManagementFilter now support * (wildcard) for the IN operator and the = operator.
OnDemanThe OnDemand configuration wizard now supports AzureAD App authentication.
Version 3.78May 4, 2023
Job LogNew Export to Excel and Export to Notepad of all or selected rows (Multi select of rows is supported). Also, when positioned on an embedded file like Source.txt the report will be opened in the same way as if you r-click the report row. This combined with the global search across all sessions and jobs and also combined with the filter gives you additional analysis capability.
Call treeNew buttons in the variables tab to : Copy , Notepad , Refresh
UIMany minor refinements.
Version 3.73April 26, 2023
SharePointNew system requirement: We are now using PnP.Core and the MS Graph and that requires .Net 6.0 to be installed on the computer.
SharePointSupport for download of files > 2GB
SharePointRedesigned “CAML query” encapsulation to bypass the SharePoint Online 5000 list item threshold limitation. The redesign is done due the the fact that Microsoft is now much more likely to throw the fatal List View Threshold exception when querying indexed columns.
DashboardNew server status check to inform you if the Job server is not running correctly.
Windows Event LogNew fatal error event reporting : The Job Azure “pending files” scanner was unable to connect to the database. No migration results from pending files will be retrieved. Notice that the Azure scanner runs within the Job Server. This event fires when the job server is incorrectly configured, typically the Windows Service account
Version 3.70March 29, 2023
Mapping listsRedesigned to be flat (no dialog box)
Version ControlRedesigned to be flat (no dialog box)
AccountRedesigned to be flat (no dialog box)
Version 3.65March
DashboardQuery across all job session can now run in parallel so you can start a long-running query and continue doing other tasks while it runs.
File reportingQuery for migrated files run in parallel so you can start a long-running query and continue doing other tasks while it runs.
SharePointConnections: New interactive data CRUD tool
Database CSV ExcelConnections: New interactive data CRUD tool. - Simple alternative to SQL Server Management Studio - Supports the local firebird database repository
System UIThe bottom menu have has been moved to the “Repository Tab”
AccountNew button to backup the local database repository.
Version 3.57March
Transformations5 new expression functions to return NULL in SQL Insert/Update Right-click menu Paste function. Use these functions to force a NULL value into a database column when performing INSERT/UPDATE. Example: If( [mycol] = 0 , NullNumber() , [data] ) - NullString() - NullDate() - NullDateTime() - NullNumber() - NullTime()
DataETLLoad into Excel: New option to use an Excel file as template when creating new Workbook reports. Use this feature to make report files that has predefined layout. The layout options includes setting fonts and colors, defining calculations, adding graphics, adding pivot tables, reordering of table columns and much more. This article shows how to reorder table columns Note that you can populate an Excel workbook report with multiple sets of data to different tables and set single values to individual cells. And when populated then copy/move the Workbook report somewhere including to SharePoint.
Version 3.54March
DashboardQuerying for session matching a criteria now supports || (OR) and && (AND) operators in the query string.
DashboardQuerying can now be canceled so you are not forced to wait for the query to complete.
DashboardOpening and querying for sessions in the Dashboard is now x3 faster when using local repository (non SQL Server)
DashboardSessions: New filter as you type option
DashboardWindow Event Log: New filter as you type option along options for filtering event types such as your own or system events
SessionLogNew GLOBAL query option allows you to query the entire repository across all jobs and sessions. This makes it significantly easier to locate log rows not having to know what job it was and to examine logs across entirely different jobs. You can cancel a query so you do not have to wait for it to complete.
SystemIn the account window you can now reset the status for Do not stow again messages so they appear again.
Version 3.52March
SharePointManaged Metadata: Browse terms via the Select Value button now show the Termstore anchor tag GUID. This is useful when troubleshooting sites referenced via dynamic URLs where a given MangedMetadata tag cannot be resolved.
SharePointManaged Metadata: When a Term cannot be resolved the column’s TermStore anchor tag (GUID) is now part of the error message making it easier to troubleshoot. Use the SelectValue on the button on the columns assignment to make sure the anchor tags match.
PowerShellNew option to disable the PowerShell command line argument -NoProfile. Disabling this argument will instruct PowerShell.exe to load whatever is configured in your user profile.
PowerShellNew option to disable automatic TryCatch code encapsulation. Disabling automatic TryCatch encapsulation is NOT recommended unless you are debugging code and want to eliminate any outside interference.
ExpressionsNew function: IsServerJob() It retunes true if the job is running on the Job Server and false if it is running in the UI. This enables you to use different values for accessing external resources such as FileShares when the Job Server runs as a Windows Service on a different account than the UI.
Version 3.51February, 2023
NotificationNew option When error to run the notification ONLY if an error happens.
DataETLLoad into Database – new option: Update records. If nothing is changed then exit Group Step. When the DataETL Step is placed in a Group Step and the database update do not change any data then the flow exits from the Group. This enables you to make sure the current job made the update to the database table, not another parallelly running job. This is useful for designing “Thread Safe” applications where multiple simultaneously running jobs, pick tasks from a database table for example by querying a column [ReadyForProcessing]=1 and then update the column to 0 signaling that “This job has that task”.
Job ServerYou can now Hold a job while it is running. The job status will change to Running to Hold and continue to run and when finished put it self on Hold.
Job ServerNew scheduling option Continuously that forces the job to repeat it self after it finishes. Typically the computer overhead is 1-2 seconds for the repeated job start.
Job ServerFaster job start. The jobs are now started and fast as possible. This becomes important when you run multiple jobs that take advantage of the new Continuously scheduling option.
Job ServerAccount window: New configuration of job server job start. [x] Delay each job start by 10 seconds If checked, the machine workload associated with job start will be reduced. Also, flows can be designed to take advantage of a delayed job start in a multitasking setup. HOWEVER it is strongly recommended to use the new DataETL option: Update records. If nothing is changed then exit Group Step to achieve a thread safe setup.
For-EachGroupBy: When having large groups (>10000 rows) the group call is split into blocks of 10000 to sustain performance for data transformation.
SystemSMART Migration now writes to the Windows Event Log when lost database connection happens, job termination due to server reboot etc. This feature enables you to better track what is going on and feed information to IT Infrastructure Monitoring systems.
NotificationYou can now enable the notification to write to the Windows Event Log. This feature enables you to better track what is going on and feed information to IT Infrastructure Monitoring systems.
DashboardNew tab “Windows Event Log” where you can view the information written to the Windows Event Log eighter by the system or by the job notifications.
UIThe Job Tree now shows “Notification” on the notification tree node
UIFileManagement & DataETL: The assignment now shows more clearly what columns are not been signed.
DevOps Version ControlSeparate environments for Development, Test and Production. Make changes only in the Development environment and cerate a version you then deploy to the Test environment and then when approved deploy to the Production environment.
DataETL ForEachSQL data source: New button Refresh drop-down menu It does a refresh of the columns names from SQL SELECT statement in the destination mapping drop-down menu.
Session logNew step-instance scroll buttons in Job tree. This makes much easier to navigate large session logs.
Session logStep icon at all rows to show what Step class the row relates to
UIRefreshed Step-Icons for easier recognition
Step browserNew r-click menu to Cut/Copy. This makes it easier to find Steps across all folders and copy/move to different folder.
SharePointYou can now use Variables in SharePoint Connections URL and authentication columns to dynamically point to relevant sites in a DevOps Version Control setup.
TransformationsRearranged UI for easier understanding of the transformation sequence.
UIJob tab new <All sessions> in the session dropdown. This enables you to examine all logs consolidated. You can specify a date range filter and a message filter. The message filter supports AND and OR. This is an example of a criteria that look for this or that and must at the same time contain important: this||that&&important
UIJob tab the FILTER and FIND now supports AND and OR. You can use || to specify OR and use && to specify AND. This is an example of a criteria that look for abc or xyz and must at the same time contain 123: abc||xyz&&123
Version 3.34January 2023
DataETL ForEachFileShare discovery: - If a filter is defined a visual indication will show it. - The number of files filtered is added as row in the session log - Enhanced error handling and dynamic error threshold to avoid useless results on large scans when the for example permissions are insufficient. - New Source.txt embedded report holding the scan result.
UIJob session log on Local default repository is now x4 faster in session retrieval and incremental refresh SQLServer based repositories do not benefit. They are already optimized. On both local and SQLServer repository, the Session Log does no longer retrieve incremental log information while not visible making the application more responsive especially when running multiple heavy jobs simultaneously.
UINew “Job tree is replacing the previous “Call tree”. The Job tree shows the entire job-structure where a given Step is referenced and lets you easily navigate the structure. Also, you can se the different Variables used for each Step in the Job. It is open by default but you can close it if you like.
UIJob tab status area rearranged.
UIJob tab Redesigned Filter/Find functionality
UIJob tab New icons added to easier log analysis
UIJob tab Indenting to show sublevels by > > > > is replaced by the new Job tree. this makes it the log easier to read.
UIAccount window rearranged.
UINew keyboard shortcuts: - Ctrl+O Open Step - Ctrl+D Dashboard - Ctrl+J Scheduled Jobs - Ctrl+L Variables - Ctrl+M Value mapping - Ctrl+S Save current Step - Ctrl+R Run current Step
DataETL & ForEachM365: 2 new virtual lists for retrieving user and group information at the tenant level. Requires Azure AD App authentication in the SharePoint connection. Advanced filtering is supported also with Variables. - Tenant users - Tenant groups
SharePointUser mapping now have a drop down menu listing all users. - M365: The Microsoft Graph is used to retrieve the user names (connection using Client/Secret or Azure AD App). If there are insufficient permissions, the users associated with the connection site are retrieved. - On-Prem: the users associated with the connection site are retrieved.
SharePointNew button to export all Active Directory users in M365 to Excel or Notepad.
SharePointUser mapping now have hyper link to documentation: SharePoint user mapping
FileManagementGroupBy: Enhanced error handling and reporting when files in the GroupBy data set does not exist or has invalid URL/UNC.
FileManagementGroupBy: New embedded report file named MissingFiles.txt in the job log for the source files referenced in the GroupData that could not be located.
FileManagementGroupBy: New progress information feed in the phase where the GroupBy data set is initially being analyzed so you better can follow what’s going on.
FileManagementCopy from SharePoint or FileShare: Removed dependency on .Net 3.5
FileManagementCopy from SharePoint or FileShare: New option to Create metadata CSV file
TransformationEnhanced tag extraction from XML
Version 3.19December 2022
VideoNew video that explains about For-Each looping
DataETLRead CSV – automatic detection of UTF8 encoding in CSV files without BOM header. Often CSV files are saved as a standard text files without encoding header (No BOM) even the files contains UTF8 encoded characters. This leads to corrupted international character output when processed by database drivers.
DataETLCSV now supports ¤ as separator character
DataETLLoad Into Database using SQL: Enhanced keyword coloring
TransformationsNew function: ReplaceAll( ‘string1’, ‘searchfor’, ‘replacewith’ ) added to the expression Right-click menu Paste function.
Version 3.16November 29, 2022
YouTubeNew button to open the SMART Migration YouTube channel
YouTubeVideo: Read CSV files – Output to For-Each or SharePoint list/Database table
Variables browseAuto resize window height according to screen resolution
Session dashboardAuto resize window height according to screen resolution
FileManagementThe option [x] Delete folders if empty now also works even if the folders referenced are initially empty e.g. there are no files to process. This feature enables you to clean up empty folders in a hierarchy
PowerShellScripts can now run in both Restricted and Unrestricted environments. To set the policy on the computer start PowerShell in administrator mode and run the following command: Set-ExecutionPolicy -ExecutionPolicy Restricted This is the built-in CMDLETs available to interface your PowerShell scripts
TransformationThe Replace action can now process multiple lines having the same “search value” thereby creating a list og values.
Data ETLThe Load into Database now has a Delete table toolbar button. Much easier that having to issue a DROP TABLE command
version 3.10November 2022
Transformation3 new powerful transformation actions: - Expression. Allows you to have cascading expressions. - Validation. Allows for validation of individual values and adding of validation message to the Session Log. - Tag extraction. Extract any tag from a JSON or XML string.
Mapping ListYou can now use the DataETL with Mapping lists including the SharePoint users mappings list. (Select/Insert/Update/Delete)
For-EachA Mapping list can now be the source for a For-Each loop
Session LogFor-Each/Group-By is now adding a full report to the Session log for each Group call. This enabled detailed tracking and analysis of complex setups.
VariablesNew data type Date
All StepsThe Call-Tree button now show a dropdown window instead of a dialog box. This makes it quicker to use.
Session logM365/File transfer: New button Cancel pending files that becomes visible when the job has completed but Azure has yet to complete the operation. This will signal to Azure to stop the async transfer rom Blob Storage to SharePoint. The status for files not yet confirmed by Azure is updated to “Canceled” and removed from the consolidated reporting.
All StepsNew toolbar button in the upper right corner: Open Steps
RepositoryNew button in the Home section: Browse all Steps . A convenient way to find and open a Step without having to navigate the folders
NotificationThe Options tab now has help text that explains the way notifications work
TransformationsReplace-action has been enhanced with an option to output the matched values within the source string separated by ;# This can be useful for creating single-value or multi-value metadata based on keywords located within for example folder name or file name. (Previously you could only replace the matched values within the source string and return the converted string.)
M365(Preview feature – disabled by default) Asynchronous Metadata Read API (AMR) has been implemented to eliminate throttling caused by reading large Lists & Libraries for example when doing incremental file migration. AMR is an advanced API that Microsoft is enforcing to replace legacy CSOM for bulk read of list item data and library item metadata. According to Microsoft: If this API is not used then the entire Tenant can be at risk of being throttled, thereby severely impacting all users normal online work.
M365(Preview feature – disabled by default) New connection property: Use Asynchronous Metadata Read API The default is Autiomatic however you can disable the API. This very powerful and rather complex Asynchronous Metadata Read API (AMR) is being enforced by Microsoft to avoid throttling. On rare occasions when dealing with lists having few items (<2000) it can make sense to disable it.
FileManagementDynamic URL/UNC when called from GroupBy (ForEach) can now mix multi-value variables, single-value variables and static text.
FileManagementFile duplicate auto-renaming when called from GroupBy (ForEach) For example myfile.docx myfile(1).docx myfile(2).docx ...
DocumentationJob notification. How to create a Power Automate notification flow
Options on all StepsNotification using Power Automate now has a link to the article How to create a Power Automate notification flow
Version 3.00October 2022
File OpenNew button to “Browse all Steps”. This makes it easy to quickly locate Steps across folders by start typing the name.
DashboardNew column Custom info that shows the latest assigned value as specified on the Options tab of any Step in your flow setup. This makes it easy to track and find specific sessions based on the data in context.
DashboardThe query now searches the columns Custom info and Message
DashboardMade larger to display more rows.
VariablesThe Variable’s values are now saved form UI session to UI session. So when you have a variable having a specific value the this value is reloaded the next time SMART Migration launched. On the Variables window you can disable this behavior.
Job notificationWindows notification (sliding window above the system-tray) is triggered when a job is completed. This is not the case for scheduled jobs.
Session logThe status “Pending” is now automatically changed when the file has been processed. So you no longer need to “Retrieve latest”
GeneralWindows-F8 is a HOT-Key to bring SMART Migration in focus
GeneralProgram icon in the Windows System Tray. Click it to bring SMART Migration in focus.
AzureNew real-time statistics columns on the Session log and Session dashboard that show how many migration packages that has been created and submitted to Azure and also how many packages that Azure has picked up for processing.
Version 2.92October , 2022
Microsoft 365SharePoint Connection: Azure AD App authentication type added. The new authentication type enables you to more securely manage the permission given to the SharePoint account used by the connection. Also, here is a step-by-step guide for how to create a self signed certificate, add the Azure AD app and configure it all. The new authentication types are named: - Azure AD App (MyCert.pfx) - Azure AD App (Installed cert)
DocumentationNew article: How to create & configure Azure AD app for authentication. This article is a step-by-step guide for how to create a self signed certificate, add the Azure AD app and configure it all. It is generic and has no content related specifically to SMART Migration
Scheduled jobsYou can now Hold alle jobs from the top panel.
Job tabThe buttons layout in the UI have been updated for optimal use
Job tabYou can now cancel a job running on the Job Server from within the Job tab
Job tabThe job schedule is now showing in the top panel and a hyperlink takes you to the schedule.
DashboardNew column Location that show where the job is running (Window/Job Server).
RepositorySQL Server repository: Upgraded to use Microsoft ODBC Driver 18 for SQL Server released by Microsoft August 8, 2022 The driver is automatically installed by the SMART Migration installer
Version 2.80September 2022
DataETLFileShare discovery: x2-20 scanning performance enhancement depending on number of files. The more files the more significant the permeance enhancement is.
DataETLFileShare discovery: New option to include permissions. (AccessControlList entries also known as the ACL)
DataETLFileShare discovery: New column [AD owner]
DataETLFileShare discovery: New “Data extraction filter” option in toolbar. The filter allows you to do advanced parameterized filtering identically to the “File Selection Filter” in FileManagement. The filter supports Master-Detail filtering when referencing the permissions columns (AccessControlList/ACL). This allows you to find files having specific ACL entries.
InstallerMicrosoft OLE DB Driver for SQL Server v19 is now automatically installed
DatabaseSupport for Microsoft OLE DB Driver for SQL Server v19. Using that driver you will be able to leverage all the latest SQL language features in SQL Server 2022. You can also use it to connect to older versions of SQLServer.
FileShareThe FileShare filter window now has help hyperlink to the documentation for the advanced Match string function that enables “light regular expressions” in the filter criterias
Version 2.69August 2022
GeneralNew Save All button in the top right toolbar to save all open Steps.
GeneralThe Call Tree dialog box is enhanced to also show FROM references. (The Steps from where the current Step is being called from)
For-EachNew ability to create a transformation on a GroupBy assignment. This simplifies the process of making calculated grouping values by eliminating the need for a separate Variable assignment with associated and transformation.
DataETLLoad into Excel: x11 Performance enhancement
SessionLogx14 Performance enhancement when opening reports in Excel
SessionLogx29 Performance enhancement when storing reports in the session log.
SessionLogNew r-click menu item “View report” on embedded reports like Source.txt
SessionLog Data tabData set viewer now shows row info: Current row, Filtered rows, Row count
Data tabWhen changing the limit the new result is automatically extracted.
Performance90% performance enhancement for UI and calculation processing in jobs. This enhancement will make the UI more responsive and make all jobs execute faster. Please note that the transfer to and retrieval from SharePoint is not affected in any way as that depends on network bandwidth and the SharePoint backend, but everything else runs faster.
DataETL/For-EachExtract from Custom delimited file has a new option to specify if the file has a header line that specifies the column names or not.
For-EachNew option for the GroupBy processing [x] Group all rows into 1 and call Step once Each [<#MyCol>] Variable will contain the column values from all rows separated by ;# Use the GroupBy functionality for high performance File-Cherry-Pickling and for combining values for use with multivalued SharePoint columns.
For-EachVariable assignments: Drag&Drop the sequence of the Variable assignments. This is useful when Variables are cross referencing each other and the calculation order matters. Also useful when using the GroupBy processing where the grouping assignments must be the last assignments.
DocumentationNew article how to use of the PowerShell Connect-PnPOnline CMDLET
DocumentationNew article with some simple PowerShell samples
PowerShellSimplified management of PowerShell references to SharePoint connections
Job ServerWindows service account: enhanced error handling of missing account permissions.
DatabaseSupport for Microsoft SQLServer 2022 and all it’s new features. Please make sure to use v19 of the Microsoft OLEDB Driver
WindowsSupport for Windows Server 2022 and Windows 11 2022 update.
InstallerFacelifted and upgraded installer
Version 2.56July 2022
Job ServerSimplified UI for automatic job rescheduling when a fatal error happens. [x] On-error reschedule job This option is default checked.
DocumentationNew article that describes the For-Each GroupBy feature
SQLSupport for the SQL keyword WITH in conjunction with SELECT. This enables advanced result stet generation like a path hierarchy derived from a linked list of records. Useful for migration of folder structures from legacy Document Management Systems (DMS) like IBM FileNet. See syntax examples here. This article describes more detail about selecting Parent-child relational rows.
File ManagementUNC file specification support added for GroupBy arrays
FileManagementWhen uploading files to SharePoint Online the log rows now initially has the values “Pending create” or “Pending replace”. This makes it clear what actions are pending to be completed by Azure
FileManagementSupport for hardcoded destination setup including Rename File and Alternative Path when using GroupBy values from a ForEach Step
Data ETLLoad into Database: Using value mappings is now significantly faster.
Value mappingPaste values copied from Excel is now x50 faster and an hour glass is displayed during the import
Value mappingNew keywords you can mix with text you type in the replacement value. - <today> Converts to todays date. You can use a format string to force a specific format like: <today:yyyy-mm-dd> - <now> Converts to current time. You can use a format string to force a specific format like: <now:hh:mm:ss> - <tab> Converts to TAB character - <newline> Converts to carriage return + line fed Char(13) + Char(11) - <cr> Converts to carriage return character Char(13) - <lf> Converts to line feed character Char(11) - <space> Converts to space character - <semicolon> Converts to semicolon character - <pipe> Converts to pipe character
Version 2.49July 2022
UIFilter as you type in dropdown menus for Variables and columns
FileManagementM365 – Apply users that do not exist to [ModifiedBy] or [CreatedBy]: Performance enhancement of x50 when >100 users do not exist.
DocumentationUpdated documentation for the consolidated reporting views: - report_migratedfiles - report_validatedsharepointusers - report_datavalidationwarnings
Run jobYou can now specify Session name and Custom info when manually starting a job. This information is useful when selecting different job sessions and when querying the the consolidated reporting views.
Extract dataTooltip on the result table generated from the “Extract data button” (upper left toolbar). Useful when querying the the consolidated reporting views
Session LogThe default retention schedule is set to 3000 so that you will by default store 3000 job sessions in the consolidated reporting views.
SharePointEnhanced CAML support for query of nested Managed Metadata tags.
SharePointEnhanced support for creating Document Sets from a For-Each Step using inherited content-types and variables.
DataETLCSV/Excel source: If the MS Access runtime engine is not installed a massage now offers to download from Microsoft and install it.
SystemSteps are now auto reopened when starting the program
Version 2.45June 2022
Selective FirstTimeMigration ReMigration RertryMigrationMajor new feature to selectively: ReMigrate, RetryMigrate, FirstTimeMigrate The selective migration feature is based on the compressive For-Each Step that has access to the new consolidated migration reporting data source that contains reference information and results for all processed files from all SharePoint tenants, farms, sites and fileshares across all jobs and sessions. The selective migration features include: - ReMigration of incorrectly migrated files for example when metadata calculations and user mappings needs to be revised. - RetryMigration files that for some reason failed for example network issues, faulty provisioned sites, bad URLs etc. - FirsTimeMigration of files based on advanced criteria. - Ultra high transfer performance using parallel transfer - Job results are automatically “merged” into the consolidated reporting data source for incremental and successive actions either for reporting or further migration. There are no useless CSV or Excel sheets floating around. - Run a job in test-mode to simulate the transfer and thereafter investigate the results and selectively decide what files to process for real. And in the process make changes for dynamic metadata, URLs, users and mappings etc. - Selection is done using the simple UI in the For-Each Step (source) and you can use raw parameterized SQL if you desire. - Embedding of the selective migration feature in any Step flow (automation) since it is based on a standard For-Each Step - Any data source can be used for the selective migration. Not only the internal consolidated reporting data sources.
GeneralRevised Tabbed UI for easier navigation between open Steps.
SharePointManaged metadata sub term query support. You can now query for a specific Managed Metadata sub term using the PnP ParentTerm|SubTerm syntax. Also direct term GUID is supported using {my-term-guid}. Sub term criteria and GUID criteria works using the operators: = (Equal) and <> Not Equal The GUID based query is activated only when specifying | (pipe). So if you just specify a term label then all terms matching the label regardless of position in the term hierarchy are considered.
SharePointManaged metadata sub term path extraction support. When you retrieve values from Managed Metadata columns the full term path is now returned.
For-EachGroupBy in combination with FileManagement now support subfolders. That enables you to GroupBy an entire Library. So you can use a grouped variable (multi value) in the destination Dynamic URL/UNC. To rename a file use the assignment to “rename file” as in all other scenarios.
GeneralNew button Computer performance index located in the Account screen that will give you an index value for how fast the computer and database is performing. Performance index value of 1.0 is minimum required for running SMART Migration with acceptable speed.
FileManagementSharePoint Online: New Move folders option allow you to move folders around super fast within the same Document Library or between libraries on the same site. The MoveCopyUtil api is used and it is super fast as no binary transfer is needed. Also, any Check-out status is retained and the file GUID does not change. When working within the same library there are no limitations but moving folders between libraries on the same site is subject to the SharePoint API ListViewThreshold limitation of 5000 items (subfolders + files being moved in 1 operation).
Data typesSet data types in DataETL and For-Each now has a new button to reload the original data types from the source table. This is useful when you want to reset what ever changes you have made to the data types or the schema for the table. Useful also when the schema has been changed externally.
SharePointNew Variable support for retrieving items starting at a given folder level (right-click menu in the folder column). This is useful for example when dynamically retrieving all folders at a given level to move them elsewhere.
PowerShellNew default script that includes Try Catch and session logging. This makes it easier to get started.
PowerShellNew CMDLET for connection to SharePoint
Migrated files reportPerformance enhancement x50
DatabaseView support for the Internal repository
Session LogNew column [Custom Info] to display the custom logging information defined either on the Options-Tab or assigned to the system Variable [<SessionLog-CustomInfo>] from a For-Each Step.
Session LogFind and Filter now also considers the new column [Custom Info].
File reportingAbility to cancel a running report.
File reportingNew filter on Custom info
File reportingNew filter on Copied or Moved
File reportingNew ability to do reporting on files subject to “Test run”
File reportingNew database reporting view report_migratedfiles located on the internal repository that enables you to do reporting across all jobs/steps/sessions/individual For-Each iterations. The output can be loaded into Excel, CSV, SP-List or database just like any other data source. Then the reporting can become become part of flow setup.
SharePoint user validation reportingNew database reporting view report_validatedsharepointusers located on the internal repository that enables you to do reporting across all jobs/steps/sessions/individual For-Each iterations. The output can be loaded into Excel, CSV, SP-List or database just like any other data source. Then the reporting can become become part of flow setup.
Data validation reportingNew database reporting view report_datavalidationwarnings” located on the internal repository that enables you to do reporting across all jobs/steps/sessions/individual For-Each iterations. The output can be loaded into Excel, CSV, SP-List or database just like any other data source. Then the reporting can become become part of flow setup.
DataETL/ForEachSharePoint: Override URL no longer needs | (pipe character) to specify subfolder, you can just use / as you would normally do in a SharePoint URL
DataETLLoad into Database: New progress information added to the Session log. This is nice when for example doing 1 million inserts.
DataETL/For-EachExtract data: Added limit result set to nn rows for SQLServer, CSV, Excel (SharePoint was already supported)
PreviewEnhanced interactive filter in the preview windows for DataETL, For-Each and FileManagement. The filter now automatically follows the column you sort by and applies the filter.
JobAvailable memory is now monitored and if the threshold of 2 GB available memory is reached the job will issue a warning
TransformationReplace action has 2 new replacement options: All but first occurrence and All but last occurrence
Version 2.23April 2022
Session LogPerformance enhancement (x3) when loading large sessions on machines that have background Accessibility software running. Most windows 10 & 11 have that
SQL ServerINSERT/UPDATE raw SQL statements can now use the SQLServer specific syntax N'my string value' to support UNICODE character set for columns of type NVARCHAR. Notice tables columns created from the IDE will be of type VARCHAR. So, to support UNICODE you must create/alter your table using embedded SQL or use SQL Server Management Studio.
JobVariable values can now be returned into the IDE from a job. This is very convenient for debugging and testing jobs that generates a dynamic filenames or other values dynamic values that the next Step to test is using.
JobSharePoint as source: new option to specify how many items to retrieve from the source. This is very convenient for debugging/testing
PowerShellEnhanced longing of PS exception trace: The log generated when an exception is thrown in the above example:
M365Azure Migration API – package retry. This feature catches the errors that Azure randomly throws for no reason when transferring data from blob storage to SPO and then performs async retry of the Azure job without having to reupload the binary content and metadata. Wen a retry occurs it is indicated on the AzureJobLog.txt file (Retry)
Job notificationYou can now run a Step or series of connected Steps to perform job notification. The flow can for example update a database or SP list with notification relevant data. You can invoke your notification Step from any Step in your job also within in a For-Each loop thus enabling individual notification for each iteration in the loop Note that the notification will not be invoked synchronously when there are files pending in Azure (i.e.. files that are uploaded successfully but not yet moved by Azure from temp blob storage to SharePoint). When all files processed within the the Step is accounted for by Azure then the notification is triggered. If an error occurs asynchronously in the Azure migration import API, it is reflected in the notification even though the upload of the file succeeded. The entire Variable context is restored when the notification is triggered asynchronously. The below variables designated for the notification are also populated: - [<NotificationType>] The job result result since the last notification - [<NotificationMessage>] The job message issued since the last notification. The first 10k of log data for warnings and errors is accumulated. If you update a database you need to make sure the column width is large enough or truncate the value. - [<NotificationFileCount>] The number of files processed within the Step the notification is linked to. - [<NotificationContext>] The information provided from the Step setup by you. This is particularly useful when you use your own job tracking setup with for example a SharePoint list that specifies parameter for the job as a whole or for For-Each iterations.
SharePointCreate Folder/DocumentSet has been redesigned to avoid throttling. This may reduce the performance in terms of number of folders/DocumentSets created per minute in order to comply with the Microsoft throttling policy change.
FileManagement[x] Include subfolders is now visible also on the Home tab
Session LogDynamic colour change for expanded levels
SharePointOverride URL no longer needs | (pipe) to separate Library from folder structure. This makes it significantly easier to use/understand
FileManagementThe FileShare attribute Last Accessed can now be referenced in filters and be used for metadata assignment and calculation.
Version 2.04March
DatabaseDrop down menu now shows database VIEWS
SharePointGeneric DMS version history migration: x17 performance enhancement for major/minor version calculation.
Job ServerNew error tracking for Windows Service account permission issues. This makes it easier to figure out what is wrong with a given Windows Service account.
UX multi monitorStep windows and multitasking jobs now open on the same monitor as the parent window. Also the resize logic now use the screen resolution for the monitor in context
Data ETL/ForEach“File and Directory scan” now includes the [LastAccessed] attribute for files. This enables a variety of applications for example only migrating the most recent used files or creating PowerBI models that shows what files are actively being used and what files are ROT (Redundant Obsolete and Trivial).
Data ETL/ForEach“File and Directory scan” Performance optimized for retrieving the first level of folders. (no longer runs through the entire hierarchy)
Version 2.01February 2022
Data ETLEdit table New button to edit a database table’s column layout
Data ETLList settings New button to open the SharePoint List settings in a browser
File ManagementLibrary settings New button to open the SharePoint Library settings in a browser
File ManagementMigration of legacy DMS document major/minor version history. Supports Alfresco, OpenText, FileNet or any DMS where the metadata for document versions is stored in a database, CSV or Excel.
For-EachGroup By: New functionality to perform data grouping. It is essential for transforming multiple rows into 1 value per. column based on one or more GroupBy columns. Use cases include: - Create values for SharePoint choice-columns or multi value Manged Metadata and Lookup columns. - Apply the multi value Variables in SharePoint and Database filters using the IN filter clause. - Document Management System migration where you can use a single result set (that you can create/prepare in various ways) as Source to a FOR-EACH and automate “FileManagement”. Then you GroupBy Site/Folder and the FileManagement step will optimize and “batch” files from various sources into 1 azure session per destination. That will effectively increase performance anywhere from x2 to x50 compared to single file processing and moreover make metadata assignment and calculation much easier and transparent.This feature is extremely useful for distributing content to many sites based on a result set that defines the destination Sites, Source storage location and metadata.
SharePointManaged metadata tags in a hierarchy can now be referenced in compliance with the PnP framework using the pipe character |
SharePoint OnlineFile Management: The version for a file can now be specified even the file has no version history. use the new [Set version] column in the metadata assignment.
Session logAuto-refresh of log for running job
Session logEnhanced filtering. Also works real-time on running job. Option to chose to view errors only.
Session logx15 faster purge of outdated log sessions. Deleting specific sessions is done instantly.
Session LogNew viewer for analysing embedded reports like SourceFiles.txt This is convenient when running on a machine that does not have Excel installed. Instead of opening the report in Notepad or Excel a new tab opens within the Step where the report can be analysed. The content can be sorted, filtered and exported to Notepad or Excel.
Session logNew r-click menu option to view File Version History
Version 1.95January 2022
Language supportAdded support for the following languages: - Thai (Thailand) - Vietnamese (Vietnam)
SharePointHandling of columns level validation rules in SharePoint Lists and Libraries. The defined validation message from the actual SharePoint rule is also returned to the session log along with the data value for your context understanding.
SharePointEnhanced handling of corrupted SharePoint List column definitions.
MS SQL ServerSupport for having the repository in a SQL Server database rather than locally in the program-public folder. This is useful for compliance with data protection policies and also gives better performance for large session logs.
TransformationNew function IndexValue
VariablesThe expression test value for [data] is now reloaded when the Variable properties window is opened
Session LogCtrl-A will now select the message text
File ManagementOptimized file copy operation between FileShares
Session LogLoad/Expand 35% faster
Version 1.93December 2021
Run Job interactivelyColumns are now sized automatically to make it easier to view data
Validated usersSharePoint: New report window to view all validated users. Useful to se how users are mapped and how they resolve against the AD
Public previewThese are the features not yet ready for GA but included for you to preview
Version 1.86
SharePoint Online“Refresh Schema” is now x2-x80 faster (The bigger Termstore and number of content-types the faster)
SharePoint Online“Connection” has new option to include hidden lists and libraries
DatabaseNew option to create a connection to CSV/Excel Sheet directly from the Database connection window.
FunctionsNew system function RemoveIllegalcharactersFromFileName
Data TransformationTest transformation for SharePoint user columns: When transforming values the user mapping list is now included in the test so the actual mapped result is shown. Also, you can edit the mapping list from within the transformation window
SharePoint user validationUser name validation can now be performed in “Data Transformation” and “Edit user mapping”
Data TransformationNew “Paste value” button for SharePoint columns of type Managed Metadata, Lookup, Content-Type, Choice
Job logNew right-click “Open SharePoint version history” in Notepad or Excel”
GeneralMouse wheel now scrolls the options tab pane on all Steps
Version 1.79
GeneralDigitally signed executables and Setup
FileManagementNew method for excluding folders based on partial string matching on the columns [Path] and [URL path]. For example *forms*;*system* will exclude/include/include+ BOTH files and folders if the path string contains “forms” or “system”. It is coded to work specifically on the column [URL path] for SharePoint source and [Path] for FileShare source.
Version 1.77
DataETL & FileManagementNew Preview filter. When running a job visually with preview you can now filter the content to process based on the sort column.
DataETL & FileManagementSharePoint user validation: The default user can now specify additional domain(s) that will be used for user validation. The syntax app@sharePoint;mydomain.com will try to use mydomain.com applied to a user name if a given user do not validate. - Example1: user JOHN will first be validated using the name as it is. If that fails then validation is attempted using JOHN@mydomain.com - Example2: user ANN@abc.com be validated using the name as it is. If that fails then validation is attempted using ANN@mydomain.com
TransformationsNow the value mapping list name is displayed along side the transformation instance to give better overview. Also the mapping list can be opened directly without having to open the transformation.
TransformationsNew Final result column to show the result of the expression combined with the transformation actions.
Job tabNew visual indication on the Job tab when a running job has completed. Job completed successfully etc.
Job tabNew button on the Job tab Goto preview job will take you to the running preview session that might be hidden behind other windows. This makes it easier to find visual a running job on the machine when there are many windows open.
Version 1.67
DataETL & ForEachSQLServer: Enhanced SQL parsing of tables and columns having all sorts of characters in the names. Added support for the UNION keyword without a FROM clause.
Job schedulingNew toolbar button added on the Job ribbon tab: Managed scheduled jobs for easy access to the job scheduler. Also the job scheduler opens immediately after a job has been submitted for easy review and status monitoring.
Scheduled jobsNew toolbar button to easily select a Step and schedule it to run without first having to open the Step and go to the Job tab.

Other changes

DataETL & ForEachSQL Source: The editor window is now dynamically resized to give more editor space.
ReportingPerformance enhancement: Retrieve file migration report is now x3 faster.
Data ETL & FileManagementSupport for SharePoint system Libraries SitePages and SiteAssets
Session LogLoad time performance enhancement when having >75000 log rows in one session.
Variables & AssignmentsVisual redesign of “Edit” window and new column Description added.
VariablesNew “Referenced” list box where all references to the selected Variable are listed. You can se the Step name, Step type and the location where the variable is being used.
VariablesMulti-select is now supported for deleting. (Ctrl-A, Ctrl-Click, Ctrl-Shift-Click)
VariablesNew check box “Show only unreferenced”. Applies a filter so only Variables that are not being used anywhere are listed. This is very useful for cleaning up by deleting unused variables
All StepsNew option to specify a Custom log message (Options tab). Useful for adding context specific information to the Session Log for job tracking and analysis. The information is globally searchable across all sessions and all Steps from the Dashboard.
All StepsNew option to specify what to do when an error occurs. valid options are: - Log a warning and continue (the default) - Log a warning and exit current Group Step - Stop the job
DataETL & ForEach & FileManagementNew option to specify what to do when no data or files are retrieved from the Source. Valid options are: - Ignore and continue (The default) - Log a warning message - Log a message and exit the current Group Step - Log a warning and exit the current Group Step
Group StepNew hyper link to open embedded Step.
Group StepNew condition return value keywords: - "ExitGroup" , "ExitGroupWarning" to not call the current Step or any subsequent steps within the Group. - "StopJob" , "StopJobWarning" to stop the job immediately.
Data ETLSharePoint Item Update: New option to react when not all source keys matches a destination Item. You can trigger a warning, exit the current Group Step or stop the job. This is very useful when your business rules dictate that every source key must match a destination Item. Note that the condition is detected before any action is taken so no data is harmed.
Data ETLSharePoint Item Update: New option to react when multiple items are updated based on a single source key. You can trigger a warning, exit the current Group Step or stop the job. This is very useful when your business rules dictate that only 1 item must be updated for each source key. Note that the condition is detected before any action is taken so no data is harmed.
Data ETL & FileManagementNew option to react when an error occurs. You can trigger a warning (the default), exit the current Group Step or stop the job. This is very useful when for example a dynamic URL is wrong and it makes no sense to continue processing the current iteration in a For-Each or the entire job.
Session LogNew hyper link to open nested Step. The link has tooltip that shows the storage location of the Step
Session LogMessage can now be 8k (previously 2k) and is globally searchable across all sessions and all Steps from the Dashboard.
PowerShellAdd-FlowLogRow new arguments to -LogTypes: - -LogType StopJob Stop the job - -LogType ExitGroup Exit the Group Step from where the PowerShell Step is called.
CalculationNew functions: - RemoveIllegalCharactersFromFileName() - ReadTextFile() - FileExist() - GetFileNamePart()
Data ETLSharePoint: Support for extraction of custom list item version history to database(Insert/Update), Excel, csv, ForEach. You can specify criteria for the extraction: Date of oldest version to include, hard coded or using a variable. When previewing the retrieved SharePoint Item data you have the option to visually also show the version history. (Please note that the version history cannot yet be applied to new SharePoint custom list items. Version history is 100% supported in File Management)
File ManagementSharePoint version history: You can now specify criteria for the extraction: Date of oldest version to include, hard coded or using a variable.
Column mappingData-ETL & File Management: New Ribbon bar button to automatically map all columns & reset mappings
DocumentationNew article SharePoint user mapping. There is a link from Options user mapping to the article within the product.
FileManagementAllow any user name (Only O365) – New validation message to verify that the conditions for this option to function are met
FileManagementThe width of Override URL has been extended to 300 characters to allow for long URLs consisting of multiple Variables
Data ETLnew link to open CSV folder and Database connection properties
Data ETL & FileManagementNew visual indication when a “From” or “To” filter is defined. The indication also serves as a shortcut link to the filter definition
File ribbonNow navigates to the location of the Step in context.
SharePointData-ETL & File Management: Emoji support (❤️ Resource for copy/paste Emojis ). Item properties (including folder names) can now contain Emojis. You can even query for items having a specific Emoji.
Job LogNew ribbon bar button to open the Dashboard
Job LogFind/Filter: Enhanced visual highlighting and keyboard navigation from the find field.
Job LogNew filter option to apply “find criteria” as a filter to limit the rows visible.
Job LogFind and filter criteria is now applied immediately when you type.
Job LogMessage column is now permanently expanded but only when clicked.
Job LogNew “Open button” to open the Step the current log line belongs to.
Data ETLTo SharePoint Document Library: New action Check out will check out files to the account defined in the SharePoint Connection.
Data ETLTo SharePoint Document Library: New action Discard check out will remove the checked out status from on files.
Data ETLPreview window: New maximize button to expand the preview Window
File MigrationFrom FileShare: Automatic conversion of file names that are URL encoded. For example My%20file.docx is converted into My file.docx before transfer to SharePoint
File MigrationPreview window: New maximize button to expand the preview Window
NotificationPowerAutomate and E-Mail: Delayed notification for SharePoint Online. Notification is not sent until all files have been verified by Azure.
NotificationPowerAutomate and E-Mail: New field “FileSummary” that shows the number of files Ok, Warning, Error
File MigrationAzure/SPO destination. Notification (if enabled) is not triggered until all files for a given step has been processed by Azure.
AccountNew “Diagnostics” setting to enable logging of the raw data sent to SharePoint and databases.
StepRevised toolbar at top right corner. All commands from TOOLS are now here also.
StepCtrl+S in all steps does a SAVE
Data-ETLNew input type “Variables”. Useful for SQL DDL statements and operation that are entirely based on variables hence no input result set.
File Reporting3x Faster retrieval of reports. (Revised indexing and SQL WHERE syntax)
File MigrationRevised detection of the condition where the computer system is out of memory and many things begin to fail.
File MigrationAzure pending files: Optimized visual status for how many files are pending asynchronously in Azure. New article that describes that scenario
Migration OnDemandNew article that describes what you can do with the template plus a comparison to MS Mover and MS Migration Manager
Data transformationNew UI for data transformations in Variables and column assignments
DashboardVisual indication of files awaiting Azure Transfer to SharePoint
DashboardNew column Job PID that show the Windows Task Manager PID
ExpressionsNew mechanism to test expressions