Importing and Exporting Data in Microsoft Dynamics NAV 2013 R2 CU 8

Two weeks ago Microsoft released CU8 for Dynamics NAV 2013 R2.  This upgrade included the long-awaited Import and Export that is replacing the old FBK backup in the Classic Client.

There are two ways to do import and export.  One is through the Windows Client

ImportExport

and the other is with PowerShell commands Export-NAVData and Import-NAVData.  You can read all about this on the Microsoft Dynamics NAV Team Blog by clicking here.

Compared to the old Classic Client FBK backup this is very fast.  On the other hand the new import functionality is very strict on the database schema.  It needs to be excactly the same in the database used for the export and import.

I was in the process of upgrading a company from NAV 5.0 SP1 and had planned to add the company to an existing NAV 2013 R2 database wich already included a few companies.  This was not as easy as you might think.  Here are the issues that I encountered and how they were solved.

After I finished the upgrade I made sure that all objects in the upgraded database where identical to the destination database.  This I did by exporting all objects from the destination database to the upgraded database.  I used a merge tool on the exported text object file to make sure.

At this point I was not able to import the data file and asked for help from Microsoft.  Got all the help I needed and the first thing that I needed to make sure was that all the tables must be identical on a SQL level.  This was done with this SQL command.

[code lang=”sql”]select db1.[Object ID] "DB1 Table ID", db1.Name "DB1 Table Name", db1.Hash "DB1 Metadata Hash", db2.[Object ID] "DB2 Table ID", db2.Name "DB2 Table Name", db2.Hash "DB2 Metadata Hash"
from [Destination Database].[dbo].[Object Metadata Snapshot] db1
full outer join [Source Database].[dbo].[Object Metadata Snapshot] db2 on db1.[Object ID] = db2.[Object ID][/code]

I needed to make sure that the “DB1 Metadata Hash” what identical to “DB2 Metadata Hash” for all tables.  This was not true even if all objects where identical.  After recompiling all tables in both databases I saw that I now had identical hashes.

This was not enough.  Since I upgraded the source database from an earlier NAV version the database still held all the old virtual table snapshots.  We needed to remove them with this SQL command.

[code lang=”sql”]delete from [Source Database].[dbo].[Object Metadata Snapshot]
where [Object ID] in
(2000000007, 2000000026, 2000000049, 2000000101, 2000000102, 2000000103, 2000000020, 2000000055, 2000000009, 2000000029, 2000000038, 2000000058, 2000000041, 2000000028, 2000000063, 2000000022, 2000000048, 2000000040, 2000000043, 2000000044, 2000000039, 2000000045)
[/code]

After this I successfully exported and imported the upgraded company into the destination database.  One more thing that I saw is that I needed to restart other service instances in order to see the newly imported company.

This new functionality opens a lot of possibilities for developement and testing.  We can now easily with PowerShell scripts create a backup of our live company data and import them into our test and/or developement environment.

In both the Windows Client and in PowerShell it is possible to copy companies, rename and delete.  I know that the import process takes a longer time than the export process.  I tried the export process on a 500GB database and it took 2 hours and 27 minutes.  In the Classic Client this would have taken days.

I expect to see this functionality improve in future releases and also expect Microsoft or some of the community leaders to publish scripts to automate a data refresh in a testing environment.

Backup your NAV Database to Microsoft Azure

I am not sure that you will find safer place to place your SQL backups than on a Microsoft Azure Storage.  Azure keeps three copies of all the data within the data center and if you enable Geo-Replication another three copies will be kept in another data center somewhere else in the world.

Running the NAV SQL Server on a virtual machine within Azure also means that the SQL database is also in six places, just to make sure.

I moved my NAV services to Azure and wanted to make sure that the backup would be as good as everything else.  To backup directly from SQL to Azure Storage you will need SQL Server 2012.  There are most likely other third-party solutions to move you backups to Azure.  There even is a program from Microsoft – SQL Server Backup to Windows Azure Tool – that will pick up files a local folder(s) and upload them to the Azure Storage.  I am using this tool to upload backup from my MySql server.

If you have SQL Server 2012, on Azure or on premise you can backup directly to your storage account.  You will need Cumulative update package 2 for SQL Server 2012 to update your installation to the required version to get this feature supported.

How it works is being shared in many places, for example on SQLServerCentral.com.  I followed these instructions and created a storage account in USA.  My SQL server is running in North Europe and I wanted to make sure that the backups where on another continent.

I created the credentials in my master database and then started to test the backup.  After a succesful backup I started to think about automating this process.  As usual a dynamic script was needed, I did not want to have to manually do anything.  Even if I removed or added a database.  This is the script

[code lang=”sql”]declare @yea varchar(10)
declare @wk varchar(10)
declare @dw varchar(10)
declare @hh varchar(10)
declare @url varchar(250)
SET @yea = (SELECT datepart(yy,Getdate()));
SET @wk = (SELECT datepart(wk,GetDate()));
SET @dw = (SELECT datepart(dw,GetDate()));
SET @hh = (SELECT datepart(hh,GetDate()));

DECLARE @TableName VARCHAR(256)
DECLARE @DBName VARCHAR(256)
DECLARE @varSQL VARCHAR(max)
DECLARE @getDBName CURSOR
SET @TableName = ‘$ndo$dbproperty’
SET @getDBName = CURSOR FOR SELECT name FROM sys.databases
CREATE TABLE #TmpTable (DBName VARCHAR(256), SchemaName VARCHAR(256), TableName VARCHAR(256))
OPEN @getDBName
FETCH NEXT FROM @getDBName INTO @DBName
WHILE @@FETCH_STATUS = 0
BEGIN
SET @varSQL = ‘USE [‘ + @DBName + ‘];
INSERT INTO #TmpTable SELECT ”’+ @DBName + ”’ AS DBName, SCHEMA_NAME(schema_id) AS SchemaName, name AS TableName FROM sys.tables
WHERE name LIKE ”%’ + ltrim(rtrim(@TableName)) + ‘%”’
EXEC (@varSQL)
FETCH NEXT
FROM @getDBName INTO @DBName
END
CLOSE @getDBName
DEALLOCATE @getDBName

DECLARE @DBName_Cursor CURSOR
SET @DBName_Cursor = CURSOR FOR SELECT DBName FROM #TmpTable;
OPEN @DBName_Cursor
FETCH NEXT FROM @DBName_Cursor INTO @dbname
WHILE @@FETCH_STATUS = 0
BEGIN
IF @dw = 1 AND @hh < 6
BEGIN
SET @url = ‘https://kappisqlbackup.blob.core.windows.net/sqlbackup/’ + REPLACE(@dbname,’ ‘,’_’) + ‘_’ + @yea + ‘_’ + @wk + ‘.bak’
BACKUP DATABASE @dbname TO
URL=@url
WITH CREDENTIAL=’kappiAzureCredential’, STATS = 10, FORMAT, COMPRESSION
END
SET @url = ‘https://kappisqlbackup.blob.core.windows.net/sqlbackup/’ + REPLACE(@dbname,’ ‘,’_’) + ‘_’ + @yea + ‘_’ + @wk + ‘_’ + @dw + ‘_’ + @hh + ‘.trn’
BACKUP LOG @dbname TO
URL=@url
WITH CREDENTIAL=’kappiAzureCredential’, STATS = 10, NOFORMAT, NOINIT, SKIP, NOREWIND, NOUNLOAD, COMPRESSION
FETCH NEXT FROM @DBName_Cursor INTO @dbname
END;
CLOSE @DBName_Cursor;
DEALLOCATE @DBName_Cursor;
DROP TABLE #TmpTable
GO[/code]

You can see that my company name, Kappi, is in the script. If you want to use this script you need to update the storage URL and the credential name.

What happens is that all databases are read and within them the script looks for the table ‘$ndo$dbproperty’.  If this table exists then I add this database name to a temporary table.  I then loop through the temporary table and start a backup for each database.

My plan is to do a full backup once a week and transaction backup every six hours.  The line ‘IF @dw= 1 AND @hh < 6′ triggers a full backup on Sundays before six in the morning.  Then I created a job that executes every six hours to start the backup.

[code lang=”sql”]USE [msdb]
GO

/****** Object: Job [Kappi_Backup] Script Date: 7.12.2013 14:16:04 ******/
BEGIN TRANSACTION
DECLARE @ReturnCode INT
SELECT @ReturnCode = 0
/****** Object: JobCategory [Database Maintenance] Script Date: 7.12.2013 14:16:04 ******/
IF NOT EXISTS (SELECT name FROM msdb.dbo.syscategories WHERE name=N’Database Maintenance’ AND category_class=1)
BEGIN
EXEC @ReturnCode = msdb.dbo.sp_add_category @class=N’JOB’, @type=N’LOCAL’, @name=N’Database Maintenance’
IF (@@ERROR <> 0 OR @ReturnCode <> 0) GOTO QuitWithRollback

END

DECLARE @jobId BINARY(16)
EXEC @ReturnCode = msdb.dbo.sp_add_job @job_name=N’Kappi_Backup’,
@enabled=1,
@notify_level_eventlog=0,
@notify_level_email=3,
@notify_level_netsend=0,
@notify_level_page=0,
@delete_level=0,
@description=N’Database Backup’,
@category_name=N’Database Maintenance’,
@owner_login_name=N’ONAZURE\kappi’,
@notify_email_operator_name=N’Gunnar Þór Gestsson’, @job_id = @jobId OUTPUT
IF (@@ERROR <> 0 OR @ReturnCode <> 0) GOTO QuitWithRollback
/****** Object: Step [Start Backup] Script Date: 7.12.2013 14:16:05 ******/
EXEC @ReturnCode = msdb.dbo.sp_add_jobstep @job_id=@jobId, @step_name=N’Start Backup’,
@step_id=1,
@cmdexec_success_code=0,
@on_success_action=1,
@on_success_step_id=0,
@on_fail_action=2,
@on_fail_step_id=0,
@retry_attempts=0,
@retry_interval=0,
@os_run_priority=0, @subsystem=N’TSQL’,
@command=N’declare @yea varchar(10)
declare @wk varchar(10)
declare @dw varchar(10)
declare @hh varchar(10)
declare @url varchar(250)
SET @yea = (SELECT datepart(yy,Getdate()));
SET @wk = (SELECT datepart(wk,GetDate()));
SET @dw = (SELECT datepart(dw,GetDate()));
SET @hh = (SELECT datepart(hh,GetDate()));

DECLARE @TableName VARCHAR(256)
DECLARE @DBName VARCHAR(256)
DECLARE @varSQL VARCHAR(max)
DECLARE @getDBName CURSOR
SET @TableName = ”$ndo$dbproperty”
SET @getDBName = CURSOR FOR SELECT name FROM sys.databases
CREATE TABLE #TmpTable (DBName VARCHAR(256), SchemaName VARCHAR(256), TableName VARCHAR(256))
OPEN @getDBName
FETCH NEXT FROM @getDBName INTO @DBName
WHILE @@FETCH_STATUS = 0
BEGIN
SET @varSQL = ”USE [” + @DBName + ”];
INSERT INTO #TmpTable SELECT ”””+ @DBName + ””” AS DBName, SCHEMA_NAME(schema_id) AS SchemaName, name AS TableName FROM sys.tables
WHERE name LIKE ””%” + ltrim(rtrim(@TableName)) + ”%”””
EXEC (@varSQL)
FETCH NEXT
FROM @getDBName INTO @DBName
END
CLOSE @getDBName
DEALLOCATE @getDBName

DECLARE @DBName_Cursor CURSOR
SET @DBName_Cursor = CURSOR FOR SELECT DBName FROM #TmpTable;
OPEN @DBName_Cursor
FETCH NEXT FROM @DBName_Cursor INTO @dbname
WHILE @@FETCH_STATUS = 0
BEGIN
IF @dw = 1 AND @hh < 6
BEGIN
SET @url = ”https://kappisqlbackup.blob.core.windows.net/sqlbackup/” + REPLACE(@dbname,” ”,”_”) + ”_” + @yea + ”_” + @wk + ”.bak”
BACKUP DATABASE @dbname TO
URL=@url
WITH CREDENTIAL=”kappiAzureCredential”, STATS = 10, FORMAT, COMPRESSION
END
SET @url = ”https://kappisqlbackup.blob.core.windows.net/sqlbackup/” + REPLACE(@dbname,” ”,”_”) + ”_” + @yea + ”_” + @wk + ”_” + @dw + ”_” + @hh + ”.trn”
BACKUP LOG @dbname TO
URL=@url
WITH CREDENTIAL=”kappiAzureCredential”, STATS = 10, NOFORMAT, NOINIT, SKIP, NOREWIND, NOUNLOAD, COMPRESSION
FETCH NEXT FROM @DBName_Cursor INTO @dbname
END;
CLOSE @DBName_Cursor;
DEALLOCATE @DBName_Cursor;
DROP TABLE #TmpTable
GO
‘,
@database_name=N’master’,
@flags=0
IF (@@ERROR <> 0 OR @ReturnCode <> 0) GOTO QuitWithRollback
EXEC @ReturnCode = msdb.dbo.sp_update_job @job_id = @jobId, @start_step_id = 1
IF (@@ERROR <> 0 OR @ReturnCode <> 0) GOTO QuitWithRollback
EXEC @ReturnCode = msdb.dbo.sp_add_jobschedule @job_id=@jobId, @name=N’Every Six Hours’,
@enabled=1,
@freq_type=4,
@freq_interval=1,
@freq_subday_type=8,
@freq_subday_interval=6,
@freq_relative_interval=0,
@freq_recurrence_factor=0,
@active_start_date=20131205,
@active_end_date=99991231,
@active_start_time=0,
@active_end_time=235959,
@schedule_uid=N’dce606ac-0803-43a2-a98a-c8ffedb1df09′
IF (@@ERROR <> 0 OR @ReturnCode <> 0) GOTO QuitWithRollback
EXEC @ReturnCode = msdb.dbo.sp_add_jobserver @job_id = @jobId, @server_name = N'(local)’
IF (@@ERROR <> 0 OR @ReturnCode <> 0) GOTO QuitWithRollback
COMMIT TRANSACTION
GOTO EndSave
QuitWithRollback:
IF (@@TRANCOUNT > 0) ROLLBACK TRANSACTION
EndSave:

GO

[/code]

Session Event already exists during a full restore

In NAV 2013 distribution Microsoft is not delivering a database backup.  You will need to create the backup file from the SQL demo database.  This is what I did.

I created a new database and started a full restore and got the error.

So I cleared the Session Event table before making a new backup and the problem is solved.

SessionEventDelete