ReFS volume using DPM becomes unresponsive

In the case you notice that the ReFS volume for DPM backup storage becomes unresponsive and stalls the entire backup solution Microsoft has provided a fix within July 18 cumulative update (KB4035951) that contains three tunable values that you set in the registry.Important: Before you follow these steps, make sure that you have read and implemented the three registry parameters as described in KB article 4016173. If these don't adequately address any issues that you encounter, do not disable these registry parameters. These parameters and the ones described in this section do not overlap functionally, so they can be used together.This update describes additional registry parameters that help address the latency issues described in the "Symptoms" section. These parameters can be used in any combination. Warning: Serious problems might occur if you change the registry incorrectly by using Registry Editor or by using another method. These problems might require you to reinstall the operating system. Microsoft cannot guarantee that these problems can be resolved. Change the registry at your own risk.
Important

• A restart is required for these parameter changes to take effect.

• These parameters must be set consistently on every node of a failover cluster.

Option 1:
This option disables cached pins, which were a major cause of the large active working set.

Specify the indicated values in the following subkey:

• HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem

• Value Name:  RefsDisableCachedPins

• Set RefsDisableCachedPins = 1

• Value Type: REG_DWORD

Option 2:
This option adds a heuristic to ReFS checkpointing logic, which causes ReFS to checkpoint when the delete queue reach a certain size. IOs are stuck on ReFS because the checkpoint logic would get stuck processing a large delete queue.Specify the indicated values in the following subkey:

• HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem

• Value Name: RefsProcessedDeleteQueueEntryCountThreshold

• Set RefsProcessedDeleteQueueEntryCountThreshold = 2048

• Value Type: REG_DWORD

Note: Setting RefsProcessedDeleteQueueEntryThreshold to lower values causes ReFS to checkpoint more frequently. Set the value to 2048, then reduce the value to 1024, then 512.Option 3:
Large duplicate extents calls introduce latency into the system, as other operations will have to wait until these long-running operations are complete. This option reduces the size of the duplicate extents call.

• Note: DPM will set this registry key change as the default value as part of UR4, which will release within August 2017.

Specify the indicated values in the following subkey:

• HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft Data Protection Manager\Configuration\DiskStorage

• Value Name: DuplicateExtentBatchSizeinMB

• Set DuplicateExtentBatchSizeinMB = 100. (Default is 2000 [2GB]. Any value from 1 - 4095 is accepted).

• Value Type: REG_DWORD

Option 4:
This option extends the TimeOutValue.Specify the indicated values in the following subkey:

• HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Disk

• Value Name: TimeOutValue

• Set TimeOutValue (in seconds) = 0x78

• Value Type: REG_DWORD

Note: The default value for TimeOutValue is 0x41 (65 decimal). 0x78 translates to 120 decimal.x
How to set the tunable parameters
Tunable parameters

Upgrading from LTSC to SAC

As you may have understood its possible to upgrade your System Center family from LTSC to SAC but lately I have got a lot of emails with questions regarding going back to LTSC from SAC.

The intention of this blogpost is to make it perfectly clear that when you choose to upgrade from LTSC to SAC there is no way you can go back. Keep that in mind folks. 

MVP renewal 2018

It’s with great pleasure that I can announce that Microsoft has chosen to give me the MVP reward in the Cloud and Datacenter Management category for the eight year in a row which I’m very humble and grateful for.

I will continue to do what I do to assist the community and also be part of supporting Microsoft.

Disable the timeout value for Online jobs (MARS)

In some scenarios it would be a good idea to change the timeout value for the MARS agent that resides on both DPM or MABS servers. This can be done by changing the following registry key.

This will actually stop the MARS agent from cancel the job due to a time out.

HKLM\Software\Microsoft\Windows Azure Backup\DbgSettings\DisableThreadTimeout REG_DWORD=1

Optimized vs. Unoptimized online jobs

Sometimes you find your DPM or MABS server running their online jobs and it takes forever to finish. It can go on for hours and sometimes days, the big question I get about this is why...

As we all know DPM and MABS can loose track of a protected data source and when that happens the protected data sources journal information is not passed on to DPM or MABS. This is an example of what can causes the replica to be inconsistent.

The same goes with the online jobs, the scenario differentiates a bit though. In the scenario of an online job being canceled it should (in the best of worlds 😊) not trigger an unoptimized online job. But if the online job was terminated unexpectedly due to a restart and loose track it must verify all the data that resides within the Recovery Services Vault in comparison to the DPM or MABS server.

This can take a very long time but the most important part here is to let the job finish or it will never be able to continue archiving your data further down the road. According to Microsoft there have been situations where the unoptimized online job has taken over 300 hours to finish…something to keep in mind.  

MMS 2018 coming up!

Heading over to the MMS conference this year to meet the community in person and deliver six individual sessions with the focus of Azure and a touch of System Center. I will be presenting the following sessions:

• To SCOM or Not to SCOM
• Better, Stronger, Faster - Your Business Apps in Azure
• A World Without IT Silos and Proper Management Focus...Let's Go
• Soup to Nuts of Azure Site Recovery
• You are the next backup guy “tap on the back” good luck!
• Business Continuity and the Microsoft Cloud

I’m excited to meet up with all the attendees and the great people of MMS once again 😊

Defender exclusions using PowerShell

When installing System Center Data Protection Manager or Azure Backup Server it always comes down to the facts of excluding the real-time protection of Defender so it may not interfere with the backup/restore process of either MABS or DPM.

I always use PowerShell to script an automation regarding this and its more convenient and quicker to do it this way rather than manually.There are some main criteria that you must exclude on a DPM or MABS server, those are:

• Default actions
• File extensions
• Paths
• Processes

When and if a DPM or MABS server finds a malicious code, it should by default remove it rather than anything else. To set that option via PowerShell you simply run the following PowerShell code:

Set-MpPreference -LowThreatDefaultAction Remove

Set-MpPreference -ModerateThreatDefaultAction Remove

Set-MpPreference -HighThreatDefaultAction Remove

Set-MpPreference -SevereThreatDefaultAction Remove

The file extensions that I always exclude from the realtime scanning is the .mdf and .ldf files using the following syntax:

Add-MpPreference -ExclusionExtension .mdf

Add-MpPreference -ExclusionExtension .ldf

I also exclude the following paths:

Add-MpPreference -ExclusionPath C:\Windows\Microsoft.NET\Framework\v2.0.50727

Add-MpPreference -ExclusionPath D:\Program Files\Microsoft System Center\DPM\DPM\bin

Add-MpPreference -ExclusionPath D:\Program Files\Microsoft System Center\DPM\DPM\Temp\MTA

Add-MpPreference -ExclusionPath D:\Program Files\Microsoft System Center\DPM\DPM\Temp

Add-MpPreference -ExclusionPath D:\Program Files\Microsoft System Center\DPM\DPM\Volumes\Replica

Add-MpPreference -ExclusionPath D:\Program Files\Microsoft System Center\DPM\DPM\Volumes\ShadowCopy

Add-MpPreference -ExclusionPath D:\Program Files\Microsoft System Center\DPM\DPM\Volumes

Add-MpPreference -ExclusionPath D:\Program Files\Microsoft System Center\DPM\DPM\XSD

Add-MpPreference -ExclusionPath E:\Program Files\Microsoft SQL Server\MSSQL13.DPM\MSSQL\DATA

Add-MpPreference -ExclusionPath X:\

Where the last part is the processes:

Add-MpPreference -ExclusionProcess dpmra.exe

Add-MpPreference -ExclusionProcess csc.exe

With this base you are able to create a decent PowerShell script of your own altering the paths and the volume letters.

SAC vs. LTSC

Microsoft announced a while back that they will start to release their software in a so-called Semi-Annual Channel (SAC) and Long-Term Servicing Channel (LTSC).

The intention of SAC is to have two release of the System Center software per year making it possible for clients to get the new functions and features more quickly to the users. This is a great step enabling the possibility to get on track with faster supported software scenarios.

One this that is important is that Semi-Annual Channel will only be available for client who has software assurance.

The Long-Term Servicing Channel (LTSC) will still be around and service releases of major System Center releases.

Experts Live 2018

I very happy to announce that I will be speaking at Experts Live in Prauge this fall.

Go to the Experts Live webpage and sign up 😊 https://www.expertslive.eu/

Hope to meet you there

#Communitypower

Unhandled Exception: System.Runtime.InteropServices.COMException (0x800706BE)

Don’t you just love those really simple error codes that DPM can provide you 😊

Today I bumped into a little challenge that I thought I would like to share with the community. Doing a standard verification process of a DPM servers actual recovery points I noticed that there where something fishy going on with a protected data source.

For some reason the data source was not able to create a recovery point and just synchronized a few MB of data and then just…gave up. In the Windows Application log it stated that the protected data source has lost its disk from this error alert.

Since the data source was also archived to an Azure recovery services vault I felt that it would be better to recreate the local backup and after that trigger a new online job for that data source.

After deleting the protected data source, but retaining the online recovery points so the archiving wouldn’t get lost, I recreated the data source. Everything looked awesome and after fetching a new cup of coffee I went back to my screen an noticed that the DPM console has chrashed…no good.

Doing some investigation in the log I found a classic event….*drum whirl*…Event ID 999. For those of you that are not familiar with this event this is a generic event that could describe almost anything so it REALLY important to read the actual logged data within the event.

Still a bit of a challenge when the DPMDB are referring to objects that are not there….hmm…there is only one way of solving this. Make the DPMDB consistent with the protected workloads via the DPMSYNC command.

Running the DPMSYNC -SYNC from the DPM management shell started out great…until that command TOTALLY crashed…*sigh*… If the DPMSYNC fails the DPM console will not be able to access the DPMDB since the DPMDB will be in a “interesting” state.

So the current status was that the DPM servers database ended non-operational and the access to the DPM console was smoked…but…there is always a way! Since Disaster Recovery is always a nice feature to have I simple restored a earlier version of the DPMDB. If you don’t have a DPM-DPM-DR setup made I highly recommend you to get started with that!

After restoring the DPMDB the DPMSYNC -SYNC command worked fine however all storage became unavailable….will this never end!!!!????

Never fear…Hedblom is here! By using the switch -ReallocateReplica (yes it works on DPM 2016 also…) I got all the replicas back in place and finished the restore process by running the DPMSYNC -SYNC again…now I was back on track! I could access the DPM console and make small adjustments to the MBS storage pool. All data that was archived to Azure was still accessible (GREAT JOB MICROSOFT!!!) and after performing a Consistency Check for the protected workloads my client now was back with ZERO data loss.


I hope this could give some answers or support if anyone in the community ends up in the same situation I did 😊

Upgrading 2016 -> 1801

I had a few who has been meeting some challenges after upgrading to DPM 1801 regarding DPMDB protection.

When performing an upgrade of System Center Data Protection Manager, the old DPMDB is transformed into the 1801 configuration. What actually happens is that the upgrade process will add a “GUID look alike” sequence of characters after the DPMDB name.

The challenges that people has notified is that their DPMDB protection (in a DPM-DPM-DR perspective) has started to fail. The reason is that after the upgrade and the new name of the DPMDB the GUID will change and therefore the possibility to continue protecting the DPMDB.

To solve this you simple modify your protection group hosting the protection of the DPMDB and be very sure that you hit the refresh button at the bottom of the guide so you will see the DPMDB with the new name. Select the new DPMDB and click through the Wizard.

Moving the DPM filters between users DPM 1801

Creating filters within the DPM console is a great way to manage visibility when it comes to present current status of jobs within DPM.

Filters is also user specifics so there is no way to create a single “filter repository” that all users can leverage…or…there is always a way 😊

The secret recipe is the local roaming profiles of users located in the “C:\Users\USERNAME\AppData\Roaming\Microsoft\Microsoft System Center Data Protection Manager” catalogue.

When you create filters within DPM a small XML file is created that is called JobsFilters.XML. Copying this to other users makes it possible to distribute administrative filters to other DPM administrators within the organization.

MVP Summit coming up -> Community power 😊

Within two weeks I’m of to Redmond to meet up with my MVP peers from all over the world and also the great people at Microsoft to discuss Microsoft visions, roadmaps and…. provide community feedback! 

I have already scheduled a few meetings with the Product Group managers with the intention to discuss and provide feedback and here is YOUR chance to also provide feedback.

If you have any thoughts on DPM, Azure Backup, Azure Backup, ASR or other feedback on Microsoft products please send me an email to Robert.hedblom@outlook.com and I will personally deliver YOUR thoughts and feedback 😊

2W with TrueSec

I got the great pleasure to deliver a Second Wednesday session regarding host-based backups with my friend Mikel that works at TrueSec Infra.

The session is in Swedish and you can watch it here

What's new in System Center 1801?

Bala Rajagopalan wrote a great article of what is new for System Center 1801 and I can highly recommend this article.

For DPM 1801 the biggest takeaway is the ability to backup and restore Vmware virtual machines. All features that was released within DPM 2016 will also be available within DPM 1801 that now applies to the Semi-Annual Channel for releses.

DPM 2016 features:

• Modern Backup Storage 
• Resilient change tracking (RCT)
• Continued protection during cluster aware updates
• Shielded VM Backups
• Hyper-V with Storage Spaces Direct
• Hyper-V with ReFS SOFS Cluster
• Upgrading a DPM production server to 2016 doesn't require a reboot

MMS 2018

I’m very happy to announce that I’m heading back as a speaker for MMS 2017 in Minneapolis. This is one of my favorite conferences since the boundaries between me as a speaker and the attendees are gone.

The community feeling for MMS is fantastic and this year I will deliver six sessions and do community meetup Monday, Tuesday, Wednesday and Thursday during the entire conference.

I will deliver session together with Steve Buchanan, Cameron Fuller, Dieter Wijckmans and John Joyner.

Join us for this awesome conference and be part of a wonderful community!!

Go and signup before its sold out https://mmsmoa.com/

Build Numbers for DPM

Just bumped into this great site where you can find the build numbers of your DPM servers. This is very useful information since the KB articles for the build are also provided.

https://social.technet.microsoft.com/wiki/contents/articles/4058.system-center-data-protection-manager-list-of-build-numbers.aspx#TwentySixteen

DPM 2016 UR4

This article describes improvements and issues that are fixed in Update Rollup 4 for Microsoft System Center 2016 Data Protection Manager. This article also contains the installation instructions for this update. See more information about this update.

Note Existing Data Protection Manager to Windows Azure customers should upgrade to the latest agent (version 2.0.9085.0 or a later version). If the latest agent is not installed, online backups may fail.


Features


Backup storage migration

With this update installed, you can now migrate your backup storage between volumes. This enables you to optimize storage usage, as you can choose to move and store data sources on volumes that are best suited to optimize performance. You can also migrate data sources when a volume is filled with backups and can't be extended.

Volume exclusion

DPM 2016 with Modern Backup Storage accepts volumes to store backups, and formats the selected volumes to ReFS. While DPM excludes system volumes in the list of volumes that are available to be formatted, additional volumes can be configured to be excluded from the list based on drive letter or mountpoint path. This avoids accidental selection of these volumes as backup volumes.

Support for TLS 1.2

TLS 1.2 is a secure method of communication that is recommended by Microsoft. For DPM, TLS 1.2 is applicable for certificate-based authentication and protecting workloads to cloud.

For more information about how to set up, configure, and run your environment with TLS 1.2, see the following article in the Microsoft Knowledge Base:

4051111 TLS 1.2 Protocol Support Deployment Guide for System Center 2016

Custom size allocation

When you create a Protection Group to back up files and folders, size calculation can take a long time if the folders contain many files. You can changing a registry key to instruct DPM to accept the volume size as default instead of calculating the size of each file, thereby saving time.

Optimized CC for RCT VMs

Optimized consistency check for Hyper-V RCT based VM backup eliminates the storage bloat caused by CC operation of RCT VM and also improves the performance of this operation.


Improvements and issues that are fixed

BMR and System State data source backups consume more storage when you use Modern Backup Storage.
When you configure Tape Library sharing, SetSharedDpmDatabase.exe command crashes when SQL 2012 is used as the DPM database.
If a disk backup is in progress, tape backup jobs of the data source may fail.
Alternate synchronization jobs for files and folders on a deduped volume raise failed files alerts incorrectly. Failed files log links in failed files alerts doesn't show the failed files.
If many data sources are protected by the primary DPM server, enabling protection from the secondary DPM server may fail and return error 36.
When you try to recover files and folders from Azure, some items in the online recovery point are not visible in the user interface when the item has a path length longer than 256 characters.
Console crashes occure while enabling protection of a file share that is protected by a primary DPM server from a secondary DPM server.
The user interface stops responding for several minutes when triggering a Consistency check job.
Severity of the alert that is raised when an online backup fails due to new disk backups present since the last online backup is lowered from “Critical” to “Informational.”


How to obtain Update Rollup 4 for System Center 2016 Data Protection Manager


Update packages for Data Protection Manager are available from Microsoft Update or by manual download.
Microsoft Update

To obtain and install an update package from Microsoft Update, follow these steps on a computer that has a Data Protection Manager component installed:
Click Start, and then click Control Panel.
In Control Panel, double-click Windows Update.
In the Windows Update window, click Check Online for updates from Microsoft Update.
Click Important updates are available.
Select the Update Rollup package, and then click OK.
Click Install updates to install the update package.
Manual download

Go to the following website to manually download the update package from the Microsoft Update Catalog:

Download the Data Protection Manager update package now.

For information about how to download Microsoft support files, click the following article number to view the article in the Microsoft Knowledge Base:

119591 How to obtain Microsoft support files from online services

Microsoft scanned this file for viruses. Microsoft used the most current virus-detection software that was available on the date that the file was posted. The file is stored on security-enhanced servers that help prevent any unauthorized changes to the file.


Installation instructions for Data Protection Manager


To install this update for Data Protection Manager, follow these steps:
Before you install this update, make a backup of the Data Protection Manager database.
Install this rollup package on the server that's running System Center 2016 Data Protection Manager. To do this, run Microsoft Update on the server.
In the Data Protection Manager Administrator Console, update the protection agents. To do this, use one of the following methods.

Method 1: Update the protection agents from the Data Protection Manager Administrator Console
Open the Data Protection Manager Administrator Console.
Click the Management tab, and then click the Agents tab.
In the Protected Computer list, select a computer, and then click Update in the Action pane.
Click Yes, and then click Update Agents.


Method 2: Update the protection agents on the protected servers
Obtain the update protection agent package from the following directory on the System Center 2016 Data Protection Manager server.


The installation packages are as follows:


For x86-based updates: i386\1033\DPMProtectionAgent_KB4043316.msp


For x64-based updates: amd64\1033\DPMProtectionAgent_KB4043316_AMD64.msp
Run the appropriate DPMProtectionAgent.msp package on each protected server, depending on the architecture of the agent.
Open the Data Protection Manager Administrator Console on the System Center 2016 Data Protection Manager server.
Click the Management tab, and then click the Agents tab. Select the protected server, update the information, and then verify that the agent version number is 5.0.342.0.

Mandatory Update for the MARS agent

Microsoft just released the information that you must update your MARS agent with the new released version 2.0.9099.0 since it contains a bunch of fixes.

Download the MARS agent and update your DPM, Azure Backup Server MARS agents.

The entire article can be accessed here

TechDays 2017 – A blast!

Just got back from TechDays 2017 and as always it was a great pleasure meeting up with the community, Microsoft and my MVP peers.

The best part of delivering sessions is (as always) the talks you have after with the attendees. I’m grateful for this opportunity and looking forward to next year!

Migration between MBS disks

Modern Backup Storage, or MBS, doesn’t just provide better performance and resiliency, from UR4 Microsoft made it possible to migrate your VHD files from one ReFS volume to another. This is done per protected datasource but can easily be automated using PowerShell.

This is a feature that the community has longed for a long time so its GREAT (😊) that Microsoft could make that possible!

You can either migrate your DPM storage via GUI or PowerShell. To be able to move the storage you need to associate another volume first to the DPM 2016 UR4 server.

If you want to migrate using PowerShell its done via three simple steps:

1. Make the Protection Group modifiable

2. Get the data source you wish to migrate and the volumes you wish to migrate to

3. Modify the disk allocation for the datasource and save the new Protection Group configuration



$pg = Get-DPMProtectionGroup

$mpg = Get-DPMModifiableProtectionGroup $pg[0]



$ds = Get-DPMDatasource $mpg

$vols = Get-DPMDiskStorage -Volumes



Set-DPMDatasourceDiskAllocation -ProtectionGroup $mpg -Datasource $ds[0] -TargetStorage $vols[0] -MigrateDatasourceDataFromDPM

Set-ProtectionGroup $mpg


If you don’t want to use PowerShell you can simply click the data source you want to migrate and from the Data Source Operations part in the DPM console choose Move Disk Storage. You can also right-click the data source and perform the same operations.

DPM 2016 UR3...or?

Microsoft just released the UR 3 for System Center 2016 but as many if you have noticed there is no update for DPM 2016 within this release.

The reason is that there is no fixes made from the DPM team yet that can be released. This doesn’t mean that they have given up, got that question via email from a concerned user. Features and fixes will be released within the UR 4 release.

For more information regarding the UR3 update can be found here

Direct download link for the MARS agent

I’ve got a lot of questions about how to get a hold of the latest MARS agent from Microsoft in an easy manner.

The best part of this is that Microsoft has provided a direct link to the latest MARS agent build that works like a charm. You can access the latest version of the MARS agent using the link below.

http://aka.ms/azurebackup_agent

How to disable the timeout value for the MARS agent

DPM and Azure integration via the MARS is a great feature since we can ship production data to an Azure Recovery Services Vault and be able to restore that either via the DPM console or PowerShell.

I noticed that there is a timeout value that could be interesting for the community to take part of. As I mentioned in my earlier blogpost regardingOptimized VS. Unoptimized Online jobs the MARS agent needs to verify the data integrity and all check all the files that resids within Azure and the DPM diskpool. Running an Unoptimized job will take a very long time and in some cases the timeout value of the job will be there canceling the job.

To avoid this you are able to adjust the following registry value:

HKLM\Software\Microsoft\Windows Azure Backup\DbgSettings\DisableThreadTimeout REG_DWORD=1

This will disable to timeout for the MARS agent and the online job till finish successfully and you can after that go back to run optimized online jobs

Optimized VS. Unoptimized online jobs

Recently I noticed an “interesting” thing that I would like to share with the community. When a DPM or Azure Backup Server archive data to Azure its known that the initial online job will take some time to finish since it will ship all the data to Azure from the on-premises server.

All the data that is replicated to Azure after the initial online job will run in an optimized state. This means that only the changed data will be shipped to Azure from your DPM or Azure Backup Server and it will be finished quite fast depending on the actual data churn.

If that online job fails the MARS Agent will lose track of what is shipped to your Azure Recovery Services Vault and needs to verify the data and the history that resides within the Azure Recovery Services Vault to be sure what data has changed from the last archived Recovery Point provided to Azure.

This is a very time-consuming job and it can take over 100 hours to complete, just had a client that had an online job running for forever and looking at the job history in the DPM console the last Online job had failed. This means that the DPM or Azure Backup Server needs to perform an Unoptimized online job.

I hope this article sorts out some question marks for you 😊

Provide feedback...Microsoft will listen

The intention of this blog article is to update those who may be new in this community 😊

You are ALWAYS welcome to send me a direct email @ Robert.hedblom@outlook.com but there is also other ways to provide feedback to Microsoft and the Product Group for DPM, Azure Backup and Azure Backup Server.

At this feedback site you are available to post your thoughts and also vote on other community members feedback. And guess what…the Product Group really reads this feedback and takes it into considerations.

https://feedback.azure.com/forums/258995-azure-backup

Cannot backup data sources – Volsnap error 39

This week I bumped into a new challenge that I thought I would like to share with the community. A customer reached out to me regarding an error that they couldn’t backup a drive of a server. The D drive was simple not able to be backed up. I looked into the problem and noticed a Volsnap error in the System Log of the DPM server.

The error stated that when DPM tries to create a snapshot of the drive there’s no Shadow Copy Storage available on the drive that DPM protects.

This can easily be solved by using VSSADMIN. Open an elevated commad prompt and use the syntax

Vssadmin add shadowstorage /For=D: /on=D: /maxsize=unbounded 

What VSSADMIN will do is to create a new Shadow Copy Storage for DPM to use.

Experts Live 2017 – Berlin

Experts Live 2017 is over and it was (as always) a blast meeting up with the community, deliver sessions and be part of the “Ask The Experts”.

Together with my fellow MVP Janaka Rangama we delivered a Business Continuity session describing how to enable Business Continuity using ASR.

If you want to get your hands on the slide-desk just pop me an email 😊

Ignite...here I come 😊

It’s with a big pleasure I’m writing this blog post today, I just got my session accepted for Ignite in Orlando!

I will deliver a Business Continuity session that will go through how you can use the Microsoft portfolio, both on-prem and Azure, to accomplish a true Business Continuity approach for your business.

During the week I will schedule a bunch of meetings with my peers and other Product Group Managers so if you have any feedback from the community pleases feel free to provide that to me @ Robert.hedblom@outlook.com

Release Notes for System Center Data Protection Manager (DPM) Technical Preview 1711

For those of us that tries the 1711 preview version Microsoft released a great article regarding how to determine if the server that you installs or update the DPM agent will restart or not.

For those of us that plays around with 1711 its great info 😊

https://docs.microsoft.com/en-us/system-center/dpm/dpm-release-notes-1711?view=sc-dpm-1711

Size do matter…

Archiving to Azure is a great feature that Microsoft made possible a while back. With new technology also comes new challenges, one is when an online job stops working and how to troubleshoot that.

When DPM 2012 R2 creates a Online Recovery point its important that the Replica and the Recovery Point volume within the DPM disk pool is more than 3 GB in size. If not you will get the error ID 100034.

The way to solve this is really simple. Just increase the size of your Protected Item within the Protection Group and increase the size to more than 3GB and then rerun your Online job.

TechDays 2017

Just got word that I’m about to deliver my “Business Continuity & Azure” session at TechDays in Stockholm that will take place between the 25th – 26th of October.

Really looking forward to present this topic and describe how you can leverage Azure as a public cloud to gain Continuity in your datacenter 😊

Looking forward to see you in the audience!!

#COMMUNITYPOWER

DPM MMC consoe won’t open

A while back i bumped into a problem with a DPM console that all of a sudden started to crash. The situation was only affected for one user and not other DPM administrators so the first conclusion was that this is user-specifics.

DPM (and other System Center products) creates a version of the MMC in the user profile. If this version is damaged or corrupt you will get an error stating

“MMC cannot open the file… This may be because the file does not excist, is not an MMC console, or was created by a later version of MMC. This may also be because you do not have sufficient rights to the file.”

To resolve this you simple need to delete the version file of the MMC. This file is located @ “:\Users\\AppData\Roaming\Microsoft\MMC\Microsoft System Center 2016 Data Protection Manager.msc” and restart the console and you are back on track.

Error 948

Recently I got many questions regarding the same error so I choose to write a blogpost about it. A few of you has reached out to me and asked what to do when the DPM service is inaccessible and you end up with Error ID 948.

The reason why this error could occur is due to DPM not being able to access the DPMDB. The reason why is endless and this is DPM’s way of saying that the communication to the DPMDB is lost.

The way to fix this is however simple! Open the DPM Management Shell and run the DPMSYNC command with the switch -SYNC. This will verify the DPMDB with the protected objects and make “adjustments” so you will be able to access your DPM server again. When the command is done you can access your DPM console (meaning that the DPMDB is now accessible) and ALL of your protected items will be in the same state….Replica Inconsistent. 

Simply run a Consistency Check on your protected items and you will soon be back on track!

Experts Live 2017

The Swedish summer starts of great, just got chosen to be a speaker at Experts Live in Berlin the 23rd – 25th of August.

During the conference I will as usual sit down with the community members and talk regarding your thoughts and share my knowledge for the good of the community.

I will deliver two sessions. The first one is an Breakout session that I will do together with my fellow MVP Janaka Rangama, we will talk about how to provide Disaster Recovery Strategies based on the Microsoft Azure platform leveraging Azure Site Recovery and other System Center solutions.

The second session is an Ask-The-Experts where we hope that as many of you can attend to sit down and discuss Disaster Recovery 😊

Hope to see you all in Berlin!

MVP renewal 2017 - 2018

Thank you Microsoft for giving me the great honors of being a MVP for yet another year. I’m deeply moved and proud of being part of this elite program of fantastic people that I get to collaborate and provide the community voice regarding Backup, Restore, Disaster Recovery and Business Continuity.

I will try my very best to continue to provide and be the community voice and share my experience.

😀

Update Rollup 13 for SCDPM 2012 R2

Microsoft has just released Update Rollup 13 for DPM 2012 R2 and with that comes three fixed issues:

Issues that are fixed

• Individual file location recovery (ILR) for VMWare VMs fails for the latest scheduled recovery point
• After the upgrade to Update Rollup 12, backup agent upgrades and installations fail on Windows 7 clients
• While synchronizing a deduped volume, alternate synchronizations raise an alert that synchronization of a few files failed, which is resolved for the subsequent synchronization

Download UR13 for DPM 2012 R2 here

Update Rollup 2 for SCDPM 2016

Microsoft has just released Update Rollup 2 for DPM 2016 and with that comes two announcements and seven fixed issues:

Announcing backups of SQL Server 2016 and SharePoint Server 2016

After you install this update, you can seamlessly back up your SQL Server 2016 and SharePoint Server 2016 workloads.

Announcing enhanced security for backups

This update includes the following improvements to security features of Azure Backup. These improvements provide greater resilience and recoverability from adversarial attacks on your important business data.

Retention of deleted backup data

Azure Backup can now be configured to keep, for 14 days from the day of deletion, any backup data that is deleted because of adversarial attacks. This makes it easier to recover your data after an attack.

Minimum retention range

Minimum retention range checks make sure that more than one recovery point is available after an attack.

Azure Portal-based authentication

Azure Backup can now be configured to request Azure Portal-based authentication for important operations that affect backup and recovery of your organization’s data.

Security alerts and notifications

You can set Azure Backup to receive security alerts and notifications when you run important operations that affect your data-protection plan. This lets you keep a close watch on any intrusive operations that are made by unauthorized personnel.

Note To access these features, make sure that your computer or server is registered to an Azure Recovery Services vault.

Issues that are fixed in this update rollup

• When the length of the file path is long, file names aren't displayed on the Recovery tab
• Backup of VMs with RCT fails after CCs
• BMR backups for Windows Server 2008 R2 Service Pack 1 fail
• Backups fail with an error stating that the VHD that contains the replica or a snapshot of the replica could not be mounted or unmounted
• Data Protection Manager crashes while migrating workloads from or to Modern Backup Storage
• Auto-protection of SQL Server databases is enabled for backups to Azure in addition to disk backups
• VM backup from the DPM server fails and generates the following error message:

The replica of Microsoft Hyper-V on is not consistent with the protected data source.

DPM has detected changes in file locations or volume configurations of protected objects since the data source was configured for protection. (ID30135)

Download UR2 for DPM 2016 here

Midwest Managment Summit 2017

I’m very happy to announce that I will be presenting at the MMS (Midwest Management Summit) 2017 in Minneapolis at the Radisson Blu in Mall of America.

Midwest Managment Summit 2017

I will present five sessions that you can find more information about here

The sessions I will present is:

• Gain Super Insight with OMS and a proper SCOM Design
• Backup is dead! Restore is Born in the Cloud! 
• Business Continuity and the Microsoft Cloud
• IT Department Staffing without Silos in a Worldwithout Datacenters
• How to Save Your Money and Get a Pay Raise

I will have the great pleasure to co-present with some close friends:

• Cameron Fuller
• Steve Buchanan
• John Joyner
• Andrew Craig

I really hope to see you at the MMS 2017 since this is a great community conference with the possibility to hang out and talk about a lot of cool Microsoft technologies and get some answers to your questions!

I'll see you there!

System Center Data Protection Manager 2016, Best Design Practices

The purpose of this blog article is to inspire and inform you as a reader what you should consider before you start implementing System Center Data Protection Manager 2016 as the restore-feature within your datacenter.
Throughout the years, I have noticed a lot of different designs that hasn’t played out well. So hopefully readers of this article will avoid any of those pitfalls that either leads to reinstalling System Data Protection Manager or realizing that you are not able to do that restore scenario that you first thought of.
Having sorted out those questions I can assure you that the road to a successful DPM implementation based on the BaaS concept will be closer at hand for sure.

The start, building the restore scenario

The most important part when it comes to backup has always been, and will always be one thing, restore. This should always be your major and primary focus in any design regardless of technology.

Having a prober recovery-plan will provide you with an optimal backup design and strategy that will meet the company need for recovery. However, this is just the thing that the majority of companies misses out on. Building the restore scenario first will provide you a more optimal design when it comes to, for example, the number of DPM servers needed, time for recovery point creations and also archiving/long-term protection and more. The best way to get started with this strategy is to identifying two things:

• Infrastructural Services
• Business Services

Your Infrastructural Services is for example your Active Directory meanwhile your Business Services could be your CRM solution that consumes the Infrastructural Services that builds-up the Business Services. The initial step in building a true restore scenario is to breakdown all the Infrastructural Services that cooperates to deliver the Business Services. What Windows Servers are involved? Does the Business Services use SQL Server and is there any IIS also dependent for a successful delivery of the Business Service from an end-user perspective and so on.

Having a detailed breakdown of a Business Service will also help you identify and understand how you could consume Azure services like Azure Site Recovery and others IaaS, PaaS or SaaS services in the most optimal way for your restore-process of the Business Service.

Virtual vs. physical DPM servers

This is a very common discussion, even today, where people tend to keep the physical concept for the DPM servers instead of virtualizing them. The most important part when it comes to providing restore concepts for a company is being able to simply scale and provide more resources to your BaaS (Backup-as-a-Service) service that you provide and deliver for your company regardless of the company size.

This is a simple task when you adapt a virtual concept for your DPM servers where you are able to provide a deployment plan for new DPM servers that will be associated with your BaaS.

The most important take-away; if you haven’t virtualized your DPM servers yet and thinking of deploying DPM 2016, Microsoft highly recommend you to virtualize them using Hyper-V to achieve the possibility of providing scale. 

DPM disk setup

To get the most out of your DPM servers, both form a performance perspective but also from a disaster recovery perspective (restoring the DPM server itself), you should use the following disk setup.

• OS disc                  (your %systemdrive%)
• Program disc      (Where you install all software)
• DPMDB disc        (dedicated disk for the DPMDB)
• Azure Scratch     (A disc dedicated for the MARS agents scratch catalogue)
• Recovery disc     (A dedicated disc dedicated for the prestaging procedure for the MARS agent)

All discs should be in the VHDX format and the dedicated DPMDB disc should be fixed in size due to performance; all other discs could be dynamic discs.

SQL Server installation

There are some key points when it comes to delivering an optimal SQL Server installation for System Center Data Protection Manager 2016. The importance of having dedicated service-accounts for your DPM server is a must but also the fact of using the correct collation of your SQL instance hosting the DPMDB. The only collation you should use are SQL_Latin1_General_CP1_CI_AS

All other collations are unsupported, so keep in mind to use the right one or you end up reinstalling your DPM servers from the beginning.

Also, remember to setup the amount of memory your SQL Server should consume. This should be set accordingly to the number of GB you have, spare at least 4-6 GB of RAM for the operating system. The SQL memory configuration is defined in the SQL instance that hosts the DPMDB.

Having a poor or wrong setup or configuration will give you a negative impression of DPM. Keep in mind that the SQL Server is the engine behind it all; If its configured poorly the engine will perform badly, simple as that.

Antivirus exclusions DPM and the MARS agent

The most common performance challenges there is regarding DPM is the fact of not setting up the real-time protection on the antivirus software correctly. If you don’t configure the exceptions for catalogues and services, you will end up with possibly corrupted data.

For System Center Data Protection Manager you should exclude the following catalogues that resides within the DPM installation catalogue:

• XSD
• Temp\MTA
• Bin

The following process should be excluded from real-time scanning:

• CSC.EXE
• DPMRA.EXE

For the DPM servers that has the MARS agent installed and pushing data to Azure it's important to exclude the following catalogues:

•  Microsoft Azure Recovery Agent\Bin
• Scratch folder

The following processes must be excluded from real-time scanning:

• CBEngine.exe
• CSC.EXE (Goes for both DPM and the MARS agent)

Not having the correct exclusions will end up with your local antivirus scanning your DPM disk pool or the or scratch area for example.

However, there is one more thing you should keep in mind. In the case where your antivirus software does find a threat you shouldn’t quarantine it (which is the most common policy), you should have it deleted by default.

Protection Group Designs

This is one of my favorite topics from the field. I have seen a lot of interesting designs when it comes to designing Protection Groups throughout the years and here is some of my thoughts that has played out very well for a large number of companies.

The first thing to keep in mind is the Protection Group name. The best staring point is to build your Protection Groups designs according to your Business Services RTO, RLO and SLA (if any). The Recovery Time Objectives (RTO) is the definition if how fast you should be able to be back-on-track with your Business Service. System Center Data Protection Manager can either synchronize your data changes every fifteen minutes for workloads like SQL, Exchange and File. For other workloads like SharePoint, Hyper-V etc., DPM can make Recovery Points every thirty minutes. So, having a clear understanding of your Business Services and your Infrastructural Services are crucial since the Protection Group is where you set the actual backup strategy or plan that should correspond to your restore plans.

Regarding naming of Protection Groups there are a few tips that could hopefully inspire some DPM administrators. To get a decent start you should consider using the following naming convention for your Protection Groups:

• Workload + “number of recovery points per day or week” + time + SYNC info + Azure + time

An example for a Protection Group having this naming convention would be:

• File (1RP/d 01:00PM | 6h Sync | Azure 01:00 AM)

In some cases, you could also provide the retention range for your on-prem disc and also Azure to make it even more clear. An example would be:

• File (1RP/d 01:00PM 30 days|No Sync|Azure 01:AM 180 days)

In this latter example, you get a clear understanding of what kind of members should be associated with this Protection Group but most importantly your restore capabilities. Let’s break it down shall we. The members of this Protection Group are for the File workload. Everyday a recovery point is made 01:00 PM that has a Retention Policy that states that the data should be available for 30 days on-prem meaning in the DPM disk pool. There is no extra synchronization made for the members of this Protection Group and all protection data will be sent to Azure 01:00 PM where it will be stored 180 days back in time.

Replace your tapes, use Azure since it really works…

There are many companies that has now started their journey to using Azure instead of tapes for their long-term retention. In many Business Cases I have made, Azure is far most the most optimal and cost effective solution when it comes to long-term retention for System Center Data Protection Manager protected workloads.

More than 50% of the cases; using Azure will cut the cost in half even for those companies that uses TaaS (Tape-as-a-Service) solutions or VTL.

One important fact to point out is the possibility to restore data from a Recovery Service Vault that is shared between DPM servers. Let’s say that you have two or more DPM servers associated with the same Recovery Services Vault. If one DPM server fail you are still able to restore the data that the DPM server pushed to Azure by adding the Azure Passphrase generated in the MARS setup. This could be done via the Add External DPM feature from the Recovery pane in DPM. This means one simple thing, as long as you have your protected data in Azure, you will always be able to restore it.