One important thing regarding backup of system states is that the local installed feature of WSB is operational. If your WSB is having trouble this will lead to problems with your backup in DPM.
In WSB there’s a global catalogue used by the WSB feature. This could enter a corrupted state and causing the System State backup (and BMR) to fail.
Here’s an example of the event log:
The backup operation that started at '2011-06-21T10:34:12.022776900Z' has failed with following error code '2155347969' (An operation on the global catalog failed because the catalog is corrupted.). Please review the event details for a solution, and then rerun the backup operation once the issue is resolved
To correct this, open the local WSB on the affected server and choose to clear the Global Catalogue in the right action pane.