Project Server 2010: Orphan baselines breaking the reporting publish

This problem has been around for a while and I know some customers were running into it very soon after the release, but we had been struggling to get a repro and understand exactly what was causing it.  We now understand the root cause and have a fix coming hopefully in the June 2012 Cumulative Update for Project Professional 2010 (no promises – but that is the current target) and there are some ways of working that can limit your chances of running into this – so decided we should share this to avoid continued inconvenience until we get the fix out there. First lets take a look at the symptoms.  The most usual indication of the problem, as the title suggests, is orphan baseline values leading to the error when publishing – a Failed But Not Blocking Correlation problem on a Reporting (Project Publish) job that will show several of the following errors if you click through for the error details: ReportingProjectChangeMessageFailed (24006) - The INSERT statement conflicted with the FOREIGN KEY constraint "FK_MSP_EpmTaskBaseline_ProjectUID_TaskUID". The conflict occurred in database "ProjectServer_Reporting", table "dbo.MSP_EpmTask". The statement has been terminated.. GeneralQueueJobFailed (26000) - ReportingProjectPublish.ReportProjectPublishMessageEx These failures are for the reporting job – so will mean that reports based on the reporting database, and any fresh OLAP cube builds could be missing data. Sometimes there may also be a crash on saving, either with a fairly generic MSSOAP 16 Send Incomplete error from Project Professional 2010 (though a subsequent save will work fine), or from PWA a queue error - GeneralQueueException (9131) A Project Operation failed due to a Queue Exception. Sub Job ID is: . Exception details are: System.NullReferenceException: …at Microsoft.Office.Project.DataEdit.Assignments.AssignmentCalendarUpdateHelper.ConvertActualContourToElapsed(,,, There may then be issues with users accessing timesheets – The view failed to load.  Press OK to reload this view… (and OK will not help). The error that will be found in the ULS logs will refer to a Calendar whose UID cannot be found… Exception occurred in method Microsoft.Office.Project.Server.BusinessLayer.Statusing.StatusingGetMyWorkForGridJson System.InvalidOperationException: CacheProjectBaseCalendars could not find project calendar for project. CalUid=0c13de33-2a07-4310-b091-c77990d9dd6a    The root of all these issues is that when you use any of the Save & Send options (XML, CSV, Excel etc.) that we are incorrectly changing some of the GUIDs associated with entities such as the tasks and calendars.  Now this isn’t affecting the main tasks and assignment GUIDs as these bad values are not persisted back to the database – but we do however create a new baseline for these non-existent new task GUIDs, and can also save a bad calendar GUID – which leads to the Timesheet problem. First the best way to avoid this issue, and then on to the detection and clean up at the database level. If you do need to use Save & Send then the best practice until we release the fix for this is to first save the plan to the server, and publish if you need to.  Then do whatever you need to with Save & Send, and then immediately after this – close and check in the plan – but do not re-save to the server.  Discard changes if it asks – but of course you will have needed to save BEFORE you did the Save & Send (just making sure you are paying attention) to avoid losing any changes you really needed.  As the bad stuff will also get persisted to the local cache, this is one of those rare occasions when you will find me suggesting that the project is removed from the local cache – after ensuring that the save and check-in completed successfully. WARNING – the following steps are direct queries against the Project Server databases – please be sure you are working against the right databases when using these – and have a database backup should any problems occur. The detection of this condition is pretty straightforward, as we are just looking for baselines that exist for a task that does not exist, so the following query executed against the Draft database will do this (Change the name to match your specific DBs – the default ProjectServer_ names are used below: -- Detect for orphan baseline task records that can cause reporting publish job failures. USE ProjectServer_Draft -- specify the appropriate draft database select PROJ_NAME, MTB.PROJ_UID,TASK_UID,TB_BASE_NUM from MSP_TASK_BASELINES MTB inner join MSP_PROJECTS MP on MTB.proj_uid=MP.proj_uid where TASK_UID not in (select TASK_UID from MSP_TASKS) This will return rows if the condition exists – and identify which projects – as before clean-up you will probably want to get them removed from the PM’s local cache as otherwise they could be re-introduced. The next scripts do the cleaning up in the DB, and they are simply deleting baseline records where the tasks are non-existent. -- Script to run on the draft DB USE ProjectServer_Draft -- specify the appropriate draft database delete from MSP_TASK_BASELINES where TASK_UID not in (select TASK_UID from MSP_TASKS) -- Script to run on the published DB USE ProjectServer_Published -- specify the appropriate published database delete from MSP_TASK_BASELINES where TASK_UID not in (select TASK_UID from MSP_TASKS) I hope this helps to understand the nature of the issue and ways to avoid it until the fix comes along.  Our apologies for the inconvenience I know this has caused many of our customers – and hopefully for those who have needed to re-run the clean-up scripts regularly this may give a way to reduce the pain. If you need any assistance with these steps then feel free to open a support incident – and when I say free I mean free – this is a bug and we do not charge for incidents that are due to bugs (or we will refund – which amounts to the same thing). The ULS log entry associated with the initial Queue errors above (for the benefit of the search engines): 05/01/2012 11:57:55.67    Microsoft.Office.Project.Server (0x1D74)    0x335C    Project Server    Reporting    atwj    Critical    Standard Information:PSI Entry Point:   Project User: REDMONDbrismith  Correlation Id: e1f4e953-7dea-448a-a528-709075c698bf  PWA Site URL: https://brismith8100/PWA   SSP Name: Project Server Service Application  PSError: ReportingProjectChangeMessageFailed (24006) RDS: The request to synchronize change(s) to project Project UID='216733b0-e194-469a-afc3-9235da4ce4c1'. PublishType='ProjectPublish' failed.  Message: 'ReportingProjectChangeMessageFailed'. Message Body: The INSERT statement conflicted with the FOREIGN KEY constraint "FK_MSP_EpmTaskBaseline_ProjectUID_TaskUID". The conflict occurred in database "ProjectServer_Reporting", table "dbo.MSP_EpmTask".  The statement has been terminated. Error:(null)    e1f4e953-7dea-448a-a528-709075c698bf and for the Timesheet error: 05/01/2012 12:13:29.65    w3wp.exe (0x2444)    0x23D8    Project Server    Task Statusing and Updates    btw9    High    CacheProjectBaseCalendars: could not locate data for calendar 0c13de33-2a07-4310-b091-c77990d9dd6a for project 216733b0-e194-469a-afc3-9235da4ce4c1    e5dd4eaf-551a-469b-a3e0-1f60e2f3d1af 05/01/2012 12:13:29.85    w3wp.exe (0x2444)    0x23D8    Project Server    General    0000    Exception    Exception occurred in method Microsoft.Office.Project.Server.BusinessLayer.Statusing.StatusingGetMyWorkForGridJson System.InvalidOperationException: CacheProjectBaseCalendars could not find project calendar for project. CalUid=0c13de33-2a07-4310-b091-c77990d9dd6a     at Microsoft.Office.Project.Server.BusinessLayer.TimePhasedDataAccess.CacheProjectBaseCalendars()     at Microsoft.Office.Project.Server.BusinessLayer.TimePhasedDataAccess..ctor(StatusingPageLoadDataSet dataset)     at Microsoft.Office.Project.Server.BusinessLayer.Statusing.ReadStatusTimephasedDataForResource(IList`1 gridChanges, Guid[] vAssnUids, IDictionary`2 assn2proj, StatusingTimephasedPeriod[]
tpdPeriods, DateTime tpStart, DateTime tpEnd)     at Microsoft.Office.Project.Server.BusinessLayer.Statusing. c__DisplayClass57. b__56(IEnumerable`1 Keys)     at Microsoft.SharePoint.JSGrid.GridSerializer.BuildOutput()     at Microsoft.SharePoint.JSGrid.GridSerializer.ToJson(Serializer s)     at Microsoft.SharePoint.JsonUtilities.Serializer.SerializeToJson(Object o)     at Microsoft.Office.Project.Server.BusinessLayer.Statusing.GetMyWorkForGridJson(JsGridSerializerArguments gridSerializerArgs, String gridChangesJson, String projectAssignmentsMap, Guid viewUid, String timephasedStart, String timephasedEnd, Byte pane, Int32 durationType, Int32 workType, Int32 dateFormat, Boolean clearPersistedProperties, Nullable`1 rowFilterType)     at Microsoft.Office.Project.Server.Wcf.Implementation.PWAImpl.StatusingGetMyWorkForGridJson(JsGridSerializerArguments gridSerializerArgs, String gridChangesJson, String projectAssignmentsMap, Guid viewUid, String timephasedStart, String timephasedEnd, Byte pane, Int32 durationType, Int32 workType, Int32 dateFormat, Boolean clearPersistedProperties, Nullable`1 rowFilterType)    e5dd4eaf-551a-469b-a3e0-1f60e2f3d1af

Read more:
Project Server 2010: Orphan baselines breaking the reporting publish


Leave a comment!

You must be logged in to post a comment.