Blog

As announced by Jan last week, we will have a dedicated Project/Project Server pre-event ahead of SharePoint Conference 2011 on Sunday October, 2nd, 2011 : SharePoint Partners – Interested in growing your business with Project 2010? A great opportunity for SharePoint/non Project Portfolio Managers to learn about our stack. More info here . There will also be plenty of Project sessions during SPC11 so don’t wait and register, looking forward to see you in California in early October:

Almost all Project Server administrators know that when you have an enterprise custom field using a lookup table that you can pick one of the values from that lookup table to be the default value for new entities.   But what about a case where the field is set to NOT have a default value but some new projects are being created where when the Save button is hit on the New Project PDP a value is being entered for that field? What might cause that? There are (at least) two possible answers. Templates on Enterprise Project Types When you create your enterprise project types you can associate an enterprise project template so that when a new project is created using an EPT it will automatically use that template. It is possible for a template to contain values in some enterprise fields as part of the template. If the template associated with the EPT has a value in that custom field then it can look like there was a default value set for the field, even when the field itself does not have a default set. Two things lead to this being overlooked: EPTs are a new feature to Project Server 2010 and it does not always occur to users that clicking “New” in the Project Center could call up a template to begin with. Many organizations tend to strip out custom field values when saving a project as a template so the idea of there being a value in the fields for a template might not occur to everyone. I have seen this issue at 2 different locations now and it can cause some pretty severe head-scratching.   Workflow Code or Custom Event Handler Code Through workflow coding or custom event handler code there could be situations where the code would examine the values of certain custom fields when the project is initially saved and then make decisions about the values of other custom fields. An example of this is if a project is created and the user picks “Washington” as the value for the ‘Location’ field. The code might be set to assign a certain value for an Account Manager” field based on the project being located in Washington state. For some users that do not know about this coding it might appear as if there were default values on some fields. This type of coding is much more common now in 2010 and in most cases users and certainly the administrators are likely to know about these kinds of changes. In my experience the most common cause for confusion about mystery ‘default’ values is going to be from a template on an EPT.

While we realize that many BPOS customers are eager to move onto the new Office 365 service and start using all the new features, we strongly recommend that you wait until we are ready to transition your tenant for you. We have received a number of Support calls from customers and partners who have tried to migrate their data from BPOS to Office 365 by themselves. Please be aware that if you attempt a manual migration, emails sent to your domains will be returned to sender as undeliverable until Support can delete your domain from BPOS and you can re-create in Office 365. This email outage could last as long as 24 hours (possibly longer if many customers are asking Support to delete their domains at the same time as you). What should someone do who is eager to move from BPOS to Office 365? Ideally, you would wait to be transitioned by the Office 365 team. However, if you are an expert in Exchange server migrations, comfortable with PowerShell scripts, and just can’t wait a couple of months, here’s what you need to do. (Please note that manual migration is not supported, manual migration will stop mail flow for at least 24 hours, and customers with Exchange Hosted Archiving will lose all archived data). Ensure all of your computers meet the Office 365 system requirements. Sign up for an Office 365 account. Do NOT add any other domains at this point Run the Office 365 online desktop setup tool on all desktop PCs. Back up all of your users’ mail to .pst files using Outlook. Remove all vanity domain information from BPOS. Remove all production domain e-mail addresses from any accounts using them. Here’s an example. Your production domain is contoso.com. You have a user, Joe, with the e-mail addresses Joe@contoso.microsoftonline.com and Joe@contoso.com. Remove the Joe@contoso.com e-mail address, as well as any other e-mail addresses in the system that end with contoso.com. Remove your production domain from BPOS. Contact BPOS Support, and Tell them you wish to have your production domain removed from FOPE. Ask them to check in the Office 365 environment to see if Microsoft has already copied your domain(s) and users to a pre-transition tenant. If this has happened, your domain(s) will need to be removed from Office 365 as well Wait 1 day for Support to remove your domains. You will receive no inbound email for this period. Sign in to your Office 365 admin account and add your production domain to Office 365. Add your users to Office 365. Import the .pst files you made in step 3 into the appropriate user mailboxes using Outlook. There will be a feature allowing domains to be deleted in FOPE without a call to Support, but it will take some time before this option is available. Please keep an eye on this blog for further updates. Final note: I have personally seen some creative workarounds in the BPOS community to the email outage that inevitably occurs while FOPE is deleting your domain. They’re not perfect, and Microsoft doesn’t support them (nor have we tested them).

Now that Office 365 has launched, we’re preparing to start transitions in September 2011, and we’ve updated the Transition Guide and Transition FAQ to provide you more information about what to expect between now and September 2012. Here’s a summary of the key actions you should take. 1)   Review the transition guide. This comprehensive overview covers all aspects of the transition process. Download the transition guide for in-depth information on everything from system requirements to the specific steps required for administrators and current users. 2)   Learn the key changes. Some of these system requirements may require you to upgrade your PCs before you transition. Make sure your business is ready for Office 365 and the new Microsoft Online Services by noting the following key changes: Outlook 2003 / Office 2003 is not supported Internet Explorer 6 is not supported Office Communicator 2007 R2 must be upgraded to Microsoft Lync Office 365 Desktop Setup is required (replaces the Sign-In Application) Review all of the Office 365 and Microsoft Online Services system requirements . In addition, make sure all your desktops are up to date for the necessary end-user requirements. You can find this information in the transition checklist for administrators. 3)   Understand the Office 365 password policy To make the transition to Office 365 as seamless as possible for your users, we will synchronize their current passwords with Office 365 whenever they change their password. This means that the new Office 365 password policy now applies to your existing subscription. Review the Office 365 password requirements to understand the changes. 4)   Download the transition checklist Start your transition prerequisites – The tasks required to configure your desktop environment can be found in the detailed transition checklist for administrators . 5)   Check out your new subscription offer The following table summarizes how Microsoft Online Services and BPOS subscriptions will map to the new Microsoft Online Services offerings after transition. All active subscriptions will be transitioned to the new offers while maintaining your current price through the end of your subscription term.  Learn more at the Office 365 website . Current Microsoft Online Services  Subscription   New Office 365 or Microsoft Online Services subscription BPOS Standard Suite >   Office 365 (Plan E1) BPOS Deskless Worker Suite >   Office 365 (Plan K1) Exchange Online >   Exchange Online (Plan 1) Exchange Online Deskless Worker >   Exchange Online Kiosk SharePoint Online >   SharePoint Online (Plan 1) SharePoint Online Deskless Worker >   SharePoint Kiosk (K1) Live Meeting Standard >   Lync Online (Plan 2) Office Communications Online >   Lync Online (Plan 1) What happens next? Microsoft will begin contacting customers and partners this summer with September and October 2011 transition dates. Each month, we’ll send out more transition scheduling notices. For more information about scheduling, please check out the transition FAQ . Processing timeframe The transition schedule is designed to best accommodate all customers and their needs, and that could mean your transition won’t be scheduled for several months. Don’t worry if you don’t receive a transition date immediately. Stay up to date 
 The transition center is the place to learn anything and everything about the transition process and all that Office 365 can do for your business. Microsoft Online Services is ready to support you through a smooth transition to Office 365. If you have questions, contact our IT-level support , which is available 24 hours a day, 7 days a week. Also, be sure to take a look at the transition forum – the answers you need are likely already available.

Microsoft is updating the Microsoft Online Services Transporter Tools (aka Microsoft Online Services Migration Tools), adding commandlets to allow Administrators to manage password expiration for user accounts, permissions for Exchange Online mailboxes, and server-side forwarding rules for Exchange Online. The following commandlets are included as part of this update: Set-MSOnlineUserPasswordNeverExpire . This commandlet allows Administrators to set user account passwords to never expire. While Microsoft does not recommend non-expiring passwords due to security best practices, non-expiring passwords are often needed for service accounts that support line of business applications, unmonitored mailboxes, and mail-enabled hardware. Administrators setting non-expiring passwords for user accounts in their organization will need to be aware of actions required when the organization transitions from BPOS-Standard to Office 365. Required steps for organizations with non-expiring passwords can be found in an earlier post to this blog on the topic. Add-MSOnlineMailPermission . With this commandlet, Administrators can establish alternate permissions for an Exchange Online mailbox, such as granting full access to a delegate or granting send-as and send-on-behalf permissions to a delegate. Set-MSOnlineAlternateRecipient . This commandlet allows Administrators to set server-side rules to forward Exchange Online messages to an alternative e-mail address, including forwarding to a distribution list (DL). The updated Transporter Tool supporting these PowerShell commandlets are now available via the Microsoft Download Center .

When: Thursday, September 01, 2011 at 9:30 AM – Friday, September 02, 2011 at 5:00 PM (GMT) Where: Technology House Shottery Brook Office Park Timothy’s Bridge Road CV37 9NR Stratford Upon Avon United Kingdom Hosted By: Technology Associates International Limited Technology Associates International Limited is one of the leading global project management consultancies specialising in Microsoft Office Project and Enterprise Project Management Solutions. Since 1990, Technology Associates have provided deployment, consultancy, development and training services to more than 500 organisations including some of the world’s leading companies, and we have an international presence throughout the world.     Technology Associates have deployed hundreds of EPM solutions and deployed more than 2,000 Microsoft Project solutions. With ten Microsoft Competencies under our belt, including ISV status, we have built a solid reputation for delivering high quality services and solutions, and providing exceptional value for money. Headquartered in the United Kingdom, with Offices in New York, San Francisco, India and Barcelona, and a strategic partner network covering AsiaPac, Middle East and ROW, we work internationally in delivering EPM and SharePoint solutions to our customer base in over 39 countries. Register for this event now at: https://taips2010admin010911-rss.eventbrite.com Event Details: Course Description: This course is designed to cover all the features and aspects of Project Server 2010 Administration. Attendance on the Introductory Project Professional 2010 course is an essential pre-requisite. Attendance on an advanced course with at least two to six weeks as a consolidation period is highly desirable . A ttendees ideally will have some experi ence of SQL Server, IIS, and SharePoint Server . Knowledge of network permissions, active directory and security models would be an advantage. Who Should Attend: Users who have a good knowledge of Microsoft Project 2010 and will be responsible for managing, maintaining and updating the Project 2010 Server database and user information. Course Content: Upon completion of the course, delegates will be able to: Understand EPM 2010 Administration Concepts & Setup ·          Understand the different areas of the PWA Home page and links from it. ·          Understand authentication – types and logins. ·          Understand how to set up e-mail notifications for users and their teams. ·          Demonstrate how to save Microsoft Project information offline. ·          Demonstrate how to view, edit, and update projects & tasks ·          Understand how to reject, create, and delegate task assignments. ·          Understand how to link documents , issues, risks and documents to tasks. ·          Describe the Outlook integration. ·          Show how to view projects in the Project Center & how to create and maintain the views. ·          Show how to view resources in the Resource Center & how to create the views. ·          Demonstrate how to view resource and assignment data related to one or more projects. ·          Understand how to edit enterprise resource information. ·          Describe how to revi ew an archive of object data. ·          View and update task changes to a Microsoft Project plan. ·          View and update calend ar changes. ·          Set up rules to automatically update projects. ·          View a historical archive of task updates. ·          Understand Check In/Check Out of Projects and resources. ·          Understand Outlook integration. ·          Understand Categories/Groups/Permissions & Security Templates and how to use these to configure the system. Essential for analyzing the business and how EPM can be adopted. ·          Understand how d ata is stored in Project Server Databases and in SharePoint. Project 2010 Server Administration ·          Understand the different Administration options. ·          Understand the communication and messaging process. ·          Understand the Task Views, Time Periods, Fiscal Periods and options. ·          Describe the provisioning process for creating workspaces. ·          Understand how to view and upload documents for projects and public documents. ·          Demonstrate searching of documents in the document libraries. ·          Understand how to create and edit an issue or risk. ·          Understand how to customize Issue and risk fields and views. ·          Describe what Microsoft Project Server is and what it is used for. ·          Demonstrate connecting Microsoft Project to Microsoft Project Server. ·          Understand the functionality of Microsoft Project Server. ·          Understand the different user types and the how they interact with Microsoft Project Server and the functionality of each, as well as how to modify permissions. ·          Understand the responsibilities of a Microsoft Project Server Administrator and adopt best practices. ·          Discuss the various views available, their purpose and how to customize them. ·          Understand the integration of SharePoint fr om the Microsoft administration perspective. ·          Understand the Databases in Project Server and how they interact. ·          Understand Reporting and Business Intelligence capabilities in EPM 2010. ·          Understand the Dashboard capability. ·          Understand customization of the Microsoft Project Web Access Client. ·          Describe license manag ement within Project Server 2010 ·          Discuss maintenance of clean data in the Microsoft Project Server database. ·          Understand how security works within Microsoft Project Server. ·          Demonstrate the process for setting permissions. ·          Under stand the process for time track ing and progressing updates. ·          Understand and edit the Resource Breakdown Structure. ·          Understand the Administration options in Project Server and how to use them. ·          Demonstrate and understand the significance of Outline code fields at Project, Task and Resource level. ·          Understand and apply the Portfolio capabilities and administer the settings in Project Server. ·          Understand ULS, Messaging logs, and the Queue services in Project Server. ·          Describe, understand and be able to apply Multi-value fields. ·          Recognise what the Active Cache is and how it works . ·          Understand the concept of web parts and SharePoint as a platform for Project Server.

Thanks to Paul Andrew for his permission to re-post this issue for the Project Server audience.  Probably not everyone has Visual Studio on their Project Server 2010 system – but I know I do, and get this error regularly (every 24 hours as Paul mentions  – and have heard of this issue from others too.  It will also pop up when you run the configuration wizard too, so with SP1 and the June CU out I thought it worth bringing this to your attention.  Here is the reason and some potential workarounds.  Thanks again Paul! Have you seen this popup exception message from the Visual Studio Just-In-Time (JIT) Debugger? Here’s the resulting event log error: An unhandled exception occurred and the process was terminated. Application ID: DefaultDomain Process ID: 6148 Exception: System.Security.Cryptography.CryptographicException Message: Keyset does not exist StackTrace: at System.Security.Cryptography.CryptographicException.ThrowCryptogaphicException(Int32 hr) at System.Security.Cryptography.SafeProvHandle._FreeCSP(IntPtr pProvCtx) at System.Security.Cryptography.SafeProvHandle.ReleaseHandle() at System.Runtime.InteropServices.SafeHandle.InternalFinalize() at System.Runtime.InteropServices.SafeHandle.Dispose(Boolean disposing) at System.Runtime.InteropServices.SafeHandle.Finalize() If you see this error message from SharePoint 2010 you can relax, nothing bad is happening. When SharePoint 2010 and Visual Studio 2010 are both installed on the same machine you may see this error every 24 hours. This occurs when the OWSTimer service has a regular process recycle and in the shutdown of the old process an exception is raised. The exception doesn’t interfere with the normal process shutdown and recycle and is only ever seen if you have a JIT debugger installed on the machine. You should never see this error on a production SharePoint 2010 server, because you should not be installing Visual Studio 2010 on those servers. You can safely ignore these exceptions and close the window, or leave it there. You actually cannot debug the process, because it will already have been closed by the time you click the button and start your debugger. You can avoid the popup exception message entirely by disabling JIT debugging on your machine. Bear in mind this will disable JIT debugging for all applications on the machine. To configure the server to no longer show a dialog when an unhandled exception occurs, use the registry editor to delete the following registry keys: · HKEY_LOCAL_MACHINESOFTWAREMicrosoftWindows NTCurrentVersionAeDebugDebugger · HKEY_LOCAL_MACHINESOFTWAREMicrosoft.NETFrameworkDbgManagedDebugger On a 64-bit operating system also delete the following registry keys: · HKEY_LOCAL_MACHINESOFTWAREWow6432NodeMicrosoftWindows NTCurrentVersionAeDebugDebugger · HKEY_LOCAL_MACHINESOFTWAREWow6432NodeMicrosoft.NETFrameworkDbgManagedDebugger More details on JIT Debugging are at: https://msdn.microsoft.com/en-us/library/5hs4b7a6(VS.80).aspx

I came upon this bug while working on my earlier posting regarding Reporting Database Refresh https://blogs.msdn.com/b/brismith/archive/2011/07/14/project-server-2010-reporting-database-refresh-failing-with-large-resource-pools.aspx – and specifically trying to bulk edit lots of resources at once in the Resource Center.  It appears we throw a very complex and large query over to SQL and it has some issues parsing it.  The failure is pretty silent for the user – who just gets the Resource Center page back and might assume everything worked ok.  However there will be no jobs in the queue, the change will not have been made – and the resources will all be left check-out (as you will find out if you try the process again!).  The answer is to check all the resources back in again (or at least the ones you checked out – someone else may be doing work with other resources) and then reduce the number you have selected and try again.  The size and complexity of the query depends mainly on the number of resources, but also will be affected by the number of custom fields at the resource level – but my finding were that you could edit around 400 resources at once with bulk edit if you had around 24 resource custom fields.  Your mileage may vary! The errors that are seen are interesting – in that it appeared to depend on the severity of the issue.  With just over the triggering number of resources the failure was pretty silent in the USL logs and SQL Server – exceeding by a few more (20+) then gave errors in SQL Server of Error: 208, Severity: 16, State: 0 (which tends to mean object not found, and I’m guessing it was not finding the temporary table it wanted to use in part of the query) – but still nothing much in the ULS logs – and it was only when increasing the number of resources close to 500 did I see the following in SQL Server Exception – Error: 191, Severity: 15, State: 1 User Error Message – Some part of your SQL statement is nested too deeply. Rewrite the query or break it up into smaller queries. and then the ULS logs gave me an exception too: 07/15/2011 15:49:02.07 w3wp.exe (0x19B8) 0x220C Project Server General 0000 Exception Exception occurred in method Microsoft.Office.Project.Server.BusinessLayer.Resource.ReadResources Microsoft.Office.Project.Server.DataAccessLayer.FilterDal+FilterException: Error during filter query execution. Query: declare @ResUid UniqueIdentifier; set @ResUid = eb736432-cd8d-4db2-8d9b-ad57bb3f0085; declare @EntUids NVarChar; set @EntUids = e162554e-45f0-496a-a86e-0010ef91ae13,bb57a927-1b4b-42e9-b29b-001c63e0b53c,… followed by hundreds more GUIDs I’ll be logging this bug internally too so we can consider a fix for it – or at least a limit on the resource selection so you don’t run in to this one.

Some of you may have noticed that my blog now prompts if you want to pin it to your Task Bar and then it will also give you some Jump List items to go to the recent posts directly, or over to the Project Administration blog.  I hope you find this useful.  If you haven’t seen these features or the prompt then perhaps you are not running IE 9?  Or have IE 9 running in compatibility mode?  Now is the time to take a look! You can either use the prompt in the floating bar or just drag the tab down to your task bar in Windows 7 (Server 2008 R2 with the Desktop Experience feature – as I am using in the first screen shot) and it will look something like this: and if you ‘right-click’ then the jump list looks like this (this time from Windows 7) Let me know if you think there are other useful Tasks to add to the jump list.  If you want more details on IE 9 then the Beauty of the Web site is a good starting place – and if you want to do something similar then take a look at https://www.pinmywebsite.com/ .  Thanks also to Stephanus Schulte who shared some of the details on doing this for our MSDN/TechNet blogs – as he has done at https://blogs.technet.com/b/sieben/ Enjoy!

If you have a large(ish) resource pool, with over about 1500 resources then you may run in to an issue with the Reporting Database Refresh not completing correctly – with the initial symptom that many of the Reporting (Resource Sync) and all of the Reporting (Project Sync) jobs may just say Cancelled and if you click through to the errors from the queue you will see: Reporting message processor failed: ReportingResourceChangeMessageFailed (24008) – A RDS message that was spawned during a RDB refresh operation attempted to execute outside of the time range in which the refresh operation run.. Details: id=’24008′ name=’ReportingResourceChangeMessageFailed’ uid=’e3caf6d3-cc85-4078-90d3-1ce1ad929776′ QueueMessageBody=’Resource UID: ‘7d536aef-9be1-46c7-971a-286e384918c8′. ChangeType=’Add’. ResourceChangeType=’All” Error=’A RDS message that was spawned during a RDB refresh operation attempted to execute outside of the time range in which the refresh operation run.’. Or Reporting message processor failed: ReportingProjectChangeMessageFailed (24006) – A RDS message that was spawned during a RDB refresh operation attempted to execute outside of the time range in which the refresh operation run.. Details: id=’24006′ name=’ReportingProjectChangeMessageFailed’ uid=’bfd6372f-b545-4368-8b8e-00807de566f0′ QueueMessageBody=’Project UID=’65cae55a-48e3-44fe-826e-f4f7fa478cc8′. PublishType=’All” Error=’A RDS message that was spawned during a RDB refresh operation attempted to execute outside of the time range in which the refresh operation run.’. and if you look in the ULS logs then you will see something like: 07/14/2011 12:19:04.56    Microsoft.Office.Project.Server (0x2084)    0x2538    Project Server    Reporting    auhd    High    PWA:https://Server/PWA, ServiceApp:Project Server Service Application, User:DOMAINUser, PSI: [RDS] ULS Event: ReportingResourceChangeMessageFailed was associated with exception: Microsoft.Office.Project.Reporting.ProjectReportingPublic.ReportException: A RDS message that was spawned during a RDB refresh operation attempted to execute outside of the time range in which the refresh operation run.     at Microsoft.Office.Project.Server.BusinessLayer.ReportingLayer.RDSBaseMessageProcessor.CheckIfAllowedToProceed(ReportingBaseMessage msg, MessageContext msgContext, Group messageGroup, JobTicket jobTicket)     at Microsoft.Office.Project.Server.BusinessLayer.ReportingLayer.ResourcesChangedMessageProcessor.HandleMessage(Message msg, Group messageGroup, JobTicket jobTicket, MessageContext mContext)    ae4c9ca3-2a2c-44d1-b7d3-3d54589fcc8a 07/14/2011 12:29:16.10    Microsoft.Office.Project.Server (0x2084)    0x3868    Project Server    Reporting    atwr    High    PWA:https://Server/PWA, ServiceApp:Project Server Service Application, User:DOMAINUser, PSI: [RDS] ULS Event: ReportingProjectChangeMessageFailed was associated with exception: Microsoft.Office.Project.Reporting.ProjectReportingPublic.ReportException: A RDS message that was spawned during a RDB refresh operation attempted to execute outside of the time range in which the refresh operation run.     at Microsoft.Office.Project.Server.BusinessLayer.ReportingLayer.RDSBaseMessageProcessor.CheckIfAllowedToProceed(ReportingBaseMessage msg, MessageContext msgContext, Group messageGroup, JobTicket jobTicket)     at Microsoft.Office.Project.Server.BusinessLayer.ReportingLayer.ProjectPublishMessageProcessor.HandleMessage(Message msg, Group messageGroup, JobTicket jobTicket, MessageContext mContext)    ae4c9ca3-2a2c-44d1-b7d3-3d54589fcc8a The first message will appear for cancelled resource sync jobs, and the second for projects.  If you don’t happen to notice this issue, then the next thing you might see are failed reporting publish jobs such as the resources that assignments belong to may not exist in the reporting database (if they were among the cancelled ones.  These errors will look something like this: Reporting message processor failed: ReportingProjectChangeMessageFailed (24006) – The INSERT statement conflicted with the FOREIGN KEY constraint “FK_MSP_EpmAssignment_ResourceOwnerUID”. The conflict occurred in database “ProjectServer_Reporting”, table “dbo.MSP_EpmResource”, column ‘ResourceUID’. The statement has been terminated.. Details: id=’24006′ name=’ReportingProjectChangeMessageFailed’ uid=’6b806909-7a24-4409-8b80-b898f4a904a9′ QueueMessageBody=’Project UID=’79bf1075-a46a-467c-828a-24a1dc00ebbb’. PublishType=’ProjectPublish” Error=’The INSERT statement conflicted with the FOREIGN KEY constraint “FK_MSP_EpmAssignment_ResourceOwnerUID”. The conflict occurred in database “ProjectServer_Reporting”, table “dbo.MSP_EpmResource”, column ‘ResourceUID’. The statement has been terminated.’. Queue: GeneralQueueJobFailed (26000) – ReportingProjectPublish.ReportProjectPublishMessageEx. Details: id=’26000′ name=’GeneralQueueJobFailed’ uid=’29aca4ff-eb33-46ac-b6e6-1126f8532dae’ JobUID=’8ed284d5-7015-4e6e-a344-6380aae4b0d1′ ComputerName=’ServerName’ GroupType=’ReportingProjectPublish’ MessageType=’ReportProjectPublishMessageEx’ MessageId=’1′ Stage=”. For more details, check the ULS logs on machine ServerName for entries with JobUID 8ed284d5-7015-4e6e-a344-6380aae4b0d1. To better understand the reason for the failure it may help to understand more about the reporting database refresh process.  It is a process that gets automatically started based on certain conditions that would mean the data in the reporting database would not be consistent – and the most common of these is that some metadata such as the custom fields have been restored from archive.  Once this happens then the reporting data may be incorrect – as it may have custom field data that no longer makes sense – for CFs that were not in the archive so they no longer exist.  If you monitor the queue you will see an initial job that says Reporting Database Refresh and this will soon change to Reporting Database Refresh(Sleeping) – which is ok.  You will then see several sets of jobs getting added at intervals.  It will look something like this: Reporting (Fiscal Period Sync) – Immediate Reporting (Resource Capacity Range Sync) – Immediate Reporting (Lookup Table Sync) for each lookup table, and added to the queue 5 minutes after the Reporting (Resource Capacity Range Sync) Reporting (Custom Field Metadata Sync) for each custom field, and added to the queue 5 minutes after the Reporting (Lookup Table Sync) Reporting (Entity User View Refresh) for each view and added at the same time as the Reporting (Custom Field Metadata Sync) Reporting (Resource Sync) for each resource and added to the queue 5 minutes after the Reporting (Entity User View Refresh) Reporting (Workflow Metadata Sync) for stage, phase etc. – added 5 minutes after Reporting (Resource Sync)’s finish Reporting (Enterprise Project Type Sync) for the EPT’s – added at the same time as the Reporting (Workflow Metadata Sync) Reporting (Project Publish) for each project and added 5 minutes after the Reporting (Enterprise Project Type Sync) Reporting (Timesheet Assignments Refresh) – added 10 minutes after the Reporting (Project Publish) Reporting (Timesheet Project Aggregation) one for each timesheet period – added at the same time as the Reporting (Timesheet Assignments Refresh) While all this is going on the initial job will wake every few minutes to make sure things are still going along OK – and it is at this point that the bug this blog is about can break things.  It is a timing issue and it is possible that when the waking job checks to see if all is OK it gets a response it is not expecting so it thinks things have failed – writes a failure message to one of our database tables along with a failure time.  Subsequent jobs are checking this table too and will see that things have apparently failed so they will just cancel (or some of the jobs may show “Failed but not blocking correlation”).  This is the reason for the error message saying “A RDS message that was spawned during a RDB refresh operation attempted to execute outsid
e of the time range in which the refresh operation run” – as the current time does not fit between the apparent start and end time of the reporting database refresh. So how to recover?  There will be a fix coming along – and my hope is that it will make the August 2011 Cumulative Update – but there is a workaround if you should find yourself in this situation.  You will need to force a reporting update for all the resources and all the projects.  For the resource you can open them in Resource Center and Save (no change needed), and you will see the Reporting (Resource Sync) and the Reporting (Timesheet Project Aggregation) jobs kicked off for this user.  The reason you need to refresh all resources and not just the ones that failed is this second group of jobs – the Reporting (Timesheet Project Aggregation) – which puts data from the admin lines of the timesheets into the MSP_EpmAssignments and MSP_EpmAssignmentsByDay tables.  You can use the Bulk Edit option to enable you to do multiple resources at once but for this to work you need to make a change (you could add a new CF and just type some text into it)  – and you also need to be sure all your resources are checked in.  You may find the page a little unresponsive when you hit Save (you might want to limit it to a few hundred at once – I did around 200 at a time and that worked for me .  A trick that might help – if you select a group of rows using the shift-click you can then check the select box for all of the selected rows.  I’m still playing around with the options here – I will update the post if I find some new tricks and tips. For projects you will need to re-publish all the plans.  You can do this either using ProjTool from the SDK https://msdn.microsoft.com/en-us/library/gg446880.aspx or use a PowerShell commands made available by the code gallery sample at https://archive.msdn.microsoft.com/pj14PowershellPSI .  The getting started guide on that page has a sample that can do a check-out, publish and check back in.