Project Server 2010: How to best manage large numbers of resources

This posting follows on from the one yesterday concerning how Project Server makes use of SharePoint permissions and features – but concentrates on some potential issues you can run into if you have a very large user base and also have projects that have very large teams.  We are also authoring a TechNet article explaining this in more depth – I will add a link once it is published.  This isn’t going in to the usage of the RBS or the other internal feature – but concentrates more on the technical issues of large user populations.  If you fit in this category then read on… As mentioned in the previous post Project Server 2010 uses the normal SharePoint permissions infrastructure to set access control both to the Project Web App (PWA) site and also any Project sites that are created for the individual project plans held in Project Server.  At the PWA site level the users are added to certain groups depending on their permissions levels within Project Server, so you will generally see SharePoint groups for Project Managers, Readers, Team members, Web Administrators and finally Workflow and Project Detail Pages Administrators.  Each of these groups will show the individual PWA users as appropriate.  This is a change from 2007 where individuals were added to the PWA site with specific permission levels.  You may have seen issues in 2007 if you had large numbers of users as whenever changes were needed in the member permissions the users would be removed and then added back – so some users would get “Access Denied” until they were added back after a change.  We had some workarounds for this scenario involving turning off the user synchronization.  In 2010 we made a couple of changes to avoid this problem – firstly the change to using groups at the PWA site level, and secondly we now remove then add back each individual as opposed to removing everyone and then adding back everyone.  So getting an Access Denied in the same scenario in 2010 is very unlikely. At the Project site level however we do not use the group approach and manage the users on an individual basis.  In most scenarios this is not an issue as the number of resources assigned to a project, and hence added to a site, is generally low compared to the total number of users in the system.  However there could be some scenarios where customers wish to have many or all of their users accessing many or all of their project sites.  This could either be achieved by adding many users to a project – or by giving the “View Project Site” permission at the team member level in a category that included many or all projects.  Either way this would then add very many individual users with permissions to the project sites.  And why is this a problem?  If the numbers of users is large then it is possible for the recommended software boundaries and limits of SharePoint Server to be exceeded – and this can lead to performance issues.  Each user added individually to a site would be considered a security scope – and the recommended maximum number of unique security scopes per list is 1,000 (SharePoint Server 2010 capacity management – Software boundaries and limits - https://technet.microsoft.com/en-us/library/cc262787.aspx ).  So each list and library in the site would be inheriting from the parent site permissions – and would exceed this limit if more than 1,000 user had access to the site (as they are individually added). In our experience the performance issues would then relate to any change in the site membership caused by changes in the categories or groups – or following such actions as adding a user or inactivating a user.  For example this last action of inactivating a user will actually remove that user from all sites they have access to – and the reason the limit is imposed is that when it is exceeded the process of removing a user can become very slow – particularly if this same user is also being removed from very many sites each of which is also way over the limit.  In extreme cases with multiple user inactivations it is possible that the server will become unresponsive and unable to authenticate users. I will include some of the error messages you might see, and the corresponding ULS entries at the end of this posting so that this aids finding this potential cause. If you are following along (and I’m sure some of my readers are way ahead of me…) you will realize there is a Catch-22 here.  Your server could become unresponsive whenever you need to manage users because you have too many users with permissions on the sites.  So remove some users… which will then make the server unresponsive...  How to escape from this loop?  There are some quick ways to get this resolved – but before rushing in to that it is better to review what it is you are really trying to achieve.  If the desire is that most people can access most projects then managing the permissions outside of Project Server using groups and inheritance from PWA is the way to go.  If however the fact that many users had access to many sites was really a mistake then you need to correct that issue – and either remove the “View Project Sites” from the offending category or reduce the number of resources assigned to the plans – but first of course you need to stop the synchronization of users to the sites otherwise any action may make your server very slow.  As mentioned in the previous post this can be achieved by us of the UserSyncSettings method – and just repeating that here to save you having to open that post up: The setting can be changed using the PSI and the Admin Web Service and the UserSyncSettings method. The enumeration of values that can be set are detailed at https://msdn.microsoft.com/en-us/library/websvcadmin.usersyncsettings_di_pj14mref.aspx , and the method described at https://msdn.microsoft.com/en-us/library/gg229480.aspx . Turning off the Project Site sync is achieved by the enumeration DisablePWS.   Member name Description Enabled Value=1. Enable all synchronizations. DisablePWA Value=2. Disable synchronization with Project Web App. DisablePWS Value=4. Disable synchronization with project sites for the user. DisableEmailSync Value=8. Disable email synchronization. DisableAll Value=16. Disable all synchronizations. This relates to settings in the MSP_WEB_ADMIN table of the Published database in the WADMIN_USER_SYNC_SETTING column. So for example a query such as: Update [ProjectServer_Published].[dbo].[msp_web_admin] set [WADMIN_USER_SYNC_SETTING] =4 would do the same as using the method to set the enumeration: int syncSettings = (int)SvcAdmin.UserSyncSettings.DisablePWS; We’d certainly prefer you to not touch the DB correctly – but I’m guessing that many of you would find it much easier to execute a SQL query to update the value than to write the code necessary to do the same (I certainly would!). Once that is turned off then you can safely do user management without causing further performance problems – but of course it would still be possible to trigger the same issues if you tried removing the users directly from the project site, using the out of the box SharePoint functionality. One way to remove the users very quickly without triggering the individual deletion that causes the problem is to inherit permissions from the parent.  This can be done via the UI on the Site Actions, Site Permissions page of the individual sites: This will lose any custom permissions.  If your end goal is to give most users access to most sites then this may be how you want to keep things long term – so before taking this action you would probably want to be sure that the PWA site has the right permissions for all the users who need access – as that will be where this site will start inheriting from once you click the button.  Obviously if you have thousands of sites (and you probably have if this is causing you problems) then PowerShell can automate the change for you. Run the following PowerShell command in the SharePoint 2010 Management Shell $site = Get-SPSite “ ” Foreach ($web in $site.AllWebs)
        $web.Update()         $web.ResetRoleInheritance()         $web.Update()         $site.Dispose() This (or the UI method) would also need to be run for any new sites created to avoid the problem coming back – but if you had ‘corrected’ your categories and team memberships then all would be ok going forward. If you do decide that leaving everything inheriting is right for most projects then you may also want to have certain projects that are more ‘secret’ and for these you will need to continue to manage the permissions and user on an individual level.  One thought I had was to set a property on these ‘special’ sites via PowerShell and then you could use this property to filter out in a modified version of the above PowerShell command and ensure you didn’t reset the role inheritance accidentally.  I should also point out that if you use the Synchronize option on the Project Sites page then this would re-break the inheritance – so should be avoided. As a guide we have seen the issue with a customer with around 3000 users where they are nearly all added to each of their 1500 sites.  And as promised, here are the error messages and ULS entries.  Different users may see different symptoms – but the user who initiates the issue, perhaps by inactivating a couple of resources, will see the ‘Save’ button on the page apparently stick on the ‘clicked’ position and eventually get a “An unexpected error has occurred.” message.  The correlation ID will be found in the ULS logs and will have several rows all relating to a SQL deadlock and the Critical level one will look like: 08/10/2011 12:17:02.85    w3wp.exe (0x2178)    0x314C    SharePoint Foundation    Database    5586    Critical    Unknown SQL Exception 1205 occurred. Additional error information from SQL Server is included below.  Transaction (Process ID 80) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.    886d9cdd-5c0c-4f3a-8f89-f4e8c92acde3 Another High level one that gives more information on the query causing the issue will be something like: 08/10/2011 12:17:06.97    w3wp.exe (0x2178)    0x314C    SharePoint Foundation    Database    tzkv    High    SqlCommand: 'SET NOCOUNT ON; DECLARE @DN nvarchar(256),@LN nvarchar(128),@@DocUIVersion int,@@S uniqueidentifier,@@Level tinyint; DECLARE @ItemId int; DECLARE @@iRet int; DECLARE @ExtraItemSize int; SET @@Level = 1; SET @@S=@wssp0;  EXEC @@iRet = proc_SecRemoveUserFromSite @@S, @wssp1, @wssp2  SELECT @ItemId = @wssp3  IF @@iRet 0 BEGIN  GOTO DONE; END  ;BEGIN TRAN IF NOT EXISTS( SELECT tp_ID FROM UserData WHERE tp_ListId = '06C8C9BB-B10B-4042-8859-9F9985E73E76' AND tp_ID = @ItemId  AND tp_Level = 1 AND tp_RowOrdinal =0) BEGIN  SELECT @ExtraItemSize = 0  EXEC @@iRet = proc_AddListItem @SiteId…. I have shortened it considerably – but the key piece is the proc_SecRemoveUserFromSite.  Finally the ‘Unexpected’ one: 08/10/2011 12:17:06.97    w3wp.exe (0x2178)    0x314C    SharePoint Foundation    Runtime    tkau    Unexpected    System.Runtime.InteropServices.COMException: Exception from HRESULT: 0x80131904    at Microsoft.SharePoint.Library.SPRequestInternalClass.UpdateMembers(String bstrUrl, UInt32 dwObjectType, String bstrObjId, Guid& pguidScopeId, Int32 lGroupID, Int32 lGroupOwnerId, Object& pvarArrayAdd, Object& pvarArrayAddIds, Object& pvarArrayLoginsRemove, Object& pvarArrayIdsRemove, Boolean bRemoveFromCurrentScopeOnly, Boolean bSendEmail)     at Microsoft.SharePoint.Library.SPRequest.UpdateMembers(String bstrUrl, UInt32 dwObjectType, String bstrObjId, Guid& pguidScopeId, Int32 lGroupID, Int32 lGroupOwnerId, Object& pvarArrayAdd, Object& pvarArrayAddIds, Object& pvarArrayLoginsRemove, Object& pvarArrayIdsRemove, Boolean bRemoveFromCurrentScopeOnly, Boolean bSendEmail)    886d9cdd-5c0c-4f3a-8f89-f4e8c92acde3 Once the sever is in the condition – which could last 15-30 minutes, then other users will get timeouts on their pages and the ULS logs may show the following: 08/10/2011 12:20:22.30    w3wp.exe (0x1228)    0x1454    SharePoint Foundation    Monitoring    b4ly    High    Leaving Monitored Scope (ExecuteStoredProcedureDataReader -- MSP_AUTH_GETUSERBYNAME). Execution Time=120002.728838442    2be0491a-a64b-4237-8cfc-40342a374d49 08/10/2011 12:20:22.30    w3wp.exe (0x1228)    0x1454    Project Server    General    8ym5    Monitorable    PWA:https:// /PWA, ServiceApp:Project Web App Service Application, User:, PSI: SqlException occurred in DAL:  0 0 -2     System.Data.SqlClient.SqlError: Timeout expired.  The timeout period elapsed prior to completion of the operation or the server is not responding.         at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection)     at … I should also point out that use of the Project Site Provisioning Settings page option to not automatically synchronize users may avoid you getting in to this situation – but you still need some process to control access – and if most sites are unrestricted then the inheritance option from PWA may be worth a try.  Just as a reminder – the option on the Project Site Provisioning Settings page looks like this: and un-checking will stop the automatic addition of Project Server users to sites (but will not remove ones who are already there). Hopefully the workarounds given will assist in avoiding these types of issues if you really need to have very large numbers of users accessing each of a large number of project sites. As promised – once we have a TechNet article out in the wild I will link to it.

See the article here:
Project Server 2010: How to best manage large numbers of resources


Leave a comment!

You must be logged in to post a comment.