Gert Lievens - Blog

“Physics is the universe’s operating system.”
A little update

As you probably have noticed, a long time has gone by before I could create a new blog post. It has been a very busy time lately. I now work for another company and have broadened my scope to SCCM 2007 R2, SCOM 2007 R2 and Image Deployment. So instead of writing blog posts, I need to read a lot of them to stay up to date myself.

I'm also setting up my own Exchange 2010 environment to understand the key concepts of that technology and to prepare for my Microsoft exam.

Besides those technologies, I'm planning to follow an ITIL V3 course.

All in all, I will not be able to maintain this blog at a regular basis, but I do hope being able to do so in the nearby future.

Access Denied page when adding users to SharePoint

Last week I was visiting a client to troubleshoot the SharePoint environment. A strange error they were experiencing was that when they were trying to add users in SharePoint, they received an Access Denied page. However, the user(s) were actually added succesfully. The same happened when removing users.
Users who had Full Control in Policy for Web Application did not experience this behavior, but that is not the way you want to handle security in SharePoint.

The only difference with the web application that had no issue, was that the Edit Personal User Information right checkbox was cleared in Central Administration => User permissions for Web Application.
Checking this box solved the problem.

 

FAST Search Server 2010 Wizard : Username or domain is incorrect

When implementing a new product, you're bound to stumble on new issues (or at least I thought it was an issue). I had installed FAST Search Server 2010 for SharePoint on a dedicated server. Everything went fine until I wanted to follow the configuration wizard for FAST Search Server. When I wanted to enter the user name and password in the dialog box, I received a "User name or domain is incorrect." error.

Let's say our domain was named domain.local. Using these formats gave us an error:
DOMAIN\fast_service
fast_service@domain.local

After I was searching for an hour what the problem could be, the customer himself apparently succeeded in getting through the configuration wizard by using this format:
domain.local\fast_service

In Microsoft products - including SharePoint 2010 - I never had to use this use name format. It are the little things...

SharePoint Farm Administrator gets access denied in Central Administration

I had a very annoying problem last week trying to add a user as Farm Administrator in SharePoint Server 2007. When accessing Central Administration, the user got an 401 Unauthorized page when pressing on the Operations or Application Management tabs.

In the Eventviewer on the server I saw the following error:

Event Type:       Failure Audit
Event Source:   Security
Event Category:               Logon/Logoff
Event ID:             529
User:                     NT AUTHORITY\SYSTEM
Computer:         SERVERNAME
Description:
Logon Failure:
                Reason:                               Unknown user name or bad password
                User Name:       mossca
                Domain:                              …
                Logon Type:       3
                Logon Process: NtLmSsp
                Authentication Package:             NTLM
                Workstation Name:       …

After a while, I came across something weird in the authorization section of the web.config file under C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\12\TEMPLATE\

    <authorization>
      <allow roles="BUILTIN\Administrators" />
      <allow roles="SERVERNAME\WSS_ADMIN_WPG" />
      <deny users="*" />
    </authorization>

As the user account wasn’t in WSS_ADMIN_WPG, I added him.

    <authorization>
      <allow roles="BUILTIN\Administrators" />
      <allow roles="SERVERNAME\WSS_ADMIN_WPG" />
      <allow roles="DOMAIN\mossca" />
      <deny users="*" />
    </authorization>

Now everything works! Keep in mind that you need to do this on every server of the farm...

SQL Server Reporting Services 2008 on Windows 7 / Windows Server 2008 R2

This week I installed SQL Server 2008 R2 November CTP on my Windows 7 workstation. Because I want to delve somewhat deeper in building reports for SQL Server, I also installed the SQL Server Reporting Services. The installation went incredibly smooth and the configuration through Reporting Services Configuration Manager didn't throw any errors.

When opening a browser and browsing to http://localhost/reportserver I received - (it has to happen sometime) - an error:
The permissions granted to user ... are insufficient for performing this operation. (rsAccessDenied)

I spent a lot of time searching and checking the Reporting Services configuration files for hints to what could be going wrong. After all, my account was a local administrator AND the service account under which Reporting Services was running.

Finally I found the solution! It was way too simple actually... Just open your Start menu and locate the Internet Explorer icon. Right click on the icon and choose "Run as administrator"... Problem solved! :)

SharePoint Designer Server error: There is no Web named " ".

When using SharePoint Designer to open a site directly from it's directory (like E:\site\ ) via the "Open Site" button, it is possible to receive the above error.

We experienced this issue with one directory, while all the others worked just fine. The strange part was that I was able to open the subdirectories of the failing directory. With this in mind, I made a new directory and began copying separate files to the new folder. After a few files, I tried to open it with SharePoint Designer and this was always succesfull. When I copied one of the last folders - which was a hidden folder - I got the error again! From that moment I knew what the cause of the problem was.

It was the hidden folder "_vti_pvt". I don't know what exactly is going wrong, but after deleting the folder everything went like it should. Watch out however! In my case the folder was empty, but this is not always the case. Do not delete the folder if files are located in it!

IIS crash Windows Update December - Windows Server 2003

SharePoint and IIS users in general, beware! One of the latest Windows Updates initiates a shutdown of all IIS 6.0 application pools. The following hotfix is the culprit:

http://support.microsoft.com/kb/973917

It is not possible to start the application pools again after the installation. You will probably get a lot of unspecified errors when you do:

"A process serving application pool ‘...’ terminated unexpectedly"
"Application pool ‘...’ is being automatically disabled due to a series of failures in the process(es) serving that application pool."

The quickest solution is to uninstall the hotfix. Microsoft has published a workaround for the problem, stating that Windows Server 2003 Service Pack 2 should be re-applied. Out of experience, I can say that it does indeed solve the problem :)

http://support.microsoft.com/Default.aspx?kbid=2009746

Move SMS 2003 database to new SQL Server

Recently, I was responsible for the migration of a SQL Server 2000 server to a new installation of SQL Server 2005. This installation resided on another server. Most applications were easy to adjust to the new SQL Server name. For SMS 2003, it should be easy to configure a new database server by using the setup wizard. Since I had no experience with SMS back-end before the migration, I was baffled to see that although the SMS saw the change, it could not connect to the database.

The initial error was "connection failed" but this seemed illogical to me because I had succesfully moved all security using the scripts provided by Microsoft. On the SQL Server I received the "Login failed for user 'NT AUTHORITY\NETWORK SERVICE'". Adding the SMS 2003 server object to the security on the SQL Server did not solve the problem.

After a lot of searching I found the following KB which explained that I should create the appropriate Service Principal Names (SPN) for the SQL Server service accounts:
http://support.microsoft.com/kb/829868/en-us

Hurray! Although the KB states another error - not associated with a trusted sql server connection - this also fixed my problem! So bottom line of the story: if you move SMS 2003 databases follow http://technet.microsoft.com/en-us/library/cc180494.aspx together with the KB mentioned above.

Update Center for Microsoft Office Products

This week I discovered the new Update Center for Office products like SharePoint:

http://technet.microsoft.com/en-us/office/ee748587.aspx

It seems a great site that especially system administrators should bookmark. All the most recent updates are listed there which makes it easy to track the latest versions in a central location. Microsoft already had a similar site like that (can't immediately remember the URL) but it didn't get updated. I hope they will do a better job maintaining this site.

SQL Server 2005 cluster failover failed

We had an annoying problem while performing Windows Updates on the Windows Server 2008 Failover cluster at a customer. While testing the failover, we saw that the SQL Server 2005 cluster resource failover did not succeed. The strange thing was that nothing in the configuration had changed since previous succesfull tests. Also, the other resources failed over without any issues.

We did encounter the following errors in the Windows eventviewer and in the cluster log file under C:\Windows\Cluster\cluster.log

[sqsrvres] ODBC sqldriverconnect failed

[sqsrvres] checkODBCConnectError: sqlstate = 28000; native error = 4818; message = [Microsoft][SQL Native Client][SQL Server]Login failed for user 'NT AUTHORITY\ANONYMOUS LOGON'.

Login failed for user 'NT AUTHORITY\ANONYMOUS LOGON'. [CLIENT: IP]184800000E0000000A00000043004C0044004200310050002D00490030000000070000006D00610073007400650072000000

[sqsrvres] OnlineThread: Error connecting to SQL Server.

The client was unable to reuse a session with SPID 110, which had been reset for connection pooling. The failure ID is 88460000140000000A00000043004C0044004200310050002D0049003000000000000000. This error may have been caused by an earlier operation failing. Check the error logs for failed operations immediately before this error message.

SQL Server cannot accept new connections, because it is shutting down. The connection has been closed. [CLIENT: IP]

[sqsrvres] CheckServiceAlive: Service is dead

[sqsrvres] OnlineThread: service stopped while waiting for QP.

I had begun troubleshooting the most basic elements in a SQL Server cluster and started with the Browser Service. The Browser Service seemed to have no problems running but after a lot of searching, I found one inconsistency on one of the nodes.
The Browser Service runs under the Network Service. On each node of the cluster is a local security group that is used to give the required permissions to run the SQL Server Browser Service:

SQLServer2005SQLBrowserUser$ComputerName

Logically, the Network Service should be in this group, so it has the privilege to run the Browser Service correctly. Well, on one node the Network Service account was not in the local security group!

After adding the Network Service account to the local group and restarting the SQL Server Browser Service on both nodes the failover was working again... It was a long search for something as simple as that. It is unexplainable why this setting has changed. (Or rather - which I find more plausible - who had changed it)

SSL wildcard certificate on multiple servers

In the recent past I have seen many people having problems with implementing a wildcard certificate in their enterprise. People usually request a certificate from a certificate authority. When they deploy the certificate on all the necessary servers, only the server from where the certificate request is generated, works.

What's the problem? Really simple actually. The certificate has been created with the private key of the server used in the certificate request proces. When deploying the certificate, all the other servers have another private key which does not match with the certificate.

The solution is equally simple. Just export the private key from the original server and import it on the other servers. This way, the private key on those servers is the same and match the certificate.

There is an excellent post at Digicert.com about this:
http://www.digicert.com/ssl-support/pfx-import-export-iis-7.htm

How to configure load-balanced SQL Server Reporting Services on a SQL Server cluster

As most of you will know, Reporting Services is not fully supported on a SQL Server cluster. More information can be found here:
http://www.microsoft.com/dynamics/crm/using/deploy/clusteringssrs.mspx
However, not supported does not mean not possible. I decided to experiment in my testing environment at home.There are some ways to achieve this goal. Be cautious however, do not try/do this in a production environment! In no means do I encourage people to implement this! It can be interesting in a testing environment.

Install two separate instances of SQL Server Reporting Services on each node of your SQL Server cluster. Do NOT use the same instance name! Configure the first SQL Server Reporting instance to create a database on the cluster (it is still possible to use another SQL Server). After that, configure the second SQL Server Reporting instance to use the database created by the first instance. Be sure to configure the RSReportServer.config file on every node. See that all values are correct. For example the ReportServerURL value needs to be set to the value you have configured in the Reporting Services Configuration Manager.

One big issue will mostly be: how do we load balance the requests to the Reporting Services URL?

There are some possibilities, each with it's own disadvantages:
1. Use a IP cluster resource
2. Use a dedicated hardware or software load balancer
3. Use DNS Round Robin (least fault-tolerant in case of failure)
4. ...

I have succesfully done (and tested) this using the first method. Of course, in this case requests will not be load balanced. The best choice is still using a dedicated load balancer or a separate reporting services farm. It is technically possible to install NLB on a cluster, but I would never recommend this.

I know, it has been ages since I wrote my first blog post. I hope to update my blog more often in the coming future!

w3wp high memory and cpu usage in SharePoint

This week we had a huge performance issue with SharePoint at a customer. The w3wp process took a huge amount of memory for itself (1,4 - 1,5 gigabyte) without any user activity. Another symptom was that while performing a full / incremental crawl, the server suffered from very high CPU usage. Since I already installed the Infrastructure update and latest hotfixes, it was really hard troubleshooting the problem.

Some things to check if you experience the same symptoms:

  1. Windows Server 2003 Service Pack 2 => http://support.microsoft.com/kb/916984

  2. Services 3.0 and MOSS 2007 Service Pack 1 => http://www.microsoft.com/Downloads/details.aspx?FamilyID=875da47e-89d5-4621-a319-a1f5bfedf497&displaylang=en and http://www.microsoft.com/Downloads/details.aspx?familyid=AD59175C-AD6A-4027-8C2F-DB25322F791B&displaylang=en

    Before SP1, there were some issues regarding the search crawl. For example, the memory usage doesn't increase as much as before while the crawl is running.

  3. Check your SharePoint permissions: in my case, I solved the issue with setting the permissions on the following key HKEY_LOCAL_MACHINE\Software\Microsoft\Shared Tools\web server extensions\12\WSS and HKEY_LOCAL_MACHINE\Software\Microsoft\Shared Tools\web server extensions\12\Search. I added the Everyone group and gave them "Read" rights.

Very strange, but the latter seemed to solve my problems! Web applications reached sizes of 650 MB till 900 MB max, which was a big difference with the 1,4 gigabyte we used to encounter. I will try to research which permissions for which group or user needs te be set. The Everyone group is too general. However, the problem is very difficult to reproduce :(

Hope this helps someone out,
Gert

 

Do not use the Shrink maintenance task

Just one recommandation: do NOT use this.

Lately I was at a customer who had scheduled the Shrink task every morning by using a maintenance plan. Since their SQL Server installation earlier that year they suffered from decreasing performance. Deleting this task and performing a rebuild of the index of all databases seemed to solve the issue.

I am amazed of how many people still use the (Auto-)Shrink option. Together with Auto-Close, this option is widely considered to be one of the worst features in SQL Server. I must say I understand why so many non-DBA's find these options interesting. Let's face it, who wouldn't like a self-sustaining database management system? Wouldn't it be great if SQL Server releases unused resources? Or if unused space is released? Well, it would be great, if only there were no penalties in using these features.

1. Auto-Close:
it does effectively close existing connections in case of inactivity. This can cause performance problems for a database that is used on a regular basis. This is because every time the last user exits the database, all open connections are closed. So every time a user tries to reconnect, the connection needs to be restarted. Even if this is a second later.

2. Auto-Shrink:
again, this option does it's job by releasing the unused space once this reaches a certain treshold. However, shrinking a database fragments the database files on disk causing performance loss.

The shrink task should be unnecessary when a good backup plan is in place because in that case the database log file will be truncated at regular intervals. It should only be used in a specific scenario.

Introduction

Hello there,

Before ranting about IT, a proper introduction seems fit.

My name is Gert Lievens and I'm here to share some of my experiences related to SharePoint and SQL Server ... stuff :)

Besides IT, I spend a lot of time:
- Watching movies
- Playing games
- Going out
- Listening to music
- Reading books/comics/...

At the moment I work for RealDolmen as a system engineer.

Enjoy!
Gert