This week was Microsoft's usual patch Tuesday. In the swag of updates was one for Sharepoint KB934525. Foolishly we installed that on both our internal and external Sharepoint site and broke both of them.
If we had read Wayne's blog entry or Vlad's then at least we would have been aware of the potential problem and chosen not to install the patch. Typically, we applied the patch before check our RSS reader and got burned.
After some initial too and frowing we tried Vlad's suggested fix but that didn't solve the issue. Seems that the reason for this was because we had installed the Sharepoint Groupboard that also caused problems.
Ok now things are getting serious with our two production Sharepoint sites down, time to call Microsoft Product Support. Now as a certified partner we were disappointed with the slow response to the issue. It took a whole day before we got anyone from Microsoft to get 'hands-on' with our problems.
In the meantime we attempted to do complete server restore of our internal Sharepoint site (since we have backups) and that failed (twice). The server restored but we got all sorts fo wierd errors. So not wanting to waste more time on that we built a new server and did a Sharepoint data restore (since we backup Sharepoint data as well seperately). A day later we had our internal Sharepoint working again just a Microsoft started working on our external Sharepoint site.
While Microsoft continued to work on the server in background we started the process of building yet another server in case we needed to do another Sharepoint restore for our extenal site. Watching the process that the Microsoft Tech had to go through there was NO WAY we would have ever been able to recover from the problems ourselves.
Anyway, after a full two days we finally had both Sharepoint sites working. After an initial frustration at waiting for service we have nothing but praise for our Microsoft Tech who stayed on the job and keep us up to date and then hung around after we left for the day to make sure it was all working after hours.
There are still a few bugs to iron out in the recovered site but as you can see (if you are reading this) then the site is operational. We'll post again shortly with some lesson learned about this experience but all we can say about KB934525 is DON'T DO IT unless you know that it may TOTALLY BREAK Sharepoint V3.0!