## Tuesday, October 13, 2009

### Forcing the SharePoint object model to reload the 12 hive

Or: A bit of insight into the inner workings of the SharePoint object model

This isn't necessarily something many have found themselves battling, but I most certainly did yesterday. As I was trying to deploy a feature containing a site definition, and subsequently create a site using the site definition, using my new SPDeploy application, it all abruptly came to a nasty halt as I added the site (using SPSiteCollection.Add()):
File or arguments not valid for site template 'SomeTemplate#0'

Having triple-checked the template, feature, solution and just about everything else (yes, SPDeploy also does the equivalent of stsadm -o execadmsvcjobs), it eventually dawned on me how SharePoint goes about things when you create a new site.

Just about any call in SharePoint will go through whatever object model class you're dealing with, such as SPSiteCollection, then to a SPRequestManager class, on to an SPRequest, SPRequestInternalClass and eventually into an arcane COM class (which appears to be part of the old SharePoint 1.0 structure, when COM, ISAPI and such was the shizznit). This COM class exposes hundreds of functions, ranging from list administration to tepmlate lookup.

The issue in my case was two-fold. First of all, any thread in an application which uses the SharePoint object model, will access a set of SPRequest objects (based on whether you need an authenticated or non-authenticated instance) from the SPRequestManager class. These are per-thread request objects, which execute on a first-come, first-served basis. In figuring this out, I began experimenting with releasing the SPRequests for the main thread in the SPDeploy tool - which would cause them to be reinitialized in a subsequent call. I verified that this would not hurt other stateful objects, such as already lookup'd SPWebApplication instances; as all functions in these classes will acquire their SPRequest instance from SPRequestManager. And as I said: the SPRequestManager would create a new instance if an old one wasn't found.

After typing up the reflection madness to clear the current thread's SPRequest objects, I gave the application another go. But still got the error - doh.

This confirmed my worst fears: the SPRequestInternalClass COM object, which is obviously native code, did its own caching. Pulling out Sysinternals' ProcMon and watching loads from the filesystem beginning with "web server extensions\12" backed this up further: web templates, and the rest of the hive, is only loaded once - on demand - then cached.

In realizing this, the solution became pretty clear, and given the nature of SPRequestManager; not too shabby at all. For tools which require adding stuff to the 12 hive (such as deploying features with web templates), and then *use* those templates without re-executing the tool: you have to recycle the SPRequests in SPRequestManager, and recycle the OWSSVR.dll COM library.

This *also* applies for PowerShell scenarios, where you during one session call into one of the SharePoint objects, such as e.g.:
([type]"Microsoft.SharePoint.Administration.SPWebApplication")::Lookup($uri); Doing so will load the OWSSVR.dll into your current session, and if you happen to e.g. trigger a function from the object model which scans the 12 hive for tepmlates; any subsequent script based addition to this folder, and then attempted usage of the additions, will fail with the error noted above. The workaround I apply in SPDeploy, which may look intrusive, but is completely harmless in the single-tool scenario, is: [DllImport("kernel32.dll")]private static extern IntPtr FreeLibrary(IntPtr library);[DllImport("kernel32.dll")]private static extern IntPtr GetModuleHandle(string lpModuleName);[DllImport("kernel32.dll")]private static extern IntPtr LoadLibrary(string lpFileName);public static void RecycleOwssvr(){ PropertyInfo threadCtx = typeof (SPFarm).GetProperty("ThreadContext", BindingFlags.Static | BindingFlags.NonPublic); var tbl = (Hashtable) threadCtx.GetValue(null, null); PropertyInfo requestProp = typeof (SPFarm).GetProperty("RequestNoAuth", BindingFlags.Static | BindingFlags.Public | BindingFlags.NonPublic); requestProp.GetValue(SPFarm.Local, null); tbl.Remove(typeof (SPFarm).Assembly.GetType("Microsoft.SharePoint.SPRequestManager")); IntPtr p = GetModuleHandle("OWSSVR.DLL"); FreeLibrary(p); string stsadmPath = SPUtility.GetGenericSetupPath("ISAPI"); p = LoadLibrary(stsadmPath + @"\OWSSVR.DLL");} And that will work beautifully. One note, though; the timer service has to be running for this recycling to work, otherwise some other internal component will freak out come second-go. ## Sunday, October 11, 2009 ### SPDeploy update CodePlex link: http://spdeploy.codeplex.com I just uploaded a new binary and source package to the CodePlex site, with some structural, config and visual improvements. First of all, web application extensions are now supported, meaning that web applications you either lookup or create as part of the configuration can be extended to intranet / extranet / custom zones. A custom rolemanager, membershipprovider and ssl usage can be specified through the attributes. See the xsd schema in the Schemas folder for more info on which attribues are available for the WebApplicationExtension node. Further, the logging now looks a lot better, thanks to clearer language and some more elaborate code behind the output formatting. I've also refurbished a lot of the general processing code, to make it easier to extend for possible project helpers. I've got more changes coming later, but with a pretty limited amount of time on my hands, I can't be specific as for when. There are still some hidden general behavior settings being tossed about in the background, which aren't obvious unless you turn to the code. I'll document and clear those up in a future (probably the next) release. ## Friday, October 9, 2009 ### Introducing SPDeploy - a quick way to (re)deploy dev and test structure to SharePoint CodePlex link: http://spdeploy.codeplex.com About SPDeploy will, based on schema validated xml configuration, lookup or create content in a local SharePoint farm, such as WebApplications (with content database and iis site / application pool), Sites, Webs. It will also handle wsp solution installations to the farm; further deployments of installed wsps into web applications; activation of features on all feature-accepting levels of the structure. Folder, list item and document support is planned, but not as a replacement for WSPs, features or plain common sense. With SPDeploy you can quickly, and painlessly, reset and redeploy your test or development environment, as well as quickly add development / third party features to the mix - without incorporating them into your production ready wsps, or extending large powershell / batch scripts. The point is to encourage proper integration testing in your development environment, and bringing structure to your favorite SharePoint build server. Important! Deployment or upgrades into a production or even UAT environments should be made with as strict a setup as possible, with (if possible) all the structure definition, creation and library deployment packaged into definitions, features and solutions. I do not suggest using SPDeploy for this. Test with SPDeploy - ship with wsps and upgradable features! Developer / extension notes All data classes in the project are generated from XSD, and generic configration runners make it easy to expand with custom sub nodes. Add an accepted sub node to the XSD, rebuild the project to generate new data classes, and create a Processor derived class to deal with the new sub node type. A new sub node processor will automatically get the result from the lookup or creation of the node above (like a SPWeb, if you're adding processors under the XML's Web node). Sample xml configuration data If you reference the schema provided in the source package, Visual Studio will even give you autocompletion and validation of the configuration as you go. <?xml version="1.0" encoding="utf-8" ?><Deployment xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="Schemas\SPDeploy.xsd"> <Solutions> <Solution FileName="C:\path\to\a\Solution.wsp"/> </Solutions> <WebApplications> <WebApplication Action="Lookup" Name="SharePoint - 80"> <SolutionDeployments> <SolutionDeployment Name="Solution.wsp"/> </SolutionDeployments> <Sites> <Site Action="Create" Url="/" Title="RootWeb" LCID="1044" Template="STS#1" OwnerLogin="somedomain\user"> <RootWeb> <Features> <FeatureActivation Guid="fba9e8df-b758-4197-8768-169ef727d5bb" /> </Features> <Webs> <Web Action="Create" Title="RecursionWeb" LCID="1044" Template="STS#1" Url="RecursionIsFun"> <Webs> <Web Action="Create" Title="RecursionWeb" LCID="1044" Template="STS#1" Url="RecursionIsFun"> <Webs> <Web Action="Create" Title="RecursionWeb" LCID="1044" Template="STS#1" Url="RecursionIsFun"> <Webs> <Web Action="Create" Title="RecursionWeb" LCID="1044" Template="STS#1" Url="RecursionIsFun"> </Web> </Webs> </Web> </Webs> </Web> </Webs> </Web> </Webs> </RootWeb> </Site> </Sites> </WebApplication> <WebApplication Action="Create" Name="TestApp" Header="vm-dev" Port="8009" Database="TestContent" PoolName="TestPooly" CreatePool="true"> <SolutionDeployments> <SolutionDeployment Name="Solution.wsp"/> </SolutionDeployments> <Sites> <Site Action="Create" Url="/" Template="STS#1" LCID="1044" OwnerLogin="somedomain\user" Title="TestWeb"> <RootWeb> <Webs> <Web Action="Create" Title="SubWeb" LCID="1044" Template="STS#1" Url="SubWebUrl"> <Features> <FeatureActivation Guid="fba9e8df-b758-4197-8768-169ef727d5bb" /> </Features> </Web> </Webs> </RootWeb> </Site> </Sites> </WebApplication> </WebApplications></Deployment> ## Monday, October 5, 2009 ### Assigning dynamic ids to HTML and Script elements - Client Side! This question popped up on Twitter yesterday: How do you usually handle duplicate IDs for html elements and scripts in SharePoint Web Parts, scripts and controls? Which is a natural concern, with all the different web parts and scripts that's being injected SharePoint portals these days. To end Monday off (in my neck of the woods anyway), I thought I'd make a quick post on what I do in my projects. Script naming convention For scripts, I have a few guidelines. First of all, my scripts follow a naming convention based on my organization handle, library concern, project name and so forth. Looking back to my previous couple of posts, e.g. the Script Loader (http://www.codefornuts.com/2009/10/pieces-of-my-core-javascript-library.html), this name would be grep.scriptloader - as that's a core piece of script for me. For a script specific to a web part, I'd name it similar to the webpart, e.g. grep.webparts.navigationcontrol. As for the actual script definitions, two matters apply: is it a class, or a single instance? For single instances, such as the Script Loader above, I declare it like this: if (!window.grep) window.grep = {};if (!grep.someSingleInstanceClass){ grep.someSingleInstanceClass= { instanceVar: "baz", foo: function() {}, bar: function() {} }} So basically I check if it's already been assigned, and if it hasn't, I assign it to a global variable. The above code would then be accessed such as grep.someSingleInstanceClass.foo(); For multi instance classes, I follow regular javascript oo syntax, with the same has-it-been-defined-clause, such as: if (!window.grep) window.grep = {};if (!grep.someClass){ grep.someClass = function(ctorVariable) { this.someVar = ctorVariable; this.DoWhatever(); } grep.someClass.prototype.DoWhatever = function() { // ... } grep.someClass.prototype.AnotherFunction = function(input) { // ... }} Then access that such as var foo = new grep.someClass("some value for the ctor");foo.AnotherFunction(42); HTML element lookups, and dynamic ids For elements, I use another trick. Since giving the elements a specific id can easily crash with other web parts of the same kind, or even other web parts / controls, I tend to avoid using ids as much as possible, and rather rely on class lookups (where the class has been named similar to the web part). For such lookups to be possible, however, they must be based somewhere - you can't just search for all instances of 'css class foo' in all of the body tag. Imagine the following scenario: <div>This element will be added several times, by other instances of the same web part</div class="webPartInstanceFrame"><script type="text/javascript"> new grep.webparts.someFrameManager(id of the previous div);</script> The script convention noted further up would make sure that the class' code will only be added once, and we feed that the id of the div, so that the class can make css class based jQuery (or similar) lookups to find the elements it needs within the frame. But what's the id of the div? Assigning a static id to it would crash with other instances of the same web part, and using a class would be just as hopeless. That's where the trick applies, and the following is a fully functional example: <script type="text/javascript" src="http://ajax.googleapis.com/ajax/libs/jquery/1.3.2/jquery.min.js"></script><script type="text/javascript"> function renamePrevious(tagType, id) {$(document.body.lastChild)            .prev(tagType + ":first")            .attr("id", id);        return id;    }</script><div>I want a dynamic ID!</div><script type="text/javascript">    var dynamicId = renamePrevious("div", "SomePrefix" + new Date().valueOf());    /* renamePrevious should be passed as an argument in dynamicId's place, to avoid all variable assignments! */    document.getElementById(dynamicId).innerHTML += " - OK!";</script>

The trick works by looking up the script block's element in the html dom. This element is the last added element, as the script block executes, even if it's not at the bottom of the page. Based on this script element, it searches its previous siblings for the first match of a given tag type, then assigns a new id.

You could do a lot better with the actual dynamic id than the millisecond postfix shown here, such as an incrementing (or even random) number which is added to a string, then checking if that already exists (in which case we increment / re-random and try again). I leave that to your imagination, though.

As for the example further up, the frame could be identified and put into action such as:
<script type="text/javascript">    new grep.webparts.someFrameManager(        renamePrevious("div", "E" + new Date().valueOf()));</script>

And that would enable us to avoid any and all name crashes for our component.

## Friday, October 2, 2009

### Connecting SharePoint UI Elements: SPDrag and the DropBox

So the last few posts have focused on some of my core javascript libraries. Nothing of the fancy stuff, just basic paper and glue type things. To make it all come together, I'll bring back up something I mentioned on Twitter about a month ago, but haven't spoken much of since: Two libraries of mine called SPDrag and DropBox.

SPDrag

The first of the two essentially makes standard SharePoint ui elements, such as files, list items and so forth, draggable. With a library building on jQuery, jQuery UI and my own Lambda framwork, it allows a developer to connect various of the SharePoint elements to his own widgets. An example of such a widget would be the Hierarchical Navigation replacement I've mentioned time and again (jQuery ajax powered, lazy-loaded SharePoint quick launch replacement). Another example is the DropBox.

The Lambda Framework, described here is a key player in making this work.

DropBox

One of the basic features missing in SharePoint 2007 has for me been the possibility of picking things up, taking them to another location, and dropping them into place. Be that files, folders, tasks or whatnot - I've wanted to pull them about. It's no joke that file copy / move in default SharePoint is as fun as watching grass grow. Even with webdav, it's nowhere near as pleasant as it should have been.

Enter the DropBox. Pick up a file. Pick up an item. Pick whatever you want, and put it in the DropBox. Then poke around the portal at your own whimsical liking; pull the item from the DropBox and place it in a compatible container. File copy made smooth, task assignment made friendly.

So am I going to release this now? No, I'm afraid not. I still have some work to do on both libraries, and I've yet to decide in which shape and form the release will take on. I'm obviously thinking OSS, but that's futile unless someone actually wants to contribute ;-)

Anyhoo, here's the screencast of several of the said libraries acting together. Please watch, and supply me with ideas if the concept appeals to you!

### Pieces of my core javascript library: The Lambda Framework

Following up on my previous post, about the Script Loader piece of my core library, here's a couple more central pieces for the stuff I'm about to tell about in posts to follow.

On the spot in this post is the lambda library. Simply put, a lambda here means an expression with placeholders for parameters to come. As such, it's not unlike the closures we already have in javascript, but this does up the stakes a little bit.

Closures allow us to define inline functions, and bind expressions to said function's parameters as well as those of the outside scope, ex:
function doSomething(){    var someGreeting = "Hi";    var closure = function(name) { alert(someGreeting + " " + name); } closure("yourself"); // Should alert "Hi yourself"}

... which is all old news.

With the following lambda library, I could construct the following fulent expression tree:
 var verification = _.first().is(10); alert(verification(10)); // Should alert 'true'  // The same as a closure: // var verificationClosure = function(x) { return x == 10; }  // A more advanced verification (although hardly the best we can do): var advancedVerification = _.and(_.first().is(10), _.second().isnt(_.first())); alert(advancedVerification(10, 10)); // Should alert 'false' alert(advancedVerification(10, 20)); // Should alert 'true'

This may seem like utter gibberish, but in certain cases it can be useful in order to define a relatively clear set of rules for an operation to take place. Specifically I'm using this in a library which establishes relationships between abstracted gui elmements. What I'm able to do is say, with an expression tree, that:
Given squares A and B; B can be dragged into A if B isn't already a child of A, and it has a certain inherent property.

This is actual code from my DropBox SharePoint plugin (I'll showcase that later), which uses another library of mine, SPDrag (that essentially makes standard SharePoint UI elements drag'n'droppable).