VSIX with the VS2010 Beta 2 SDK

I'm working on an extension for VS2010 that I'll definitely be talking more about in the future, but I wanted to start by discussing the difficulties involved in even getting started.

I'd heard a lot about how VS2010 was much easier to extend than previous versions, particularly due to the use of MEF for writing extensions. What I didn't take into account before coming up with my idea is that MEF is only used for extensions to the editor (which my idea doesn't involve). That's how I found myself writing a regular extension.

Optimizing the Editor extension experience seems to have influenced the available project templates as well. The Beta 2 SDK comes with project templates for Editor Viewport Adornment, Editor Margin, Editor Classifier, Editor this and that, WPF and Winforms controls, and then a default VSIX Project. You'd think, then, that the one most applicable to my extension would be the VSIX Project, but that thought was where the difficulty began.

I'm going to suggest up front that the best way to write a non-Editor extension is to find a sample that most closely matches what you want to do on the MSDN Code Gallery for VSX and just modify it instead of starting with the blank project like I did. The main issues seem to stem from the SDK providing UI to edit many of the involved files, but in a totally incomplete way leaving many of the necessary settings only discoverable via editing the files in the XML editor or something.

The first thing you'll find when comparing the VSIX Project to any of the samples is that the VSIX Project has a special project type that gets you another tab in project properties. While this tab does have a few useful options, it doesn't expose the settings necessary for making sure a PkgDef file gets generated and that it contains your assembly dll and pdb and that they all get deployed to the right place for the Experimental Instance of VS to find them for debugging. The way I got all this to work was to manually edit my csproj file and change the following elements from "false" to "true".


<GeneratePkgDefFile>true</GeneratePkgDefFile>
<IncludeAssemblyInVSIXContainer>true</IncludeAssemblyInVSIXContainer>
<IncludeDebugSymbolsInVSIXContainer>true</IncludeDebugSymbolsInVSIXContainer>
<IncludeDebugSymbolsInLocalVSIXDeployment>true</IncludeDebugSymbolsInLocalVSIXDeployment>
<CopyBuildOutputToOutputDirectory>true</CopyBuildOutputToOutputDirectory>
<CopyOutputSymbolsToOutputDirectory>true</CopyOutputSymbolsToOutputDirectory>


In order to make sure your package actually gets registered with the Experimental Instance and that the dlls are referenced out of the local folder instead of the GAC you have to add this PropertyGroup section to your csproj as well


<PropertyGroup>
<RegisterOutputPackage>true</RegisterOutputPackage>
<RegisterWithCodebase>true</RegisterWithCodebase>
</PropertyGroup>


The next thing you'll notice is that the samples all have .vsct files in them that define all the Menus, Groups, Buttons, Bitmaps, etc. These files basically control how your extension is integrated with Visual Studio. Since your project doesn't have this, you'll have to add it. Unfortunately there isn't an item template for this, so the closest match is just an XML file. Of course it can't be that easy. There's actually a compilation step that happens against this file, too (it isn't just data). No problem, let's just change the Build Action to the same thing the sample has. Wait, why isn't it in the dropdown? Time to open the csproj file manually again and look for your newly added vsct project. In the ItemGroup, instead of a Content or None element, you want a VSCTCompile element that looks like this:


<VSCTCompile Include="Filename.vsct">
<ResourceName>1000</ResourceName>
</VSCTCompile>


That magic number in the ResourceName element must match up with the value you put in the ProvideMenuResourceAttribute you put on your package.

Now the VSCT is set, the project is configured properly, and everything works, right? Not yet. If you compile now, you'll get a weird error in the VS SDK targets file like "No resource file set the resource merger". I guessed that this meant we needed a resource file so I added one. That didn't quite fix it so I had to go back into the csproj and change how the resource file was listed there. Underneath the EmbeddedResource element, add an element called "MergeWithCTO" with the text "true" and that problem should go away.

Ok now everything should run, the breakpoints set in the package will be hit and the real work can begin. See why I suggested starting with a sample and just changing it?

I went ahead and created connect items for these: VSCT, Incomplete Properties UI, and Resource files.

Back from PDC

The PDC was fun (who doesn't like free laptops), but not as energizing as previous years. Many of the announcements were just refinements of things previously announced.

The new Silverlight 4 stuff looks promising, as do the new features in Entity Framework 4.0. I'm also interested in trying to pry the IQueryable serialization stuff out of RIA so I can make use of it in a more typical distributed system situation. RIA is explicitly designed to try and hide the distribution tier (which I understand for the scenarios they are targeting), but the data services stuff is too cool to leave it limited to that.

I had several good talks with different Microsoft folks about EF, MVC, and F# among other things. Now that all the session videos are online almost immediately, these in-person interactions are the main draw of the conference (well, that and the laptop).

Having the conference end later in the day Thursday was good, since it made me stay until Friday and gave us a chance to head to Santa Monica Thursday night and dip my toes in the ocean. We arrived right as the sun was setting and immediately forgot about all the stupid traffic on the way there.

Stepping into methods with yield return

I had a failing unit test (Rhino Expected Call wasn't happening) so I ran with the debugger to try and investigate. When I got to the method in question I tried to step into it but the debugger acted like I had pressed Step Over. The output window had something like this in it: "Step into: Stepping over method without symbols".

After some investigation I figured out the issue. The method in question was returning an IEnumerable and was implemented using "yield return". My test had code like:

var result = callTheMethodInQuestion();
repository.VerifyAll();

Can you see the issue yet?

I was never enumerating result, so the method in question was never actually running, which is why my expected call didn't happen. Slapping a .ToList() onto result triggered the method evaluation and everything worked fine.

AddFromTemplate has misleading documentation

When building a custom solution template, it's sometimes useful to programmatically execute other templates. In our case, we have a unit test project template that we'd like to call from the domain template. To do this, you use a method off of ENVDTE90.Solution3 called AddFromTemplate, which takes among other things, the path to the template, the output directory, and the new project name.

The documentation for this method says this about the project name parameter:
"The name of the project file in the destination directory. This should include the extension."

Seems pretty clear about whether or not to include the extension, right?

The first version of our wizard included the extension. Unfortunately, this resulted in the $safeprojectname$ parameter ending up with the extension in it as well. Since this parameter is also used to define the default namespace, we ended up with test classes whose full names were something like MyApplication.MyProjectTests.csproj.MyClassTests. Yuck.

The solution appears to be to completely disregard the documentation and don't include the extension. So far this appears to be working.

How to access a Virtual PC from the Host

Most of the documentation I've seen for Virtual PC describes how to setup the virtual machine itself to be able to access the network. I have an application server installed on a virtual, though, that I'd really like to be able to access from the host machine.

This took some digging to find, but this article was exceptionally helpful. The only extra step I had to take was to disable Windows Firewall on the virtual machine so it would let the network traffic through.

SSRS 2005 in Vista with IIS 7

There are many articles online covering getting Sql Server Reporting Services 2005 working on Vista with IIS 7, but none that I found covered the particular problem I was having. Everything installed ok and the configuration manager was giving me all green checks, but accessing /Reports gave me an error about "unable to connect to the remote server" and accessing /ReportServer gave me a 403.1 Forbidden.

While poking around the IIS Management tool I noticed that in the Http Handlers section, virtually nothing was listed for the ReportServer web application. No ASP.NET, no nothing. I clicked "Revert to Inherited", causing it to pick up the settings of the parent site (that had ASP.NET enabled), and everything started working fine. Who knows how the handlers got overridden, but at least it's working now.

EDIT: Also, to fix some other nastiness, you have to follow the advice of "rmeans" here

Without the wildcard mapping you get a fun "For security reasons DTD is prohibited in this XML document" error.

EDIT2: And to actually call the web services via WCF you have to take the obvious step of switching the generated basicHttpBinding to pass the windows credentials, like this:

<binding name="ReportingService2005Soap" ... >
  <security mode="TransportCredentialOnly">
    <transport clientCredentialType="Windows" proxyCredentialType="None" realm="" />
      <message clientCredentialType="UserName" algorithmSuite="Default" />
  </security>
</binding>

but also the totally not obvious step of allowing the client credentials to impersonate, even if you're using a shared data source with a defined id and all that. You add a behavior to the endpoint for the SSRS web service like this:

<endpointBehaviors>
  <behavior name="allowImpersonate">
    <clientCredentials>
      <windows allowedImpersonationLevel="Impersonation"/>
    </clientCredentials>
  </behavior>
</endpointBehaviors>

Writing a Compiler with F# Part 4

Now that we've defined the AST for our language and written a simple evaluator, let's make it easier to construct ASTs. Obviously we don't create C# programs by directly constructing the syntax trees; instead, we type in text which is lexed and then parsed into the tree.

It turns out that creating lexers and parsers is such a common language task that there are many tools available to generate them for us from some basic definitions. One common lexer generator is called Lex and F# ships with a version of it called fslex. Likewise, fsyacc is the F# variant of the Yacc parser generators. I'm not going to talk about the different types of grammars as those are well documented elsewhere, suffice to say that fsyacc is capable of generating parsers that can handle languages you're probably familiar with.

Recall from last time that lexers take a stream of characters and produce tokens. The do this by matching a regular expression against the input to identify each character. If we assume we have some tokens defined for the elements of our language like PRINT, INT, SEMI (for a ; to separate statements in a statement list), PLUS, MINUS, LPAREN, and RPAREN, we can easily define the regular expressions that produce these tokens from the input.

Adding an .fsl file to the project allows us to define the lexer rules. When the project is built, fslex will run against our file and produce a generated F# file that contains our lexer. Here's what the definition looks like:

{
open System
open Parser
open Lexing
}
let num         = ['0'-'9']+
let integer     = '-'? num
let whitespace  = ' ' | '\t'
let newline     = '\n' | '\r' '\n'

rule token = parse
  | whitespace  { token lexbuf }
  | newline     { (lexbuf: lexbuf).EndPos <- lexbuf.EndPos.NextLine; token lexbuf }
  | "("         { LPAREN }
  | ")"         { RPAREN }
  | "+"         { PLUS }
  | "-"         { MINUS }
  | ";"         { SEMI }
  | integer     { INT (Int32.Parse(lexeme lexbuf)) }
  | eof         { EOF }
  | "print"     { PRINT }

The initial { } contains F# code that is directly inserted into our generated file. The tokens themselves (anything in all caps) are defined in the parser which we'll see in a minute so we have to "open Parser" to be able to see them. Next we set up some basic patterns including "num" for strings of digits.

Then, we define the token production rules. The left side (immediately following "|") defines the pattern we are matching, while the braces contain the action to execute (and token to return) should the pattern be matched. Note that "whitespace" results in simply calling our "token" function again with the remaining input, effectively ignoring the whitespace. Also, "newline" updates the current line number so should there be a problem the error can correctly report the line it was on. The only other point of interest is that the INT token carries along a piece of integer data (the actual number that was typed).

Armed with our lexer, we can now create an fsy file and build our parser. It looks like this:

%{
open Ast
%}

%start start

%token <int> INT
%token PLUS MINUS LPAREN RPAREN
%token PRINT SEMI EOF

%left PLUS MINUS

%type <stmt> start

%%

start: StmtList EOF        { Seq(List.rev $1) }

Expr: 
    | INT                 { Int $1 }
    | Expr PLUS Expr      { BinOp (Plus, $1, $3) }
    | Expr MINUS Expr     { BinOp (Minus, $1, $3) }
    | LPAREN Expr RPAREN  { $2 }
    

Stmt: 
    | PRINT Expr            { Print $2 }    

StmtList:
    | Stmt                { [$1] }
    | StmtList SEMI Stmt  { $3 :: $1 }

Like the lexer, the parser starts with a code block that simply opens the Ast module with the Ast types defined in it. "%start" indicates the name given to the generated parsing function, while "%type" later on indicates that the function will return a "stmt". Next we have a list of tokens (note that INT is defined to carry a piece of data of type int) and an indication that PLUS and MINUS are left associative. This also defines precedence so we can do things like correct multiplication and division later. Finally we have a list of patterns to match. Like the lexer, the left side matches the input (although now we're matching tokens rather than raw characters) and the right side (in the braces) indicates which type defined in Ast should be returned. The "$x" escapes refer to the tokens matched. For example, Expr PLUS Expr results in a BinOp where the left hand side is the value of the first token ($1), i.e., the first "Expr" and the right hand side is the value of the third token ($3), or the second "Expr". The PLUS token itself is "$2". The last trick is that we end up building the StmtList backwards, for efficiency sake (prepending an element to a list is O(1) while adding to the end is O(n)), so in the "start:" pattern we have to reverse $1 before returning the Seq.

Now we can use our parser with something like this:

let parseText text =
    let lexbuf = Lexing.from_string text
    Parser.start Lexer.token lexbuf

Calling "parseText" with a valid program ("print (1 + 1)") will result in an instance of "stmt" that we can pass to the evaluator we build in the last article.

Now we have a pretty stable base to work from. Next time we'll add variables and functions and then after that we'll add types other than "int" so we can start to worry about type checking.

The C# coalesce operator short circuits

Though it isn't mentioned in the help, the C# coalesce operator ( ?? ) does short circuit. If the left hand side isn't null, then the right hand side isn't evaluated.

F# Steals Show Smart Tag Keyboard Shortcut

Over the last few days I've noticed that pressing Ctrl-. no longer expanded the Smart Tags in the text editor (used to Refactor Rename or implement interfaces). It turns out FSI steals this combination for Canceling Evaluation.

To rebind it, find View.ShowSmartTag in the keyboard options and set it back to Ctrl-.

Edit: It turns out this is mentioned in the release notes that I neglected to read. Oops.

Writing a Compiler with F# Part 3

Now that we can read user input and pipe it into our evaluator, we need to start implementing the main parts of the compiler.  First it's helpful to understand what compilers actually do.  In a rough sense, they take the text files containing the code and output an exe or a dll or something, i.e., they transform human readable text into machine code.  This is a simplified definition, but it will suffice for our purposes.

It may be tempting to go directly from text into code, but for anything beyond the simplest of languages this ends up being very difficult both to write and maintain.  Most compilers divide up the work into several stages that form a pipeline of sorts.  Some common stages are:

Lexing

The raw sequence of characters is transformed into a sequence of Tokens. For example, the Lexer would consume 'w' 'h' 'i' 'l' 'e' from the input file and return the WHILE token

Parsing

The tokens from the Lexer are consumed to produce an Abstract Syntax Tree. An AST is a convenient in-memory representation of the program. More on these in a moment. For example, the Parser might consume VAR IDENTIFIER('i') COLON TYPENAME('int') and produce an instance of our VariableDeclaration class with the Name property set to 'i' and the type set to 'int'

Type Checking

Once we have the AST, we can make several passes over it to check the semantics of the language, including that programs are well typed. For example, when looking at an IfStatement we'd want to make sure that the Expression for the if clause was of type bool.

Other backend phases

We'll talk more about backend phases like instruction selection, register allocation, and optimizations if we ever get to building the backend. Ultimately these phases involve walking over the AST and finally spitting out machine code (assembly language, IL for .NET, etc.)

What you should see here is that the actual textual representation of the language plays a very small part in this whole process. A large portion of the compiler depends solely on the AST. Only the Lexer and Parser really care whether you end statements with semicolons or have syntactically significant whitespace, for example. Because of this, it's useful to start thinking about the AST first, and then worry about Lexing and Parsing.

In F#, the most common way to model ASTs is through the use of discriminated unions. These are roughly the functional programming equivalent of a polymorphic class hierarchy, although much simplified. Let's look at an AST for a very simple language that lets you print arithmetic expressions (provided you only want to use parentheses and addition/subtraction).

type binop =
    | Plus
    | Minus

type expr =
    | Int   of int
    | BinOp of binop * expr * expr
    
type stmt =
    | Print of expr
    | Seq of stmt list

The keyword 'type' introduces the declaration of a discriminated union. Each option is followed by a |. binop, for example, resembles an Enum that can either be Plus or Minus. The main difference between these unions and enums becomes apparent when you look at 'expr': each of the options can carry data. We can have an expr Int that contains an integer value, or a BinOp that contains a binop (remember Plus or Minus) and two other expressions. We would represent "print 5 + 3" in our AST as Print(BinOp(Plus, Int(5), Int(3)))

Discriminated unions really become powerful when combined with pattern matching. Given the AST above we can easily build a simple evaluator. Here's all we need:

let rec evalE (env,fenv) = function
    | Int i -> i
    | BinOp(Plus, l, r) -> (evalE (env,fenv) l) + (evalE (env,fenv) r)
    | BinOp(Minus, l, r) -> (evalE (env,fenv) l) - (evalE (env,fenv) r)
    
and eval (env,fenv) = function   
    | Seq stmts -> List.fold_left eval (env,fenv) stmts
    | Print e -> printfn "%d" (evalE (env,fenv) e); (env,fenv)    

Ok what's going on here? We've defined two mutually recursive functions, 'eval' for evaluating statements, and 'evalE' for evaluating expressions. We can safely ignore (env,fenv) for now; those will be useful when we add variables and functions. The body of 'eval' says: If I'm handed a Seq of statements, fold the eval function over each statement. Since we don't have variables yet this is could easily just be 'for each statement, evaluate it'. It also says: If I'm handed a Print of some expression, evaluate the expression and then print it out. Evaluating expressions works similarly. If I'm handed a BinOp(Plus) of two expressions, evaluate each one and then add them together.

In these relatively few lines of code, we've built a simple arithmetic evaluator. If you type this stuff into fsi, you can call the evaluator by doing something like:

eval (0,0) (Print(BinOp(Plus,Int(5),Int(6))))

Which will evaluate 5 + 6 and print out 11. Next time we see how to build a simple Lexer and Parser for the language so we don't have to type the AST out manually any more.

Using a custom base class for a generated ObjectContext

We're adapting our framework to support the Entity Framework in addition to its current support for Linq to SQL and running into issues. The Linq to SQL approach involved, among other things, a Base Data Context class that we could easily instruct the built in code generator to use. The LINQ to SQL designer has first class support for specifying that we want the generated data context to inherit from our BaseDataContext instead of the built-in DataContext directly.

Not only does the Entity Framework designer not have first class support for this, it appears that the Entity Framework tool chain itself has no support for this at all. I hope I'm wrong about this, but thus far I've been completely unable to find a way to specify that I want my generated ObjectContext to inherit from my BaseObjectContext instead of just ObjectContext.

Not impressed.

Writing a Compiler with F# Part 2

Since the ideas I want to mess with are more related to type systems than, say, instruction selection or optimizing passes, I'm going to completely neglect the backend of the compiler for awhile. Initially I'm not going to compile the language at all. Instead I'll just build a simple interpreter to evaluate the type-checked parse tree.

The easiest way to play with this is to make a basic REPL (Read Evaluate Print Loop). Basically a prompt that you can type language snippets into and have them run and the results printed. For F#, the REPL is called fsi.exe. Other languages have this feature as well (in Ruby it's irb.exe).

To get started, I'm going to forget about all the lexing and parsing stuff and get my feet wet with F# by building the framework for a simple REPL.

First, I'll need to print a prompt and get some input. Here's an F# function to handle that:

let getInput (prompt:string) =
 (Console.Write(prompt); Console.ReadLine())

Pretty basic stuff. I had to specify the type for "prompt" since Console.Write is overloaded.

Next, I want a basic loop to get input and evaluate it. Functional programming and F# in particular work naturally with sequences as a first class element. To that end, I'm going to model the user input as a sequence of lines. The F# Seq type has a generate method for this purpose. It takes 3 functions: an initializer that returns any data you might need, a generator that given the initial data produces either another element or None to signify the sequence has ended, and a cleanup function. Initially, I don't need any supporting data so both the initialize and cleanup functions can do nothing. Here's what it looks like:

let input = Seq.generate
 (fun () -> ()) // init
 (fun () ->
   match getInput(">") with
   | "#quit" -> None
   | inp -> Some(inp))
 (fun () -> ()) // cleanup

The middle function uses getInput that was defined above and asks the user for input. If they type "#quit" then the sequence ends, otherwise the input is returned.

Now we'll take this sequence and do something with it. Ultimately we'll want to evaluate the input and then keep prompting. Since the statement run might result in new variables being declared or existing ones modified, we'll need some way to remember what happened. The state of our world is referred to as an environment, so we'll write a dummy eval function that takes an environment, evaluates input, and returns the new environment to pass along to the next statement. For now let's use this:

let eval (env:int) (x:string) =
 (Console.WriteLine("Evaluating: {0}", x);env)

We ignore the environment, just print out the input, and return the environment unchanged.

The last step is to apply this eval function to our sequence of user input, passing the modified environment forward to the next element. This is a classic case of a fold. We start with an initial environment, call eval on the first item in the sequence, and then pass the resulting environment in when calling eval on the second item. Then the environment returned by eval'ing the second item gets passed into the third, etc. Here's what this looks like:

Seq.fold eval 0 input |> ignore

That's it. We don't care about the final value of the environment so we just pipe it to ignore. With only a few lines of F# we have a basic REPL going and we're ready to start with the real language work next time.

Writing a Compiler with F#

I've been dabbling with F# off and on since it was first publicly available, but I've never put enough time into it for it to feel natural to me.  The language itself has also evolved considerably since the early days.

To take my knowledge of the F# from its current state (simple snippets typed into FSI used to befuddle and amaze people who have never seen a functional language with strong type inference before) into a solid working understanding I'm going to write a compiler for a new programming language.  Languages like F# excel at solving problems like this, so this should give me a good exposure to what it's really like.

I also have some ideas regarding programming languages that I want to try out, but I'll talk more about those as they arise.  I'm going to start pretty simply for the target language and as time goes by I'll hopefully transition from worrying about the low level stuff like F# syntax into the really interesting stuff like the type system for my language and how it interoperates with the CLR.

My intent is to make a series of posts highlighting the process I'm going through including interesting things about F#, compilers and compiling, and type systems.  I'm not getting any Big Ideas about the resulting language - my formal PL background is limited to a solid CS education, reading Pierce's Types and Programming Languages, and lurking Lambda The Ultimate.  That is to say, I'm expecting to do things...less than optimally.

Now since I've posted this introduction I actually have to work on this thing.  Next time I'll talk about some basics of F# and also compilers.

Type.IsAssignableFrom seems backwards

Every time I've ever had to use Type.IsAssignableFrom it has always felt backwards. I usually use this method when checking if a given type implements a specific interface or derives from a certain base class.

This has come up recently when writing Windsor Interceptors and also methods that use reflection to walk my assemblies and automatically register components with my container. In these scenarios I end up with the concrete type in a variable of type Type, while the interface or base class I'm checking for is statically known. In the midst of my "where" clauses like "where type.IsGenericType && type.IsPublic" it's tempting to write "type.IsAssignableFrom(typeof(IMyInterface))", but this is, of course, incorrect. You have to reverse the order to get "typeof(IMyInterface).IsAssignableFrom(type)". There's a small amount of cognitive dissonance for me every time I have to do it.

What I'm really thinking is "make sure 'type' is an 'IMyInterface'. Why not make it so the API reflects that? Here's a little extension method that will let you write "where type.IsA<IMyInterface>()" instead. This reads a lot more naturally to me.

public static class TypeExtensions
{
  public static bool IsA<T>(this Type type)
  {
    return typeof(T).IsAssignableFrom(type);
  }
}

Extension Methods and CodeDom

A developer on my current project is creating a code generator and wondered how to emit extension methods. It turns out this is really easy.

All you have to do is emit a System.Runtime.CompilerServices.ExtensionAttribute on your static method and the class that contains it. Then just construct the method like a regular static method where the first parameter is the type you want to add the extension to. For example, to emit something like:

public static class StringExtensions {
public static string SayHi(this string instance)
{ ... }
}

you can emit:

[Extension]
public static class StringExtensions {
[Extension]
public static string SayHi(string instance)
{ ... }
}

Then make sure you reference System.Core.dll in your compiler options and everything will work fine.