Monday, April 30, 2007

"Astoria" Enables RESTful Data Services

Pablo Castro's new blog's first post is Codename "Astoria": Data Services for the Web describes the ADO.NET team's new incubation project codenamed "Astoria."

Astoria is the topic of Pablo's MIX07 presentation:

Accessing Data Services in the Cloud
Speaker: Pablo Castro - Microsoft
Audiences: Designer, Developer

Come learn about new Microsoft technologies that enable you to make your data available over the Web through a simple REST interface and using open formats such as plain XML, JSON or even RDF. We also discuss the underlying entity framework that makes it easy to model, publish, and program against your data over the Web.

Update 5/3/2007: The video of Pablo's presentation is XD006 - Accessing Data Services in the Cloud.

Note: That's one of the members of LINQ-Related Sessions at MIX 07. I mentioned that Pablo Castro promised "to put a Web spin on the Entity Framework, which might include a side trip for LINQ to Entities." It turns out that I was right.

The project currently offers an Astoria toolkit for Orcas Beta 1 and a read-only demonstration of Astoria online services for various data sources, such as the Northwind and AdventureWorks sample databases, Encarta articles, and anonymous TagSpace items.

In the future, you'll be able to create your own experimental online data services and temporarily host them on the Web site.

According to the documentation, which you can download from the Astoria Web site:

Project Astoria consists of a combination of patterns, libraries and an online service that explores the concept of data services for the web. Currently, Astoria data services use relational databases as the underlying store, but in general the nature of the store does not surface in the Astoria interfaces.

The goal of Astoria is to facilitate the creation of flexible data services that are naturally integrated with the web. As such, Astoria uses URIs to point to pieces of data and simple, well-known formats to represent that data, such as JSON and plain XML. This results in the data service being surfaced to the web as a REST-style resource collection that is addressable with URIs and that agents can interact with using the usual HTTP verbs such as GET, POST or DELETE.

In order for the system to understand and leverage semantics over the data that is surfacing, Astoria models the data exposed through the data service using a model called the Entity Data Model (EDM), an Entity-Relationship derivative. This organizes the data in the form of instances of "entity types", or "entities", and the associations between them. ...

Currently Astoria can represent data in plain XML, JSON (JavaScript Object Notation) and in a subset of RDF+XML.

The default representation format is XML. This is not a specific format; it is a fixed mapping of the entity structure to XML elements, with some added semantics to make use of the information provided by the EDM schema.

The team is considering adding RSS and ATOM to the data formats.

Astoria data services rely on the Windows Communication Framework (WCF), Entity Data Model, and Entity Framework. Here's a brief excerpt from the FAQ page:

Q: What is the relationship between Astoria and Windows Communication Foundation (WCF)?
A: All Astoria data services are essentially WCF services. You can see Astoria data services as a special form of WCF service that is aware of the nature of the data that is exposing.

Q: What is the relationship between Astoria and the Entity Data Model (EDM)?
A: Astoria data services uses the EDM as the model for the data exposed through its services. Having a high-level data model allows Astoria to provided added semantics to the service, such as having a clear definition of what an “entity” is for a given service or how to navigate from one entity to another associated entity.

Q: What is the relationship between Astoria and the ADO.NET Entity Framework?
A: Astoria exposes data modeled as EDM models. The ADO.NET Entity framework is the natural choice for an infrastructure that already supports the EDM and through mapping technology can present relational stores in EDM terms. The Astoria runtime that sits in the web servers and serves the requests to data services is built on top of the ADO.NET Entity Framework.

I plan to test the toolkit later this week and will add links to posts with my results here.

This is the second of the promised MIX07 announcements of projects that take advantage of the Entity Framework, which was recently cut from the Orcas release of Visual Studio. The first is here.

Update 5/31/2007: Granville Barnett has posted a new article on, "Interacting with Astoria Data Services," which describes Astoria Web services, explains their impact on rich Internet applications (RIA, "rich interactive applications" in Microsoftese), and provides examples of using Astoria with ASP.NET AJAX.

Update 5/7/2007: Channel9 offers a Pablo Castro: Astoria Data Services video segment with Pablo at the whiteboard diagramming a RESTafarian view of data services.

Update 5/2/2007: Techmeme added two links to this post on 5/1/2007.

Alex Barnett's Microsoft Codename "Astoria": Data Services for the Web provides a basic description of Astoria and its URI syntax, including its more popular operators. He's also posted an interview with Pablo on YouTube and some pithy quotes from industry luminaries about Astoria. (Alex was working on Astoria when he left Microsoft in 2006.)

Update 5/1/2007: Read what Andrew Conrad has to say about Project Astoria and how it might integrate with Project Jasper.

"Jasper" to Deliver Dynamic ADO.NET 3.0

During Monday morning's MIX07 keynote, Andrew Conrad spilled the beans about the project he's been working on for the past few months: "Project Jasper." Also known as Dynamic ADO.NET, here's the official description of this new data-access technology from Andrew's blog post:

Microsoft Codename “Jasper” is a set of components aimed at fulfilling the need for a rapid and iterative development experience for data. With Jasper, you are able to just point at a database and immediately begin coding against its data using intuitive, domain-specific data classes. No configuration and no source code generation are required. Jasper works with existing application frameworks (including ASP.NET, WinForms, and WPF) and existing, real-world databases.

Jasper provides a programming interface to your data that is well-suited to rapid development. Jasper provides two main services:

  • Data Access: query and update facilities for reading and writing data, exposed via an Object-Relational (O/R) API.
  • Data Binding: presentation of data in controls that form the visual user interface of the application.

And here's a second description from the download page for the MIX07 Community Technical Preview (CTP):

Jasper leverages the power of dynamic languages and the concept of convention over configuration to provide a programming surface for data that enables rapid development of data-bound applications. While most other rapid data access frameworks are only capable of working against simple databases, Jasper can scale to almost any database, regardless of size or complexity. This is possible because Jasper takes advantage of the ADO.NET Entity Framework’s significant investments in mapping and conceptual data modeling. [Emphasis added.]

Jasper's dependence on the ADO.NET Entity Framework (EF) is significant, considering Saturday's announcement that Orcas has dropped the EF from the RTM, after an earlier postponement of the Entity Data Model (EDM) Designer.

According to Andrew:

For CTP release, we are specifically targeting Visual Basic 9 (Orcas version of VB.NET) and Iron Python 1.1. In other words, these are the languages we did a bulk of our testing for the CTP release. The reality is any CLR language that supports late binding can be used with the Jasper framework

I plan to give the MIX07 CTP a workout this week and will add links to reports of my results here.

Update 4/30/2007 14:17 PM: The ADO.NET Team posted Project Codename "Jasper" - Announced at Mix 07 which describes Jasper as an "ADO.NET incubation project for building quick, clean and iterative applications." The post says:

Using “Jasper”, developers start with a connection string and can immediately begin programming the core of an application. “Jasper” uses a set of new technologies to make this happen:

  • Dynamic generation of data classes so there is no configuration or design time code-gen to carry around.
  • Rich query and O/R capabilities because “Jasper” is built on top of the Entity Framework.
  • Auto-binding capabilities for ASP.NET, WinForms, and WPF to make binding data to a UI simple and automatic.

Carl Perry offers additional details about Jasper in his Announcing Project Jasper - A RAD vision for the Entity Framework post.

Shyam Pather adds his interpretation of Jasper and excerpts from the Jasper Technical Overview in Microsoft Codename "Jasper" CTP Released!

Note: Andrew also mentioned that the Dynamic Language Runtime (DLR) will be announced at MIX07.

Update 5/2/2007: The video for the Sam Druker and Shyam Pather "Jasper" presentation is available for viewing or download at the following link:

DEV18 - Rapidly Building Data Driven Web Pages with Dynamic ADO.NET

Speakers: Samuel Druker - Microsoft, Shyam Pather - Microsoft
Audience: Developer

Come learn about how new technologies from Microsoft bring together the concepts of dynamic languages and ADO.NET and allow you to quickly develop data driven Web pages using ASP.NET dynamic data controls, scalable to even the most complex databases.

For more information on the DLR, listen to Jon Udell's podcast: A conversation with John Lam about the dynamic language runtime, Silverlight, and Ruby.

Update 5/5/2007: Andrew Conrad's Jasper and C# post asks "Is Jasper usable from C#?"

The short answer is – We designed Jasper specifically for CLR languages with late-bound facilities such as VB and IronPython.  C# doesn’t currently support late-bound calls and hence the answer would be no though some aspects of Jasper may still be applicable.

He then goes into an explanation of how C# might be used. The real question is "Why would anyone want to use C# in a dynamic, late-bound environment?" Andrew then moots the issue with:

[W]e are actively working on moving Jasper over to the dynamic language runtime - the freshly announced .Net addition to the runtime designed to extent the CLR for dynamic typed CLR languages (VBX, IronPython, IronRuby).  Based on early analysis it is a strong possibility that we move away from the base class mechanism and support a more dynamic model where data class types (or even specific instances!) can be modified at runtime to allow the user to add their own business logic.

Update 5/8/2007: Andrew Conrad reports that Channel 9 has posted a pre-MIX07 interview with the Jasper team:

Project Jasper is geared toward iterative and agile development. You can start interacting with the data in your database without having to create mapping files or define classes. You can build user interfaces by naming controls according to your model without worrying about binding code. In this interview Shyam Pather, Carl Perry, Andrew Conrad and Jeff Derstadt, go deep whiteboarding and demoing Jasper.

Sunday, April 29, 2007

Déjà Vu All Over Again: Entity Framework Cut from Orcas

Microsoft software architect Mike Pizzo announced in the ADO.NET Entity Framework Update post of 10:09 PM, Saturday, April 28, 2007:

[W]e have decided to ship the ADO.NET Entity Framework and Tools during the first half of 2008 as an update to the Orcas release of the .NET Framework and Visual Studio.

Mike does his best to spin the bad news by starting the announcement with:

Microsoft is deepening its investment in the ADO.NET Entity Framework as a critical piece of Microsoft’s Data Platform vision.

Many Microsoft observers will read that sentence as:

Microsoft is deep-sixing its investment in the ADO.NET Entity Framework as a critical piece of Microsoft’s Data Platform vision.

It's too soon to make that judgment, but there's no question that Mike's announcement delivers yet another blow to those developers (like me) who followed Microsoft's primrose path into its previous object/relational mapping (O/RM) tool void.

Microsoft MVP Scott Bellware avoided the most recent O/RM trap by abandoning a planned short-term implementation of the Entity Framework (EF), as described in his April 24, 2007 Falling Back to NHibernate - A Phone Call with the Entity Framework Folks:

We made a decision today to put an early end to our exploration of the Entity Framework for our current product work, and to fall back on NHibernate. ...

When the EDM designer started to drop out of the CTP's (post-August 2006), we had hoped that it would reappear around beta 1, and would be fairly solid around beta 2. We're scheduled to ship our current work product in coincidence with Orcas. We expect to start customer previews when a go-live license is offered. Without an editor for the metadata, we really can't justify the use of the EF to our customers.

The phone call [to the ADO.NET EF team] took place in two parts. The second part of the call got into some NDA-covered material that Kevin wasn't able to stick around for. Tim and Dan shared some info that gave me with a definitive and conclusive basis for making a sound call for our product. I'm very grateful to the EF team to have been enabled to make such a clear cut decision.

I have the feeling that Scott might have received advance notice (under NDA) that EF and its components were going back to the drawing board. Regardless, he said in an April 22, 2007 comment to my Persistence Ignorance Is Bliss, but Is It Missing from the Entity Framework? post:

I won't remain committed to the Entity Framework if the EDM Designer doesn't arrive by Orcas RTM.

I see the EDM designer as one of the most compelling features of EF in its current incarnation. It's the piece that translates well to how we expect our customers will expect to work with the entities in our product.

If the EDM designer is not in place by RTM, then I don't expect that we can ship a product to our customers if part of the product's customization story is based on the presence of the EDM.

We would need to fall back on NHibernate, and we have had this contingency plan in mind since we began the foray into EF.

Scott implemented his contingency plan two days later. (He insists he's not a member of the "NHibernate Mafia".)

Note: There still might be hope for EF with the Domain Driven Development (DDD) crowd. Interknowlegy's Tim McCarthy is writing .NET Domain-Driven Design with C#: Problem-Design-Solution to be published by Wiley. You can download the slides and code for Tim's April 19, 2007 "Domain-Driven Design using the ADO.NET Entity Framework" presentation to the International Association of Software Architects' Southern California chapter (IASA SoCal). The code is based on Paul Gielens original code for "Organizing Domain Logic" which compares Microsoft's Three-Layered Services Architecture (MS-TLSA) approach against DDD. (The three layers are the traditional data, business and presentation, not the EDM's storage, mapping, and coneptual layers.)

Does Microsoft Have a Data Access Strategy?

Mike posted Microsoft’s Data Access Strategy to the Data blog three minutes after the preceding bombshell. He answers the question with:

Yes, it turns out we do. Microsoft envisions an Entity Data Platform that enables customers to define a common Entity Data Model across data services and applications. The Entity Data Platform is a multi-release vision, with future versions of reporting tools, replication, data definition, security, etc. all being built around a common Entity Data Model.

Mike disclosed another part of the data access strategy in the first post:

Microsoft will be leveraging the Entity Data Model in future versions of Microsoft products such as SQL Server. This Data Platform vision enables customers to leverage their investment in a common conceptual entity model across product lines. [Emphasis added.]

Frans Bouma, the developer of LLBGen Pro, a commercial O/RM tool, has a different (and sinister) take on Microsoft's data access strategy:

The announcement speaks about Microsoft building applications on top of Entity Framework and also that the Entity Framework is moving towards SqlServer. Now, if you can add 1 and 1 together, you of course realize that if you have an application, say Sharepoint vNext, build on top of Entity Framework and you have one of your major cash cows SqlServer enabled with Entity Framework, you have a combination to earn more money: people who want the next version of your application also want your new database system.

The Entity Framework in ADO.NET vNext was designed to be database independent, and it might still be database independent when it eventually ships (I personally don't believe their H1 2008 mark). If you have one of your other major cash cows, say Sharepoint vNext, build on top of ADO.NET vNext, you then open up the road to host Sharepoint on say Oracle, DB2 or an open source database. This then would alienate one of your other cash cow products, SqlServer which will have the Entity Framework build-in.

Personally, I don't buy Frans' theory at present. Microsoft's "major cash cows" are Windows operating systems, the Office suite, and well-entrenched server-based applications, such as Exchange Server and SQL Server. By default, Windows SharePoint Services (WSS), which is a component of Windows Server 2003, uses freely distributable SQL Server 2005 Express (SSX) as its data store not as its host operating system.

Paul Wilson, the developer of the commercial WilsonORMapper (a simpler competitor to LLBGen Pro) says "Great News: Only One O/RM Shipping in Orcas." Wilson is a fan of LINQ to SQL because it's simpler but "good enough for the vast majority of cases."

Mike contrasts LINQ to SQL and LINQ to Entities:

LINQ to SQL supports rapid development of applications that query Microsoft SQL Server databases using objects that map directly to SQL Server schemas. LINQ to Entities supports more flexible mapping of objects to Microsoft SQL Server and other relational databases through extended ADO.NET Data Providers.

Despite the comments regarding support by LINQ to SQL for databases other than SQL Server by Dinesh Kulkarni, Scott, Guthrie, Matt Warren and Keith Farmer quoted in my Future LINQ to SQL Support for Multiple Databases? post of April 19, 2007, Mike has put that issue to bed. It's clear as part of Microsoft data access strategy that LINQ to SQL is LINQ to SQL Server (only).

Shades of WinFS, ObjectSpaces, Project Green, and the Microsoft Business Framework

The previous indication of trouble in the Entity Framework camp was the announcement at VSLive! San Francisco that the Entity Data Model (EDM) Designer had been cut from the Orcas release.

Earlier, a major controversy about EF's lack of "Persistence Ignorance" and its need to support Plain Old CLR Objects (POCOs), test-driven development (TDD), and agile programming techniques erupted at Microsoft's MVP Summit conference in Redmond on March 12 - 15, 2007.

But the real problem is that .NET developers might equate the result of EF's integration into the next version of SQL Server (code-named "Katmai") with the demise of ObjectSpaces, EF's ill-fated predecessor OR/M tool, when it was integrated into WinFS. My June 26, 2006 Microsoft Bites the Bullet: WinFS Bites the Dust post delivers the grisly details of the deaths of WinFS, ObjectSpaces, Project Green, and the MBF.

Here's Andrew Conrad's statement of the official status of ObjectSpaces in Visual Studio 2005 (then codenamed "Whidbey") as of April 6, 2004:

ObjectSpaces is not participating in Whidbey beta 1, however, it remains part of the Whidbey/Yukon wave and will be made available as a downloadable add-on pack for the .NET Framework Whidbey shortly after Whidbey ships. This additional development and stabilization period will be focused on improving the overall ObjectSpaces programming experience and providing tighter integration with WinFS, the next generation file system in Longhorn. The schedule for the ObjectSpaces mapping tool is also being adjusted accordingly.

His post concludes with:

Bottom line - this will mean some change in the release schedule, but should not adversely effect anyone's plans to deploy the ObjectSpaces framework. In fact, I believe the tighter integration with WinFS will significantly justify the delay.

Does that language sound familiar? Compare the comments with those to Mike's initial post.

Even more ominous is Barbara Darrow's April 26, 2007 speculation in CRN that "Katmai, Orcas Link Could Delay Longhorn Release." If true, which I doubt, Katmai's EF dependency could delay it and Longhorn Server until probably late 2008.

Mike replies with a Sunday-afternoon comment regarding spin, ObjectSpaces, and WinFS:

I apologize for the “PR-spin” feel people have associated with this post. Even a short delay of the Entity Framework is a difficult message, especially to convey through the imprecise nature of written communication. How do we convey that we believe this to be the right decision, without people losing confidence in the technology, particularly in light of Microsoft’s less-than-perfect record in shipping ORM solutions, and in the wake of WinFS? I struggled with the wording. With the approach. With the message. Yes, marketing was involved in reviewing the content (of course). I regret now accepting some of their input, but the message is the same. The Entity Framework will not ship in Orcas. It will ship shortly there-after. The slip isn’t about Microsoft’s lack of commitment to the technology. It does give us more time to address some key feedback we’ve received to date from internal and external customers, including providing at least CTP-quality tools for defining flexible mapping between the store and the objects. There’s no way I can convince you that this isn’t another ObjectSpaces, or another WinFS. You’ll have to look at how we continue to deliver over the next few months. You’ll have to look at the quality of the bits. You’ll have to look at the other investments Microsoft is making around the Entity Data Platform, including announcements and demos this week at Mix.

This year I celebrate my 20th year at Microsoft. In that time I’ve seen Microsoft invests heavily in a number of technologies that, for one reason or another, never come to market. This isn’t one of them.

I'm inclined to believe Mike's comment. NetDocs also comes to mind as an example of a well-funded Microsoft technology that never came to market.

"Those who cannot remember the past are condemned to repeat it."

George Santayana (1863 - 1952, The Life of Reason)

Mike says in his initial post, "Stay tuned for announcements starting next week at Mix around new products building on the ADO.NET Entity Framework." I plan to do this, but with trepidation.

Update 5/3/2007: Redmond Developer News senior editor Kate Richards quotes me in her April 30, 2007 ".NET Entity Framework Slips Beyond Orcas" article:

Roger Jennings, a developer and technology writer for OakLeaf Systems, doesn't think the later ship date is going to be a big deal. "They decided that they couldn't release the Entity Framework without the Entity Data Model Designer, which was my contention to start with," he said. "And they've also just announced a couple of incubation projects that depend on it that also might have an influence on the delivery date." ...

"Almost everybody is considering [Entity Framework] to be ObjectSpaces revisited, but I don't think that's the case," says Jennings.

The "incubation projects" are the ADO.NET Team's "Project Jasper" (a.k.a., Dynamic ADO.NET) and "Project Astoria" (a.k.a., "RESTful Data in the Cloud.)

Note: This item was linked from the Discussions section of the Techmeme entry for the original ADO.NET post. There's also a Discussions link to a Channel9 thread.

Saturday, April 28, 2007

Stealth Dynamic Languages Presentation at MIX07

According to Beth Massi's April 27, 2007 So, you're going to Mix? post on the Visual Basic Team blog:

Jim Hugunin and John Lam [will be] talking about dynamic languages on .NET. (11:45am, Tuesday) This one has so much revolutionary stuff in it that they actually pulled it from the session descriptions on the Mix site. Long story short, search the web on musings about why John and Jim are at Microsoft, add to that the fact that the VB team has been working with them heavily, and you might be able to figure it out.

The original name of their currently anonymous presentation was Just Glue It! Dynamic Languages in "WPF/E": Jim Hugunin - Microsoft, John Lam - Microsoft.

Update: 5/2/2007: The final session blurb that appeared yesterday is:

DEV02 - Just Glue It! Ruby and the DLR in Silverlight

Speakers: Jim Hugunin, John Lam
Abstract: The web was built using dynamic languages. Their plain-text format made it easy to mash up scripts to create the next great app. Similarly, dynamic languages will find a home in Silverlight applications where plain-text formats are common. Silverlight can be easily deployed, which means that a wider range of dynamic languages will be used in building browser-hosted applications. In this demo-centric talk, you will see this happen before your eyes as we rapidly create an application by combining code and markup from existing samples in Ruby, Python, JavaScript, and Visual Basic. This unprecedented level of integration is possible since all of these languages are implemented on top of the new Dynamic Language Runtime (DLR).

Catch a preview with a few screen captures from the video in this post.

I'm guessing that the "Jim and John Show" will extend their March 26, 2007 MTS07: Jim Hugunin and John Lam on Dynamic Languages and the CLR presentation at the Microsoft Technical Summit and describe what Mary Jo Foley called the "Dynamic Language Runtime" (DLR) in her Microsoft to roll out dynamic-layer for .Net post of April 24, 2007.

In an August 6, 2006 eWeek interview, Darryl K. Taft quotes Jim:

"I joined Microsoft because I was so inspired by the possibilities of .Net," Hugunin said. "The chance to work with the Visual Basic team is wonderful. The chance to make changes into .Net to make it more amenable to dynamic languages is phenomenally exciting."

[W]ith Visual Basic, Hugunin said Microsoft is looking at leveraging some of the simplicity that Basic has. "That feels a lot like what people say they love today about dynamic languages," he said. "People say they love a language like Ruby because it has this simplicity and this dynamism."

And Visual Basic, in its formative years, "was state of the art in providing this simple, dynamic experience. So we see that there's this great potential for Basic to be another great dynamic language, where we keep all the great benefits that we put into Visual Basic .Net, but we have some of this dynamic feel that people love."

Hugunin demonstrated how, with simple changes in the code, Visual Basic could be made to behave like a dynamic language. "But that's just, at the moment, an experiment," he said with a smile.

So there's sure to be a VB hook in Jim and John session.

It appears that Andrew Conrad (one of the original ObjectSpaces devs and an early LINQ proponent) has been working on dynamic languages code for a MIX07 session or two. Andrew says:

More information to come in the new few weeks, but all I can say at this point is that the experience has been very dynamic. And if you can’t make it to Mix07, we should have some screen casts and white papers available some time in the next several weeks. [Emphasis Andrew's]

Plain Old Visual Basic (POVB) as a Dynamic Language?

The following MIX07 session underwent a dramatic metamorphosis recently:

Rapidly Building Data Driven Web Pages with Dynamic ADO.NET
Speakers: Samuel Druker Polita Paulus - Microsoft, Shyam Pather - Microsoft
Audience: Developer

ASP.NET dynamic data controls are part of a powerful, rich new framework that lets you create data driven ASP.NET applications extremely easily. ASP.NET dynamic data controls do this by automatically discovering the schema at runtime, deriving behavior from the database and finally creating an ASP.NET page. Anything that can be inferred from the schema works with almost no user effort. If needed, the page can be further customized either by using static languages such as Microsoft Visual C# or Visual Basic .NET, or dynamic languages such as IronPython, Visual Basic, Jscript, etc. In this talk, we show you how to build rich, database driven Web applications from scratch, such as TaskList along with other demos. [Emphasis added.]

Come learn about how new technologies from Microsoft bring together the concepts of dynamic languages and ADO.NET and allow you to quickly develop data driven web pages using ASP.NET dynamic data controls, scalable to even the most complex databases.

Notice that "dynamic ... Visual Basic" in the preceding description is plain VB, not VB .NET, which I'm betting is a reincarnation of VBScript for the CLR. Polita is the developer of Blinq, so there might be a LINQ hook in the session.

One of the topics of MTS07 was "When is the veil of silence going to be lifted?" It's been almost eight months since Jim added a post to his Thinking Dynamic blog. The last post announced the release of IronPython 1.0 on CodePlex and included a brief history of the three-year project. Microsoft released IronPython 1.1 with little fanfare on April 17, 2007.

Andrew Conrad's April 27, 2007 post starts with: 

Over the last several months, because of my current project I have had to program in a bunch of different languages.   (The reason for this will become crystal clear early next week).  Basically, I have had to code in C#, VB, and IronPython while at the same time I have been trying learning Haskell and occasionally Ruby - and some Javascript just for fun on the weekends. 

What's the chance of an announcement that IronPython (and perhaps even RubyCLR) will support LINQ?

Microsoft's Shri Borde wrote in a May 12, 2005 IronPython list message:

IronPython 1.0 will not support LINQ. We have started to look at it, but will seriously look at it after the summer. It is a meaty issue given that many of the concepts in LINQ are based on static typing.

Keith J. Farmer, now with Microsoft, started a [IronPython] Extension methods... thread on the list that compares Python's list comprehensions with LINQ expression trees, along with LINQ and Python extension method syntax. In his September 20, 2005 message, Keith says:

I think the interesting bit for us, now, is that all this runs on the [.NET Framework] 2.0 runtime. From what I can tell, all it'd take would be some compiler modifications to recognize extension methods (marked by attribute) and to create an easy way to create expression trees, and IronPython would be set for it.

Yes, I'm excited about LINQ.. I've seen many gems come out of .NET (and a few blunders). This, I think, is just plain beautiful.

LINQ is the big thing for C# and VB, as are dynamic languages for .NET. They deserve one another. Unfortunately, LINQ integration with .NET dynamic languages isn't one of Miguel de Icaza's guesses for MIX07 announcements.

Note: Go to LINQ-Related Sessions at MIX 07 for updates to current LINQ sessions. 

Thursday, April 26, 2007

More Issues with VB 9.0 Samples in the Updated LINQ to SQL Documentation

It's evident that no one tech-edited or double-checked the VB 9.0 code examples in the "LINQ to SQL: .NET Language-Integrated Query for Relational Data" whitepaper Dinesh Kulkarni, Luca Bolognese, Matt Warren, Anders Hejlsberg, Kit George that was updated in March 2007.

I encountered the most egregious example of the authors' disdain for VB in the following code snippet from the "Stored Procedures Invocation" topic:

<StoredProcedure(Name:= "VariableResultShapes")> _
  <ResultType(typeof(Customer))> _
  <ResultType(typeof(Order))> _
public VariableResultShapes(shape As Integer?) As IMultipleResults
  Dim result As IExecuteResult = ExecuteMethodCallWithMultipleResults(Me, _
    CType(MethodInfo.GetCurrentMethod(), MethodInfo), shape)
    return CType(result.ReturnValue, IMultipleResults)
End Function
The example uses C#'s typeof() instead of VB's GetType() function in lines 2 and 3, is missing the Function keyword on line 4, and uses the unsupported ? symbol for Nullable(Of T) on line 4. There are obvious errors on 50% of the lines of the example. Further, the following required Imports statements are not to found anywhere in the document:
Imports System.Data.Linq
Imports System.Data.Linq.Provider
Imports System.Reflection
It would behoove authors of whitepapers with a penchant for C# to at least check to see if their VB sample code compiles.

By the way, Kit George is a program manager on the VB team and presented the LINQ Overview Webcast on April 25, 2007.

Updated 4/27/2007: Dropped initial (off-topic) paragraphs.

Wednesday, April 25, 2007

Bart de Smet Releases LINQ to SharePoint Alpha

The LINQ to SharePoint implementation promised in Bart de Smet's Coming soon - The return of IQueryable - LINQ to SharePoint announcement has arrived as an alpha version for the Orcas March 2007 CTP: LINQ to SharePoint - Alpha release available now from CodePlex with documentation, samples, and a quick-start video.

Here from Bart's blog are its featured highlights:

  • Supports writing LINQ queries for SharePoint lists in both C# 3.0 and Visual Basic 9.0
  • The SpMetal entity type creation tool supports both C# 3.0 and Visual Basic 9.0
  • Talks to SharePoint over web services or though the SharePoint Object Model
  • Most CAML query constructs are available via LINQ

Bart promises to update the alpha to Orcas Beta 1 shortly. Check the B# blog or here for availability.

Querying SharePoint with a Web services and supporting updates in the obscure XML dialect called Collaborative Application Markup Language (CAML) isn't a piece of cake, as I demonstrated in a December 2003 Visual Studio Magazine article, "Manage SharePoint Lists."

P.S. Thanks for supporting VB and C#, Bart.

The .NET Standard Query Operators (SQOs) from LINQ Update

Bart released an April 6, 2007 update to the 0.9 Release Candidate for A custom implementation of the .NET Standard Query Operators from LINQ on CodePlex. This version is intended for use with VS2005 or Orcas March 2007 CTP. The custom SQOs implement a clone of the LINQ to Objects API (the System.Linq.Enumerable class).

After you build the project, you can add a reference to BdsSoft.Linq.dll in a test project and then replace using System.Linq; with using BdsSoft.Linq; to work with the custom implementation.

Note: If you uncomment the CS30 definition to use C# 3.0 features, you'll need to add a reference to the System.Core namespace to build the BdsSoft.Query class.

Tuesday, April 24, 2007

LINQ and LINQ to SQL Webcast

Marc Schweigert of Microsoft's Public Sector DPE (Developer and Platform Evangelism) Team has produced a 90-minute Webcast entitled Introduction to LINQ + LINQ to SQL. The presentation aired at 2:00 PM EST on 4/24/2007, immediately after the VB team's Orcas Overview Webcast and the day before its LINQ Overview presentation. Registration is required to view the archived Webcast.

Following is the abstract:

Modern applications operate on data in several different forms: Relational tables, XML documents, and in-memory objects. Each of these domains can have profound differences in semantics, data types, and capabilities, and much of the complexity in today’s applications is the result of these mismatches. In this talk, we will explain how the Orcas release of Visual Studio aims to unify the programming models through LINQ capabilities in C#, a strongly typed data access framework, and an innovative Application Programming Interface (API) for manipulating and querying XML.

Database-centric applications have traditionally had to rely on two distinct programming languages: one for the database and one for the application. In the second part of this talk we will introduce LINQ to SQL, a component of the LINQ project designed to help integrate relational data and queries with C# and Visual Basic. LINQ to SQL enables developers to express queries and updates in terms of their local programming language without sacrificing the server-side execution model of today’s high-performance SQL-based approaches. Using these advances, database queries that previously were stored as opaque strings now benefit from static type checking, CLR metadata, design-time type inference, and of course IntelliSense. LINQ to SQL also supports a rich update capability that lets you save changes to an object graph back to the database using optimistic concurrency or transactions.

LINQ to SQL receives about the last 45 minutes of the presentation, most of which uses C#.

Saturday, April 21, 2007

Installing Orcas Beta 1 from the 5.32 GB ISO Image

Conventional single-layer DVD-R/RW and DVD+R/RW discs hold 4.7 GB. So unless you have a double-layer DVD burner and media, you won't be able to run the 5.32-GB Visual Studio Orcas Beta1 Setup program from a physical DVD.
Setup fails before completion of .NET Fx 3.5 when run from a networked drive. Believe Setup's Start Page where it says:
Some components require that network connections be temporarily suspended during setup.

Click for full-size image.

If you try installing from a network drive, which includes third-party apps that mount ISO images to virtual CD or DVD drives, you might experience what appears to be a non-fatal timeout message while Setup is copying its initial set of files. Even if not, you'll undoubtedly see this message during the installation of .NET Fx 3.5:

and end up with this screen when installation fails:

Click for full-size image.

This means that your double-layer DVD burner must be local to the machine on which you want to install Beta 1. Virtual machines connect to the host OS's physical drives as networked drives, so you're out of luck if you want to install Orcas Beta 1 on Vista as a guest OS.

The Solution

Use Daemon Tools' free Virtual Daemon Manager (VDM) to mount the en_visual_studio_orcas_beta_1_professional_dvd_23591.iso image file to a virtual DVD drive. The current VDM version (4.09) works fine under Windows Vista as a host or guest OS, despite what you might have heard on the Web.

Note: When you install VDM on virtualized Vista, save and run the setup file. You must reboot to finish VDM's installation. Under Virtual Server 2005 R2 RC1 with Vista as the guest OS, you must explicitly run daemon4091-x86.exe a second time to complete the installation after rebooting.

Update: 4/24/2007: Commenters have suggested using WinRAR or IsoBuster to extract the image to executable files. WinRAR isn't free (it has a 40-day trial license) and IsoBuster requires the PRO version ($29.95 personal, $49.95 professional) to open an image of this size. Some commenters have suggested other virtual DVD drive apps, but VDM is free and has proven itself under Vista.

Thursday, April 19, 2007

Future LINQ to SQL Support for Multiple Databases?

In my recent tests of the Orcas March 2007 CTP's implementation of LINQ to SQL and the Object/Relational Designer, I tried substituting connection strings to SQL Server Compact Edition (SSCE) and Oracle 10g Express Edition with managed providers, and other databases with OLE DB drivers. Most resulted in "unsupported product" messages and all threw exceptions when attempting to open a connection with the DataContext object. One DataContext constructor accepts an IDbConnection object, which implies at least some degree of database independence.

SqlMetal.exe generates an XML external mapping file (NwindLINQ.xml for this example) that you can use in lieu of decorating entity classes with LINQ to SQL-specific attributes. This feature enables generating POCO (plain old CLR objects), which many developers prefer. The "LINQ to SQL: .NET Language-Integrated Query for Relational Data" white paper's "External Mapping" section indicates that the top-level <database> element of NwindLINQ.xml should have a ProviderType="System.Data.Linq.SqlClient.Sql2005Provider" attribute. It's missing from the mapping file I created with SqlMetal.exe, but it did point me to the System.Data.Linq.SqlClient namespace.

Inspecting SqlClient in Object Browser shows it contains about 167 classes having interesting names such as Funcletizer, which contains Friend Shared Function Funcletize(ByVal expression As System.Linq.Expressions.Expression) As System.Linq.Expressions.Expression. I remembered Funcletizer.Funcletize from Mike Taulty's blog and found it near the end of his Deconstructing LINQ to SQL (Part 2) post:

internal QueryInfo BuildQuery(Expression query, SqlNodeAnnotations annotations)
    query = Funcletizer.Funcletize(query);
    QueryConverter converter = new QueryConverter(, this.typeProvider, this.translator, this.sqlFactory);
    converter.ConverterStrategy = (this.Mode == ProviderMode.Sql2005) ? ConverterStrategy.SkipWithRowNumber : ConverterStrategy.Default;
    SqlNode node = converter.ConvertOuter(query);
    return this.BuildQuery(this.GetResultType(query), node, annotations);

You have to love the naming here :-) Note that the provider looks to have a SQL 2000 and a SQL 2005 mode on it which makes sense but I hadn't thought about it before even though I had thought that functions such as Take might use something 2000/2005 specific.

From there on in, it gets "really hard, really fast" or at least it did to me. From BuildQuery onwards there are a lot of classes (SqlSelect, SqlBinder, SqlRetyper, SqlMultiplexer, SqlFormatter and a whole lot more) that look to be involved in taking what we have as a tree and turning it into T-SQL.

It's obvious that it won't be a walk in the park to write the corresponding "really hard" extension code to implement another managed data provider and SQL flavor, such as Oracle PL/SQL or IBM DB2 SQL.

Update 4/21/2007: David Sceppa, an ADO.NET program manager and author of Programming Microsoft ADO.NET 2.0 Core Reference, posted ADO.NET Orcas: Sample Provider to the ADO.NET Team blog on March 16, 2007. The entry begins:

Last month, the ADO.NET team hosted a number of ADO.NET Data Provider writers on campus to present information on how to enhance an existing provider to support the Entity Framework features in the upcoming Orcas release. To help provider writers who were unable to attend, we’re publishing the material we presented to our team blog. In this blog post, you’ll find information about how to extend an existing provider. The blog also has a [Resources].zip attachment that contains the slide decks presented at the event, a help file that covers the CommandTree class, and the source code for a sample ADO.NET Data Provider.

The file contains OrcasSampleProvider.exe, but the source code snippets in "Extending an Existing Provider" specification don't appear to me to comprise the complete source code for an extended provider.

According to David's ADO.NET vNext Provider Writer Event Summary post of August 25, 2006, representatives from DataDirect, IBM, MySQL, SQLite, and Sybase attended the event. PHX Software's Robert Simpson writes about the conference in this short-term blog.

According to the April 2007 "Looking to LINQ" story in Redmond Developer News:

Microsoft is working with other database vendors to build interoperability into their repositories as well. Oracle Corp. officials would only say they're monitoring LINQ's progress, but IBM Corp. says it plans to support LINQ in both its DB2 database and the Informix IDS platform.

Curt Cotner, an IBM Fellow and CTO for database servers, says LINQ could very well be critical in eliminating the fragmentation that exists today among programmers and database developers. Still, Cotner expects LINQ to have its share of both ardent supporters and critical detractors. "It's going to be popular for a significant segment of customers, but there will be another segment that will see it as not down the path they're trying to follow," he says. ...

But Cotner doesn't see [the Java Persistence API,] JPA and LINQ as competing specifications per se. "It addresses a different segment of the population," he says. And frankly, he admits from a developer standpoint, JPA can't touch LINQ in terms of its ability to build native queries into code from the language and development environment.

VistaDB Software, Inc. announced the intent to support LINQ in a future version of VistaDB. VistaDB 3 is an embedded SQL database for .NET, the .NET Compact Framework, and Mono that supports datatypes compatible with SQL Server 2005 (except xml) and uses T-SQL syntax. Jason Short wrote in an April 7, 2007 "LINQ and VistaDB" blog post:

I feel quite strongly that LINQ support in the future will be a vital feature of VistaDB. Since we are already a fully managed database for Dot Net, the ability to make fully type safe queries will definitely set us apart from other database systems. And the fact that we will not have to drop to SQL code to perform that queries will mean a faster running database as well (and that is what you really wanted to hear, right?). ...

LINQ support is on the list for research right now. When will be release it? I don’t know. I would really like to hear from you about your plans to support LINQ. I think it is quite obvious that LINQ is going to change the way a lot of applications are written to query their database.

Update 5/16/2007: SQLite

I recalled seeing references to future support for databases other than SQL Server 200x [Express/MSDE], so I ran searches on "LINQ to SQL" Oracle, "LINQ to SQL" "Compact Edition", "LINQ to SQL" MySQL, etc. Here are the most topical responses I found from Microsoft folks (in chronological order):

Dinesh Kulkarni, LINQ to SQL Sr. Program Manager (6/15/2006, in answer to LINQ Project General forum request to "Compare DLINQ, NHibernate and Typed DataSet"):

Out of the box, [LINQ to SQL] will support SQL Server 2000/2005 but will be usable with other databases once providers for those databases are available.

Scott Guthrie, a general manager in the Microsoft Developer Division (1/28/2007, response to a blog comment re LINQ to SQL support for Oracle and MySQL):

"LINQ has a provider model, which will allow other databases to be plugged-in. So you could add support for both Oracle and MySql."

Steve Lasker, SQL Server Compact Edition program manager (3/1/2007, reply to blog comment):

We don't yet have a LINQ to SQL (DLINQ), but we do have ADO.NET V3 Entities working over SQLce. If you install the Sync Services CTP, you'll see we install a Entity dll for SQLce as well. This CTP isn't directly compatible with the Orcas Feb CTP, so don't expect to do anything with it just yet. It's possible the Feb CTP that was just released would enable Entities over SQLce, but I have to check to see what made it into that build.

Terry Adams, LINQ to SQL development lead (3/29/2007, LINQ Project General forum):

"LINQ to SQL does not currently support Oracle clients."

A Microsoft program manager (4/2/2007, private correspondence):

"Our Orcas plans do not include 'LINQ to SQL' Server Compact Edition in Orcas. LINQ to SQL is only SQL Server."

Matt Warren, Microsoft product manager for LINQ to SQL (4/12/2007, response to Julie Lerman's post in the LINQ Project General forum about LINQ to SQL being for SQL Server only):

LINQ to SQL actually stands for LINQ to 'databases that use SQL' or in other words LINQ for the relational data model. LINQ to Entities means LINQ for the Entity-Data-Model which is a kind of a relational++ model.

Dinesh Kulkarni, LINQ to SQL Sr. Program Manager (4/13/2007, Microsoft Connect feedback re use of IDbConnection to create a DataContext, resolved as "Postponed"):

"Thanks for your interest in the LINQ project and feedback. Support for other providers indeed is a very important area that we are looking into. Because of schedule reasons, we were not able to fit it in V1.

"As for OLEDB/ODBC, the problem is that these standards really don't fully address the semantic differences between SQL dialects and database type systems. So I suspect that we will have to emphasize managed ADO.NET providers (e.g. Oracle provider, DB2 provider etc.) over ODBC/OLEDB. Unlike in case of ADO.NET v2 APIs where the user decides the SQL semantics by creating appropriate string, LINQ to SQL (aka DLinq) has to understand the semantics of the generated string. Hence, we need separate providers that understand the corresponding SQL flavor and type system.

"That said, there may be enough value in handling a common subset. However, such a 'generic' provider will likely have limitations."

Keith Farmer, (4/18/2007, LINQ Project General forum response to my post about LINQ to SQL dependency on SQLClient):

Actually, LINQ to SQL is rather decoupled internally. Between scheduling and other external factors, however, we've only been able to show you a SQL Server provider.

Dinesh Kulkarni (4/19/2007, Microsoft Connect feedback answer to a follow-up question about using third-party managed .NET data providers with LINQ):

"Unfortunately we haven't had enough time to do a provider SDK or even a public provider API in Orcas. This is definitely a high priority post-Orcas release item for us. Thanks again for your feedback."

Update 4/27/2007: Elisa Johnson, Program Manager, Microsoft Data Programmability (4/23/2007, comment in the ADO.NET Team's ADO.NET Entity Framework - Orcas Beta 1 post):

We are working closely with many DB vendors, including IBM, to help them update their ADO.NET providers to support the Entity Framework. Unfortunately, I do not have any information as to when they may release a new provider.

Update: 5/7/2007: Jay Hickerson of the Visual Basic Team responding to a LINQ Project General forum question about the purpose of the O/R Provider property of LINQ to SQL's DataContext object:

The O/R Provider was originally placed there so that we could generate code that would work correctly with databases other than MS SQL Server at runtime. However, the runtime support for this did not materialize in time to get it into the designer for Orcas. The property has been removed for Beta 2 and there will not be support for different providers from the designer. We are considering adding some level of support back in a future release, possibly a service pack.

I doubt that the ADO.NET team would devote the resources necessary to port LINQ to SQL to competing RDBMSs, especially those for which Microsoft and the database vendor have released managed .NET providers, such as Oracle, but also encompassing products such as IBM DB2 9 and VistaDB 3.0 for which managed data providers are available from the vendor RDBS only.

Presumably, the Entity Framework (EF), which was designed from the get-go as the foundation of database-agnostic O/RM, and LINQ to Entities is the solution for Oracle, DB2, Informix IDS, and PostgreSQL. The ADO.NET team is supporting developers with extending providers for the EF's CommandTree class, which isn't likely to be the same as that for LINQ to SQL. However, the EF and LINQ to Entities seems to me to be overkill for embedded and other lighter-weight RDBMSs such as SSCE, VistaDB 3, MySQL, and Firebird (formerly Interbase). These databases are more likely to be used in projects with simpler object graphs that match the underlying persistence database's schema.

I'm not sanguine about the prospects for a future "provider SDK or even a public provider API" for LINQ to SQL from Microsoft. A future Microsoft LINQ to SQL provider for SSCE might arrive, but SSCE already has a LINQ to Entities provider that, hopefully, will be operable in Beta 1.

I think it's a good bet that an IBM DB2 LINQ to SQL EntityClient provider will arrive around the same date as the Orcas RTM in 2008H1 (?) that Microsoft finally releases EF and its components. I'm not holding my breath for the announcement of an EntityClient provider from Oracle.

Update 4/29/2007: Corrected spelling of Keith Farmer's first name, error, and rescheduled EF delivery date in the preceding (last) paragraph.

Tuesday, April 17, 2007

LINQ to SQL Video Series from Mike Taulty

Mike Taulty, a member of Microsoft UK's Developer & Platform Group who frequently blogs on LINQ-topics, has created a series of short (about 15-minute) video segments devoted to LINQ to SQL as implemented by the Orcas March 2007 CTP. Mike plans to post a new segment each business day until he exhausts his collection.

Following is a complete list of all segments (only two of which were available when I first posted this):

  1. Introduction to LINQ to SQL (13:00, April 16, 2007)
  2. A Tour Around the DataContext (18:00, April 17, 2007)
  3. Mapping Between Types and Schema (13:00, April 18, 2007)
  4. Tools for Generating Mapping Information (14:00, April 19, 2007)
  5. Inserting Data (09:00, April 20, 2007)
  6. Deleting Data (15:00, April 23, 2007)
  7. Updating Data (20:00, April 24, 2007)
  8. Concurrency (19:00, April 25, 2007)
  9. Joins (13:00, April 26, 2007)
  10. Dealing with NULLs (09:00, April 27, 2007)
  11. When Do Queries Execute? (08:00, April 30, 2007)
  12. Deferred Loading in LINQ to SQL (12:00, May 1, 2007)
  13. Querying with Stored Procedures in LINQ to SQL (24:00, May 2, 2007)
  14. Modification with Stored Procedures in LINQ to SQL (18:00, May 3, 2007)
  15. Calling Standard T-SQL Functions in LINQ to SQL (05:00, May 4, 2007)
  16. Calling Custom T-SQL Functions in LINQ to SQL (13:00, May 8, 2007)
  17. Inheritance in LINQ to SQL (18:46, May 9, 2007)
  18. Transactions in LINQ to SQL (06:51, May 10, 2007)

The presentations are well-edited and have surprisingly high production value for this genre. 18 segments are close to a full TV season (26 weeks).

Update: 5/11/2007: Mike has posted a trio of LINQ to XML video segments in C# and VB.

Update 4/24/2007: Mike's added posts about System.Linq.Expressions and LINQ to SQL - Dynamically Constructing Queries recently.

Technorati tags: , ,

LINQ-Related Sessions at MIX 07

Four of the 122 sessions of Microsoft's MIX 07 conference, which runs from April 30 to May 2 at Las Vegas's Venetian Hotel, have LINQ-related content. Here's are the details from the Sessions page:

Building Silverlight Applications using .NET (Part 1 of 2) Building Silverlight Applications using .NET (Part 2 of 2) Speakers: Jamie Cool - Microsoft, Nick Kramer - Microsoft Audience: Developer

This session demonstrates building a rich interactive application (RIA) using Silverlight and managed code. We'll cover how to use Visual Studio to create applications, how to create UI using XAML markup and code, how to build a custom control, how to retrieve data from a Web service, how to manipulate data with XML and LINQ, and how to use the available programming languages in Silverlight.

Note: Microsoft appears to have redefined RIA from rich Internet application (the common usage) to rich interactive application.

Deep Dive on Data Driven Experiences Speakers: Aaron Dunnington - Microsoft, Tim Scudder - Microsoft Audience: Developer

Update 4/28/2007 from Channel9 Buzzcast #13: In this episode, Aaron Dunnington, Program Manager on the Data Programmability XML Team talks about harnessing the power of the semantic web using Microsoft XML technology stack.

Come learn how technologies like Silverlight, Language INtegrated Query (LINQ), and SQL Server 2005 can help developers build impactful, dynamic applications that reach the broadest possible audience.

Comment: OK folks, what is this session really about? SQL Server, XML, or SQLXML?

Using LINQ to Dramatically Improve Data Driven Development in Web Applications Speaker: Anders Hejlsberg - Microsoft Audience: Developer

Modern applications operate on data in several different forms: Relational tables, XML documents, and in-memory objects. Each of these domains have profound differences in semantics, data types, and capabilities, and much of the complexity in today's applications is the result of these mismatches. Anders Hejlsberg, Microsoft Technical Fellow and Chief Architect for Microsoft Visual C# and LINQ, will explain how LINQ (Language Integrated Query) unifies these programming models and dramatically improves the experience of creating data intensive Web applications. Anders promises to have few slides and lots of live code demos!

Note: Anders appears to have expanded his title of Chief Architect for Microsoft Visual C# to include LINQ. This might explain the dearth of LINQ sample code in Visual Basic.

Update 4/28/2007: This session should shed some light on the forthcoming LINQDataSource for ASP.NET. Beth Massi says, "Be sure not to miss Anders typing the 'Dim' keyword!"

Bonus Session: Pablo Castro promises to put a Web spin on the Entity Framework, which might include a side trip for LINQ to Entities. Will Pablo disclose more details about CloudDB or Blue/Cloud, a.k.a. Pacifica? Don't miss:

Accessing Data Services in the Cloud Speaker: Pablo Castro - Microsoft Audiences: Designer, Developer

Come learn about new Microsoft technologies that enable you to make your data available over the Web through a simple REST interface and using open formats such as plain XML, JSON or even RDF. We also discuss the underlying entity framework that makes it easy to model, publish, and program against your data over the Web.

Office Pool: How many years ago was Pablo's mug shot for DevTeach taken?

Update 4/29/2007: Pablo's presentation should be even more interesting considering that Microsoft announced last night that Orcas has dropped "the underlying entity framework."

Andrew Conrad, one of the members of the original ObjectSpaces team, says he's been working on the following project for the last year:

Rapidly Building Data Driven Web Pages with Dynamic ADO.NET Speakers: Samuel Druker Polita Paulus - Microsoft, Shyam Pather - Microsoft Audience: Developer

Update 4/28/2007 New description: ASP.NET dynamic data controls are part of a powerful, rich new framework that lets you create data driven ASP.NET applications extremely easily. ASP.NET dynamic data controls do this by automatically discovering the schema at runtime, deriving behavior from the database and finally creating an ASP.NET page. Anything that can be inferred from the schema works with almost no user effort. If needed, the page can be further customized either by using static languages such as Microsoft Visual C# or Visual Basic .NET, or dynamic languages such as IronPython, Visual Basic, Jscript, etc. In this talk, we show you how to build rich, database driven Web applications from scratch, such as TaskList along with other demos. [Emphasis added.]

Come learn about how new technologies from Microsoft bring together the concepts of dynamic languages and ADO.NET and allow you to quickly develop data driven web pages using ASP.NET dynamic data controls, scalable to even the most complex databases.

Notice that "dynamic ... Visual Basic" in the preceding description is plain VB, not VB .NET. Polita is the developer of Blinq, so there might be a LINQ hook in the session.

Andrew says:

More information to come in the new few weeks, but all I can say at this point is that the experience has been very dynamic. And if you can’t make it to Mix07, we should have some screen casts and white papers available some time in the next several weeks. [Emphasis Andrew's,]

Update 4/18/2007: Minor edits and major additions. Update 4/24/2007: Read Mary Jo Foley's ‘Volta’: Microsoft’s dev platform in the Cloud? article about the "Live development platform" that Soma Somasegar plans to divulge at MIX 07.

Friday, April 13, 2007

Consolas Font Pack for VS 2005 and Orcas

One of the reasons for adopting the new template for the OakLeaf Systems blog was gaining more horizontal space for code samples. Here's what Microsoft says about Consolas and its downloadable installer:

Consolas is intended for use in programming environments and other circumstances where a monospaced font is specified. All characters have the same width, like old typewriters, making it a good choice for personal and business correspondence. Optimizing the font specifically for ClearType allowed a design with proportions closer to normal text than traditional monospaced fonts like Courier. This allows for more comfortable reading of extended text on-screen.

The package will change the default text face in Visual Studio to the Consolas family.

This package is only intended for licensed users of Microsoft Visual Studio 2005.

Another plus is that Consolas has a narrower pitch than Courier New/Courier, similar to the difference between the traditional Elite and Courier typewriter fonts, but Consolas is more readable than Courier New in a smaller height and width. Here's a bit-map example of code from

I've modified the CSS for the code samples in the blog to use Consolas as the default. Windows Vista, Windows 2003 Server R2, and most Longhorn builds install the Consolas font family. Strangely, Orcas doesn't install it.

If you don't have it installed, you'll see line wrapping in some examples because of the greater width of Courier New characters.

Technorati tags: , ,

Thursday, April 12, 2007

Entity Change Tracking in LINQ to SQL and the Entity Framework, Round 2

Entity change tracking is required to detect and resolve conflicts that can occur when more than one user updates an instance of a particular entity, such as a SalesOrder or MedicalRecord object, concurrently. Thus the common term for this problem, which occurs only with multi-user systems, is concurrency conflict.

There are two methods of handling concurrency conflicts: pessimistic locking and optimistic locking. If you're familiar with optimistic locking and handling concurrency conflicts, jump to the "Resolving Concurrency Conflicts with LINQ to SQL and the Entity Framework" topic.

Pessimistic Locking to Prevent Concurrency Issues

If the application's data store, usually a relational database, uses pessimistic locking, the following actions occur in response to an entity update request:

  1. User 1 requests a copy of the entity from the data store.
  2. The data store locks the record(s) associated with the entity to prevent other users from editing them. (For a SalesOrder, there is typically a record for a SalesOrderHeader and one for each SalesOrderLineItem.)
  3. User 1 edits the entity.
  4. User 1 submits the changes to the data store directly (two-tier) or through a business services layer (n-tier).
  5. The business services layer, the data store, or both determine that the changes don't violate either business policies or relational integrity.
  6. The data store completes the transaction, updates the record(s) and releases its locks on the records.
  7. User 2 repeat steps 1 through 6.

Pessimistic locking prevents concurrency conflicts at the expense of data accessibility. If User 1 goes to lunch or home after completing step 1 but before performing step 4, other users can't edit (and applications might not even be able to read) the locked data until User 1 returns to work and completes step 4 or an application-imposed time limit expires, which cancels the editing operation.

Optimistic Locking and Handling Concurrency Issues

Optimistic locking maximizes data accessibility but introduces uncertainty into data values when multiple users edit the same records simultaneously. The degree of uncertainty increases when multiple users edit independent copies of the data and submit the changes to the data store. This problem arises in any application that makes use of object/relational mapping techniques or ADO.NET DataSets. Data uncertainty is further exacerbated as the copy grows older; that is, the original data becomes stale.

If the application's data store uses optimistic locking, the following actions can occur in response to User 1 and User 2 obtaining a copy of the same entity concurrently:

  1. User 1 requests and obtains a copy of the entity. The data store doesn't lock the associated records.
  2. User 2 requests and obtains a copy of the entity before User 1 submits her edits.
  3. User 1 submits her edits which conform to business rules and don't violate relational integrity.
  4. A short time later, User 2 submits his edits which also conform to business rules and don't violate relational integrity.
  5. If the client, business layer (if present), and data store support concurrency management, a concurrency exception occurs. The exception is needed because User 2 might have made no changes or different changes if he was aware of the changes made by User 1.
  6. User 2 (or a supervisor/manager) resolves the concurrency conflict or User 2 cancels the edit.

The conflict must be resolved in step 6 because User 2 has captured original member (property) value(s) or a timestamp value in step 2. The data store detects concurrency conflict(s) by comparing original values with corresponding member values in the corresponding table cells or a timestamp cell. If any pair of values differ, the data store rolls back the pending transaction and reports that no records were affected by the submission. In this case, the data store, business layer, or client code throws an exception. Repeated attempts by User 2 to update the data with his values will fail until User 2 either abandons the editing attempt or synchronizes his original or timestamp value(s) with those of the data store.

Note: My "Optimize Update-Concurrency Tests" article from the October 2003 issue of Visual Studio Magazine compares the performance of concurrency management by testing member values of typed DataSet tables or SQL Server 2000 timestamp values. The tests results showed timestamp-based test with relatively large objects, such as a SalesOrderHeader, outperform member value tests by a factor of about 2.1.

Resolving Concurrency Conflicts with LINQ to SQL and the Entity Framework

Concurrency conflicts can be resolved by blindly allowing the last update to prevail (called the "last in wins" approach) or by presenting the user with resolution options and (hopefully) sufficient information on which to make an informed judgment.

Orcas March 2007 CTP's LINQ to SQL implementation offers developers the capability to easily create a conflict resolution message box, such as this:

The conflict occurred because the original ContactName value (Maria Anders) didn't match User 1's change to the database value (Maria Anders-Brown). User 2's option descriptions are taken directly from the Orcas March 2007 CTP's online help file. The database value is retrieved with the exception raised by the database (SQL Server 200x only for LINQ to SQL). Current database values are needed to make an informed conflict resolution decision. (Timestamp values aren't meaningful; they only indicate that underlying data has changed while an edit was in progress.)

Here's the VB 9.0 code to process changes tracked by the DataContext object (dcNwind) and resolve concurrency conflicts:
Private Sub ProcessChanges()
  'Process changes with concurrency conflict resolution
  sbLog.Length = 0
  Catch ccExc As ChangeConflictException
    'Enumerate the change conflicts for updates and deletions
    For Each ccObject As ObjectChangeConflict In dcNwind.ChangeConflicts
      For Each ccMember As MemberChangeConflict In ccObject.MemberConflicts
        Dim strMsg As String = dcNwind.ChangeConflicts.Count.ToString + _
          " concurrency conflict(s) have been detected in Object '" + _
          ccObject.Object.ToString + "', "
        With ccMember
          strMsg += "Member '" + .Member.Name + "'" + vbCrLf + _
            "Original Value = " + .OriginalValue.ToString + vbCrLf + _
            "Current Value = " + .CurrentValue.ToString + vbCrLf + _
            "Database Value = " + .DatabaseValue.ToString + vbCrLf + vbCrLf + _
            "Click Yes to force the Refresh method ... values." + vbCrLf + _
            "Click No to force the Refresh method ... modified." + vbCrLf + _
            "Click Cancel to force the Refresh ... database."
          'Resolve each member conflict
          Dim intResolve As Integer = MsgBox(strMsg, MsgBoxStyle.Exclamation, _
            Or MsgBoxStyle.YesNoCancel, _
            "Concurency Conflict Resolution for " + ccObject.Object.ToString)
          If intResolve = MsgBoxResult.Yes Then
          ElseIf intResolve = MsgBoxResult.No Then
          End If
        End With
    'No exception should occur at this point
  Catch exc As Exception
    MsgBox(exc.Message, MsgBoxStyle.Exclamation, _
      "Non-Concurrency Exception Submitting Changes")
    txtLog.Text = sbLog.ToString
  End Try
End Sub

Conflict resolution by user intervention is much more practical in a conventional, 2-tier (client/server) environment than with n-tier or service-oriented architecture (SOA) architecture.

The Entity Framework's ObjectStateManager detects concurrency conflicts but doesn't offer a built-in handler to resolve them at the member level. The ObjectStateManager provides all current and original values (but not database values) for comparison:

Following is the code that produces the preceding message box:

Private Sub btnSaveChanges_Click(ByVal sender As System.Object, _
    ByVal e As System.EventArgs) Handles btnSaveChanges.Click
  'Save changes and report optimistic concurrency errors that occur
  Dim timSave As New Stopwatch
    Dim intSavedEntities As Integer = edmNwind.SaveChanges
  Catch ccExc As OptimisticConcurrencyException
    Dim osEntries As IEnumerable(Of ObjectStateEntry) = ccExc.StateEntries
    Dim strMsg As String = Nothing
    Dim intCtr As Integer = 1
    For Each osEntry As ObjectStateEntry In osEntries
      strMsg += "Concurrency exception " + intCtr.ToString + " in entity set "
      strMsg += osEntry.EntitySet.Name + "." + vbCrLf + vbCrLf + "Current values: "
      For intCol As Integer = 0 To osEntry.CurrentValues.FieldCount - 1
        strMsg += osEntry.CurrentValues.Item(intCol).ToString + "; "
      strMsg += vbCrLf
      strMsg += "Original values: "
      For intCol As Integer = 0 To osEntry.OriginalValues.FieldCount - 1
        strMsg += osEntry.OriginalValues.Item(intCol).ToString + "; "
      strMsg += vbCrLf + vbCrLf
      intCtr += 1
    strMsg += ccExc.Message
    MsgBox(strMsg, MsgBoxStyle.Exclamation, "Optimistic Currency Exception")
  Catch sysExc As Exception
    MsgBox(sysExc.Message, MsgBoxStyle.Exclamation, _
      "Error Saving Changes to Database")
    txtTime.Text = Format(timSave.ElapsedMilliseconds / 1000, "0.000")
  End Try
End Sub

The Entity Framework developer must add procedural code to resolve the conflicts; retrieving database values requires a round-trip to the server with a DataReader to retrieve the current row values.

The Current Need for Do-It-Yourself Change Tracking

The key to detecting concurrency conflicts is to maintain original member values or the timestamp value. The DataContext object handles change tracking for LINQ to SQL; the ObjectStateManager does the same for the Entity Framework's Object Services layer. Neither object tracking approach is well suited to n-tier, SOA, or ASP.NET Web applications that update data because the object copy being edited in the client tier is detached from the change tracker in the middle tier or service layer. Currently, the developer must write change tracking code to track changes to the detached object and pass the change information along with the edited object to the middle tier. I use the term detached (from the change-tracker) so as not to confuse the problem with disconnected (from the database.)

Note: See "Disconnected Operation" and the Entity Framework for an example of confusing disconnected with detached.

Anders Borum started on 4/9/2007 a Entity Change Tracking - questions LINQ to SQL design thread in the LINQ Project General forum that proposes tracking changes to entities in the entities themselves instead of the DataContext:

Why does LINQ to SQL not implement a change tracking mechanism on an entity level?

Without the entity tracking mechanism, it would seem like the framework is primarily targeted at developers working in a truly connected state (i.e. always use a DataContext to query for entities and remain connected (to a DataContext) throughout the lifetime of the entities).

I consider it a really big obstacle to fully accept LINQ to SQL as a mature ORM if I have to implement the change tracking mechanism every time I design an architecture with more than 2 tiers (e.g. a standard ASP.NET application that creates a DataContext for each request, makes changes and disposes resources).

Note: Anders started the Compiled Queries in LINQ to SQL - performance considerations and entity caching thread that I mentioned in my earlier LINQ and Entity Framework Resource Updates post. This thread is definitely worth reading in its entirety.

Anders' issue is the same as that Frans Bouma raises in his Why change-tracking has to be part of an entity object post:

I used ASP.NET as an example to show the stupidity of this EDM/Linq to Sql design flaw, but any setup where an entity object gets disconnected from the session/context which fetched it, will result in this [requirement for] 'DIY changetracking' code. And for what? Why isn't this solved properly? Beats me... you'd think that after all these years, Microsoft would come up with a framework for data access / entity management which would really help the developer with taking care of the plumbing crap and let the developer focus on what really matters: the application code.

Although Frans' post is directed primary to the Entity Framework, his complaint about loss of original values in n-tier applications and the requirement for developers to roll their own value tracking code echo's Anders' complaint.

Do-it-yourself change tracking becomes more practical when you compare timestamp values, as Frans observes in this comment. Timestamp and primary key values don't change (unless your table has a composite primary key), so you can use these values as original values in most cases. If you choose the timestamp approach, you must make a roundtrip to the database to compare current values with database values to determine which member values have pending changes. However, database values don't necessarily correspond to original values at the time editing began.

The LINQ to SQL Team's Proposed Change Tracking Solution

I found it strange that Anders' 4/9/2007 post hadn't elicited a response from members of the LINQ to SQL team by 4/12/2007. It turned out that the thread had moved to an earlier and somewhat related update existing disconnected objects (dlinq) thread that started on 2/1/2007. Microsoft's Matt Warren delivered this initial response on 4/11/2007:

The reconnecting API's you have available to you at this time do not support the scenario you guys are discussing. There is more work coming that makes 3-tier scenario's easier. The re-attach and playback API's are included because they are the bare minimum required to make any reconnecting work at all, however cumbersome it may seem at this time. There is an upcoming feature that automatically databinds LINQ to SQL objects directly to webforms that use these API's to communicate changes back into the DataContext. There is also a technology in the works that automatically serializes LINQ to SQL objects to other tiers, change tracks and data binds the objects for you on that tier and serializes the objects back, solving the cyclic reference serialization problem and the indescribable schema problem for objects with change sets; which seems to match what many of you are describing. [Emphasis added.]

Matt's subsequent message clarifies the technology in the works:

The mechanism that does the change tracking on the client is similar to a mini-connectionless DataContext. It is a type that packages up all the objects, lists of objects and graphs that you want to send to the client. It serializes itself and everything you've given it automatically. (It implements IXmlSerializable.) On the client, the same class also manages change-tracking for all the objects that were serialized with it. When serialized again (on the trip back) it serializes both the objects and their change information that it logged. Back on the middle tier, you just make one call to re-attach the whole package to a new DataContext instance and then call SubmitChanges.

The remaining question is how the Entity Framework folks plan to deal with this issue.

Another concurrency-related issue is the necessity to store and test the values of all related objects in the graph when undetected modifications to dependent objects might have serious medical or economic consequences. I'll reserve that issue for a future round of the Entity Change Tracking in LINQ to SQL and the Entity Framework quartet.

Updated 4/12/2007: Fixed errors due to uploading a draft instead of the final version of the initial post and added responses from Matt Warren in the LINQ Project General forum.