What's New in DekiScript for MindTouch Deki 9.02 "Lyons"

Statement expressions

Sometimes it's necessary to evaluate several statements in places where only an expression is allowed.  It's one of those things where you may not know what you're missing until you encounter it and then the workaround is to build some horribly complex expression to circumvent it.  However, for "Lyons" we've addressed this limitation by now allowing multiple statements to be combined into a single expression.  The syntax is reasonably intuitive too.

The following line assigns the outcome of a sequence of statements as if they were an expression:

let x = (var y = [ ]; foreach(var i in num.series(a, b)) let y ..= [ i ]; y);

To enable this capability, we allow the use of ; inside parentheses to indicate that the expression is a composition of statements.  The parentheses also create a scope for new variables, which means that the variable y will not be accessible from outside the statement expression.

The value of the statement expression is computed by accumulating all values that are produced.  In the above example, we only produce the value of y as a side-effect since all other statements return nil as a result, which is discarded by the accumulator.  If multiple values are produced, they are concatenated together into strings or XML documents.  The progression looks a bit like this:

1. Start with nil.
2. Take on value of first produced result that is different from nil.
3. If new value is not a URI and not XML, and the accumulated value is not a URI or XML, then convert both values to strings and concatenate them.
4. Otherwise, convert both values to XML and merge them (note: the XML merge rules are the same as those used by Deki when embedding XML into pages).

Using statement expressions is especially useful when working with functions that take DekiScript code, such as list.sort and map.select as more complex expressions can now be built.

Insight: Why didn't we use curly braces (i.e. { }) instead?  After all, that would have looked fairly intuit too.  The problem is that curly braces are already used by maps and would have caused unnecessary ambiguities in the parser. Using parentheses instead avoided this issue.

XML Literals

So far there have been two ways to create XML elements in DekiScript.  Either switch to HTML mode and create effectively HTML elements with DekiScript attributes or use web.html to parse HTML text into an XML document.  Both approaches are still valid and have their uses.  The former is useful to annotate content with dynamic properties, and the latter can be used to embed HTML that was obtained as a text fragment.  However, relying on the web.html function to parse a string that was built using concatenation is asking for trouble.  It's too easy to forget to XML-encode a character and that breaks the document then, but until now, there wasn't a viable alternative.

To address this need, we've added XML literals to DekiScript.  It's now possible to write XML documents directly in DekiScript without resorting to concatenation and the web.html function.

Let's look at this code to embed the title of a page with bold formatting.  This example will fail if page.title contains invalid XML characters, such as "<" or ">".

web.html("<strong>" .. page.title .. "</strong>");

Now, with XML literals, we can rewrite our code as follows:

<strong> page.title </strong>;

Not only is this code much easier to read, but the page.title value is now properly XML-encoded automatically.  Also, DekiScript can now report an error if we mispelled the closing tag for our element; something that web.html could not warn us of until it was too late.  However, with DekiScript we can also avoid this whole issue by using a shortcut notation to close tags as shown in the following code.  Now we only need to worry about closing all the elements, but not about their names.

<strong> page.title </>;

XML literals are just like other values in DekiScript, such as lists and maps.  You can assign them, you can pass them as arguments, and so forth. All the following uses are legal:

<ul>
<li> 1 </li>
<li> "2" </li>
<li> var x = 1; x + 2 </li>
</ul>

<ul>
foreach(var i in num.series(1, 5)) {
  if(i % 2 == 0) {
    <li> <strong> 'Number ' .. i </strong> </li>;
  } else {
    <li> 'Number ' .. i </li>;
  }
}
</ul>

xml.text(<xml attribute="123">456</>, "@attribute")

<a href=(page.uri) title=(page.path) >page.title</>;

xml.format(<("a".."b".."c") ("a"..1)="text">page.title</>);

Example #1 is rather boring and just shows that XML literals do pretty much what you would expect of them.  The one noteworthy thing is that contents between elements is DekiScript, not text.  This makes it simple to create XML with dynamic values.

Example #2 builds on the previous example by showing how XML literals can be nested to enable more sophisticated XML construction.

Example #3 shows an XML literal with a static attribute, while Example #4 shows a dynamic attribute.  Attributes can be followed by either a string using single or double-quotes, or by parentheses containing an expression.  One more reason to support statement expressions!  The attribute is remove if the expression evaluates to nil.  Otherwise, the expression is cast to a string and becomes the value of the attribute.

Example #5 shows how the parenthesis notation is used to give dynamic names to elements and attributes.

XML literals provide a powerful new tool for creating dynamic content in pages.  It combines the benefits of correctness enforcement by the DekiScript parser with the flexibility dynamic documents on the fly.

URI append operator

Working with URIs can be tricky.  For example, relying on string concatenation to build new URIs can cause subtle, but annoying errors.  Let's consider the following cases where we want to append something to the variable abc which holds a URI value.

We could use uri.build to avoid these issues and thus make sure we always get it right.  In the following table, we replace our previous statements, with code that is always correct.

Unfortunately, this code is also a lot more verbose and difficult to use as we have to remember what paremeter goes where.

What if we had a notation for appending something to URIs just like we have for concatenating strings?  Then life would be a lot simpler.

We did just that and defined the & operator to provide append semantics for URIs.  When using the & operator, an implicit call to uri.build is used based on the type of the argument on the righthand side of the operator.

Here are the rules for x & y, where x is always a string or URI, and

• y is a string: uri.build(x, y, _)
• y is a map: uri.build(x, _, y)

Let's revisit our examples to illustrate.

As you can see, the new & operator allows for a concise and always correct way to build new URIs.

The uri.build function still remains useful though when appending multiple segments and parameters at once, as it can do the entire operation in one invocation.  So, if efficiency is a concern, keep using uri.build when needed.

Generators

In Lyons, we've made the first step towards supporting generated sequences of values.  Generators are the generalization of the basic "var x in collection" pattern used by foreach statements. Generators can be used in foreach statements, HTML foreach attribute, dynamic lists, and dynamic maps.

Here are examples of the generator notation:

var x in collection

var x, y in collection

var x, y in collection where x > y

var x, y in collection, if x > y

var x, y in collection, var z = x + y, if z > 10

var x in collection1, var y in collection2

var x in collection1, var y in collection2 where x.id == y.id

var key : value in map

Generators are extremely powerful and versatile, making it simple to iterate over complex sequences and only introduce only one new keywords (where).

The generator notation is availabl to both foreach statement and HTML foreach attributes.

foreach statement example:

foreach(var k : v in { a: 1, b: 2, c: 3 })  {
    ...
}

HTML foreach attribute example:

<ul>
    <li foreach="var p in page.subpage where #p.tags != 0">{{ p.title }}</li>
</ul>

The ability to specify constraints directly using the where statement makes the HTML where attribute redundant.  However, the attribute continue to work for backwards compatibility.  When present, it merely becomes an additional constraint for the generator.

Dynamic List Construction

When working with lists, it's often necessary to either create a sub-selection or create a new list from an existing one.  With the introduction of statement expressions, this need has been partially addressed, but the mechanism is still cumbersome.  Other modern languages, such as Python and C# 3.0 offer list comprehension and LINQ respectively, to give developers a more natural and efficient notation for building lists.

Let's revisit the sample that was used when statement expressions were introduced and rewrite it using the new list notations. 

let x =  [ i foreach var i in num.series(a, b) ];

The new notation is much more compact and builds the list in one fell swoop.  List construction is not limited to iterating over a single collection, multiple collections can be iterated, conditions can be used to skip over elements, and even local variables can be used during the construction process.  In short, all of the new capabilities introduced by generators are available in list construction.

The notation for generating dynamic lists has two parts.  The first part is the production.  This part is the same as in previous releases and contains items that are listed in a comma-delimited fashion.  Then comes the optional foreach keyword and the generator part.

[ PRODUCTION foreach GENERATOR ]

The foreach keyword indicates that the production expression to its left must be evaluated in the context of the generator expression.  Any variables defined by the generator expression can be used in the left production.  The production must contain at least one item, but may contain more.  All items in the production are appended to the list for each iteration.  The foreach keyword must be followed by a generator expression.

Here are a few samples of list constructions:

[ c foreach var c in customers where c.City == "London" ];

[ { Name: c.Name, OrderID: o.OrderID, Total: o.Total } foreach 
    var c in customers, var o in c.Orders 
]

[ { OrderId: o.OrderID, Total: t } foreach 
    var o in orders, 
    var t = list.reduce(o.Details, "$value + $item.UnitPrice * $item.Quantity", 0),
    if t >= 1000
]

[ { Name: c.Name, OrderCount: n } foreach
    var c in customers,
    var co = [ x foreach var x in orders where c.CustomerID == x.CustomerID ]
    if #co >= 10
]

[ { Name: c.Name, OrderDate: o.OrderDate, ProductName: p.ProductName } foreach
    var c in customers,
    var o in orders where c.CustomerID == o.CustomerID,
    var d in details where o.OrderID == d.OrderID,
    var p in products where d.ProductID == p.ProductID
]

var noprimes = [ j foreach var i in num.series(2, 8), var j in num.series(i*2, 50, i) ];
var primes = [ x foreach var x in num.series(2, 50) where x not in noprimes ];

Dynamic Map Construction

Maps can now also be dynamically built, just like lists.  For example, the following expression builds a map of all tags on the current page that have an associated definition page:

let x =  { (tag.name) : tag.define foreach var tag in page.tags where tag.define is not nil };

Just like for lists, the map is built in one fell swoop.  Map construction is not limited to iterating over a single collection, multiple collections can be iterated, conditions can be used to skip over elements, and even local variables can be used during the construction process.  In short, all of the new capabilities introduced by generators are available in map construction.

The notation for generating dynamic maps has two parts.  The first part is the production.  This part is the same as in previous releases and contains items that are listed pairwise in a comma-delimited fashion.  Then comes the optional foreach keyword and the generator part.

{ PRODUCTION foreach GENERATOR }

The foreach keyword indicates that the production expression to its left must be evaluated in the context of the generator expression.  Any variables defined by the generator expression can be used in the left production.  The production must contain at least one item, but may contain more.  All items in the production are merged into the map for each iteration replacing entries with the same name, if one already existed.  Entries with nil as name are omitted.  The foreach keyword must be followed by a generator expression.

The in and not in operators

The in operator checks if the item on its left occurs inside the collection on the right.  For lists, it checks if the item is contained in the list.  For maps, it checks if the map contains an entry with the value of the item (not the key, since that can easily be checked by looking up the key in the map).

Example using the in operator:

if(x in [ 1, 2, 3 ]) {
    ...
}

Example using the not in operator:

if(x not in [ 1, 2, 3 ]) {
    ...
}

The in operator is different from list.contains in that it performs an equality (==) check instead of an identity (===) check.

The is not operator

Ok, this one is barely worth a mention, but for some the is not operator will prove a nice, aesthetic addition.  Now instead of writing:

if(!(item is list)) {
    ...
}

You can write this instead, which is a little more pleasant and easier to read:

if(item is not list) {
    ...
}

New List Functions

We added a few new list manipulation functions, which should make quite a few people happy: OrderBy, GroupBy, Reduce, and Random.

var orders = [ { Name: c.Name, OrderID: o.OrderID, Total: o.Total } foreach var c in customers, var o in c.Orders ];
let orders = list.orderby(l, "Total descending");

let orders = list.orderby(orders, [ "Total descending", "Name" ]);

var customersByCountry = list.groupby(customers, "$.Country");

var total = list.reduce(order.Details, "$value + $item.UnitPrice * $item.Quantity", 0)

var randomItem = list.random([ 1, 2, 3, 4, 5 ]);