The Wallace Line

Weather Data on the Web

2009-05-24T17:03:00.003+01:00

In preparation for our cruise up to Scotland this summer, I'm setting up some SMS services so I can get weather reports on board, provided we're in mobile phone range. This is based on the two-way service rented from Clickatell. I recently rewrote a PHP/MySQL router which routes MO calls to an application based on the first word of the message, and returns the reply, if any, to the originator. Much simpler in XQuery because the routing table is now just a simple XML configuration file and the XQuery code is much cleaner.

So far I've written services to get the UK shipping forecast, the UK inshore waters forecast and the latest weather conditions at weather buoys. Each has presented different challenges to acquire the raw data, both technical and legal. In the domain of weather information at least we seem a very long way from an integrated, easy to use, web of data.

First the inshore waters forecast. The only publicly available format is this web page. The Met Office does provide a few RSS feeds but none for shipping forecasts. This web page looks technically promising for analysis even if I'm unsure of the legal status of this act. I'd like to know how the Met Office is funded currently but failed to discover from a Google quick search. I'd like to know the extent to which this is 'Our Data' and despite the Met Office legal notices and Freedom of Information pages, I'm none the wiser really. I console myself with the fact that I'm only playing with no intention to produce a commercial service in competition with the Met Offices own services.

The Inshore waters page looks promising, with sections for each area split into meaningful headings. However on closer inspection the page suffers from that increasingly common bane of the scrapper, a complex mixture of data and JavaScript. The page appearance is the result of JavaScript processing of bland text. Here is the raw forecast for my bit of the coast:

Lands End to St Davids Head including the Bristol Channel

24 hour forecast:
Variable 3 or 4.
Slight becoming moderate later in southwest.
Fair.
Moderate or good.

Outlook:
Variable 3 or 4, becoming west or northwest 4 or 5, occasionally 6 later.
Slight or moderate.
Thundery rain or showers.
Moderate or good.

Well now. Firstly, this is not all the data in the displayed section; the time span and the strong winds warning(if any) are elsewhere in the HTML. The nice sections are not there: instead the four parts of the forecast separated by fullstops - so the last sentence 'Moderate or good' is the Visibility. Second, the limits of the areas are identified by place identifiers in the maplet, but these do not appear in the text, and only the full area name can be used to identify. Of course, the ardent scraper can cope with this. I've been forced to add my own area ids however to support the SMS interface:

Lands End to St Davids Head

But it's horrible, unstable and makes me wonder if this design is a form of obfuscation. I suppose if they wanted to, they could switch randomly between different HTML/JavaScript layers generating the same appearance and then scrappers would be really stuffed - thankfully that seems not be be the case.

Next stop, the shipping forecast. In this case the forecast text is not on the page at all but in a generated JavaScript file which defines JavaScript arrays and their values. In an way that's simpler because I just have to fetch the JavaScript source and parse it. This application and its design is described in detail in the XQuery Wikibook.

Over in the States, their freedom of information creates a very different data climate, and NOAA provides a wonderful array of RSS and XML feeds. However, reusing even this data is not without its problems. One set of feeds I want to tap into are the data from weather buoys around the world. Many are operated by NOAA and others by local Met services or commercial operations. The UK coverage shows the locations and identifiers for UK station and there is an RSS feed of the current conditions at a buoy. The nearest up-weather buoy to Bristol is 62303, Pembroke Buoy. Well this is certainly easily accessible and valid RSS - but ... all the useful data is CDATA text in the description element:


May 24, 2009 0700 UTC

        Location: 51.603N 5.1W

        Wind Direction: SE (140°)

        Wind Speed: 5 knots

        Significant Wave Height: 3 ft

        Atmospheric Pressure: 30.14 in (1020.8 mb)

        Pressure Tendency: +0.03 in (+1.0 mb)

        Air Temperature: 51°F (10.8°C)

        Dew Point: 49°F (9.3°C)

        Water Temperature: 52°F (11.1°C)

So to separate this into meaningful data with semantic markup requires string parsing to extract the data, conversion to standard formats (the date for example) and markup in some XML Schema. Again XQuery can do the analysis. Here is the Pembroke Buoy current data. The data is augmented with some additional derived data, the wind strength on the Beaufort scale.

Of course it would be better to use existing XML schemas or RDF vocabularies than invent my own. However there doesn't seem to be anything in use which fits the bill There is some work on XML schemas for research data interchange but nothing for simple observations and forecasts that I could find. Perhaps the most comprehensive set of element names on which to build is to be found in the NOAA observation XML feeds such as this for Central Park, New York. This is a prime example of how data could be provided and its delightfully simple use makes it a good candidate for student exercises. In this format, both formatted strings and atomic values are provided. In contrast to the use of an attribute for unit, the element name is a concatenation of measurement name and unit, which seems somewhat problematic to me. The data has an attached XML schema but curiously the data is not valid according to this schema. Instead of omitting missing or undefined values, as the schema requires, the text NA is used instead. I emailed the office responsible for this data and was informed that they decided to do this because they got too many enquiries about missing data so they added the NA to make it clear it was really missing! There certainly seems to be a genuine problem there for users who don't read the schema, but my follow-up question as to why, in that case, they didn't change the schema went unanswered.

Weather data represents a case where the domain is generally understood and of interest, large quantities of data are being generated and the data is of critical importance to many users, making it an ideal case study in the web of data for my students. Despite widespread discussion of XML and RDF standards, practical data mashups must rely on hand-coded scrapping, home-build vocabularies and data extracted on dodgy legal grounds. Surely we can do better.

Twitter Radio

2009-05-13T19:56:00.003+01:00

Thought I'd try to get my XQuery Twitter Radio application going to listen to the tweets from the Mark Logic conference. It's only a simple script, requires Opera with Voice enabled and uses http-equiv="refresh" to refresh the page. It only works if the window is active, so it rather limits my use of the computer - just need another to run the radio I guess. If I wasn't marking, I'd write an AJAX-based version. XHTML+Voice is quite tricky to get right however.

Twitter Radio on #mluc09

I rather like the idea of following the Mark Logic conference with an eXist-based mashup - perhaps we should organise an eXist conference in Bristol - with my part-time status next academic year, perhaps I should put some effort into an event in Bristol.

Matching sequences in XQuery

2009-05-06T07:56:00.004+01:00

Collation is a core algorithm in processing sequences. In XQuery, the straight-forward expression of the algorithm is as a recursive function:



declare function local:merge($a, $b  as item()*) 
        as item()* {
    if (empty($a) and empty($b))
    then ()
    else if (empty ($b) or $a[1] lt $b[1])
    then  ($a[1], local:merge(subsequence($a, 2), $b))
    else  if (empty($a) or $a[1] gt $b[1])
    then  ($b[1], local:merge($a, subsequence($b,2)))           
    else (: matched :)
             ($a[1], $b[1],  
              local:merge(subsequence($a,2),
              subsequence($b,2)))
   };

Coincidently, Dan McCreary was writing an article in the XQuery wikibook on matching sequences using iteration over one sequence and indexing into the second. The task is to locate missing items. Collation is one approach to this task, albeit requiring that the sequences are in order.

Here is a test suite comparing three methods of sequence comparison.

I also did some volume tests with two sequences differing by a single, central value. Here are the tests on a sequence of 500 items. In summary, the timings are :

* Iteration with lookup: ~~6984 ms~~ - not repeatable - average is 2600
* Iteration with qualified expression: 1399 ms
* Recursive collate: 166 ms

The collate result is surprising and rather impressive. Well done eXist!

More XQuery performance tests

2009-05-01T17:10:00.003+01:00

I noticed this morning that Dan had added an alternative implementation to an article in the XQuery Wikibook on matching words against a list. It got me wondering which implementation was preferable. I wrote a few tests and was surprised at the result. My initial implementation based on element comparisons was five times slower than comparing with a sequence of atoms, and Dan's suggestion of using a qualified expression was worse still.

Here is the test run and the Wikibook article.

XQuery Unit Tests

2009-04-27T21:09:00.002+01:00

I had a fright last week - Wolfgang asked for a copy of the test harness I'd used to evaluate different implementations of a lookup table. This is code I wrote some time ago, tinkered with, good enough for our internal use but ... well pretty bad code.

I have to confess here that as a lone XQuery programmer, my code doesn't get the level of critique it needs. The Wikibook has been disappointing in that regard: I've published thousands of lines of code there and there has not been a single criticism or improvement posted. Typos in the descriptions are occasionally corrected by helpful souls, graffiti erased by others but as a forum for honing coding skills - forget it. In my task as project leader on our FOLD project (now coming to an end), I see and review lots of my students' code as well as the code Dan McCreary contributes to the WikiBook so I do quite a bit of reviewing. However I am only too conscious of the lacunae in my XQuery knowledge which perhaps through over kindness or because everyone is so busy, remain for too long. I'm envious of my agile friends who have been pair-programming for years. Perhaps there should be a site to match up lonely programmers for occasional pairing.

Anyway the test suite got a bit of work on it one day last week and its looking a bit better.

Here is a sample test script . As a test script to test the test runner it has the unusual property that some failed tests are good since failing is what's being tested. Here is it running.
Here is another, used to test the lookup implementations and one to test the geodesy functions.

Version 1 of the test runner executed tests and generated a report in parallel. A set of tests may have a common set of modules to import, prefix and suffix code. For each test, modules are dynamically loaded, the code concatenated and then evaled inside a catch.



let $extendedCode := concat($test/../prolog,$test/code,$test/../epilog)
let $output :=  util:catch("*",util:eval($extendedCode),Compile error)

The output is compared with a number of expected values. Comparison may be string-based, element-based, substring present or absent. (I also need to add numerical comparison with defined tolerance.) A test must meet all expectations to pass.

To get a summary of the results requires either running the sequence of tests recursively or constructing the test results as a constructed element and then analysing the results. Recursion would be suitable for a simple sum of passes and fails, but it closely binds the analysis to the testing. An intermediate document decouples testing from reporting, thus providing greater flexibility in the analysis but requiring temporary documents.

So version 2 constructed a sequence of test results, and then merged these results with the original test set to generate the report. Collating two sequences is a common idiom which in functional languages must either recurse over both, or iterate over one sequence whilst indexing into the other, or iterate over a extracted common key and index into both. The reporting is currently done in XQuery but it should be possible to use XSLT. Either the collating would need to be done before the XSLT step or XSLT would have the collating task. Not a happy situation.

So last week in comes version 3. Now the step which executes the tests augments each test with new attributes (pass, timing) and elements (output) and similarly each expectation with the results of its evaluation so that one single, enhanced document is produced, with the same schema as the original [the augmented data has to be optional anyway since some tests may be tagged to be ignored]. Transformation of the complete document to HTML is then straightforward either in line,in a pipeline with XQuery or XSLT. The same transformation can be run on the un-executed test set.

Augmenting the test set is slightly harder in XQuery than it would be in XSLT. For example, after executing each test, the augmented test is recreated with:



 element test  { 
       $test/@*,
       attribute pass {$pass},
       attribute timems {$timems},
       $test/(* except expected),
       element output {$output},      
       $expectedResults

This approach means that, once again, handling the construction of temporary documents is a key requirement for XQUery applications.

But I'm still not quite happy with version 3. As so often I'm struggling with namespaces in the test scripts - now where's my pair?

Implementing a table look-up in XQuery

2009-04-19T00:27:00.007+01:00

Handling temporary XML fragments in the eXist XML db has improved markedly in version 1.3. I have been looking again at an example of processing MusicXML documents which I first wrote up in the XQuery wikibook.

The code requires a translation from the note name (A, B) to the midi note value for each note. The pitch of a note is defined by a structure like:


  <pitch>
     <step>C</step>
     <alter>1</alter>
     <octave>3</octave>
 </pitch>

One approach is to use an if -then -else construct:


declare function local:MidiNote($thispitch as element(pitch) ) as xs:integer
{
  let $step := $thispitch/step
  let $alter :=
    if (empty($thispitch/alter)) then 0
    else xs:integer($thispitch/alter)
  let $octave := xs:integer($thispitch/octave)
  let $pitchstep :=
    if ($step = "C") then 0
    else if ($step = "D") then 2
    else if ($step = "E") then 4
    else if ($step = "F") then 5
    else if ($step = "G") then 7
    else if ($step = "A") then 9
    else if ($step = "B") then 11
    else 0
  return 12 * ($octave + 1) + $pitchstep + $alter
} ;

but this cries out for a table lookup as a sequence:


declare variable  $noteStep := 
(
   <note name="C" step="0"/>,
   <note name="D"  step="2"/>,
   <note name="E" step="4"/>,
   <note name="F" step="5"/>,
   <note name="G" step="7"/>,
   <note name="A" step="9"/>,
   <note name="B" step="11"/>
);

declare function local:MidiNote($thispitch as element(pitch) ) as xs:integer
{
  let $alter := xs:integer(($thispitch/alter,0)[1])
  let $octave := xs:integer($thispitch/octave)
  let $pitchstep := xs:integer($noteStep[@name = $thispitch/step]/@step)
  return 12 * ($octave + 1) + $pitchstep + $alter
} ;

or an XML element:


declare variable $noteStep := 
<steps>
   <note name="C" step="0"/>
   <note name="D" step="2"/>
   <note name="E" step="4"/>
   <note name="F" step="5"/>
   <note name="G" step="7"/>
   <note name="A" step="9"/>
   <note name="B" step="11"/>
</steps>;

declare function local:MidiNote($thispitch as element(pitch) ) as xs:integer
{
  let $alter := xs:integer(($thispitch/alter,0)[1])
  let $octave := xs:integer($thispitch/octave)
  let $pitchstep := xs:integer($noteStep/note[@name = $thispitch/step]/@step)
  return 12 * ($octave + 1) + $pitchstep + $alter
} ;

We could also store the table in the database since it is constant.

eXist does some optimisation of XPath expressions, but it does not factor out the invariant expression $thispitch/step
in the XPath predicate.

I wrote a test suite to time these various implementations. Typically this shows that factoring the sub-expression reduces the execution time by 25%. However, even with this optimisation, the structure lookup is disappointingly slow. It is about 50% slower than the if/then expression when stored on disk, and 100% slower when in memory.

This aspect of XQuery performance is important if XQuery is to be used for general application development since data structures such as indexed and associative arrays have to be represented as sequences of atomic values or elements. This performance is not really surprising and there may be more performance to be gained by indexing the data base element.

XQuery module for Geodesy

2009-04-08T07:06:00.004+01:00

I wrote my first attempt at Mercator ~ Latitude/Longitude conversion functions about 2 years ago when working in a case study for SPA2007. Part of this was documented in the XQuery Wikibook article on UK bus stops and Ordnance Survey coordinates. At the time I did not appreciate why my coordinates were just a bit off but fudged the difference. Last month I used the same functions to map pedal cycle data but as gardens and roofs appeared to be much more dangerous than roads, I thought I'd better try harder and learnt about Helmert Transformations.

My latest attempt is now available in the XQuery Examples Google Project and the Wikibook article has been revised to use this module. The formulae come mainly from the OS Guide to coordinate systems. PHP Code on Barry Hunter's http://www.nearby.org site was also useful.The test suite for this module is still being added to. I have struggled with rounding, needed to get positions with sensible resolutions and for testing and I'm not yet happy. Some tests need visual inspection and there is a problem with heights.

The module depends on the eXist math module, a prime candidate for cross-implementation standardization by the EXQuery initiative. In the latest version (v1-3) of the module, the math:atan2() function has parameters in the correct order (y,x) but older releases had these parameters reversed,as in v1-2.

The design of the module uses elements with attributes in the same namespace as the module to define compound structures such as LatLongs, Ellipsoids and Projections. These are defined in an associated schema. Compile-time checking in eXist is limited to checking the element name since eXist is not schema-aware although full schema-awareness would be of benefit in this module.

Suggestions for additions to this module are most welcome, as of course is any review of the code.

Dashboards and Widgets in XQuery

2009-04-05T07:04:00.002+01:00

Jim Fuller's recent article on Dashboards in XQuery makes a very good case for using XQuery for mashups generally. Jim's dashboard application reminded me of work I had been doing with my students last term on a configurable web page containing widgets to display NOAA weather data, RSS feeds, Google Maps and their own choice of data source. For this we used PHP with Simple XML, but to demonstrate the power of XQuery, I needed to show them the same application built on XQuery. It also seemed to me that the business dashboard would benefit from a design which split widget code from company-specific data.

The basic architecture of the approach here is to create a set of generalised widgets as XQuery functions and to define the specifics of the widget in a configuration file.

Here are a couple of configurations : jim.xml which is based on Jim's Dashboard example

and dsa.xml which is based on our course weather display.

The second example has a page refresh rate set, but I'm working on making the refresh rate widget- rather than page- specific using AJAX.

In the demo interface code, each widget links to its XQuery source code and hence to the code of called functions. Widget executions are timed and there is a link to the configuration file itself. Here is a basic main script:


import module namespace widgets = "http://www.cems.uwe.ac.uk/xmlwiki/widgets" at "widgets.xqm";

declare option exist:serialize "method=xhtml media-type=text/html omit-xml-declaration=no indent=yes 
        doctype-public=-//W3C//DTD XHTML 1.0 Transitional//EN
        doctype-system=http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd";

let $configuri := request:get-parameter("config",())
let $config := doc($configuri)/config
return 
<html>
    <head>
        {$config/title}
        <link rel="stylesheet" href="{$config/@css}" type="text/css"/>
   </head>
   <body>
        <h1>{$config/title/text()} </h1>
         {
         for $section in $config/section
         let $component := $section/*
         return 
          <div>
                 <h3>{$section/@title/string()} </h3>
                  {widgets:render($section)}
          </div>
          }
   </body>
</html>

Run it

There are a couple of ideas used in the code.

Each widget is defined by its own section in the configuration file. As a simple example, to configure the widget to get the date and time:


<section column="c2" title="Time in Bristol">
       <datetime>
            <format>EE dd/MM HH:mm</format>
        </datetime>
</section>

The main script processes the configuration file, and each widget is rendered by calling a function of the same name. The code for this dispatching currently uses the eXist function util:eval(), within a util:catch call, to implement late binding:



declare function widgets:render($section as element(section)) as element(div) {
(:  dispatch section to the matching function   :)
     let $component := $section/*[1]
     let $widget := local-name($component)
     let $function := concat("widgets:", $widget, "($component)")
     return   
       <div>
              {util:catch( "*", util:eval($function), <span>Missing or bad widget.</span>) }
       </div>
};

A safer alternative would be to use typeswitch :


declare function widgets:render2($section as element(section)) as element(div) {
    let $component := $section/*[1]
    return
        typeswitch ($component)
            case element(datatime) return widgets:datetime($component)
            case element(SQL) return widgets:SQL($component)
            case element(monitor) return widgets:monitor($component)
            ....
            default return
                <div> Missing widget {local-name($component)} </div>
};

but this needs updating every time a new widget is added to the module.

To help with processing table data from different sources such as SQL, Excel and Google Spreadsheets, support functions transform these to a common XML structure. This standard form is then transformed to its final HTML with the XSLT included in the configuration.

For example here is the configuration for an SQL widget, reading data from a (readonly) MySQL database:


<section column="c2" title="Overpaid executives">
        <SQL>
            <username>cwstudent</username>
            <password>cwstudent</password>
            <database>jdbc:mysql://stocks.cems.uwe.ac.uk/Emp</database>
            <query>select ename,sal from emp where sal > 2000 order by sal desc</query>
            <xsl>
                <xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="2.0">
                    <xsl:template match="table">
                        <table border="1">
                            <xsl:apply-templates select="row"/>
                        </table>
                    </xsl:template>
                    <xsl:template match="row">
                        <tr>
                            <td>
                                <xsl:value-of select="ename"/>
                            </td>
                            <td>
                                <xsl:value-of select="sal"/>
                            </td>
                        </tr>
                    </xsl:template>
                </xsl:stylesheet>
            </xsl>
        </SQL>
    </section>

and its widget code:


declare function widgets:SQL($component as element(SQL)) as element (div) {
 <div>
    {
       let $connection := sql:get-connection("com.mysql.jdbc.Driver", $component/database, $component/username, $component/password) 
       let $result :=   sql:execute($connection, $component/query, false())
       let $table:=  widgets:sql-result-to-table($result)
       return transform:transform($table,$component/xsl/*,())
     }
 </div>
 };

widgets:sql-result-to-table converts the sql result to a common internal XML format:


 declare function widgets:sql-result-to-table($result as element(sql:result)) as element(table) {
<table>
  {for $row in $result/sql:row
    return 
       <row>
       {for $col in $row/sql:field
       return
           element {$col/@name}
                {string($col)}
       }
       </row>
    }
</table>
};

This simplifies the XSLT transformation to the final HTML div element.

There's more to do on this framework, including adding support for access to passworded Google SS, Calendar and RSS feeds for which Jim's code will be handy. The framework also needs to support widgets from other modules and I'm not sure about the intermediate XML format and I'd like to use this more widely to layer the weather widgets as well. Getting the right split between widget code and configuration data is always tricky of course. However this framework seems quite useful and I intend to revisit some of the WikiBook examples to restructure as widgets.

I'll post the full code to the XQuery Wikibook shortly.

Parameterised MS Word Documents with XQuery

2009-04-01T21:27:00.007+01:00

It's coming round to exam time again at UWE, Bristol and as usual I've been struggling to get mine written. The XQuery-based FOLD application (which supports staff and students in our School) generates exam front pages contain exam details such as module code and title, examination date, length and time as HTML which had to be copied (poorly) into MS Word. This wasn't very satisfactory and it would be better to generate a Word document with the completed front page and sample pages with headers and footers. I'd put this off as it seemed too complicated. The Word XML format wordml is one route but it looked daunting to generate for scratch.

However for this application I only need to make some small edits to a base document. The most obvious approach was to 'parameterise' the Word document with place-holders. Unique place-holders can be edited in with Word before the document is saved as XML. Fields which are not editable in MS Word, such as the author and timestamps can be parameterised by editing the wordml directly. To instantiate a new Word document, the place-holders in the wordml are replaced with their values.

Treating this as string replacement is easier than editing the XML directly, even if this was possible in XQuery. The XQuery script reads the wordml document, serializes the XML as a string, replaces the placeholders in the string with their values and then converts back to XML for output.

Although this is not a typical task for XQuery and would be written in a similar way in other scripting languages, it is possible in XQuery with the help of a pair of functions which should be part of a common XQuery function library. In eXist these are util:serialize() to convert from XML to a string and the inverse, util:parse().

The function needs to replace multiple strings so we use a an XML element to define the name/value pairs:


let $moduleCode := request:get-parameter("moduleCode",())
..
let $replacement :=
<replacement>
 <replace string="F_ModuleCode" value="{$moduleCode}"/>
 <replace string="F_Title" value="{$title}"/>
 <replace string="F_LastAuthor" value="FOLD"/>
 ..
</replacement>

and a recursive function to do the replacements:


declare function local:replace($string,$replacements) {
if (empty($replacements))
then $string
else
  let $replace := $replacements[1]
  let $rstring := replace($string,string($replace/@string),string($replace/@value))
  return
       local:replace($rstring,subsequence($replacements,2))
};

After gathering the parameter values and formatting a replacement element, the new document is generated by:


let $template := doc("/db/FOLD/doc/examtemplate.xml")
let $stemplate := util:serialize($template,"method=xml")
let $mtemplate := local:replace($stemplate,$replaceStrings/*)
return
 util:parse($mtemplate)

Here the generated wordml is displayed in the browser, from where it can be saved, then loaded into Word. I found out the directive at the front of the wordml:


<?mso-application progid="Word.Document"?>

is used by the Windows OS to associate the file with MS Word so the media type is just the standard text/xml. However it is helpful to define a suitable default file name using a function in eXist's HTTP response module, the pair to the request module used to access URL parameters:


let $dummy := response:set-header('Content-Disposition', concat('attachment;filename=',concat("Exam_",$moduleCode,".xml") ))
let $dummy := response:set-header('Content-Type','application/msword')

The document could also be saved directly to the database, or all documents generated eagerly ready for use.

This approach feels like a bit of a hack, but it took only an hour to develop and is a major improvement on the previous approach. Changes to the base document will need re-parameterisation, but that seems a small overhead for slowly changing standard documents. XQuery forces a recursive approach to the string replacements where an iterative updating approach would avoid copying the (rather large) string, but performance is fast enough for this task, indeed in eXist string handling is very fast. My MS Office users will be happier but I still need to think about the Unix and Mac OS users.

Review of IBM developerWorks article by Brian Carey

2009-04-01T09:56:00.006+01:00

I've just come across an article published by IBM's developerWorks "Use XQuery for the presentation layer" by Brian Carey. This illustrates the value of storing complex data in XML form and using XQuery to select and transform to HTML. Whilst the main message is well-presented, the implementation, below layers of Java, is over-complicated in a couple of ways.

Brian makes the curious observation under the heading Using XQuery prevents "cheating" that "You cannot write business logic in XQuery because XQuery focuses exclusively on querying and transformation". This rather ignores the fact that a significant number of applications are built soley on XQuery as the server-side language. The consequence is that a simple Web application retains a complex and unnnecessary Java middle tier acting as Controller in an MVC architecture.

Brian's web application provides a browsing interface to a collection of products using AJAX to update the page and XQuery to select producats and transfrom to an HTML table.

Implemented in XQuery on the open source eXist XML database as an example, we need only use the HTTP interface functions provided to couple the HTTP requests directly to the XQuery script. For eXist the additions would be:


declare variable $docName as xs:string := "lures.xml";
declare variable $configuration as xs:string := request:get-parameter("configuration",());
declare variable $usage as xs:string:= request:get-parameter("usage",());

It might be objected that this script binds the resources and interfaces too closely to the script. Indeed the only benefit of the Java controller layer is this de-coupling. We can achieve the same effect in XQuery with a local configuration file to aid portability and testing:

<ConfigurationFile>
   <lureFileName>/db/Wiki/BC/lures2.xml</lureFileName>
</ConfigurationFile>

and adding these lines to the XQuery script:


declare variable $config := /ConfigurationFile;
declare variable $docName := $config/lureFileName;

I've implemented Brian's code with eXist on the XQuery Wikibook server and the relevant scripts are here:

http://www.cems.uwe.ac.uk/~cjwallac/apps/BC/index.html

I changing only the script address in the HTML code and correcting an error in the AJAX function where request.readyState was miss-typed. (took me a while to track that down!). Middle layer all gone. Storage of the whole application in eXist would be a normal deployment but was not possible without editing because the HTML page is not valid XHTML .

One impediment to the use of XQuery as an application development language is that functions to extend the XPath function with functionality such as HTTP interfacing are implementation-dependent, limiting portability. A new inititive EXQuery seeks to remedy this problem by developing a cross-platform library.

One other feature of Brian's implementation is the structure of the XML file. The start of the file looks like

<lures>
<casting>
<minnows brand="Yohuri" style="deep" size="3">
<minnow color="midnight blue">

But since the data is intended to be searched for usage(e.g. casting) and configuration (e.g. minnow) this leads to XQuery code like



if ($configuration = 'minnow' and $usage = 'casting') then
for $minnows in doc($docName)//casting/minnows
return

else
else if ($configuration = 'minnow' and $usage = 'trolling') then

..

and because the child structures are actually all the same, this leads to unmaintableable and repetative code.

A fix is possible in XQuery using local-name to filter using the node names themselves. A better approach would be to flatten the whole file, changing the representation of the configuration and usage concepts from elements to attributes:

<lures>
   <lure brand="Yohuri" style="deep" size="3" color="midnight blue" usage="casting" configuration="minnow">

..

and the query becomes


  for $lure in doc($docName)//lure[@configuration = $configuration][@usage=$usage]
  return ..

In summary, this implementation might be a "cheat" but cheating by reducing complexity and lines of code seems a good thing to do.

Data Normalization

2009-02-23T10:47:00.004Z

My lastest teaching program is for model inference from un-normalized data. This had its inception in a PHP tool written some years ago. The new version uses my ER XML schema to integrate the output with the other data modelling tools.

Normalisation is usually taught on data base courses via the abstract concepts of first, second third normal and higher normal forms. In my introductory module I just want to get over the basic idea of reduction in data duplication through the actual factorisation of a first normal-form table (atomic values) into a set of related third normal-form tables.

Here is the tool, written of course in XQuery:

http://www.cems.uwe.ac.uk/xmlwiki/ER/normalizer.xq

I get the students to take a nominated data set, factorise it, generate the SQL table declaration and INSERT statements, load them into a MySQL database and then reconstruct the original table using a select statement joining all the tables. This allows the student to check that the factorisation is loss-less but of course it does not check that it is optimal. At present the tool allows the student to explore different possibilities and create any factorisation they like.

The state of the factorisation is the current dataset URI and the current factorisation, defined by an ER model. Currently these are passed between client and server in the URL. This limits the size of the model and i guess I should change to POST but the interface behavior will not be as clean (the back button works as a simple Undo) and I can't have simple generated links on buttons. I guess I need help in interface programming here.

For the record, the code is 600 lines split into 27 functions and using two functions in the larger er module to transform the XML model to DDL and an ER diagram. Code by request until I find a suitable home.

Data Modelling Tutor

2009-02-06T07:39:00.003Z

The SQL tutor is now in use and seems to be finding favour with students and other tutors. There is a long list of things to add, like the ability to discuss an exercise but the course moves on and now I want to apply the same ideas to the teaching of data modelling. Students often find this rather difficult.

For the past few years we have used an excellent case tool called QSEE, developed by Mark Dixon at Leeds Metropolitan University. We have mainly used this multi-diagram tool for the ER diagrams. QSEE supports conceptual ER models and handles foreign keys, link tables and week entities when generating the SQL DDL. I have a running battle with some other tutors over the use of conceptual ER diagrams versus Relational ER Diagrams, complete with foreign keys and link tables. In my multi-paradigm teaching, conceptual models which treat the later as artefacts of a relational data model makes more sense. Of course I'd like to see a few improvements but sadly development of this tool seems to have ceased. Pity that it hasn't been open sourced.

My teaching emphases the difference between a model and its various representations as diagrams, as text and as code. Since we have already studied XML, it is natural to think of representing the conceptual data model as an XML document and writing the transformations in XSLT or XQuery. Having used Graphviz for a number of years the XML can be transformed to the dot notation to create diagrams in different diagraming conventions. Moreover the goal of providing an interactive data modeling tutor seems more easily realised by processing textual descriptions.

So this weekend, snowed in from work on the boat, I've been working on this tutor and data modelling tool. The really hard part has been to write the model comparator so that differences between a student model and the 'master' model can be detected and explained. This has to take account of variations in the names the student may use as well as differences in order of definition, so a straight XML diff isn't enough. What I have now is not quite right but it will have to be good enough if I want to get this tutor out to students this week.

So here is the link to the index of worksheets so far written:

http://www.cems.uwe.ac.uk/xmlwiki/ER/index.xq?tag=ER

The transformations currently supported include Chen diagrams, ER diagrams with or without attributes, SQL DDL, SQL DDL with foreign keys and many-many resolution, and a rather clunky English text representation.

One feature which was unplanned and just emerged as an obvious addition, was the ability to provide a default model template so the student could solve initial problems by filling in the blanks rather than starting with a blank page.

There is still a lot to do, but I'm pleased to have got the prototype of the ground - a long-held idea finally coming to fruition - made possible by the power of XQuery and the eXist XML database, for which I give thanks to Wolfgang and and the guys every day.

2009-01-23T22:57:00.004Z

Some years ago I wrote an SQL workbook which was used on a couple of courses, but although it had an interactive interface so that that a student could test their SQL statements against the example database, the results were not checked. I planned to create an interactive site which would present tasks to the student, accept the SQL statement input, execute the statement and compare the result with that of the model answer. Since I was teaching PHP/MySQL at the time, in good dog-fooding tradition, I started on an implementation using this platform, but it got sticky and stalled. Then I discovered XQuery and the other day, wrote an XQuery/ eXist implementation.

http://www.cems.uwe.ac.uk/xmlwiki/SQLTutor/help.html

[I'm really supposed to be marking but I find my most creative streak when marking's about - the root perhaps of the love/hate relationship I have with my job.]

The relative ease with which this version was created well illustrates the power of the XQuery/ Native XML database development approach. This application lies in the sweet spot for this technology and here is why I think that is:

XML handles Composition

Each worksheet is represented by an XML document. The document describes properties of the worksheet - database used, title, tags and each of the steps in the worksheet. It is too weak to say that steps are part of the worksheet, they -are- the worksheet. A normalised relational implementation requires one table for the worksheet properties , another for the steps and a foreign key to link these two types together. This artifical separation into master and child records complicates the model and its processing. A symptom of the need for a composition can be found in the naming problem - what do you call the master table - worksheet perhaps? But that's not right - in domain terms a worksheet is the master record AND its children as a single entity. worksheetBody? - yeech.

XML handles order
The Steps in a worksheet are ordered. To represent this in SQL requires an additional sequence number . Then editing to insert and delete steps requires re-numbering. In XML, order is implicit in the document order of the steps.

XML handles heterogeneity
Steps in a worksheet are different types. Some are simple explanations, some are demonstrations of SQL constructs, many are exercises and others have yet to be designed. A relational approach would either coerce all types into a common structure, with lots of null fields, or use multiple tables, one for each type, and a super-type table. [Object-relational databases support a more transparent implementation but who uses those?]. In XML, different types of Step can be freely interleaved in the step sequence.

XML handles multi-valued attributes
As with most of my applications these days, I use tags to support searching and classifying resources. In a normalised relational database, I should break these out into a separate table with a foreign key, but would probably fudge the problem by putting all tags into an arbitrarily concatenated field. In XML tags are simply repeated elements with no temptation to step outside the data model.

XML supports documents
Worksheets are standalone entities, which are authored, edited, deployed, removed as units. In a relational approach, all worksheets would be stored in common database tables, and the domain concept of worksheet as a unit is lost. This is particularly a problem in development: in the SQL tutor, worksheets may either be located in the XML database but may also be accessed from anywhere on the web - it makes little difference to the script providing the interactive interface. So new worksheets can be tested before installation with ease. They can also be exchanged, encouraging reuse and standardisation.

XML Documents are human-editable
My PHP/MySQL prototype stalled on the need to develop an interactive editor for worksheets in additon to the interactive tutor. In XML this component is not required, provided of course that the worksheet author is happy to author XML. If not, there are generic editors available to assist.

XQuery handles XML natively.
Inputs - the script, the SQL table returned by the SQL interface and the XHTML conent of some step properties - and the output - XHTML pages - are all XML structures. A language which handles XML natively does not have perform the re-structuring required in , say, PHP before it can be processed [PHP- XML interfaces have made this somewhat less difficult].

XQuery handles comples processing.
XQuery is much more than the XML equivalent of SQL .The SQL tutor has the task of testing whether two results are functionally the same. Two tables are the same if the column headings are different, but the data the same, or if columns are in different orders, or if rows in an SELECT statement with no ORDER BY clause are in a different order. The sequence and element structures of the XML data model combined with the expressiveness of XQuery make this computation straightforward to code.

A Native XML database supports development
I struggle with the design of the Worksheet schema. Minor additions occur all the time as my ideas develop, but since eXist is schema-independent, I can make these changes bottom-up before I need to formalise them as a schema. In a relational database I'm forced to work top-down, schema first. Moreover I feel safe that if I have a major redesign, an XSLT transformation will do the necessary schema evolution of existing Worksheets.

Conclusion
This case study demonstrates that a relational database is a poor technology for implementing a tutorial about SQL. This asymetry is not the case for XQuery, and the next project is to modify the tutor to do the same for XQuery expressions.

The tutor goes into use with 200 students next week although the worksheets themselves could do with more revision. Since the tutor logs all input, it will be interesting to be able to see what kinds of error are made, and then be able to adjust the hinting accordingly. Wish me luck!

Listen to Twitter

2008-11-02T23:40:00.003Z

Finding myself working but wanting to know how Lewis Hamilton was getting on, I wondered if Twitter would be able to let me know. I was looking for interesting feeds for the students, so I knocked up a bit of XQuery to fetch the atom feed for a Twitter search and turn that into Voice+XML for use with Opera. Works pretty well even if it is rather unsophisticated and uses page refresh rather than AJAX. The script uses the md5 hash of the last tweet spoken to know what new tweets there are. I plan to have this running on Tuesday in the lecture. One problem is that it only works if the Opera window is active so I can't have it running in the background. However the main problem is that tweets don't indicate the language so a lot of very poor, and probably disappointed Portugese is being tweeted now on the Hamilton stream.

http://www.cems.uwe.ac.uk/xmlwiki/Twitter/vsearch.xq?search=Lewis+Hamilton&seconds=30

Wikipedia Categories for Posters

2008-10-26T18:49:00.004Z

I'd planned to extend the Alphabet maker into a site that assisted Charlie to find appropriate names by inducing the category of terms and then either warning about names not in the category, or correcting spelling based on names in that category, or even suggesting a name for a missing letter.

First I thought I should understand the categories available in dbPedia and started with the Wikipedia categories using the skos vocabulary . I wrote a small skos-based browser:

http://www.cems.uwe.ac.uk/xmlwiki/RDF/skosbrowse.xq?action=help

This has two pages: a category page showing the category, the list of resources in that category with broader and narrower categories and a resource page showing the English abstract and the Wikipedia thumbnail if there is one.

From a category, you can link to a gallery of all thumbnails for resources in that category, and hence to a random Alphabet poster based on that category. There is a significant proportion of dead links among the thumbnails however and I need to look-ahead to exclude them.

One feature of this application which I haven't seen elsewhere (I live a sheltered life!) is the use of key-bindings to perform common searches on selected text. Text selected in the abstract can, with one key-stoke, link to Wikipedia, Google, Google Maps or Google Images. I like the idea of giving more control to the user over what is linked, and I have implemented this on my prototype presentation software which I'm trialling on a couple of courses to see if students find this useful.

Browsing around dbPedia using Wikipedia categories and foaf:depiction is not without its problems. For example the category Amphibians includes:

common names of amphibians - Cave Salamander
species of amphibians - Gerobactrus
groups, families and orders of Amphibians - Oreolalax
parts of amphibians - Vocal Sac
lists of amphibians - List of all Texas amphibians
lists of related subjects - Federal Inventory of Amphibian Spawning Areas

This puts me in mind of Borges' invention of a Chinese classification of animals. Aren't categories like "suckling pigs" and "those that from a long way off look like flies" just delicious? However, erhaps a subject's other categories might help but there is no "List" category for example, so no way to disambiguate the various usages of a category.

foaf:depiction has a similar problem. The Modern Painters category shows a equal mixture of depictions of the painter and depictions of works by the painter, with a few depictions of where the artist lived. This is particularly confusing when the image is a portrait! However, these categories are much cleaner than others, if somewhat incomplete.

It has often been observed that tools based on dbPedia should help to improve Wikipedia. For example it is clear that the Painters by Nationality
should not have any Painter resources, so it would be nice to use this interface to edit the categories of the two errant painters directly from an interface like this.

Alphabet Poster

2008-10-22T23:35:00.005+01:00

Grandson Charlie (age nearly 6) rang the other night to tell me the animals he had found for the animal alphabet we had discussed the previous night. I thought it would be a neat present to make a program to create a poster by fetching images from the web for each of his words and lay it out as a poster. I like the idea of writing programs as gifts, but Charlie would prefer something real- like a climbing wall!

I thought of using Flickr, or Google images, then settled on using Wikipedia, searched via dbpedia.

There are generally two images included in the dbpedia data - foaf:img - a full size JPEG image and foaf:depiction a GIF thumbnail. The thumbnails are fine for this job.

The SPARQL query to get the thumbnail for an image is rather simple:

PREFIX : <http://dbpedia.org/resource/>
PREFIX foaf: <http://xmlns.com/foaf/0.1/>
SELECT * WHERE {
:Hedgehog foaf:depiction ?img.
}

The XQuery script parses the list of words and for each word, uses this query to get the uri of the wikipedia image. The trickiest part was laying out the poster. I struggled to do the gallery layout in CSS alone but could not get this to work with an image + caption. In the end I reverted to a table layout with a width parameter.

The functional XQuery requires the layout to be done in two stages: first generate the table cells in the right, sorted order. Then compute the number of rows required for the given number of columns and generate the table, indexing into the cell sequence to layout the cells in order. In an imperative language, or a language which did not require that every constructed element was well-formed, the two task can be merged. The approach necessitated by the functional language feels cleaner but I'd prefer to write this as a pipeline: sort the words > generate the image cells > layout the table without the need to resolve the structure clash (a Jackson Structured Programming term) between the image order and the table order via a random access into a sequence. The co-routines in Python would make a better solution I feel. XML Pipelines might help but they feel too heavyweight for this minor task.

Charlies Animals so far.

The XQuery Script is in the Wikibook

RDF Vocab work

2008-09-22T16:30:00.002+01:00

I'm off to Oxford to learn about RDF Vocabularies at the Oxford Vocamp.

My own meanderings in this field have been limited to a rather hacked Vocabulary Browser written in XQuery:

http://www.cems.uwe.ac.uk/xmlwiki/Vocab/index.xq

and my rather limited attempts to provide an RDF extract from the FOLD Information System.

https://www.cems.uwe.ac.uk/wiki/index.php/FOLD/RDF

with a current dump of the RDF

http://www.cems.uwe.ac.uk/~cjwallac/FOLD/all2008.rdf

SPARQLing Country Calling Codes

2008-03-01T09:08:00.002Z

Stimulated by Henry Story's blog entry, I wrote the equivalent in XQuery, and in doing so, bumped into some issues with the dbpedia data. In particular, there is no category I could find to identify a country, but then what constitutes a country depends on what the geographical entity is classified for, so this is to be expected.

In the end I resorted to scraping the wikipedia page which lists the codes directly.

Wikibook module

XQuery SMS service

2008-02-28T08:25:00.003Z

I've recently resurrected our two-way SMS service for use by my students in their current coursework, a site to gather and report results for their chosen team sport. I require an exotic interface to the data, for example a speech interface with Opera or an SMS interface. In my SMS installation, the first word in an in-coming message is used to determine the service to which the message is to be routed via HTTP, and the reply if any is then sent via our out-bound service to the originating phone. The framework was originally implemented in PHP, but individual services can be in any language. There are a number of mainly demonstration services implemented. XQuery is used to implement a decoder for UK vehicle license numbers. This is also a nice example of the use of regular expressions. By comparison with the original PHP script, the XQuery version is both cleaner and more general. However there is no regexp function in XQuery which returns the matched groups in an expression, so this is perhaps bodged with a wrapper around the XSLT2 analyze-string function.

http://en.wikibooks.org/wiki/XQuery/String_Analysis#SMS_service

RDF /Sparql with XQuery

2008-02-13T17:18:00.003Z

As part of my learning about RDF, Sparql and the semantic web, I thought I would take the familiar employee/department/salary grade example which I used in the XQuery/SQL comparison as a case study. To this end I wrote two XQuery scripts:

XML to RDF - a script using a generic function, guided by a map , to translate flat XML tables to RDF and RDFS
Sparql query interface - an XQuery interface to a Joseki Sparql service to allow the user to execute Sparql queries against the emp-dept RDF

This is documented in an article in the wikibook.

http://en.wikibooks.org/wiki/XQuery/RDF_and_the_Emp-Dept_case_study

AJAX, AHAH and XQuery

2008-01-14T22:38:00.000Z

Today [well some days ago now, this item got stuck in draft] , I came across the abbreviation AHAH to refer to the style of using AJAX to request XHTML fragments to be inserted into an HTML page. The example of XQuery and AJAX to search employee data in the wikibook used this pattern - like the gentleman in Molière's play, I had been speaking AHAH all these years without realising it .

I also happened on an item in Mark McLaren's blog in which he describes the use of this pattern to provide an incremental search of the chemical elements. He advocates using a JavaScript library such as script.aculo.us but I'm not sure this library is warranted for a simple task like this (tempting fate here I fear). For teaching purposes, minimal code is best I feel. So I implemented a version using XQuery and minimal JavaScript.

XQuery and AHAH make a pretty good pair I think.

GoogleChart API and sparklines

2008-01-12T19:14:00.000Z

As a long-time fan of Edward Tufte's work, I've often wanted to make use of his sparkline idea, but haven't come across a suitable tool to make them. Now the GoogleChart API can generate these and a plethora of other chart types via a web service.

Here is an XQuery script to demo the interface, using the character-based simple encoding of the data:

I have one small problem - I don't know how to get rid of the axes.

Later

I've just discovered the undocumented chart type lfi so the sparkline can be shown without the axes - I found out from Brain Suda's blog

AIS

2008-01-10T21:24:00.000Z

As we start to think about the equipment we need aboard Aremiti, the Westerly ketch we are currently re-fitting, one new item that is on our shopping list is AIS.

All vessels over 300 tons and passenger vessels over 100 tons are required to carry an AIS transmitter. This broadcasts vessel data such as identification, location, speed and course on a VHF frequency. This is picked up by shore or vessel-based receivers and decoded into NMEA sentences. The data can then be used to map the vessel on a electronic chart or radar or combined with a receiving vessel's own location and course, in collision avoidance. AIS data may also be broadcast by or on behalf of static navigational aids like lighthouses and buoys.

There are a number of manufacturers of AIS 'engines' (receiver/decoders) : NASA (misleadingly called a 'radar' system) and KATAS; and software such as Shiplotter.

Since the setup cost for an amateur shore station is minimal, anyone with line of sight of a busy stretch of water can set up their own. Some publish the results on the web.
A site which I came across tonight, http://www.aisliverpool.co.uk/index.php
is a wonderful example of what a enthusiastic web engineer can do with this data. No longer is that ship in the distance a grey blob - it's a vessel with a name, a speed, a destination, a closeup when mashed up with images from this site or http://www.vesseltracker.com/en/Home.html.
and possibly a story, a history of visits and voyages. In a small boat, that data broadcast to all and sundry could be life-or-death information to you. That distant blob on an apparent collision course is no longer anonymous, routeless and inhuman. If you are still uncertain about the ships intentions, it's so much less confusing to call up a vessel by name than some vague lat/long and bearing.

All this depends on the global unique, stable IMO number, introduced to improve the safety of shipping. On the web, it is this identifier which is the basis on any semantic web data and tools to bring this information together.

The problem for both the above sites is to garner a modicum of funds to support the engineer's passion. One key question for the semantic web is how to reward them for making their deep pot of information available as RDF. It would seem so wrong to scrape their pages, tempting though it is.

More XQuery and Semantic web mashups.

2008-01-07T20:14:00.000Z

Somewhat rested after a short, breezy holiday in Falmouth , with the server now working, I completed my two case studies of XQuery /DBpedia mashups. Both are described in the XQuery Wikibook. The implementation is still a bit hairy, but now makes use of the SPARQL Query XML Result format, although I still find it useful to transform to tuples with named elements.

The first is the mapping of the birth places of football players by club. [Wikibook]

The starting page is an index of clubs in the top English and Scottish leagues:

http://www.cems.uwe.ac.uk/xmlwiki/RDF/clubIndex.xq

The second shows the discography of rock artists and groups, shown as an HTML table and using SIMILE timeline. [Wikibook].

The starting page is an index of artists in a selected Wikipedia category, by default the Rock and Roll Hall of Fame:

http://www.cems.uwe.ac.uk/xmlwiki/RDF/groupIndex.xq

Servers down

2007-12-31T08:49:00.000Z

Wouldn't you just know it? I no sooner blog about XQuery for Semantic web mashups than the servers in my department at the university go off-line and I guess they might not be back up now till the 2nd. About time we had eXist-db server space in the Cloud I say. I wonder if Amazon would be interested?

DBpedia and Simile Timeline

2007-12-30T15:28:00.000Z

Simile Timeline provides a neat way to display events and we have been using it on the FOLD project and on the DSA module to represent the events in the life of music artists and groups, with data extracted by hand from a copy of the Rolling Stones Review. Having discovered DBpedia, it seems obvious to progress to using this data source instead. The XQuery code is presented in an article in the Wikibook

The endpoint is

http://www.cems.uwe.ac.uk/xmlwiki/RDF/groupTimeline.xq?group=

and the parameter is the wikpedia page name (with underscores)

Some examples:

Only the album cover is displayed in the pop-up, with links to Wikipedia and DBpedia. Coverage of the minimal data required is quite good, but there are gaps, and the format of the release date varies. This is partly due to the need to encode not only the data but also the accuracy with which the date is known. Some wikipedians have used xs:gYear and xs:gYearMonth. The xs:date format, being bigendian, seems to naturally support progressive accuracy, but of course partial values like 2007-12 are not valid. In this example, I 've merely hacked a year out but this is not satisfactory.

There is also an HTML page view of the same data,with added comment text: e.g.

The Allman Brothers Band

A separate query creates an index, with links to both views, derived from a Category:

The next step is to derive a set of life events for the group and group members - births, marriages and deaths - to place on a parallel timeline.

Football Teams, DBPedia and SPARQL

2007-12-29T12:44:00.000Z

Having written a number of applications which scrape information directly from Wikipedia pages, I was delighted to discover the DBpedia project. Chris Bizer and his collegues have brought the rather nebulous ideal of the Semantic web to life with this RDF database built on extracted data from Wikipedia, browsable as linked data or queryable with SPARQL. With this resource, I see how to contrast triples and SPARQL with SQL and XML in the DSA course.

My first experiment has been to try to answer a question that has been recently prompted, in part, by the appointment of Fabio Capella as the manager of the England Football team. My godson Oliver and I were wondering just how international our club sides are and what better way to find out than to use the DBpedia data to create a map of the birthplaces of the players in a team.

The result is described in some detail in an XQuery Wikibook article.
Here for example are the players in the Bolton Wanderers team shown via GoogleMap. (you may have to refresh - there's often an initial server error ??)

Being based on an extract from Wikipedia some weeks ago, this data is not quite up-to-date, there is missing data and inconsistancies in property tagging but I couldn't do this without DBpedia - thank you guys.

Pipelines

2007-11-28T08:38:00.001Z

Looks like its going to be the year of 'Pipes' in my teaching. I came across Yahoo pipes earlier this year and decided to restructure my second-level module, Data Schemas and Applications, around the 'web as database', before diving into local database technologies like Relational and XML databases. Yahoo Pipes provides an approachable starting point and has the added benefit of being a technology which none of my students, from a diverse collections of programmes, have yet encountered. Also, those slinky pipes and typed ports are -so- seductive.

However, the visual editor soon becomes awkward to use and thus leads naturally into using a scripting language instead. I started with XSLT expecting to move quickly into XQuery, but I've been surprised to find how much can be done, especially with XSLT2.0. For a transformation engine, I've set up a service (using Saxon8 via XQuery on eXist-db). This has allowed us to implement most of the yahoo pipes we'd written and also searches over an XML file with a single script containing a form and the search results. More..

Although many of the steps in a Yahoo pipeline can be handled within a single XSLT script, some of the processing I want to demonstrate involves processing HTML pages which are not XHTML, so I needed a tidy service too, and to be able to pipeline them together.

So... I need a pipeline language, a way of visualizing the pipeline and an engine to execute the pipeline. Naturally I started to write my own, based mainly on XPL which Eric Bruchez introduced me to at XML Prague. A tentative first step using an XQuery script is described in an XQuery WikiBook article.

Of course this is fine as play but I need to join the real world of pipeline languages. I suppose the main contenders are :

XProc (W3C)
XPL (in Orbeon)
DPML (in NetKernel )

NetKernel looks theoretically and practically very interesting and what's more, 1060research are a locally-based spin-off from HP labs next door.

Topological sorting in XQuery

2007-11-24T17:59:00.000Z

In the course of my attempts to implement an XQuery engine for the XML pipeline language XPL, I realized that I would need to sort the processes so that each process has its inputs available, i.e. a topological sort.

Given a sequence of nodes and references with the Relax-NG schema:



    element node {
           attribute id { xs:string },
           element ref {
               attribute id { xs:string }
           }*
       }+

The post-condition after sorting can be defined as:


declare function local:topological-sorted($nodes) as xs:boolean {
   every $n in $nodes satisfies
         every $id in $n/ref/@id
                satisfies $id = $n/preceding::node/@id
};

and the recursive function to order the nodes is:



declare function local:topological-sort($unordered, $ordered )   {
   if (empty($unordered))
   then $ordered
   else
       let $nodes := $unordered [ every $id in ref/@id satisfies $id = $ordered/@id]
       return local:topological-sort( $unordered except $nodes, ($ordered, $nodes ))
};

Sweet, eh? Even if this implementation is not the most efficient, it has the advantage of being obviously correct.

See (and improve!) the XQuery WikiBook article.

the Prime Sieve in XQuery

2007-11-18T18:05:00.000Z

I guess I've been putting my spare effort over the past few months into the Wikibook on XQuery, so the blog has been very neglected.

I've learnt a lot in writing the example code in the Wikibook. The other day I bumped in the Euler Project and started on the first few the problems. Problem 3 is about primes so I had to write a prime number generator. The Sieve of Eratosthenes in XQuery is so simple and obvious:


declare function local:sieve($primes,$nums) {
  if (exists($nums))
  then 
       let $prime :=  $nums[1]
       return local:sieve(($primes,$prime), $nums[. mod $prime !=  0])
  else $primes
};

local:sieve((),2 to 1000)

The list of primes starts off empty, the list of numbers starts off with the integers. Each recursive call of local:sieve takes the first of the remaining integers as a new prime and reduces the list of integers to those not divisible by the prime. When the list of integers is exhausted, the list of primes is returned.

Lovely but sadly not very practical for the size of numbers in the problem.

Discussion and execution are in the Wikibook.

Web 2.0 in teaching

2007-06-14T16:49:00.000+01:00

This year, I've used a few of the Web 2.0 tools to enliven my courses. This year, these techniques were used on two modules, DSA and IAD. Since I also teach web technologies, it seems appropriate that students gain some experience of these tools as part of the courses.

This is a summary of the value my students and I have found in some of them.

Background

The standard VLE at UWE is BlackBoard (UWEOnline). There are a few difficulties with the way it is administered here, particularly around the restricted access to material. Another problem is the lack of ability to link to modules or documents to UWEOnline, limiting the possibilities for integration.. Nonetheless, this the vehicle students expect to be used. Of course in this subject area much of the material is based on server languages like PHP, MySQL, and XQuery and thus resides on our faculty server.

Blogs

Blogs were used in two ways - as a means of communicating from teachers to students, with student commenting (DSA and IAD), and as a component of each students work.

Tutor blogs

The blogs are all hosted on Google's Blogger. During the year this improved in ease of use an in the ability to tag any item. Thus all items on lectures, or coursework 1 or PHP can be tagged and then browsed as a group.

Over the year there were over 50 posts to each blog. The tutors on the modules all had write access. Activity on the blogs was monitored by Google Analytics and showed the expected increase in traffic around lecture time and towards coursework hand-ins. Since the blogs were public,there was a background of hits from all over the world, and some items, such as one on the Periodic Table of Visualisation methods were linked to and gained considerable traffic. Comments on both blogs were few, from students and the public, but there were some.

I also occasionally post to my personal blog (here)and it was sometimes unclear where an item should be posted. I also tried to run one for the new 1 -year programme in Internet Application Development but this did not take off .

Lessons:

Students prefer a consistent interface to learning materials, although they also valued the narrative structure of the blog. However there are difficulties in using it to organise information which must be easily findable, even with the tagging facility. Thus, with some reluctance, I have decided to revert to putting lecture notes on BlackBoard rather than my own web site. BlackBoard will also contain the workplan. I plan to use my own blog to add my own commentary and additional material, with items tagged for consumption by one or other module even though the blog will only be able to refer to items in BlackBoard by name.

Student blogs

Students on IAD were required to keep individual blogs as a record of their reading over they year. This was not successful, partly as a result of concern for the damage to reputation that poor quality web appearances could have on future prospects. In this regard, it may be preferable to use the private blogs in BlackBoard for this purpose.

Wikis

I have used wiki's for some time, and one was used for web2.0 technologies. It is doubtful that this was beneficial, with so much excellent material on wikipedia. However for details of implementations and tools which are specific to CEMS, there is a need for a local wiki. This was established as the CEMS wiki and is being populated, though much more could be done before it becomes the first palce of enquiry about the use of a language, tool or technique in CEMS.

I also set up a student wiki in CEMS for use by students in organising and presenting their research into individual topics. This was quite successful, both as an opportunity to gain experience in wikis and as a means of organising their material. However, the student entries were more in the nature of online articles, with a low level of linkage. Students were required to hand-in a copy of their wiki material (since this is needed for external examiners) and this perhaps led to the lack of a linked structure. A more collaborative, group-based approach might be tried to get a larger network but there are obvious problems with attributing authorship.

Google docs

On one module,the tutors kept a register of attendance. We opted to use the recently available GoogleSpreadsheets for this, partly to explore the value of collaborative documents. Technically, this was very successful, and a big improvement on passing Excel spreadsheets around. However (there seem to be rather a lot of 'however's in this account :-( ) the spreadsheet feels rather cramped and lacks the ability to hold row and column headings in place when scrolling. (freeze panes)

RSS

An advantage of blogs is that they provide an RSS feed which can then be aggregated with other feeds. (unlike BlackBoard which does not) An attempt was made to use this feed to populate the announcements in Blackboard which was only partly successful, creating an additional point of failure. Take-up of RSS is still slow but the new myUWE portal supports RSS feeds so it will be suggested that the blog feed can be added by the individual student.

Podcasts

A couple of lectures were recorded on audio and posted to the blog. I also used Evoca to record short messages.
Whether these initiatives were valued is difficult to say. I think the audio alone is of limited value and integrated slideshow, video and audio is the ideal. Creating such resources for small to medium-sized classes, in such as rapidly changing field as the web, may not be economically viable.

Conclusion

These new technologies have great promise, but the fit with a corporate culture an institutional E-Learning policy creates tensions. This year's experiences are certain food for thought.

XQuery functions

2007-06-09T08:16:00.001+01:00

Over the last few weeks, I've been writing some general XQuery functions for our own system and in collaboration with Dan McCreary. There are XQuery collections to which these could be contributed, notably Priscilla Warmsley's //www.xqueryfunctions.com/xq/
but some of my functions depend on the eXist function libraries, and I would also like to prove the functions work and show examples of use in unit tests.

I'm currently working on some system tools to browse the code, develop full coverage tests and help refactor the code for the FOLD application. Part of this suite is an XQuery unit test tool. As an experiment, I've combined the two and put up a simple demo on our public eXist demo server.

The function is defined in the prolog which is prepended to each test before execution. The result can be either text or a node which is compared with the expected result as strings or with deep-equal respectively.

The test scripts themselves, viewed as XML look a bit of a mess but the source shows the formatting. CDATA sections are now preserved in stored files in eXist which is convenient when round-tripping XQuery code containing XML, but is controversial since CDATA is only intended to encode XML characters on input but is not part of the infomodel.

Links

CommonCode
The test script for a function Dan wrote
The test script for a BandMap structure

Next step is to document the test framework itself and provide the means for others to add test scripts.

SPA2007 - day 0

2007-03-25T17:57:00.000+01:00

I tinker endlessly with the eXist workshop material, still some solutions to do and the new server isn't having anything to do with the Java Webstart client. Hopefully it will get fixed before Wednesday afternoon. I'll also have to figure how to create multiple folders with the same initial contents - should be able to use a backup but not sure how to install in a different place.

FizzBuzz

2007-03-19T07:39:00.000Z

Here's a quick XQuery solution to the FizzBuzz problem posed in David Patterson's blog.

I've taken the liberty of splitting the hyphenated string into two attributes.


let $config :=
<fizzbuzz>
<range min="1" max="100"/>
<test>
   <mod value="3" test="0">Fizz</mod>
   <mod value="5" test="0">Buzz</mod>
</test>
</fizzbuzz>

for $i in ($config/range/@min to $config/range/@max)
let $s :=  for $mod in $config/test/mod
      return
         if ($i mod $mod/@value = $mod/@test)
         then string($mod)
         else ()
return
if (exists($s))
then string-join($s,' ')
else $i

Timelines

2007-03-15T00:06:00.000Z

Here are some ways in which timelines are being used in the web:

TimeSearch
GoogleEarth timeline
HyperHistory
SIMILE timeline from MIT

Dinosaur timeline

Provides a Javascript API to create a complex panning timeline.
Events are uploaded in XML format which defines for each event

start title end? isDuration? latestStart? earliestEnd? image? link? and body

Cant locate schema for the event stream
Event date format is not xs:date
body must be a string i.e. with < not <
Set the mime type to application/xml
example XML timeline

I've implemented the SIMILE timeline in my Family History. Here is an example
Links to other people load their timelines (although some are incomplete and not working).

Multiple event streams can be displayed in multiple bands - e.g. here are the lives of two family members

It would be great to get a subset of world history events from TimeSearch to include as a separate band.

To do:

Convert xs:date to Timeline format
Handle photos with missing dates
Focus timeline when no date of birth

My Family History

2007-03-14T23:52:00.000Z

Bamber Gascoigne's TimeSearch got me thinking about my family history project again.

The Family History project started with the idea of putting some family photos on the web, together with some meta data about the photo. Most had a list of subjects but only very occassionally a date. I had the idea that if the birth dates of the subjects were added, and the age of just one subject could be guessed, the system could infer the date of the photograph, and hence the age of all subjects. With birth and death data included, a timeline could be extracted for a person, including birth, deaths and marriages, photographs and even world events.

The resultant eXist/XQuery/XSLT prototype is here.

What would be nice would be be able to combine such personal histories with events in world history, or to compare the timelines for famous people with that of a family member. A simplistic approach would be to deep link into history sites and this is what the prototype has done, creating links like

These generated links are not as clever as those in TimeSearch, which I would guess are hand edited.

This is simple to accomplish, but its rather one-side - my site can link to another, but it can't mashup the data with my own. A mashup requires an API and a published format for exported content. I suppose we might look at hcalendar as one possible format but it would need extending it to support tags. RSS is another possibility.

TimeSearch by Bamber Gascoigne

2007-03-14T17:09:00.000Z

The item on the BBC's Start the Week' by Bamber Gascoigne on the history site he has created called TimeSearch sparked my interest TimeSearch is a history search portal, using event stubs to provide links onwards to a wide range of sites - from general news sites, Wikipedia and Google images to specialist on-line resources. Events can be selected using two hierarchical category systems, location and them as well as year. I confess I had some trouble deselecting categories once selected.

This is the latest form of delivery of the extensive material Bamber has created and put on line in:

History World

An online encyclopedia of history with great little quizes.

OCEAN index

One wonders how this authored material compares with the collaborative Wikipedia. It is certainly an impressive and rich resource.

SPA 2007

2007-02-13T22:32:00.000Z

I have been working on my session for this year's SPA for a while now. Seems a timely session to run according to Elliotte Rusty Harold's predictions for XML in 2007 (which mentions eXist).

Mostly I have been vacillating about which case study to use - I have so many part-finished projects. At first I revived the Whisky project, but after a lot of work, this felt like a bit of a slog - it was good as a modeling exercise but didn't reveal the power of XML and XQuery to my satisfaction. I then switched to the Bus Timetable project, which is much more realistic and one which I want to solve to replace the existing PHP/MySQL application with its hand-crafted timetables. However the data here is very complex, and the data files I have so far are incomplete. Nevertheless, this is the case study I settled on, although I did a short detour into the DVD hire example and of course there is always StudentsOnline, the main site we are developing with this technology.

Timetables come from TravelLine South-West in the form of TransXChange files. These first need processing to a more amenable structure. The actual data model is highly interlinked which makes for some XPath fun. Conversion is required to calculate absolute times (TXC has time differences between stops) and to convert from OS Easting and Northing to Lat/Long . With the simplified structures, I can present the data in various ways. HTML pages show the stops and the departures times but the neat idea is to generate kml files to overlay on Google Earth. Using a NetworkLink, I can display changing times on the map. The XPath date and time functions are a great help.

Other bits of functionality include an Ajaxian incremental Bus stop locator and membership registration rretainig favourite stops and services.

Here are some useful links for this project:

eXist database home
an XQuery tutorial from Howard Katz
A basic Timetable implementation (pending)
TransXChange documentation
TravelLine South West
Google Earth
kml description and reference

Computed synethesia

2007-01-29T14:02:00.000Z

The Cloud appreciation society lead me to the Cloud Harp project. I'm listening to a recording made in 2004 - very ethereal.

It re-generates my interest in artificial synaesthesia generally and perhaps a return to my Bristol Harbour project, using image stream and image processing to pick out the frequency and direction of wave patterns to modulate music, with self-contained stations set up around the harbour.

Information arts often has artificial synaesthesia at its heart, converting information in one medium to information in another, with the aim of thus enhancing our appreciation and understanding of the source domain and creating beauty and interest in the target domain.

Cloud Appreciation Society

2007-01-28T19:01:00.000Z

One of my Christmas presents was 'The Cloudspotter's Guide' by Gavin Pretor-Pinney and published in 2006 by Hodder & Stoughton on behalf of the cloud appreciation society.

This site includes an excellent example of a subject-specific photo site, with many excellent photographs of cloud formations, optical effects and associated phenomena. The site was interesting in the context of the DSA module because it combines the problem of creating a photo site with the problems of tagging and categorising information, so I thought I'd write a review of it here, as perhaps as an example of what might have been included in theFlickr review. Next year I think I will base the coursework on a selection of sites like this rather than a monster site like Flickr. I welcome suggestions.

Navigation

The home page is a page featuring the last photo to be loaded.

Search by category and by partial string match within the title string are provided on this page. These lead to two different album pages - a paged layout in the case of the category, a list with thumbnails and titles in front of the page from which the search was launched in the case of the title search. Its not clear why there should be two different approaches. Usefully, the category selector shows the number of photographs in that category.

Direct links to the categories to which the current image has been assigned are provided and a featured category (Cloud Lookalikes) is at the top.

Each page includes a selection of other photographs. The criteria appears to be just the adjacent photo accession numbers and is not based on any criteria of similarity, but nevertheless it does encourage serendipitous browsing. Curiously the aspect ratio of these photos is determined by that of the main photo, leading to mal-shaped images.

Clicking on the photo rather surprisingly links to the next photo in sequence. There is also a 'Previous Page' link which is functionally a back button - it is not quite clear why this is included since this is a standard browser button. Perhaps Next and Previous links (in the accession sequence) would provide a better navigational mechanism.

A search (or link) to a members photos would be useful.

Photo Data

Images are jpegs at medium and thumbnail resolution.

Meta data include the name of the copyright owner (or is it just the member?), a descriptive title, data and time (but of upload I would guess, not of the photograph itself). The title can contain links e.g. to related sites.

Each photograph is related to several categories.

It is a pity that the photographs do not appear to be geo-coded, because it would be nice to mash these up with Google Earth. There is often a description of the place which could be translated through a geo-coding service. Since some photos are taken from planes, altitude data is needed too.

Tagging and classification
Each photograph is classified into one or several of around 40 categories and hence linked to other photos in the same category through that category. The category system itself is interesting as a information construct. On the surface, it would seem that it would benefit from some hierarchy. e.g.

* All photos
** Cloud Type
*** Cumulus
**** Kelvin-Helmholtz wave cloud
***
** Cloud features
*** Contrails
***
** Time-of-Day
*** Sunrise
*** Sunset
** Optical Effects
*** Rainbow
*** Halo
***
and probably others.

It also seems that some category meta-data is embedded in the name - the numbers on Cloud types is a reference to the Chapters in the book. Clearly a general description of the category itself could also be part of the overall data structure. The book provides a hierarchical classification of cloud types which would be useful to include. There are also occasional synonyms embedded in the name -eg Mamma (also know as mammatus) which could be treated uniformly as category meta-data.

The task of classifying each photo into these categories is not open to the public or members but is seen as an expert task. However there are places, such as in the category of 'Clouds that look like things' where folksomony would seem to be appropriate. Photographs also sometimes contain objects e.g. balloons, planes, boats for which arbitrary tags seem suitable.

Upload
Rather surprisingly, photo upload, even by members, is not supported, and photos have to be emailed to the webmaster with accompanying meta data for upload and categorisation.

Rating and comments
The public can easily rate photos and add comments. The comments are generally expressions of appreciation of no interest to anyone but the photographer. There is no information on the basis of the rating - indeed all photos I've seen are rated 4.It is not clear what value this adds or indeed what is being rated - the photograph or the subject.

Compatibility and Accessibility
Images have alt tags but these are the copyright names, not the description of the image which would better aid the reader.

There are keywords in the page meta data but these are not photo specific.

XHTML compliance - main photo page shows 1 error and 17 warnings. Typically these are problems with table and div tag nesting and problems with & . The error is a non-HTML tag.

The display of some characters in the title is broken (on Firefox and IE) e.g this

API and Feeds
The site provides RSS2.0 and Atom feeds of the latest photo additions and a Feedburner link.

Technology
The scripting language is PHP, images are held as files, not held in the database. Not sure what database is used (probably MySQL?). CSS is used for styling. Small amount of Javascript e.g. for GIF rollover on the rating buttons and call to update the rating. One function is defined (flip) which is not used.

The program architecture uses a single index.php script with a parameter to determine what page type to return, rather than multiple scripts. Interesting design issue here.

Data Model

Where I've been

2007-01-20T18:18:00.000Z

Just discovered this little application which generates a GIF map of the world showing the countries you have visited.

So many places yet to see, and so much of places visited unseen like most of the States:

From Douwe Osinga, a Google engineer

Voice snippets

2007-01-18T18:12:00.000Z

I've been experimenting today with http://www.evoca.com/

A free account allows you to record and store up to 60 minutes of sound. This interface is simple to use. Sound can be captured from a number of sources but the easiest is directly from a microphone on the computer. Everything is done through the web interface. When the recording has been made, an HTML snippet is provided to paste where you need it, and this invokes a flash player.

I'm trying these as an addition to my blog for the DSA module

I suppose it will just pollute the airwaves, but maybe it offers another mode for information dissemination which may just help some students. For two lectures last year I recorded the whole lecture and mounted it as an MP3 file, but they very long and un-synchronised with the Powerpoint slides. I tried calling out slide numbers but it was reminiscent of the early soccer commentaries which used numbered squares (the origin I learnt today of the phrase 'back to square 1') .

Whisky - Laphroaig

2007-01-15T18:08:00.001Z

This entry is a sample of data sources on a specific whisky - assembled here as a notepad for the workshop at SPA2007 and for the Data, Schemas and Applications module.

Laphroaig
Laphroaig Distillery,
Port Ellen,
Isle of Islay,
Argyll,
PA42 7DU Tel. 01496 302418

Home site

General

Types of information

Chronology
Location

Gooogle Map http://maps.google.com/maps?q=PA42%207DU

Multimap http://www.multimap.com/map/browse.cgi?pc=PA427DU&scale=25000&cat=loc
Wikipedia Geohack

Pronunication - John Butler's site
Taste http://www.whiskyclassified.com/laphroaig.html
Tasting notes e.g. Mike Padlipsky
Ownership - changing ownership - see the yearbook
Bottlings http://www.laphroaig.com/whiskies/index.asp?expanded=our_whiskies
Glossary http://www.whiskymag.com/words/

Printed

Malt Whisky Yearbook 2006 p170
Collins Gem Whisky p 171

Beyond Belief

2007-01-15T10:54:00.000Z

I spent part of this weekend listening to the presentations at BeyondBelief2006 for which all sessions over the two and a half days are available as Flash video. A great resource to have provided and very well done. Ongoing discussion on the Edge.

Amongst the speakers, I was particularly taken with

Carolyn Porco - the picture of Earth through the back-lit rings of Saturn is just wonderful.
Harold Kroto - vega science trust
Mahzarin Banaji - Implicit association tests
Ann Druyan - Carl Sagan

Amongst other thoughts provoked by these sessions, I was struck by the observation that it is scientists of all breeds who are most directly trying to understand and pay attention to 'God', whilst the religous and the theologians pay attention to the works of man - the books, the practices, the historical development of religous thought by man.

Double line spacing in Dissertations

2006-11-04T22:45:00.000Z

It is standard practice to require dissertations to be double-spaced. In the age of word processors, one wonders why this practice persists. This must surely be a legacy from the days when dissertations were typewritten and students were allowed to make corrections in situ rather than re-print. Word processors have eliminated that problem. Double-spacing might also be suitable for drafts which require annotation for editing, but final copies of dissertations are not used in this way and modern techniques such as commenting do this job.

(Primo Levi's 'Chromium' in 'The Periodic Table' has several great examples of the way solutions to old probelms hang around even when the problem not longer exists.)

I have seen readability expressed as another reason for double-spacing. It's not easy to track down the evidence for this. Kruk and Muter(1984) report "single spacing produced reading that was 10.9% slower than that produced by double spacing" on a 'video' screen. Weller (2004) reports on a previous study that "It was discovered that single spacing of text requires more eye fixations per line and therefore fewer words are read per fixation, which increases reading time (Kohler, Duchnicky & Ferguson, 1981)." But Mills and Weldon (1987) report that this same study showed a 2% slowing of reading rate, hardly convicing evidence.

Other writers express the view that this convention is out-moded e.g. the author of the Tex manual : link

Princeton University now accepts single-spaced printed copies although ProQuest (an online dissertation repository) still requires double-spacing for the electronic copy ( and hence all printed copies therefrom). It is claimed that this improved on-line readability.

A PhD Student in the US has calculated that 20,000 reams of paper would be saved if ProQuest accepted single-spaced.

Double-spacing is often mandated in legal practice but even here the convention is challenged.

Declarative Processes

2006-10-23T09:11:00.000+01:00

I am getting back to the declarative process work I began a couple of years ago. Peter Marks and Ben Moseley's session at SPA2006 got me started again.

I now have a partial implementation of a DVD rental application written in a declarative process style using eXist and XQuery. The essence of the design is to retain all events and compute the current state by the evaluation of predicates over the event trace. The rules are hard-coded in eXist functions in this demo. Rules are mostly XPath expressions. For example

To find the current address for a person $p given SetAddress events

events($p)[SetAddress][last()]//address

To find the address at $date

events($p)[SetAddress][@date < $date][last()]//address

To check if a member is suspended given Suspend and Reinstate events

exists( events($p)[Suspend or Reinstate][last()][Suspend])

Some of the advantages of this approach are clear:

the state of the system can be evaluated at any time in the past by filtering the event stream for events before the specified date
if the rules are held as data, the state can be computed synoptically (i.e. according to rules applicable at the time) - but how should they be encoded - as executable XQuery expressions?
changes in rules are supported, allowing re-interpretation of the past
subjective rules are supported although only within the event language created
'what-if' analysis is supported, allowing questions to be asked about the impact of changes in the rules
rules are robust to insertion of other events
the event history is often required to understand what has happened and why so there is no addition storage requirement

Some disadvantages are also clear:

The computational cost is higher than simply fetching a simple value
Writing declarative rules is not easy and will need a higher level language
The approach requires a global model of the system, so its applicable to systems but not to distributed systems
It's not yet clear what range of behaviour is expressable

Much more work is needed to see how well this model will fit with other processes. Some applications include:

lightweight processes - e.g. academic processes such as setting and marking coursework
descriptive models of real processes and their subsequent analysis for process improvement

Prior work

The Alloy language (Daniel Jackson, MIT)

Peter Marks and Ben Moseley - Functional Relational Programming, SPA 2007, FRP site

Work on rule-based language has had a rough ride - several projects in this area (e.g. BRML) are defunct. Some activity in R2ML and RuleML.

America

2006-08-04T20:07:00.000+01:00

Time to forget about University and the project. Tomorrow we're off to the USA for adventure and catching up with friends and relatives.

Here is a link to a GoogleEarth overlay containing the main places we are visiting:

Trip on Google Earth

Photos from the Minnesota trip are now up on Flickr: http://www.flickr.com/photos/paddlers9/

eXist examples

2006-06-04T09:06:00.000+01:00

I now have somewhere to host my example XML database examples.

http://www.cems.uwe.ac.uk/chriswallace/index.xql

This supercedes the previous hosts for my teaching applications on a friend's server and on the exist-db demo server.

There are a few there, but further development will have to wait till the end of the term. Few typos I've just spotted :- (

One problem I faced was to allow a visitor to view the Xquery files themselves. I initially saw this as a eXist problem, needing an eXist solution, and indeed Adam, one of the eXist developers, put in a feature to support this. However I slowly realised that if I wanted to put up a number of examples (there are about 9 in various stages of development) , I should develop a schema for an application index page, and individual XML configuration files instead of writing unique pages. I can then use XSLT to display the page. If I then wrote an Xquery to fetch a query and display it , I could use the application configuration XML to authorise access to the scripts.

This experience prompted or reminded me of a couple of computing adages:

The only numbers of importance in computing are 1, 2 and many- with its meta counting variant: 1, 2, Schema
Use of the configuration file to generate the web page and to authorise access to the script as an example of the Shanley principle - in which one component serves multiple purposes see Michael Jackson, Software Requirements and Specifications, Addison-Wesley 1995, p29-30].

In future, I want to enhance this configuration file to be a description of the application architecture and information flow, so I can also generate, via GraphViz, a clickable SVG image of the application.

That's the problem with eXist and XQuery: this technology greatly reduces the time needed to create an application, but it increases the range of ideas for applicatins which can be easily implemented even more!

RSS

2006-03-13T11:22:00.000Z

Discovered bbc backstage this morning whilst researching for my lecture on Web services and REST. As an experiment, I thought I'd add a page of RSS-derived links to the FOLD. I shouldn't still be surprised by this, but it only took a few lines to add a primitive page.

I've put the demo up on the exist demo site. There is also a version which generates Voice+XML to make a real newsreader. It needs Opera 8 with the speech extension and there is only a selection of two American voices but it handles the task very well.

Cellular automata and Processing

2006-03-06T06:25:00.000Z

I've just re-discovered my Processing application which implements the generalised Conway Game of Life. The interface allows you to set any rule - for each of the counts of alive neighbours, whether a dead cell lives or a live cell dies. You can also run Conway's rule or generate a random rule.

Here it is

and here is my Processing page

What it doesnt have is any way to register interesting rules the viewer may discover. I discovered a nice one today: l2d=134&d2l=157 which shows constantly varying interesting behaviour - perhaps for a screen saver. Also l2d=125&d2l=3568&id=0.1 which is a slow converger to nearly all white with a few blinkers.

So two jobs to do: One to allow finds to be registered and commented on, the other to take automata specification from the URL in a form like that above. Both seem very Web2.0 - the first enabling participation, the second a unique url for every resource.

Apart from being a exercise in writing Processing, I use it in a lecture on Processes and Emergence.

'Processing' is a poor brand name (try searching for it!) [like eXist in that regard] but a wonderful little language for animations. Developed by two design students form MIT, Ben Fry and Casey Reas, it is open-source software with a really nice little IDE, Java-based (it generates a Java applet) with some great examples of both animations and information display. A much better introduction to programming than full-on Java I think and a candidate for a first language, partly because it places emphasis on algorithms and change, not structure and stasis. Its also much simpler than Flash Actionscript as well as non-propriatory.

Here are some of my favourite Processing examples

Zipcodes by Ben Fry - locates a US Zipcode
Flight Patterns by Aaron Goblin - visualising the flights of planes in the US on a single day
Dreamlines by Leonardo Solaas - a Flickr/Processing mash-up

Family Relationships

2006-02-11T07:05:00.000Z

The family album site really needs generalized deduction. The deductions about dates and ages are hand coded:

deduce the date of the photo:

photo.deduceddate =
(photo.date,
{ some subject s | s.birthday and photo.s.age | s.birthday + photo.s.age},
)[1]

deduce the age of the subjects:

all s: subject |
s.deducedage = (photo.s.age, photo.deduceddate + s.deducedage)[1]

One neat trick would be to calculate relative relationships: if the viewer is x and the subject is s then what is s to x ( great great grand uncle or even the more verbose - your father's father's father's brother)

This calls for a Prolog engine perhaps

Possibilities include

PyLog a Prolog translater for Python
A number of Prolog engines listed here but this looks very old.

Or RDF and Jena?
[sites which are undated are just terrible - I must remember that on my own site]

Exist NDX web server

2006-02-05T09:58:00.000Z

Paul has generously provided me with space on his server to install exist in its Jetty configuration. This provides me with an external site which I can use with GoogleMaps. So far I've been putting up the family history site.

Developing on this site is much more immediate that with my typical FTP enabled site. With the Java client, I can work directly on the database , editing code, uploading scripts and photos and executing test queries. There is also a rather safer web interface. Great for rapid development. I was working on improving the XSLT and looking at the problem of aliases (maiden names usually) . I'm so used to using the oh-so-useful generalised = in Xquery that I forgot that Xalan is still on XSLT 1.0 with XPath 1.0, whereas XQuery uses XPath 2.0. This generalisation is the main difference to hit me but its so useful and a pain to have to replace with the inferior contains(). In fact I'm really struggling with alias still.

One idea is to develop this family history with my brother Richard and my nephew, Reddyn, in New Zealand, who I have sadly neglected. Here we could all work together in a very Web 2.0 collaboration. Hope they run with the idea.

More Google mashups

2006-01-20T23:01:00.000Z

More play with map-based mashups. This evening, using programmableweb's listing , I found Flash Earth, a Google Earth mashup. Here's the link to a narrow siq in the Wadi Rum area in Jordan, one of several with old inscriptions carved in the sandstone.

This memory game is a neat idea, using yahoo images to populate a layout of images to play pairs (or memory).

But my favourite is Community Walk Last night I started to create a walk around the university campus - just a single, rather bland view of our building - perhaps this is not such a good idea if we want to attract students. Later I started one for our trip to Jordan. Just three photos uploaded at present, and I'm uncertain about the location of one of these - I can''t seem to find the location of Petra. I do wish I'd keep the GPS data since I had the Garmin with me to check heights and distances, but at that time, the advantages of geo-coding photographs just hadn't occured to me. Maybe a student project to make a camera and GPS lashup. What would be really nice would be to add a fluxgate compass too, so the photo is coded with the direction in which the camera was pointing as well.

Just found another UK mashup of all UK schools It shows the location of all UK schools, locates the nearest 20 to a specified location, highlights the marker from a list of school names and their Ofsted rating (at least for Independent Schools).

Finally on this bit of research, located Barry Hunter's work on at nearby.org.uk which offers a REST interface for conversion between a range of different coordinate systems, e.g. Postcode to Lat/long. Will be very handy for the FamilyHistory mashup, although the postcode is only down to sector level - not enough accuracy for this application, and I suppose there's not much extra work in hand-coding this number of locations.

Beginner's Mashup

2006-01-17T23:55:00.000Z

I've been playing with integrating maps into my family history photo album. The idea is to place each photo in time, place and (through the people in the photo) in the web of family relationships.

The first attempt uses an XML file of photo meta-data, transformed with a simple XSLT on the client-side. This transformation is to a table of data with links to the raw jpegs, map positions (courtesy of Multimap) and years (thanks wikipedia). No relationships yet. Hardly a mashup, since the context data is linked only with URLs.

The second attempt uses Google maps. Now the map is embedded in the photo page, much closer coupling afforded by the use of Javascript. The step of including the Google Javascript source is coupled with authorisation via URL parameters- a neat application of the Shanley principle. The XSLT caused me a bit of pain here when generating the Javascript and it's still not right: the transformed HTML works fine in IE6 and if generated client-side with XALAN, but its no good on Firefox which just spins.

I now have to think how the students will use and develop this example: their server is inside our firewall and so Google won't be able to issue a key. They need a server in the DMZ for this I guess but we need that for the new Internet Applications Development module for September this year.

The title

2006-01-15T18:46:00.000Z

The title is an echo of Alfred Russel Wallace's Line through the Malay Archipeligo which he postulated to explain the differences he observed in bird and animal species. The line runs between Borneo and Sulawesi, and between Bali and Lombok, Asian species (such as tigers) to the west , Australiasian species (such as marsupials) to the east. Nowadays the theory of tectonic plates and land bridges during ice ages provides an explanation for this disjuncture.

My appropriation is hardly warranted: ARW is no relation of mine, and I'm a teacher of computing, not natural history. I have crossed the Wallace line however, when a few years ago, my wife and I circumnavigated in our yacht Perdika of Bristol. It was in fact quite a struggle. The current flow through all these straits is fierce and we crabbed our way from Lombok to Bali, pointing fully 40 degrees north of our track over land, and later when motoring northwards up the strait, finding the inshore counter-current frighteningly close to the Bali shore.

The title also reflects the nature of a blog, a time-line of writing. I'm interested in time-lines and the representation of complex event-spaces at present, partly as a domain to use in my second year course on data structures which this semester will be looking at XML and XML databases, eXist in particular. I've also started to do a bit of genealogical research into my personal 'Wallace Line' and discovered how addictive this detective work is. From census records available online at scotlandspeople.gov.uk, I find that my great-grandfather (or rather one of four of course) Ebenezer Wallace was a wine merchant in Edinburgh in the 1880's, a fact which makes my own consumption of good wines now feel like a family duty.

The first word

2006-01-15T11:11:00.001Z

I teach internet computing - therefore I must blog.