XQuery SMS service

I've recently resurrected our two-way SMS service for use by my students in their current coursework, a site to gather and report results for their chosen team sport. I require an exotic interface to the data, for example a speech interface with Opera or an SMS interface. In my SMS installation, the first word in an in-coming message is used to determine the service to which the message is to be routed via HTTP, and the reply if any is then sent via our out-bound service to the originating phone. The framework was originally implemented in PHP, but individual services can be in any language. There are a number of mainly demonstration services implemented. XQuery is used to implement a decoder for UK vehicle license numbers. This is also a nice example of the use of regular expressions. By comparison with the original PHP script, the XQuery version is both cleaner and more general. However there is no regexp function in XQuery which returns the matched groups in an expression, so this is perhaps bodged with a wrapper around the XSLT2 analyze-string function.

http://en.wikibooks.org/wiki/XQuery/String_Analysis#SMS_service

RDF /Sparql with XQuery

As part of my learning about RDF, Sparql and the semantic web, I thought I would take the familiar employee/department/salary grade example which I used in the XQuery/SQL comparison as a case study. To this end I wrote two XQuery scripts:

• XML to RDF - a script using a generic function, guided by a map , to translate flat XML tables to RDF and RDFS
  
• Sparql query interface - an XQuery interface to a Joseki Sparql service to allow the user to execute Sparql queries against the emp-dept RDF

This is documented in an article in the wikibook.




http://en.wikibooks.org/wiki/XQuery/RDF_and_the_Emp-Dept_case_study

AJAX, AHAH and XQuery

Today [well some days ago now, this item got stuck in draft] , I came across the abbreviation AHAH to refer to the style of using AJAX to request XHTML fragments to be inserted into an HTML page. The example of XQuery and AJAX to search employee data in the wikibook used this pattern - like the gentleman in Molière's play, I had been speaking AHAH all these years without realising it .

I also happened on an item in Mark McLaren's blog in which he describes the use of this pattern to provide an incremental search of the chemical elements. He advocates using a JavaScript library such as script.aculo.us but I'm not sure this library is warranted for a simple task like this (tempting fate here I fear). For teaching purposes, minimal code is best I feel. So I implemented a version using XQuery and minimal JavaScript.

• The incremental search
• The WikiBook page

XQuery and AHAH make a pretty good pair I think.

GoogleChart API and sparklines

As a long-time fan of Edward Tufte's work, I've often wanted to make use of his sparkline idea, but haven't come across a suitable tool to make them. Now the GoogleChart API can generate these and a plethora of other chart types via a web service.

Here is an XQuery script to demo the interface, using the character-based simple encoding of the data:

• Random sparkline
• Wikibook page

I have one small problem - I don't know how to get rid of the axes.

Later

I've just discovered the undocumented chart type lfi so the sparkline can be shown without the axes - I found out from Brain Suda's blog

AIS

As we start to think about the equipment we need aboard Aremiti, the Westerly ketch we are currently re-fitting, one new item that is on our shopping list is AIS.

All vessels over 300 tons and passenger vessels over 100 tons are required to carry an AIS transmitter. This broadcasts vessel data such as identification, location, speed and course on a VHF frequency. This is picked up by shore or vessel-based receivers and decoded into NMEA sentences. The data can then be used to map the vessel on a electronic chart or radar or combined with a receiving vessel's own location and course, in collision avoidance. AIS data may also be broadcast by or on behalf of static navigational aids like lighthouses and buoys.

There are a number of manufacturers of AIS 'engines' (receiver/decoders) : NASA (misleadingly called a 'radar' system) and KATAS; and software such as Shiplotter.

Since the setup cost for an amateur shore station is minimal, anyone with line of sight of a busy stretch of water can set up their own. Some publish the results on the web.
A site which I came across tonight, http://www.aisliverpool.co.uk/index.php
is a wonderful example of what a enthusiastic web engineer can do with this data. No longer is that ship in the distance a grey blob - it's a vessel with a name, a speed, a destination, a closeup when mashed up with images from this site or http://www.vesseltracker.com/en/Home.html.
and possibly a story, a history of visits and voyages. In a small boat, that data broadcast to all and sundry could be life-or-death information to you. That distant blob on an apparent collision course is no longer anonymous, routeless and inhuman. If you are still uncertain about the ships intentions, it's so much less confusing to call up a vessel by name than some vague lat/long and bearing.

All this depends on the global unique, stable IMO number, introduced to improve the safety of shipping. On the web, it is this identifier which is the basis on any semantic web data and tools to bring this information together.

The problem for both the above sites is to garner a modicum of funds to support the engineer's passion. One key question for the semantic web is how to reward them for making their deep pot of information available as RDF. It would seem so wrong to scrape their pages, tempting though it is.

More XQuery and Semantic web mashups.

Somewhat rested after a short, breezy holiday in Falmouth , with the server now working, I completed my two case studies of XQuery /DBpedia mashups. Both are described in the XQuery Wikibook. The implementation is still a bit hairy, but now makes use of the SPARQL Query XML Result format, although I still find it useful to transform to tuples with named elements.

The first is the mapping of the birth places of football players by club. [Wikibook]

The starting page is an index of clubs in the top English and Scottish leagues:

• http://www.cems.uwe.ac.uk/xmlwiki/RDF/clubIndex.xq

The second shows the discography of rock artists and groups, shown as an HTML table and using SIMILE timeline. [Wikibook].

The starting page is an index of artists in a selected Wikipedia category, by default the Rock and Roll Hall of Fame:

• http://www.cems.uwe.ac.uk/xmlwiki/RDF/groupIndex.xq

Servers down

Wouldn't you just know it? I no sooner blog about XQuery for Semantic web mashups than the servers in my department at the university go off-line and I guess they might not be back up now till the 2nd. About time we had eXist-db server space in the Cloud I say. I wonder if Amazon would be interested?

DBpedia and Simile Timeline

Simile Timeline provides a neat way to display events and we have been using it on the FOLD project and on the DSA module to represent the events in the life of music artists and groups, with data extracted by hand from a copy of the Rolling Stones Review. Having discovered DBpedia, it seems obvious to progress to using this data source instead. The XQuery code is presented in an article in the Wikibook

The endpoint is

http://www.cems.uwe.ac.uk/xmlwiki/RDF/groupTimeline.xq?group=

and the parameter is the wikpedia page name (with underscores)

Some examples:

• Eric Clapton
• The Allman Brothers Band

Only the album cover is displayed in the pop-up, with links to Wikipedia and DBpedia. Coverage of the minimal data required is quite good, but there are gaps, and the format of the release date varies. This is partly due to the need to encode not only the data but also the accuracy with which the date is known. Some wikipedians have used xs:gYear and xs:gYearMonth. The xs:date format, being bigendian, seems to naturally support progressive accuracy, but of course partial values like 2007-12 are not valid. In this example, I 've merely hacked a year out but this is not satisfactory.

There is also an HTML page view of the same data,with added comment text: e.g.

The Allman Brothers Band

A separate query creates an index, with links to both views, derived from a Category:

• Rock and Roll Hall of Fame
  
• American Jazz Singers

The next step is to derive a set of life events for the group and group members - births, marriages and deaths - to place on a parallel timeline.

Football Teams, DBPedia and SPARQL

Having written a number of applications which scrape information directly from Wikipedia pages, I was delighted to discover the DBpedia project. Chris Bizer and his collegues have brought the rather nebulous ideal of the Semantic web to life with this RDF database built on extracted data from Wikipedia, browsable as linked data or queryable with SPARQL. With this resource, I see how to contrast triples and SPARQL with SQL and XML in the DSA course.

My first experiment has been to try to answer a question that has been recently prompted, in part, by the appointment of Fabio Capella as the manager of the England Football team. My godson Oliver and I were wondering just how international our club sides are and what better way to find out than to use the DBpedia data to create a map of the birthplaces of the players in a team.

The result is described in some detail in an XQuery Wikibook article.
Here for example are the players in the Bolton Wanderers team shown via GoogleMap. (you may have to refresh - there's often an initial server error ??)

Being based on an extract from Wikipedia some weeks ago, this data is not quite up-to-date, there is missing data and inconsistancies in property tagging but I couldn't do this without DBpedia - thank you guys.

Pipelines

Looks like its going to be the year of 'Pipes' in my teaching. I came across Yahoo pipes earlier this year and decided to restructure my second-level module, Data Schemas and Applications, around the 'web as database', before diving into local database technologies like Relational and XML databases. Yahoo Pipes provides an approachable starting point and has the added benefit of being a technology which none of my students, from a diverse collections of programmes, have yet encountered. Also, those slinky pipes and typed ports are -so- seductive.

However, the visual editor soon becomes awkward to use and thus leads naturally into using a scripting language instead. I started with XSLT expecting to move quickly into XQuery, but I've been surprised to find how much can be done, especially with XSLT2.0. For a transformation engine, I've set up a service (using Saxon8 via XQuery on eXist-db). This has allowed us to implement most of the yahoo pipes we'd written and also searches over an XML file with a single script containing a form and the search results. More..

Although many of the steps in a Yahoo pipeline can be handled within a single XSLT script, some of the processing I want to demonstrate involves processing HTML pages which are not XHTML, so I needed a tidy service too, and to be able to pipeline them together.

So... I need a pipeline language, a way of visualizing the pipeline and an engine to execute the pipeline. Naturally I started to write my own, based mainly on XPL which Eric Bruchez introduced me to at XML Prague. A tentative first step using an XQuery script is described in an XQuery WikiBook article.

Of course this is fine as play but I need to join the real world of pipeline languages. I suppose the main contenders are :

• XProc (W3C)
• XPL (in Orbeon)
• DPML (in NetKernel )

NetKernel looks theoretically and practically very interesting and what's more, 1060research are a locally-based spin-off from HP labs next door.

Topological sorting in XQuery

In the course of my attempts to implement an XQuery engine for the XML pipeline language XPL, I realized that I would need to sort the processes so that each process has its inputs available, i.e. a topological sort.

Given a sequence of nodes and references with the Relax-NG schema:

    element node {
           attribute id { xs:string },
           element ref {
               attribute id { xs:string }
           }*
       }+

The post-condition after sorting can be defined as:



declare function local:topological-sorted($nodes) as xs:boolean {
   every $n in $nodes satisfies
         every $id in $n/ref/@id
                satisfies $id = $n/preceding::node/@id
};

and the recursive function to order the nodes is:

declare function local:topological-sort($unordered, $ordered )   {
   if (empty($unordered))
   then $ordered
   else
       let $nodes := $unordered [ every $id in ref/@id satisfies $id = $ordered/@id]
       return local:topological-sort( $unordered except $nodes, ($ordered, $nodes ))
};

Sweet, eh? Even if this implementation is not the most efficient, it has the advantage of being obviously correct.

See (and improve!) the XQuery WikiBook article.

the Prime Sieve in XQuery

I guess I've been putting my spare effort over the past few months into the Wikibook on XQuery, so the blog has been very neglected.

I've learnt a lot in writing the example code in the Wikibook. The other day I bumped in the Euler Project and started on the first few the problems. Problem 3 is about primes so I had to write a prime number generator. The Sieve of Eratosthenes in XQuery is so simple and obvious:


declare function local:sieve($primes,$nums) {
if (exists($nums))
then
let $prime := $nums[1]
return local:sieve(($primes,$prime), $nums[. mod $prime != 0])
else $primes
};

local:sieve((),2 to 1000)

The list of primes starts off empty, the list of numbers starts off with the integers. Each recursive call of local:sieve takes the first of the remaining integers as a new prime and reduces the list of integers to those not divisible by the prime. When the list of integers is exhausted, the list of primes is returned.

Lovely but sadly not very practical for the size of numbers in the problem.

Discussion and execution are in the Wikibook.

Web 2.0 in teaching

This year, I've used a few of the Web 2.0 tools to enliven my courses. This year, these techniques were used on two modules, DSA and IAD. Since I also teach web technologies, it seems appropriate that students gain some experience of these tools as part of the courses.

This is a summary of the value my students and I have found in some of them.

Background

The standard VLE at UWE is BlackBoard (UWEOnline). There are a few difficulties with the way it is administered here, particularly around the restricted access to material. Another problem is the lack of ability to link to modules or documents to UWEOnline, limiting the possibilities for integration.. Nonetheless, this the vehicle students expect to be used. Of course in this subject area much of the material is based on server languages like PHP, MySQL, and XQuery and thus resides on our faculty server.

Blogs

Blogs were used in two ways - as a means of communicating from teachers to students, with student commenting (DSA and IAD), and as a component of each students work.

Tutor blogs

The blogs are all hosted on Google's Blogger. During the year this improved in ease of use an in the ability to tag any item. Thus all items on lectures, or coursework 1 or PHP can be tagged and then browsed as a group.

Over the year there were over 50 posts to each blog. The tutors on the modules all had write access. Activity on the blogs was monitored by Google Analytics and showed the expected increase in traffic around lecture time and towards coursework hand-ins. Since the blogs were public,there was a background of hits from all over the world, and some items, such as one on the Periodic Table of Visualisation methods were linked to and gained considerable traffic. Comments on both blogs were few, from students and the public, but there were some.

I also occasionally post to my personal blog (here)and it was sometimes unclear where an item should be posted. I also tried to run one for the new 1 -year programme in Internet Application Development but this did not take off .

Lessons:

Students prefer a consistent interface to learning materials, although they also valued the narrative structure of the blog. However there are difficulties in using it to organise information which must be easily findable, even with the tagging facility. Thus, with some reluctance, I have decided to revert to putting lecture notes on BlackBoard rather than my own web site. BlackBoard will also contain the workplan. I plan to use my own blog to add my own commentary and additional material, with items tagged for consumption by one or other module even though the blog will only be able to refer to items in BlackBoard by name.

Student blogs

Students on IAD were required to keep individual blogs as a record of their reading over they year. This was not successful, partly as a result of concern for the damage to reputation that poor quality web appearances could have on future prospects. In this regard, it may be preferable to use the private blogs in BlackBoard for this purpose.

Wikis

I have used wiki's for some time, and one was used for web2.0 technologies. It is doubtful that this was beneficial, with so much excellent material on wikipedia. However for details of implementations and tools which are specific to CEMS, there is a need for a local wiki. This was established as the CEMS wiki and is being populated, though much more could be done before it becomes the first palce of enquiry about the use of a language, tool or technique in CEMS.


I also set up a student wiki in CEMS for use by students in organising and presenting their research into individual topics. This was quite successful, both as an opportunity to gain experience in wikis and as a means of organising their material. However, the student entries were more in the nature of online articles, with a low level of linkage. Students were required to hand-in a copy of their wiki material (since this is needed for external examiners) and this perhaps led to the lack of a linked structure. A more collaborative, group-based approach might be tried to get a larger network but there are obvious problems with attributing authorship.

Google docs

On one module,the tutors kept a register of attendance. We opted to use the recently available GoogleSpreadsheets for this, partly to explore the value of collaborative documents. Technically, this was very successful, and a big improvement on passing Excel spreadsheets around. However (there seem to be rather a lot of 'however's in this account :-( ) the spreadsheet feels rather cramped and lacks the ability to hold row and column headings in place when scrolling. (freeze panes)

RSS

An advantage of blogs is that they provide an RSS feed which can then be aggregated with other feeds. (unlike BlackBoard which does not) An attempt was made to use this feed to populate the announcements in Blackboard which was only partly successful, creating an additional point of failure. Take-up of RSS is still slow but the new myUWE portal supports RSS feeds so it will be suggested that the blog feed can be added by the individual student.

Podcasts

A couple of lectures were recorded on audio and posted to the blog. I also used Evoca to record short messages.
Whether these initiatives were valued is difficult to say. I think the audio alone is of limited value and integrated slideshow, video and audio is the ideal. Creating such resources for small to medium-sized classes, in such as rapidly changing field as the web, may not be economically viable.

Conclusion

These new technologies have great promise, but the fit with a corporate culture an institutional E-Learning policy creates tensions. This year's experiences are certain food for thought.

XQuery functions

Over the last few weeks, I've been writing some general XQuery functions for our own system and in collaboration with Dan McCreary. There are XQuery collections to which these could be contributed, notably Priscilla Warmsley's //www.xqueryfunctions.com/xq/
but some of my functions depend on the eXist function libraries, and I would also like to prove the functions work and show examples of use in unit tests.

I'm currently working on some system tools to browse the code, develop full coverage tests and help refactor the code for the FOLD application. Part of this suite is an XQuery unit test tool. As an experiment, I've combined the two and put up a simple demo on our public eXist demo server.

The function is defined in the prolog which is prepended to each test before execution. The result can be either text or a node which is compared with the expected result as strings or with deep-equal respectively.

The test scripts themselves, viewed as XML look a bit of a mess but the source shows the formatting. CDATA sections are now preserved in stored files in eXist which is convenient when round-tripping XQuery code containing XML, but is controversial since CDATA is only intended to encode XML characters on input but is not part of the infomodel.

Links

• CommonCode
• The test script for a function Dan wrote
  
• The test script for a BandMap structure

Next step is to document the test framework itself and provide the means for others to add test scripts.

SPA2007 - day 0

I tinker endlessly with the eXist workshop material, still some solutions to do and the new server isn't having anything to do with the Java Webstart client. Hopefully it will get fixed before Wednesday afternoon. I'll also have to figure how to create multiple folders with the same initial contents - should be able to use a backup but not sure how to install in a different place.

FizzBuzz

Here's a quick XQuery solution to the FizzBuzz problem posed in David Patterson's blog.

I've taken the liberty of splitting the hyphenated string into two attributes.

let $config :=
<fizzbuzz>
<range min="1" max="100"/>
<test>
   <mod value="3" test="0">Fizz</mod>
   <mod value="5" test="0">Buzz</mod>
</test>
</fizzbuzz>

for $i in ($config/range/@min to $config/range/@max)
let $s :=  for $mod in $config/test/mod
      return
         if ($i mod $mod/@value = $mod/@test)
         then string($mod)
         else ()
return
if (exists($s))
then string-join($s,' ')
else $i

Timelines

Here are some ways in which timelines are being used in the web:

• TimeSearch
• GoogleEarth timeline
• HyperHistory
• SIMILE timeline from MIT
• Dinosaur timeline
  
• Provides a Javascript API to create a complex panning timeline.
  
• Events are uploaded in XML format which defines for each event
  
• start title end? isDuration? latestStart? earliestEnd? image? link? and body
• Cant locate schema for the event stream
  
• Event date format is not xs:date
• body must be a string i.e. with < not <
• Set the mime type to application/xml
  
• example XML timeline

I've implemented the SIMILE timeline in my Family History. Here is an example
Links to other people load their timelines (although some are incomplete and not working).

Multiple event streams can be displayed in multiple bands - e.g. here are the lives of two family members

It would be great to get a subset of world history events from TimeSearch to include as a separate band.

To do:

• Convert xs:date to Timeline format
• Handle photos with missing dates
• Focus timeline when no date of birth

My Family History

Bamber Gascoigne's TimeSearch got me thinking about my family history project again.

The Family History project started with the idea of putting some family photos on the web, together with some meta data about the photo. Most had a list of subjects but only very occassionally a date. I had the idea that if the birth dates of the subjects were added, and the age of just one subject could be guessed, the system could infer the date of the photograph, and hence the age of all subjects. With birth and death data included, a timeline could be extracted for a person, including birth, deaths and marriages, photographs and even world events.

The resultant eXist/XQuery/XSLT prototype is here.

What would be nice would be be able to combine such personal histories with events in world history, or to compare the timelines for famous people with that of a family member. A simplistic approach would be to deep link into history sites and this is what the prototype has done, creating links like

• TimeSearch 1946
• Wikipedia 1946

These generated links are not as clever as those in TimeSearch, which I would guess are hand edited.

This is simple to accomplish, but its rather one-side - my site can link to another, but it can't mashup the data with my own. A mashup requires an API and a published format for exported content. I suppose we might look at hcalendar as one possible format but it would need extending it to support tags. RSS is another possibility.

TimeSearch by Bamber Gascoigne

The item on the BBC's Start the Week' by Bamber Gascoigne on the history site he has created called TimeSearch sparked my interest TimeSearch is a history search portal, using event stubs to provide links onwards to a wide range of sites - from general news sites, Wikipedia and Google images to specialist on-line resources. Events can be selected using two hierarchical category systems, location and them as well as year. I confess I had some trouble deselecting categories once selected.

This is the latest form of delivery of the extensive material Bamber has created and put on line in:

• History World
• An online encyclopedia of history with great little quizes.
  
• OCEAN index

One wonders how this authored material compares with the collaborative Wikipedia. It is certainly an impressive and rich resource.

SPA 2007

I have been working on my session for this year's SPA for a while now. Seems a timely session to run according to Elliotte Rusty Harold's predictions for XML in 2007 (which mentions eXist).

Mostly I have been vacillating about which case study to use - I have so many part-finished projects. At first I revived the Whisky project, but after a lot of work, this felt like a bit of a slog - it was good as a modeling exercise but didn't reveal the power of XML and XQuery to my satisfaction. I then switched to the Bus Timetable project, which is much more realistic and one which I want to solve to replace the existing PHP/MySQL application with its hand-crafted timetables. However the data here is very complex, and the data files I have so far are incomplete. Nevertheless, this is the case study I settled on, although I did a short detour into the DVD hire example and of course there is always StudentsOnline, the main site we are developing with this technology.

Timetables come from TravelLine South-West in the form of TransXChange files. These first need processing to a more amenable structure. The actual data model is highly interlinked which makes for some XPath fun. Conversion is required to calculate absolute times (TXC has time differences between stops) and to convert from OS Easting and Northing to Lat/Long . With the simplified structures, I can present the data in various ways. HTML pages show the stops and the departures times but the neat idea is to generate kml files to overlay on Google Earth. Using a NetworkLink, I can display changing times on the map. The XPath date and time functions are a great help.

Other bits of functionality include an Ajaxian incremental Bus stop locator and membership registration rretainig favourite stops and services.

Here are some useful links for this project:

• eXist database home
• an XQuery tutorial from Howard Katz
  
• A basic Timetable implementation (pending)
  
• TransXChange documentation
• TravelLine South West
  
• Google Earth
• kml description and reference