Welcome to XML Jack
XML Jack is a research project was born from frustation about the use of current XML manipulation methods such as DOM, SAX, XSLT and XQuery. The XML Jack project tries to find new ways to manipulate XML in programming languages.
The aim is to find methods that offer powerful advanced data manipulation as in SQL en XSLT, while being easier to use than DOM and XSLT and being less verbose and more powerful than XQuery.
The XML Jack project started as an attempt to create a new language for XML manipulation comparable to XSLT and XQuery. The idea was to create a new kind of "SQL for XML", but soon I noticed:
- SQL is used against a persistent storage.
- XML is usually manipulated in memory.
- A personal dislike for yet another programming language.
- The charm of Microsofts Language Integrated Query (LINQ).
- It is impractical to switch programming language within a program.
So the next step was to look for a better DOM, the Document Object Model API of the W3C for in program XML manipulation. However, many such attempts already litter the web, what was wrong with those attempts? It took me some time to get the bigger picture.
Types of XML use
After some time it dawned on me that there are many ways to manipulate XML and I collected some use cases.
| Use case | Best tool |
|---|---|
| Read some values from an XML file. | PHP's SimpleXML Easy to use but limited update facilites. |
| Read and update values in an XML file. | DOM Read & update in one API though it is not very simple to use. |
| Fill an XML structure with values from other sources. | XQuery But then this is what XQuery does, while it doesn't do much else. |
| Streaming summary of values over XML input. | SAX XSLT can do the job but officially has to keep everything memory until your through with all the data. |
| Streaming tranform of XML input into differently structured output. | XSLT As SAX is more difficult to use. The disadvantage of XSLT is that is has a steep learning curve. |
| Create XML Files on the fly. | ? You should use DOM for it but it is usually quicker to program this yourself, while trying to avoid excessive string concatenation. |
| Seamless conversion of XML values to native programming language values. | PHP's SimpleXML But then PHP doesn't really have a type system. |
| Take XML input, reposition nodes. | ? You would have to do this with either DOM or XSLT, but DOM has an unfriendly API and in XSLT you have to code for the things you do not want to change. |
| Take XML inputs, put parts of one document in the other. | ? DOM can do this and an XSLT programming wizard can do this using XSLT. |
| Combine XML inputs using inner joins. | XQuery You can do this in XSLT, but not in a generic matter and again it takes a wizard. |
| Combine XML inputs using outer joins. | ? You can do this in XQuery, but it has no language support for it so one has to resort to boilerplating. |
| Restructure an XML hierarchy. | ? Maybe you can do it in XSLT, maybe not. |
| Merge two XML trees. | ? Forget it. |
Now remember the last time you where thinking what tool to use for accessing a database? Yes, that is right! You don't think about it at all. Either you can do it in SQL so you use SQL or you have to do a transform in the language in use, e.g. when you have to compare something to the next or previous item.
Of course you can do everything above using the DOM API, but DOM does not support these operations. DOM just allows you handle the low level changes while you have to program the actions yourself.
What we need is a single programming tools with these properties:
- Easy things should take little code.
- Changes should be as easy as reading data.
- Effortless conversion to native programming language base types
- Low- and high-level changes use a single syntax.
- The optimizer decides to stream or to handle the data in memory.
Impossible? Keep checking this site.
Outer Joins / ELSE Syntax
One thing I want is the advanced SQL feature of outer joins. As outer joins are generally not supported within programming languages this was not a trivial ambition. In the end I think I solved this by using a different method for generating outer joins which requires extensive use of sub-queries to filter the data. Looking at these sub-queries I saw that they belonged to a larger class of sub-queries whose data sources are linked to their containing query. This led to a general method for simplifying such queries that I named ELSE Syntax (Existential Logic Simplified Syntax) and that I patented.
Currently I am writing a demonstration site for ELSE Syntax and am trying to get a scientific article published, so details have to wait until after publication.