xray is a framework for testing XQuery code on MarkLogic Server. It makes it super easy to write tests as standard XQuery functions.
Check out the GitHub repository for more details.
MarkLogic XQuery performance tuning – computing facets concurrently
As seems to be quite common in search applications built on MarkLogic, our latest project has a lot of facets, varying from document language with only a handful of values to publication title with over 50,000. Facets are very fast to calculate, but with a lot of distinct values in each facet, it can easily take a couple of tenths of a second per facet. With multiple facets the time really starts adding up. Profiling our search queries revealed that over 70% of the time was taken up with faceting.
The code looked something like this. The facets are defined in XML and there is a function to resolve a single facet which is called on each using function mapping. The total time was around 0.5 seconds.
The cts:element-values function has an option called ‘concurrent’. The description from the API documentation:
Perform the work concurrently in another thread. This is a hint to the query optimizer to help parallelize the lexicon work, allowing the calling query to continue performing other work while the lexicon processing occurs. This is especially useful in cases where multiple lexicon calls occur in the same query (for example, resolving many facets in a single query).
Unfortunately simply adding the concurrent option to the facet definitions doesn’t make any difference because the function mapping is blocked waiting for the response of each call. The same would happen if we looped over the facets in a FLWOR statement.
One solution is to rewrite the resolve-facet function to recurse through the facets, only accessing the results of cts:element-values after the recursive call. This way it will loop through the entire sequence of facets first and the expensive calls won’t be blocked. Total time is now 0.2 seconds – the time of the slowest facet.
By the way, if you’re using the MarkLogic search API then there’s no need to do anything, it already computes facets concurrently – sorry for wasting your time 
It’s also worth noting that this technique isn’t limited to cts:element-values, other lexicon functions also support the concurrent option.
Tagged API, MarkLogic, XQuery
2 Comments
DQ update
It’s been a while, but DQ has finally had a bit of attention to add a couple of important missing features:
Tab renaming – double-click on a tab
Explore – new button in toolbar to explore current database
Auto-save – queries are now saved on execute instead of on exit
API interface updated for MarkLogic 4.2
MarkLogic 4.2 is finally out! (http://developer.marklogic.com/products/marklogic-server/4.2) so I’ve made a few minor changes to the API interface to better support multiple versions of the MarkLogic API.
See it in action:
http://api.xqueryhacker.com
Download/Fork:
http://github.com/robwhitby/MarkLogic-API-Interface
Tagged API, MarkLogic, XQuery
Leave a comment
Search interface to Mark Logic API now on SourceForge
Update: The project has now moved to github.
http://github.com/robwhitby/MarkLogic-API-Interface
http://github.com/robwhitby/MarkLogic-API-Interface
The search interface I built to the Mark Logic API (see Searching the Mark Logic API function reference with Mark Logic) is now hosted on SourceForge for anyone to download and run locally.
Try it out here: http://api.xqueryhacker.com
It requires Mark Logic Server v4 (community edition is fine, it’s only small). There’s a few steps to the installation, instuctions are in a readme file in the download. Please let me know if there are any problems and I’ll try to help.
Tagged API, Ext JS, MarkLogic, XQuery
3 Comments
Searching the Mark Logic API function reference with Mark Logic
I’ve always thought it odd that the Mark Logic API function reference isn’t searchable. It seems obvious that it should be loaded into Mark Logic itself. Most of the time I use it for looking up the signature of a particular function, take cts:uris() for example – can anyone remember what order the parameters go in? Finding a function detail page from the home page is a mission, so I always just hit F3 to search the home page.
So anyway, I loaded the content into Mark Logic and built a quick interface using ExtJS. We’ve been using it at work for about 6 months, and it’s now got a new home at api.xqueryhacker.com. You can choose to search just function names or all content, and browse by namespace. There’s a link in the header to switch between version 4.0 and 4.1 of the API.
In the next few weeks I’ll tidy up the code and put it up on SourceForge so if anyone’s interested they can install it locally – I’m not really intending to publicly host it permanently.
As always, all feedback welcome..
Tagged API, Ext JS, MarkLogic, XQuery
5 Comments
Early version of DQ – an alternative interface for Mark Logic’s CQ XQuery editor
Update: The project has now moved to github.
http://github.com/robwhitby/DQ
http://github.com/robwhitby/DQ
Here’s a very early version of a new interface to CQ that I’m slowly working on, imaginatively named DQ. It aims to address a number of shortcomings in CQ, the web XQuery interface provided by Mark Logic.
Current features
Tabbed code editor (based on EditArea) with XQuery syntax highlighting, search and replace, and support of tab key
Save queries and results to file system
Auto-save of all query tabs using browser local storage (no more session confusion)
Highlighting a section of code and executing will run just the selected code (inspired by SQL Query Analyzer!)
Future features
Integration of XQuery API reference
Function auto-complete
More suggestions welcome!
Requirements
Recent version of CQ installed (I’m using 4.1.2)
Firefox 3.5 or IE8 (Chrome doesn’t style XML output, Safari and Opera not yet tested)
Installation
Unzip DQ folder in root of your CQ directory
Browse to: http://SERVER:CQPORT/DQ
The project is up on sourceforge:
https://sourceforge.net/projects/mldq/
Please give it a go and let me know what you think. I’d be interested in suggestions on improvements, extra features etc.
Tagged DQ, MarkLogic, XQuery
15 Comments
Error restoring a Mark Logic forest
After it spent about 2 hours restoring a forest I got an annoying but slightly amusing error:
500 Internal Server Error XDMP-EXTIME: xdmp:sleep(3000) -- Time limit exceeded in /forest-backup-go.xqy, on line 56 [0.9-ml]
Now what do I do? Try again and cross fingers this time?
And why is restore not async in the admin ui? Backup is..
This is just the first forest, there’s another 4 to go. Not what I need on Friday afternoon 
Rant over.
UPDATE… Rookie mistake believing the error message meant the restore had failed – turns out it has worked!
Tagged MarkLogic
Leave a comment
Oracle slates "inferior" Mark Logic
Oracle have published a PDF comparing their new XML DB to Mark Logic 4.1.
It’s amazing how openly hostile they are. Instead of comparing the two products on any meaningful basis, they resort to making vague, unsubstantiated claims about their superiority. It really comes across as desperation and highlights that Oracle clearly feel threatened by Mark Logic. It’ll be interesting to see how Mark Logic respond – CEO Dave Kellogg has said he’ll post a response on his blog in the next few days.
Tagged MarkLogic, Oracle XML DB
2 Comments
SyntaxHighlighter XQuery brush
Update: Now hosted on GitHub:
http://github.com/robwhitby/SyntaxHighlighter-XQueryBrush
http://github.com/robwhitby/SyntaxHighlighter-XQueryBrush
I’ve just improved the code snippets in my posts by using the excellent SyntaxHighlighter by Alex Gorbatchev.
I had to write a new brush for XQuery, but this was pretty straightforward (it has a much better extension system than I found in Notepad++ for example). As well as XQuery 1.0 functions, keywords etc. I’ve included the MarkLogic 4.1 API.
Here’s an example..
xquery version '1.0-ml';
declare variable $URI as xs:string external;
declare function local:document-move-forest($uri as xs:string, $forest-ids as xs:unsignedLong*)
{
xdmp:document-insert(
$uri,
fn:doc($uri),
xdmp:document-get-permissions($uri),
xdmp:document-get-collections($uri),
xdmp:document-get-quality($uri),
$forest-ids
)
};
let $xml :=
sdasdasdasd
If you want to use it download SyntaxHighlighter, and then get the XQuery brush files from GitHub:
http://github.com/robwhitby/SyntaxHighlighter-XQueryBrush
Example usage:
<script type="text/javascript" src="scripts/shCore.js"></script> <script type="text/javascript" src="scripts/shBrushXQuery.js"></script> <link type="text/css" rel="stylesheet" href="styles/shCore.css"/> <link type="text/css" rel="stylesheet" href="styles/shThemeDefault.css"/> <link type="text/css" rel="stylesheet" href="styles/shThemeXQuery.css"/> <script type="text/javascript"> SyntaxHighlighter.config.clipboardSwf = 'scripts/clipboard.swf'; SyntaxHighlighter.all(); </script> <pre class="brush: xquery;">xquery version '1.0-ml';</pre>
Next job is to change the layout of this blog to make the content column wider...
Tagged MarkLogic, SyntaxHighlighter, XQuery
2 Comments
