Software and Architecture

The Lean Startup

2012-01-24T16:28:00.001-08:00

I just finished reading Eric Ries' The Lean Startup, and am happy to say that it was much better than I expected.

When I heard about the book I thought that it only applied to cloud based web startups: the kind of shops that could readily perform A/B testing over a weekend, an approach pioneered by Google early on.

My reaction to this was predictable: that's fine when you're making simple changes to web pages, but what if you're doing some heavy technical lifting, e.g., new technology (Watson), hard technology with deep foundational requirements (Dropbox), etc.

Ries describes something more subtle. He develops a general methodology that allows quantitative evaluation of assumptions, as quickly as possible, with the least possible amount of effort.

Dropbox is one of his most striking examples

The challenge was that it was impossible to demonstrate the working software in prototype form. The product required that they overcome significant technical hurdles; it also had an online service component that required high reliability and availability. To avoid the risk of warming up after years of development with product nobody wanted, Drew did something unexpected easy: he made a video.

The goal of the video was to validate the assumption that people would actually be interested in such a product. Of course in the Dropbox case the interest was there, but the book is filled with numerous examples in which it wasn't.

Ries' core approach consists of a large scale feedback loop, shown below (taken from here):

My two core take aways are

MVP--the minimum viable product, and
Measure-- the importance of making clear, tangible predictions ahead of time.

Both ideas are important in any product that has a customer focus, whether the customers are online consumers or members of an internal department.

They are reminiscent of the techniques my group used years ago to measure uptake of product features by internal departments: if we deployed a page with a particular group in mind, we could see if they continued to use it a few weeks out. If not, we'd visit them and ask what the problem was. There wasn't any need to wait for feature requests (or to hear in a budget meeting that you weren't providing any value). We sought them out and fixed the problem, if at all possible. Monitoring at page level allowed us to make these judgments for small product increments.

MVP is simply getting something in front of people as quickly as possible -- it doesn't need to be working, doesn't need to scale, doesn't need to be fully polished, but it does need to give a feel as to why someone would want to incorporate the product into their life.

Measure is more generally important. Ries is perfectly willing to start with measurable goals that seem almost trivial. The idea being that in short order these goals can eliminate easy workarounds, myths of low hanging fruit, etc.

In one of his examples, he aimed for a revenue of a few hundred dollars a month to start. A few hundred a month sounds trivial, and certainly isn't sustainable, but after a few months he couldn't even do that.

The point is that if his goal was the few million dollars a year he'd need for true sustainability, it would have taken him years to realize that he was on the wrong track. This because realistically, even in the best case, the million dollar goal would require a few years to achieve, pushing feedback out to years rather than months.

The Lean Startup doesn't just focus on startups, it includes of a number of examples from groups within large non-software companies (Proctor and Gamble) and established software companies (Intuit) etc. shows that startup is more an attitude than a corporate structure.

I highly recommend this book, I don't consider it so much a Lean Startup book, but an optimized effort book: How to decide if your efforts are wasting your time or actually taking you in the direction that you want to go.

To be clear: I don't think that this means that data measured at a fine grained level is the only way of gathering feedback, e.g., it's hard to incrementally identify the best design for a large website c.f., Goodbye, Google. However, even in these cases, an MVP is important. It is vital to ground your ideas with your target audience, sooner rather than later (even if it's with a short video of a simulation of your idealized goal).

Watir

2012-01-17T09:45:00.001-08:00

I know I'm late to this, but I'll share anyway:

If you're actively developing or supporting anything with a web front end, take a look at watir for automated smoke and regression tests. The site has a "watir in 5 minutes" page, but probably really takes more like 3, it's that well designed.

Watir stands for:Web Application Testing in Ruby. It is:

Open source
Simple to install: just a ruby library (gem)
All of ruby is available when writing test scripts
Well regarded
Widely used. Here's a partial list:

Watir Limitations

“W3C” web only!

No plug-ins (active-x, flash)

No recording

Watir Advantages

Widely used
Very easy to use
Programmer friendly
Ruby is sane
Terse without being obtuse

Interactive Development

irb

For example, this screenshot shows a section of a browser wind and an irb window about to execute a method to click on the browser's edit button:

resulting in:

Page elements are referenced by

Type and Id/displayed-text/link destination etc

@browser.select_list(:id, 'StatusSelect')

or Index

Index is obviously not the preferred approach, but it is occasionally necessary, e.g., for pages with four identical “submit” buttons

On a maintainability note: I recently had to change a script from FireFox (which I'm increasingly becoming disenchanted with) to Chrome. The change only required altering 2 lines of code

#require 'firewatir'
require "watir-webdriver"
and
# @browser = Watir::Browser.new
@browser = Watir::Browser.new(:chrome)

However, I did find the behavior a bit different between Chrome and Firefox: If a button wasn't visible on the page Chrome wouldn't click it. I never had a issue with this in Firefox.

Update:, upon further investigation it appears that may be the behavior of the newer versions of watir, rather than a Chrome compatibility issue.

Building and Submitting iOS Apps built using user created static libraries

2011-10-23T07:43:00.001-07:00

Here are my notes on getting an iOS app submittable to Apple’s app store, which includes an interesting “welcome back to header files” experience. Resolving all the issues required a fair amount of searching and futzing, so I thought I’d put together a synopsis and share:

Step 1: Compiling a project referencing a static library I created.

Background: I’m building a suite of apps centered on pdf file display, using the vfr pdf reader framework -- the developer, Julius Oklamcak, characterizes it a basic framework, but I found it both robust and easily configurable to fit my needs.
The first task was to build a library consisting of the vfr pdf reader code so that the code would be factored appropriately. Using this approach, any bugs showing up in the pdf reader require only one code change to fix all the books.
I used CarbonFive’s blog post on building and using static libraries to perform the setup.
It took a bit of tweaking to get the library incorporated/recognized by the dependent projects, and, sadly, one of those tweaks came back to bite me later.

The primary change was to put all the projects in a single Xcode workspace so that the library was readily available to the dependent project (apps). Workspace creation, followed putting the library in the frameworks folder, and making the .h files public, made the .h files visible to the dependent project.
After these changes, the dependent project could
compile and run, both on the simulator and on my test device.

Step 2: Archiving the Project
However, when I went to archive the app (a precursor to submission), the necessary .h files suddenly could not be found.
There didn’t seem to be any “build level” way around this -- meaning that I couldn’t find a way to eliminate this error that just entailed changing some configuration files. The only solution I could find was to move copies of (or links to) the actual header files being referenced.
Upon further consideration this appears to be an artifact of the build environment/ObjectiveC language, since even the foundation frameworks are packed with their header files (see below)

Step 3: Validating the Project
At this point the archive could be created without errors, but when I tired to validate and submit I got the following
"[projectname] does not contain a single–bundle application or contains multiple products. Please select another archive, or adjust your scheme to create a single–bundle application."
This was a quick issue to fix -- the answers appearing on stack overflow
Archiving project in XCode incorrectly creates multi-application bundle
The fix required changing the file visibility (back) from public to project. Changing file visibility is most easily accomplished by moving the .h files to the appropriate part of the Copy Headers section of the Build Phases tab for the target -- I will admit that this approach was advocated it the CarbonFive blog, but it didn’t appeal to my (Java tuned) sensibilities.
Step 4: Submitting the Project
The fun wasn’t quite over, since an attempt to submit the app resulted in the response
No suitable application records were found.
After setting up the signing appropriately per Apple’s guidelines http://developer.apple.com/ios/manage/distribution/index.action
-- somewhat obscurely located here

(BTW am I the only one finding it frustrating that Apple’s documentation hasn’t been fullly refreshed to reflect the Xcode4 UI?)
and reading stack overflow again,
illed out the questionnaire and I was able to submit and upload successfully.

Hope this short walkthrough proves useful.

Commenting in html, tpl.php etc.

2011-08-15T08:43:00.001-07:00

I came across a very nice commenting convention in the drupal zen theme template files:

They close all of the class and id marked sections with

<!-- /.{classname} -->

<!-- /#{idName} -->

e.g.,

<tr class="genesis-page-header-row">

is closed with

</tr><!-- /.genesis-page-header-row -->

This is nice enough in helping you to keep track of what's going on when editing the code. However, it is invaluable in understanding what might have gone wrong when examining pages in a "modern" inspector.

For example, here's how that section looks in Safari 5.1:

making it clear that the you and the browser both agree on the <tr> being terminated by this </tr> (if you and the browser don't agree, it probably indicates that something has gone awry).

Debugging Drupal Access Denied

2011-06-26T10:54:00.001-07:00

I just finished debugging an Access Denied problem in Drupal, and wanted to add a couple more tips to those in Simon Lane's Drupal Permissions Issues: A Debugging Checklist

First: assure that the users unable to access the content have been given the appropriate roles.

Second: check the permissions of a role that can access the content relative to one that can't -- the following query works in MySQL and is both quicker, and more accurate, than a visual check.

select permission, module from role_permission where rid = 5 and permission not in (select permission from role_permission where rid = 4)
Where "5" is id of the role that has access, "4" is the id of the role being denied access.

Third, and most obvious: assure that the "inaccessible" content has been published. This was the source of my particular problem (doh!).

I echo the words of many posters, when I say "It would be nice if Drupal would log the reason for denying access. It would save everyone a lot of time"

Drupal 7.0: broken image preview

2011-04-28T10:48:00.001-07:00

I recently decided to use Drupal for one of my projects, since all reports are that it is both well designed and relatively easy to extend.

I'm just getting it set up locally on my MacBook Pro (10.6.7) and ran into a problem with previewing uploaded images: nothing was getting displayed for the preview, even though the images would display fine when saved.

The following errors were being logged.

The solution, took a while to find, but was relatively simple: changing the upload location to Public files (see below), cured the problem.

(Switching it back recreated the problem) -- seemed worth sharing.

CoreData != CoreObjects

2011-04-04T19:27:00.001-07:00

I ran into a situation last week that had me (proverbially) tearing my hair out.

I was trying to call methods on an object stored using CoreData and the calls just WOULD NOT WORK. The reason was not obvious, and even the simplest method dispatch was failing: the debugger showed the objects as being of type NSManagedObject rather than the expected GTGItemLocationList, even after casting the local var as a (GTGItemLocationList *).

Rather than continue, I decided to spend some time reading Pro Core Data for iOS since, in any case, I felt that my understanding of Core Data was suboptimal. Pro Core Data is an excellent resource and greatly improved my understanding of the Core Data framework. I decided to stop trying to treat my GTGItemLocationList as an instance of the GTGItemLocationList class, and instead use the NSMutableArray pattern described in the book.

I went back and coded up what I needed in about an hour.

Everything has been going well since then, and I thought I'd write this up as a cautionary tale about CoreData not providing objects in the manner expected. However, after digging a bit more, I found that this wasn't quite accurate. After all, the method which was throwing this exception was itself a method on an object stored by CoreData, and the dispatch was to the expected class/method combination. The pattern of accessing the object was different: rather than directly through an accessor, it was via an objectEnumerator on NSArray. The object returned using this technique was a GTGItem manipulatable via conventional OO techniques as expected (and appreciated).

I will admit that I haven't determined what's going on here. I think that the moral of the story is that in exchange for CoreData's simplicity and ease of use,* you might find yourself exploring a corner case that exhibits unexpected behavior. In retrospect, the GTGItemLocationList is completely superfluous. Eliminating this class would improve functionality and move me back into the CoreData's core use cases, reinforcing one of my basic iOS heuristics: If it is very hard to do, your first reaction should be "what built in function am I ignoring that would make this easy?".

For completeness -- I'm not the only one who has seen this problem, see here, and here -- none of the suggestions cured my problem

_____________________________________

*I mean this sincerely: the Pro Core Data highlighted the built in! CoreData data migration capabilities. Given how much time the industry spends planning/managing/performing data migration projects, any capability in this space is welcome.

python, Mac, MySQLdb "wrong architecture"

2011-02-04T13:14:00.001-08:00

I thought I'd post this tip since I had a few false starts following other pointers I found on the web -- I'd like to save others the wasted effort.

The error (MySQLdb "wrong architecture") IS because of a mismatch between the version of python and the version of the OS that you're running.

If you are trying to import MySQLdb into Python and getting a "wrong architecture" error message, try the following:

First, determine the version of your OS by typing

uname -a

into a shell (Thanks OSXDaily)

Then install a current version of Python that matches this architecture from python.org, reinstall MySQLdb and see if it works.

This is all that it took for me to be up and running, and is (relatively) painless.

Note: be sure that your scripts point to the newly installed version at /usr/local/bin/python, rather than the system version at /usr/bin/python

Adding Structure to Data

2010-12-20T07:11:00.001-08:00

This post demonstrates how to add further structure to data after the initial items have been (uniquely) identified and committed to your persistent store. The core idea here is that once you have items uniquely identified, you can overlay a structure (or any number of structures) upon them as desired.

These structural overlays can also be made to interact as much (or as little) as necessary to address the question currently under consideration.

For example, given four cell lines (this is taken from a brochure from the Charles River labs web site):

Cell Line	Species	Organ	ID
SW780	Human	Bladder	CL-1
Hep3B	Human	Liver	CL-2
B16	Mouse	Skin	CL-3
Madison109	Murine	Lung	CL-4

(note: not all relationships need to be specifically listed in the parent table)

This gives us the following structure:

Upon which we can overlay a set of relationship showing the source organ

Or source species

Now, lets say we add a new cell line

Cell Line	Species	Organ	ID
SW780-1	Human	Bladder	CL-5

Giving us

We may later realize that CL-5 was derived from CL-1 and just use a separate parent child relationship table to store the information

Parent	Child	Relationship
CL-1	CL-5	"derived"

(note: "Root" cell lines are those that do not appear in the Child column or do not appear in this table at all)

This sort of thing can be generally extended and need not be a strict tree:

Parent	Parent Table	Child	Child Table	Relationship
CL-1	Cell_Line	CL-5	Cell_Line	fusion-parent
CL-2	Cell_Line	CL-5	Cell_Line	fusion-parent

Obviously the richer the relationship, the more likely you are to move to a table specifically designed to capture that information.

Mixture Component	Mixture Component Table	Mixture	Mixture Table	Amt
C-1	Compound	C-9	Compound	0.1
C-1	Compound	C-9	Compound	0.1
R-2	Reagent	C-9	Compound	0.1

As these structures build up it is easy to then interrogate the information about our available cells.

Query: What mammalian cell lines do we have? Procedure: Traverse from the mammalian node and collect all cell line instances

Query: What cell lines are derived from C-1? Procedure: Find cell lines derived from C-1, find cell-lines derived from them (recursively), collect all cell line instances.

The overall pattern is pretty straightforward and is can be processed with standard graph algorithms

Knowing Where Your Bytes Are

2010-11-08T19:32:00.001-08:00

I thought that I'd post a pointer to this analysis of the foursquare/mongodb outage (for completeness, here is foursquare's description of their outage). I'm not a foursquare user, so I didn't know anything about this outage until I was catching up on my RSS reading last week, but I think the lessons are broadly applicable.

The key point being that even on a modern/cutting-edge platform, low level details of how your data (and processing) maps onto the underlying hardware can be the source of major headaches.

In general, the number of possible details that could bring a system to its knees is too large to keep in one's head at all times. In practice, what is required is the ability to track down why the system isn't responding to your changes as expected. In the FourSquare case the question became: "why wasn't the RAM usage decreased when 5% of the data was moved to another shard?"

The solution(s) are always obvious in retrospect, and the FourSquare mongodb team appear to have determined the root cause with admirable speed. In my mind the key to this speed was their ability to quickly get to the question "why didn't my RAM usage decrease" from the question "why didn't my system speed up when I added another shard."

At some level this goes back to my post last year on the Instruments performance analysis tool. A good substrate of these tools is critical: if you don't have a performance meter that's pegged, in this case the swapping/paging meter, getting a handle on what's going on is difficult, to say the least.

I'm glad I didn't personally experience that particular debugging session.

Middle Distance Ontologies -- an Intermediate Summary

2010-08-30T17:44:00.001-07:00

I've posted a number of design exercises using middle distance ontologies including:

Motivation

Design considerations

A detailed consideration of Necessary Attributes and Opaque Identifiers

A design for storing antibody information

A design for storing assay information

At this point I think it's worthwhile to look back and see what, if anything, is different and useful about the middle distance approach.

What struck me most as I was thinking through the examples was how similar the process was to object oriented class design. Using opaque identifiers allows moving a clump of stuff (attributes, identifiers, functionality, behavior) to another class so that the objects in the system correspond to objects in the world. They can then be analyzed as a coherent whole (aka modularization). Pragmatically, such partitioning allows developers to become intimately familiar with how the stuff within that partition operates and evolves over time, permitting them to develop a high level of fluency in the domain.

A key component of this approach proved to be the use of the criterion what could change to drive the splitting off of the chunks of stuff. As such, it is an extension to/refinement of existing design techniques rather than being a replacement for them.

So, how does middle distance thinking extend current techniques?

Extension to DB design: A focus on "objects in the world," rather than just cardinality, results in a more fine grained partition of the problem space. All things which are one-to-many or many-to-one are necessarily distinct and have different potential rates of change, but sometimes things which are one-to-one are also distinct, and should be separated to accommodate future change.

Refinement of OO design: middle distance ontologies are problem space oriented rather than software oriented. As such, it is less concerned with factoring out the appropriate superclasses than OO designs. This is because software design criteria that push such refactorings are "within the system" rather than "in the world." There is nothing in the middle distance approach that pushes common functionality up to a common implementation.

One other aspect of the middle distance approach is that it allows you to pull attributes out of the design and hide them behind the opaque identifiers. This modularization allows you to change the definition of such things as validity as your knowledge of what constitutes validity in your problem domain changes.

In summary, I think the middle distance approach is useful as a design principle, but is not a distinct design technique per se. Assessing its utility in practice must await an opportunity to apply it to a real world problem.

ArchiMate/OpenGroup

2010-08-02T13:49:00.001-07:00

Andy Siegel of Genzyme, Hemant Virkar of Digital Infuzion, and I gave a talk, ArchiMate as a Communication Tool in Launching an EA Effort, at the Open Group's Boston Conference. It discussed our experiences using ArchiMate as a communication tool for rolling out an EA Effort at Genzyme.

As I've mentioned before, I have found ArchiMate to be a practical, useful, framework. This presentation delineates some of the reasons why I came to that conclusion.

Middle Distance Ontologies: assays

2010-07-19T17:32:00.001-07:00

This analysis of assays is the companion to my previous post on antibodies.

A core bifurcation within assays is between in-vivo and in-vitro assays. I'm entirely ignoring clinical trials etc. since they are a completely different conceptual space.

The main differences between in-vivo and in-vitro assays is that the measurement is more indirect/variable and the delta between planned and actual measurements is much greater in the in-vivo assays.

In both we have

the system under test
the test response(s) being measured
the measurement event (with a potential planned vs. actual component to each)
the entity whose impact is being assessed
the way this entity was introduced into the system (most important for in-vivo assays)

The system under test

This captures the technology maintaining the experimental conditions, the SOP, the "target", and the readout.
It would therefore seem useful to use four opaque identifiers here:

one for the SOP
one for the particular technology or system being used for the measurement, e.g., the animal

one for the measurement device

one for the receptor/disease

The test response(s) being measured.

In normal practice, response types are behind opaque identifiers, e.g., %INH, with an additional qualifier as to the response units. Middle distance thinking does nothing to change this. When it comes to derived data (see also my post on necessary attributes), there are two options which I call

"resulted in" (this value "resulted in" this derived result) design.

"resulted from" (this value "resulted from" an operation on these results) design in which the transformation that calculated the value is designated by an opaque identifier.

I've seen a number of systems work well in which a more basic value points to a result derived from it -- a "resulted in" design. However, my preference is for the "resulted from" design as it allows the transformation to be more open about the algorithms used and the data points which served as sources of the value. This design allows the result to point back to its source data points (via the opaque identifier), rather than forcing the source data points to designate the derived result. It also permits a many-to-many relationship rather than the many-to-one coerced by the "resulted in" design, albeit with an attendant increase in complexity.

The measurement event.

(The measurement event may include an indication that the actual measurement event deviated in some way from the planned measurement event.)

This one is surprisingly different when viewed from a middle distance perspective. As opposed to the techniques which I'm familiar with from either conventional transactional systems or warehousing efforts, the middle distance approach suggests two factors:

hiding the details of the measurement behind an opaque identifier (including equipment operator, time of measurement, deviation from plan)
surfacing a flag (again an opaque identifier) to indicate if there were any problems of significance with this measurement.

This delegates the determination of error significance (and its type) to processes more familiar with the unique characteristics of the measurement.

The entity whose impact is being assessed.

Normal practice maps these to opaque identifiers that tie back to sample lots, be they compounds, mixtures, formulations, or natural products.

The way in which this entity was introduced into the system.

In some systems this may be covered by the SOP for the response being measured. However in more complex (in-vivo) systems it is worthwhile to explicitly call this out, since it is easy to imagine the same SOP being performed either with multiple injections or an implantable device.

In summary it appears that "almost all" of the detail is hidden behind opaque identifiers.

Middle Distance Ontologies: antibodies

2010-06-14T16:51:00.001-07:00

I want to push on the whole concept of "Middle Distance Ontology" a bit harder and see how it plays out -- my current plan is to concentrate on the discovery space with two entities: Assay Results and Antibodies.

I'll cover antibodies in this post, assay results in the next.

Now, there are many different perspectives from which to view antibodies, to name a few:

As a biologist investigating antibody action

As a vendor producing antibodies to meet a specification
As a pharmaceutical company procuring antibodies from a vendor to use in an assay

For this exercise, I'll take the perspective of a pharmaceutical company storing/analyzing assay results, since it is the viewpoint I understand best. Antibodies are something that I'm not intimately familiar with, so my approach will be to generate a list of attributes and then evaluate them for inclusion/exclusion/opaque identification.

Here are some antibody attributes I came up with. Many were taken from an interesting white paper from Pierce Biotechnology/Thermo Scientific:

Basic Attributes: Primary/Secondary; Monoclonal/Polyclonal; Antigen; Vendor Location; Batch

IgG Fragments: IgG Whole Molecule; Gamma Chain of IgG; Fc Fragment of IgG; F(ab ́)2 Fragment of IgG

IgM Fragments: IgM Whole Molecule;
Fc5μ Fragment of IgM

Mu Chain of IgM
Light Chains of Immunoglobulins

We certainly care about the primary/secondary antibody distinction. This captures both the fact that the secondary antibody was used (aka the primary antibody does not have a fluorescent tag (or equivalent)) and the characteristics of this secondary antibody if it appears. Interestingly a quick search was able to discover a reference to tertiary antibodies, so the principles outlined in the scenario analysis section calls for us to provide for these in the design of the core ontology, even if they are unlikely to be used.

In our situation, the factor that couples the primary and secondary (and tertiary) antibodies is their co-occurrence in the assay. A priori, there is nothing that requires there to be anything other than the primary antibody, nor is there any necessity for the antibodies to be able to bind to each other (after all mistakes happen). We might want to say that the primary, secondary and tertiary should (must) be able to bind to each other. However it would be inappropriate to include this as part of this core ontology since we are trying to capture the run of an assay and a run may be erroneous.

Therefore for each experimental run we will have a primary antibody and perhaps one secondary+ antibodies. In addition, we might multiplex the experiment and run more than one antibody set per "container" per experiment. A quick search for multiple antibody assay made me think that "multiplex antibody" was the appropriate search term, which results in > 2,000 hits, which indicates that it is possible.

Antigen is something that we (likely) provide or at minimum specify to the vendor. Although it should consist of a unique sequence, understanding its meaning and role within the overall program would require the ability to support an arbitrary level of complexity. This clearly calls for an opaque identifier.

At the vendor level there we will need to track some vendor identifier (opaque), shipment information (again opaque) and some sort of vendor lot/group identifier (opaque). The scenario that we we wish to be able to support is one in which the vendor ships multiple lots per shipment or spreads one lot across multiple shipments. Tracking the most fine grained vendor location as the opaque identifier at this level protects you from mergers/divestitures and new location startup issues, all of which would potentially be permanently hidden by the use of a larger grained identifier (just think if you only had ONE identifier for all of Thermo!).

When it comes to the more specific characteristics of the antibodies e.g., fragments and chains, they are not attributes that present distinctions which are important to the analysis of the results from the perspective of a pharmaceutical company.

Summary

Opaque identifiers
- Vendor facility
- Vendor lot/shipment
- antigen

Primary identifiers
- quantity
- monoclonal/polyclonal
- antibody

Elided Completely (stored seperately)
- antigen/antibody hierarchy
- antibody fragments and chains

Necessary Attributes and Opaque Identifiers

2010-05-26T18:16:00.001-07:00

At first blush, the use of necessary attributes and opaque identifiers looks a lot like database normalization, with necessary attributes being the columns and opaque identifiers being the foreign keys. I will admit that there are some similarities, although I maintain there are also some strong differences.

The most obvious difference is that many of the attributes (columns) that you would put in a database table would not appear in the ontology. In particular, you would avoid those attributes that are dependent upon specific business processes or technology. This includes most everything that relates to a hierarchy and any inter-object relationships that are only germane to one particular scenario.

Nullable columns are a mixed bag: The immediate reaction might be that if an attribute can be NULL, a priori, it would not be ontologically necessary. However, pure ontological necessity can be at cross purposes with the goal of stopping our analysis at a Middle Distance. A good example of something that can be NULL which would be included is the "derived from" relation. If we stop our analysis at the instrument (which is likely), some results, having been loaded from an instrument will have no source (derived from) results. I find no compelling reason to eliminate the "derived from" attribute, since independent of business processes, most results will derive from a combination of/analysis of other results and an indication of this is necessary for determining dependencies etc, it is just that some results will have no antecedent.

Note: It is open at the moment if the "results" of an average, should be treated as a single result or a set of results of different types, all of which are created by the same operation. In any case, given that our analysis truncates at the instrument, even if we considered the instrument a transformation that created the result(s), there would be no result(s) that served as inputs for the transformation.

By the same token, some attributes that we might think of as always being present may be elided from the model since they are hidden behind opaque identifiers.
A good example here would be the use of geoposition rather than address. Using a Middle Distance approach we would structure the location of an address as an identity preserving opaque identifier (geoposition) rather than as a set of columns containing foreign keys that reference other items (in other tables in a relational model) which hold the street/city/state/country values. This opaque identifier allows us to ignore all political boundaries, variations in street names etc. when designating our location. If required, these values can be derived on a "just in time" basis.

This is the core Middle Distance ontological question:

Given the way we use an entity, can the entity exist without having a value for the attribute under consideration?

If the answer is no then some flavor of the attribute must be brought forward and attached to the entity. The question of whether or not to represent this attribute as an opaque identifier has to do both with the complexity of the attribute and its variation in practice. I think that experimental conditions are another paradigmatic example of something that should be hidden behind an opaque identifier since the level of detail that is important (e.g. include the instrument, SOP, lab location? etc.) changes depending upon the particular type of experiment conducted and the variability of these conditions within the organization -- any and all of which might change over time.

However, the fact that an experiment will have SOME experimental conditions will be invariant.

This hints at a general rule: if the information about a thing requires one or more ancillary tables/objects to represent it, it is best to wrap the information in an opaque identifier which is designed to be sufficient to disambiguate the reference, but does not contain any detail. This identifier can be expanded out to a "report specific" level of detail on demand.

My next post(s) will work through some examples.

Considerations in developing a middle distance ontology

2010-05-10T17:17:00.001-07:00

In my mind there are three essential considerations when developing a middle distance ontology

What are the entities under discussion?

What constitutes the necessary attributes of these entities?

Should these attributes be hidden behind opaque identifiers or should they be an integral part of the entity under consideration?

The first question "What entities are under discussion?" is the easiest to answer: These are the entities that you discuss when performing your activities. If something has never come up as a factor in your activities (and isn't obviously on the horizon) there is no need to consider it.

Patients, trials, compounds, assays etc. are both important and are definitely "ready to hand" in the Heideggerian sense.

The second and third questions "what count as explicit attributes" and "what are the modifiers captured by opaque identifiers" are more subtle and domain specific.

This highlights a core point about the middle distance ontology viewpoint: what's important is what matters to the activity that you are performing. If it doesn't impact what you are doing it should not be modeled in detail. Truncating the detail is what keeps the model's complexity under control.

However, there is one caveat to this "what you know is all you need to know" approach: it is critical to evaluate the likely potential changes to your current situation. Doing this well requires an identification of the scenarios that might impact your operation in the near future and thinking them through in some detail, using the scenarios to pressure test your decisions.

Such a scenario analysis is needed since the ontology (obviously) constitutes a deep structural commitment and any changes at this level are usually both costly and painful.

I would posit the following classifications of the potential changes:

Changes in the science: These can be very unpredictable, but often there are precursors consisting of some new "interesting results" in an area. Although the exact resolution of the controversy may not be known, any outline of their structure can help highlight areas of necessary flexibility.

Changes in the environment: (mergers etc.) do others in the field think of things similarly. If not, what are the most significant differences?

Changes in the business structure: are there any "nearby" functions that would require support in the face of an internal restructuring?

Changes in the technology: there are two parts to this:
- Changes in the computer technology: most likely won't impact your ontology unless you're pushing systems to their limits (more and more unlikely in my experience).
- Changes in the technology of the systems which you are analyzing: e.g., reactions now produce ten similar but not identical compounds rather than a single compound, suddenly photos become tagged with GPS information etc. Another hint is if you're starting to hear the words "high throughput" in a context in which you've never heard them before.

I will admit that a difficulty of doing this is that it spans all architectural disciplines from application to enterprise, but I don't see any way around it.

My next post will focus on when to hide (attributes) behind an opaque identifier.

MacBookPro Sleeping Problem: Solved.

2010-04-26T16:56:00.001-07:00

If you have a MacBookPro that's having trouble waking up from sleep correctly: spinning beachball, infinitely long login check etc., you might want to try the fix suggested on macrumors by jmora71.

It appears to apply primarily to systems with Solid State Disks and a fair amount of RAM (the author mentions 8G, I have 6G).

Since performing the fix, I haven't had problems in over a week. I used to have issues daily.

For various reasons I had thought the cause was Parallels, but I was happily mistaken.

Ontologies in Practice

2010-04-19T16:50:00.001-07:00

This is an extension of an old post An Extensible System for Discovery Data. As I've been thinking more about what constitute the Simplest Building Blocks, I've begun to realize that they designate something very close to an ontology in the "middle distance." That is, it isn't about an ontology down to the fundamental constituents of matter, nor is it about specifying things in sufficient detail to adequately compare and track what's going on between organizations (see Barry Smith's presentation for a discussion of these issues), rather it is an ontology of the stuff we deal with in our day to day activities.

For example, an experiment has

A protocol consisting of:
- a set of initial conditions
- one or more intermediate steps

A result set consisting of one or more members,
- any of which might be invalid for one or more reasons

A set of analysis results
- with methods
- parameters
- derived results (results that are based upon this result)
- supporting results (results upon which this result is based, such as calibration curves)

My basis for claiming that these are constituents of a "middle distance" ontology is twofold:

Each component is ontologically necessary. That is, an experiment cannot exist without these components.

When analyzing these constituents we do not need to go into further detail. We can hide that detail behind an opaque identifier and need not give it further meaning. This opacity allows us to stop our analysis at that point; we don't have to analyze down to the constituent quarks (or chiral forms, for that matter, if they don't have any impact on our current goals).

In future posts I'll cover the entities and relationships that I take as being important in this middle distance and how to identify them (which, like most modularization efforts, is a more of an art than a science).

Architects As Service Providers

2010-03-29T17:04:00.001-07:00

This paper by Roland Faber of Siemens Healthcare recently appeared in IEEE Software. It talks pointedly about how it is more effective for architecture to be structured as a service that provides value by interacting closely with project developers rather than being structured as a function that produces documentation to be followed by the projects. It even advocates that architects perform some hands on coding in the projects (mirable dictu).

It would be impossible for me to agree more, including the part about hands on coding, my fondness for which is pretty obvious given my blog posts.

The article posits that this close, ongoing interaction is a good way of assuring both that projects understand what the architects are trying to accomplish with the architecture and also that the architects develop an appreciation for the practical issues involved in building working software. There is also a side effect of this kind of interaction that they don't mention: its value in preventing the obsolescence of the person doing the architecture.

The standard scenario in this industry is that a person spends a number of years learning their craft and refining their practice at which point, if they're good, they become an architect or manager and stop coding. The architects' (or manager's) skills stay relevant for a few years (five years seems to be a recurring number), after which they become a pointy headed character in a Dilbert cartoon.

As an industry we should be learning from this anti-pattern.

It seems to me a truism that coding keeps you grounded, and current, and keeps this Dilbertization from happening. Certainly at some point other duties e.g., architecture/management require that you take yourself off of the critical coding path, otherwise the success of the project is put in jeopardy, but new technologies or core utilities that are not on a critical path timeline are all fair game.

I strongly believe in this approach and this is the first article I've seen that reflects my personal practice.

Low Level Virtual Machine (LLVM)

2010-03-15T17:05:00.001-07:00

LLVM, as described in this article on AppleInsider, stands for Low Level Virtual Machine. It is an open source project that is used and partly supported by apple.

One of the most interesting things about LLVM is a quote at the bottom of page 1 of the article

Apple also uses LLVM in the OpenGL stack in Leopard, leveraging its virtual machine concept of common IR to emulate OpenGL hardware features on Macs that lack the actual silicon to interpret that code. Code is instead interpreted or JIT on the CPU.

This approach makes it very likely that developers will use the hardware optimized instructions. Most other approaches impose significant costs upon the developers, e.g., the need to write additional code to cover every possible hardware configuration. With the LLVM there is no coding penalty, therefore using the optimized routines becomes a no-brainer, resulting in faster code for people with beefier hardware (who are also those who tend to be most worried about performance) and usable code for everyone else.

As background the article points to a presentation by Chris Lattner but I prefer his paper with Vikram Adve LLVM: a compilation framework for lifelong program analysis because it talks in terms I can understand (like "Static Single Assignment").

So here's what's cool: LLVM eats a code representation that is very amenable to optimization and analysis. It optimizes this input and outputs machine code (potentially tuned for the actual hardware which will run the code) decorated to allow low-overhead runtime profiling.

(Click on image for larger version)

This approach permits repeated optimizations based upon recent run-time data rather than generalized heuristics -- it is reminiscent of "hot spot" with larger scope but less immediacy.

I'd be remiss to not mention the strategic implications of this: it allows Apple to radically shift hardware configurations, while restricting the software impact to a relatively small chunk of code c.f. iPad.

Update 21 Aug 2010: Just noticed LLVM got a SIGPLAN award -- well deserved!

Hubs & Connectors

2010-03-03T09:24:00.001-08:00

I recently stumbled upon the composite software site and was impressed by their architecture. It is a virtualized/federated solution that reminds me of the Hub/Connector system which I had proposed as a data integration model for the drug discovery/cheminformatics space.

The advantages of such an architecture over a conventional data warehouse include:

There is no requirement to perform a complete mapping of the data. This allows focus upon solutions that address the particular problem at hand and the mappings required to solve it. Such a focus is especially important when the data structure and mapping rules are in a state of flux for part of the system. It allows the high flux areas to be avoided.

The target data store need not have a structure capable of holding all of the data simultaneously. For example, a target table that would hold all of your CDISC SDTM SUPPQUAL values could require upward of 1000 columns reaching the limits of many common relational databases. On the other hand, the solution for an incremental data set would be an order of magnitude smaller.

Only the data of interest is accessed/moved. In systems that only analyze a small set of the data at a time, server size can be reduced substantially.

Data need not be moved to a central repository, minimizing duplicative storage space.

Of course there are disadvantages

A warehouse allows the precalculation of complex results, imposing little operational delay in retrieving these results.

Warehouses can be more easily structured to handle analyses which involved large portions of the dataset.

In scientific domains, it isn't uncommon for new assays, results, etc. to break your current mappings. A virtualized approach minimizes the impact of these problems upon your system and is certainly something to look at if this sounds like your situation.

Seam 2.2 + Jboss 5.1: a resolution

2010-02-15T18:01:00.001-08:00

A few weeks ago I posted about my difficulties migrating to new versions of jboss and seam.

I was finally able to resolve these difficulties by following these steps:

I first took the suggestion of this post on seamframework.org and used seam-gen to generate a basic seam application that worked with my existing database.

I then replaced all the libraries with the ones from the newly generated application.

As a penultimate step I followed the procedure outlined in the migration guide.

I was now able to search for differences between the new and the old version.
The primary differences were in the *List.java classes

--the new version of the code followed the pattern:


    private static final String[] RESTRICTIONS = {“lower(shows.name) like concat(lower(#{showsList.shows.name}),’%’)”,};
    private static final String EJBQL = “select shows from Shows shows”;
    @SuppressWarnings(“unchecked”)
    public ShowsList() {
        setEjbql(EJBQL);
        setRestrictionExpressionStrings(Arrays.asList(RESTRICTIONS));
        setMaxResults(25);
    }

--- the pattern in the old version was:


	private static final String[] RESTRICTIONS = {“lower(shows.name) like concat(lower(#{showsList.shows.name}),’%’)”,};
	private Shows shows = new Shows();
	@Override
	public String getEjbql() {
		return “select shows from Shows shows”;
	}
	@Override
	public Integer getMaxResults() {
		return 25;
	}

The primary difference is that the restrictions used the setRestrictionExpressionStrings method. As one of the commenters on the seamframeworks post mentioned: it would have been good to have discussed this in the migration document.

Not a tremendously big deal but, as I said previously, disappointing.

popViewControllerAnimated & message sent to deallocated instance

2010-02-01T17:53:00.001-08:00

This is just an informational posting -- it is so odd I thought I'd share.

If you have any of the following:


controllerDidChangeContent

didChangeSection

didChangeObject

controllerWillChangeContent

in the same class as a save that does a
popViewControllerAnimated
when the save completes you will get a


controllerWillChangeContent:]: message sent to deallocated instance (insert your address here)

I'm not exactly sure how this happens
but essentially
these methods hold onto the old copy of self in some manner.

Here's an example:


(gdb) p self
$1 = (GroupItemSelectViewController *) 0x3a34f70
Current language:  auto; currently objective-c
(gdb) c
Continuing.

Now pop back to the old screen and then do some editing:


2010-01-05 20:34:36.052 GoodToGo[62202:207] (GroupItemSelectViewController:didSelectRowAtIndexPath) setting Item to YES 
2010-01-05 20:34:37.960 GoodToGo[62202:207] (GroupController:numberOfRowsInSection) rows: 3
2010-01-05 20:34:39.902 GoodToGo[62202:207] (GroupController:numberOfRowsInSection) rows: 2
2010-01-05 20:34:42.690 GoodToGo[62202:207] (GroupController:numberOfRowsInSection) rows: 3
2010-01-05 20:34:43.561 GoodToGo[62202:207] (didSelectRowAtIndexPath) Selecting row 0
2010-01-05 20:34:43.563 GoodToGo[62202:207] (GroupItemSelectViewController:viewDidLoad) fetchedItemCount:5, holderCount: 5

What is the current value of self?


(gdb) p self
$2 = (GroupItemSelectViewController *) 0x3c84820
(gdb) c
2010-01-05 20:34:51.690 GoodToGo[62202:207] (GroupItemSelectViewController:didSelectRowAtIndexPath): setting Item to NO 
Continuing.
2010-01-05 20:34:53.546 GoodToGo[62202:207] (GroupController:numberOfRowsInSection) rows: 3
2010-01-05 20:34:53.547 GoodToGo[62202:207] *** -[GroupItemSelectViewController controllerWillChangeContent:]: message sent to deallocated instance 0x3a34f70

As you can see 0x3a34f70 is the previous value of self, not its current value.

I do consider this a bug in the framework.

Seam + {jboss 5.0 | jboss 5.1} = ?

2009-11-09T17:21:00.001-08:00

About a month ago I completed the upgrade of all my Macs to Snow Leopard. This generally went smoothly -- not a surprise for a release that has been charactered as "more refinement than upgrade."

One exception was a jboss 4.2.2/seam 2.1 application that would hang when generating a list view of objects that included images. My initial reaction was that this provided an opportunity to upgrade to jboss 5.x and partake of whatever enhancements that offered.

This proved to be a task that ended in frustration. I spent ~ 40 hours on it and eventually gave up. I fell back to the earlier version and upgraded to jboss 4.2.3 which solved the problem (which I think was related to using Java 1.6).

I thought I'd share some of my experiences, just in case someone else finds it useful:

The first glitch was that the version of seam I was running didn't appear to work with jboss 5.x so I upgraded to version 2.2

The attendant upgrades caused me to change some of the DB mappings
e.g.,

change blob annotations (mysql specified)
From: @Column(name = “data”, length = 8000000)
To: @Column(name = “data”, length = 8000000, columnDefinition = “mediumblob”)

I also tried switching to jboss developer studio to see if that would help me uncover the problem -- this had no real impact.

The core symptom was that nothing was coming back from pages that generated a list of items in the DB and no Hibernate queries showed up in the back end stream.

I eventually tried to go in to one of the more "internal" pages

http://localhost:8080/webtwo/Competition.seam?htmlTitle=competition&competitionId=2&cid=6

(a real advantage of Seam's rest interface) and finally saw a hibernate query on the background stream with the warning:

WARN [Param] could not create converter for: competitionId javax.el.PropertyNotFoundException: Target Unreachable, identifier ‘competitionHome’ resolved to NULL

This warning was similar to the error I was getting that the authenticate method resolved to NULL.

What appeared to be happening is that the seam annotations weren't being processed correctly (specifically @Name("competitionHome") ).

After searching on this error I found this link
http://www.seamframework.org/Documentation/WhatHappensWhenYouDeploySeamAppInJBoss5
which made me think that things are basically broken.

As I said, rolling back to the original code, the problems went away in jboss 4.2.3. However, I must admit that I'm surprised that the issue exists in the newer versions of seam/jboss

Jboss
5.0.0.GA	Stable	104 MB	2008-12-05	LGPL	 134971	Download	Notes
5.1.0.GA	Stable	130 MB	2009-05-23	LGPL	 181731	Download	Notes
Seam
JBoss Seam 2.2	2.2.0.GA	Production	111 MB	30.07.2009	LGPL	Notes	Download

I know that this is "unsupported" code, but I would still think that there would be better testing. After all, 5.1 was out in May of this year and Seam 2.2 claims

Seam 2.2 examples target JBoss Application Server 5.1.

Now, I do realize I could have been anywhere from 1 minute to 1 month away from a solution for this problem (if anyone has a solution, I'd be more than happy to try it), but I have two closing thoughts:

Allocate more time than you might have expected towards making the transition

It would have been appreciated if the various teams involved paid more attention to migration tools (even documentation) and/or backward compatibility. My various searches trying to solve this problem turned up a lot of people having obscure issues with the transition: this is not the way to encourage wide uptake of a tool set.

XCode 3.x

2009-10-20T06:17:00.001-07:00

I've been coming up to speed on iPhone development and thought I'd share some of my experiences.

The first is that Beginning iPhone 3 Development is a very useful starting point. I tried a couple of other resources but finally settled on this. I do like books better than video when learning a new environment but this book also has the advantage of being up-to-date and accurate. I hate trying to learn an environment/languiage when the examples are wrong! Beginning iPhone 3 Development employed a technical reviewer who worked and verified all of the examples. Shouldn't every book like this have a technical reviewer? -- the world would be a better place.

A few observations on the development environment, which although reasonable is a bit more primitive than netbeans or eclipse, especially around refactoring
e.g.,
renaming a class/header file doesn't rename all of the imports throughout the project, and there isn't a "Refactoring" capability that I've found that does this.

The C aspects certainly harken back to an earlier era, e.g., one has to define a function in a .m file and declare it in a .h file for it to work correctly. At least Code Sense minimizes the chances for mistyping in this case.

XCode only allows you to view the interface specification (the .xib file) via the interface builder--however it is useful to realize that the .xib file is really an xml file that can be viewed in a normal text editor e.g., emacs.

Misspellings count and don't seem to generate errors:

Surprised that Code Sense doesn't prompt when overriding methods from the superclasses, which causes the the classic "why wasn't this method called debugging session"

Similarly the compiler doesn't tell if you're calling undefined methods

Misspelling accessor e.g.,
childController.tltle = prez.name;
vs
childController.title = prez.name;
gets the error: request for member ‘tltle’ in something not a structure or a union

Also can't believe that there isn’t enough introspection so that you still have to do these:


#pragma mark NSCoding
-(void)encodeWithCoder:(NSCoder *)encoder{
	[encoder encodeObject:field1 forKey:kField1Key];
	[encoder encodeObject:field2 forKey:kField2Key];
	[encoder encodeObject:field3 forKey:kField3Key];
	[encoder encodeObject:field4 forKey:kField4Key];
}
-(id)initWithCoder:(NSCoder *)decoder{
	if(self = [super init]){
		self.field1 = [decoder decodeObjectForKey:kField1Key];
		self.field2 = [decoder decodeObjectForKey:kField2Key];
		self.field3 = [decoder decodeObjectForKey:kField3Key];
		self.field4 = [decoder decodeObjectForKey:kField4Key];
	}
	return self;
}
#pragma mark -
#pragma mark NSCopying
-(id)copyWithZone:(NSZone *)zone{
	FourLines *copy = [[[self class] allocWithZone:zone] init];
	copy.field1 = [[self.field1 copyWithZone:zone] autorelease];
	copy.field2 = [[self.field2 copyWithZone:zone] autorelease];
	copy.field3 = [[self.field3 copyWithZone:zone] autorelease];
	copy.field4 = [[self.field4 copyWithZone:zone] autorelease];
}

On The Bright Side

@synthesize obviates a lot of useless typing.

Categories seem cool and I plan to explore them further. Categories let you add methods to an existing class -- the source code of the existing class is not required.

From the xcode 3.1 doc Categories

Provide a simple way of grouping related methods. Similar methods defined in different classes can be kept together in the same source file.

Simplify the management of a large class when several developers contribute to the class definition.
Let you achieve some of the benefits of incremental compilation for a very large class.

Can help improve locality of reference for commonly used methods.
Enable you to configure a class differently for separate applications, without having to maintain different versions of the same source code.
To declare informal protocols.
See “Informal Protocols ,” as discussed under “Declaring Interfaces for Others to Implement.”

The doc also contains a suitable caveat:

Although the language currently allows you to use a category to override methods the class inherits, or even methods declared in the class interface, you are strongly discouraged from using this functionality. A category is not a substitute for a subclass.

That is, categories are powerful and can blow your foot off, the "power tool" version of shooting yourself in the foot, if you're not careful.

You can schedule actions to happen in the future and then cancel them when superseded by a subsequent user action.

[NSObject cancelPreviousPerformRequestsWithTarget:self selector:@selector(singleTap) object:nil];
			[self performSelector:@selector(doubleTap) withObject:nil afterDelay:.4];

It is very nice to have that capability just "built in."

Patterns in Network Architecture

2009-09-14T17:17:00.001-07:00

I recently finished reading Patterns in Network Architecture by John Day. It's an attempt to rethink network architectures and polish up "the unfinished demo" that is the internet.

Now, I'm not a network guy, so I can't evaluate the quality of his proposed solutions in any detail, but I liked his thought process and found it a useful read for anyone interested in a good example of thinking through a hard problem and coming up with a disciplined "minimal covering" solution.

Day focuses upon discovering the appropriate layers and layer structures necessary for communication. He works up from interprocess communication on a single machine to processes communicating across multiple machines.

The implications of this analysis are interesting in and of themselves and closely resemble structures seen in other systems. His metaphors are primarily in terms of name lookup and binding, using compilers and operating systems as examples (I have to admit that this only feels partially correct to me: I think the full problem is more akin to providing the data/instructions to a processor and therefore needs to include the mapping from a "memory location" to an actual address accessible by the chip's execution unit e.g., it should take into account caching, TLBs et al.).

The first and foremost conclusion in Day's opinion is that there is one layer that provides interprocessor communication and it replicates. That is, the structure of each network layer is the same, but the policies and optimizations differ depending upon the particulars of what the layer is connected to. Every layer has three parts, data transfer, IPC (Interprocess Communication) control and IPC management -- where control is short cycle management.

In his words

"Layers have two major properties that are of interest to us: abstraction and scaling (i.e., divide and conquer). Layers hide the operation of the internal mechanisms from the users of the mechanisms and segregate and aggregate traffic. But most important, they provide an abstraction of the layers below"

When moving from a shared to a distributed environment the core new functionality is an Error- and flow-control protocol (EFCP). This protocol replaces the shared memory mechanisms of the single system to ensure reliability and to provide flow control in the environment of communication between two systems. An EFCP PM is a task of the IPC process. Although in theory, such a process could be included even when communication is on a shared processor, in practice, this communication is so reliable as to make it redundant.

I think that the core insight/technique was to frame communication from the network perspective as being from application to application and not as interacting with the network e.g., in the figure below (6-15 from the book) communication is conceptualized as being across, that is between applications at the same layer, rather than down through the network and back up to the other application.

The application concerns itself with developing a shared state with its partner application. The N-1 layer provides an an abstract aggregated API to support the application's view of the communication and performs whatever aggregation and abstraction necessary to develop a shared state with the N-1 application on the other side. It then hands off details to the N-2 layer which gets it to the N-2 layer on the other side, etc.

Yes, it is just encapsulation all over again, but as we all know, finding the right thing to encapsulate and doing it in a practical way takes a lot of work.

I'm eliding a number of the other key findings of the book such as

The observation than an address only needs to be unique within a (distributed) application layer
A connection is made only after authentication has been obtained and the connection authorized etc.

All are developed in a thoughtful way showing deep insight into the problem.

Although not the easiest read for someone without a strong networking background, it is an interesting and useful exercise to watch someone so well versed go through the process so thoughtfully.

Flavors of Architects and Analysts

2009-08-24T18:00:00.001-07:00

I was recently involved in a discussion on the difference between architects and business analysts and decided to put together my thoughts on the subject.

Here they are: for each category “Architect”, “Business Analyst” there are a number of sub-categories

I normally think of three levels of Architecture:

Application – addresses evolution and delivery of a single application (small set of highly related functions), activities include:
- Partitioning functionality within an application.
- Developing best practices.
- Assuring flexibility to meet current and immediate business needs.

Platform – addresses evolution and delivery of a multiple application for a particular business area (applications grouped by functionality/user community), activities include:
- Assuring a commonality of results.
- Providing for fine grained interoperability.
- Developing frameworks that allow multiple applications to ship on a common substrate.
- Building in flexibility to meet business developments on the planning horizon (this year/next year goals) for moderate-sized groups within the company (~100 people)
- Assuring that a substrate achieving these goals is in place so the applications can pick it up at the appropriate time.

Enterprise:
- Identify core data elements and services that will be important over the strategic timeframe.
- Assure that there is an appropriate mix of flexibility/capabilities to meet strategic goals e.g.,
  - If acquisition of companies is a strategic goal, methods for rapidly merging personnel, purchasing, and operational information systems are important.
  - If acquiring products is a strategic goal, capturing data about supply and delivery chains etc. is important.

On the business analyst side I similarly think of three levels:

Department Level:

What are the processes involved in performing a function: including as is and to be states?

Division Level:
- What is the external business goal that is being addressed?
- Is this the right way to address it?
- Should functions be merged/refactored?

Corporate/Strategic:
- What are the strategic business differentiators going to be in the business that we want to be in 3-5 years?
- What must the business look like to support them?
.

At all levels there should be some time spent to look at potential inflection points that might radically change the structure of delivery and build in flexibility to address that potential, e.g.,

For architecture think outsourcing, software as a service, location aware computing, etc.

For business think increased competition in product acquisition, competition from generic products, regulatory/legal landscape.

How these various functions are actually assigned to people depends a lot upon the scale of the problem, the level of risk/uncertainty, the talents of the people involved, and the flexibility of the organization. At one extreme, a star performer building upon a solid platform architecture (in the sense used above) can be a combination business analyst/application-architect/developer for a system serving 50+ users in a non-validated environment.

I think it important that business analysts are able to understand the business processes and vocabulary well enough so that there is a good transmission of information between those expressing the needs and the analyst. This implies a greater stickiness between the analyst and the user community than is necessary for the architecture or project management functions.

Similarly there are commonalities in the level of abstractions used in the architecture level (Application, Platform and Enterprise) that imply that levels are sticker than business areas or technology.

In theory, project management is more transferable, but the stickiness here revolves around legal requirement, diversity of end use (geography, user types), system novelty.

Open Source as an Architectural Driver

2009-07-26T07:18:00.001-07:00

Phillip Longman's post about open source products in healthcare (specifically, the VA's "health IT system") talks about how the Midland Memorial Hospital's installation of the VA system went well because it was easy to use, and it was easy to use because it was open source.

Well, the first one wasn't a surprise. Well-designed easy to use software is well, easy to use. The fact that this leads to successful system uptake/user adoption should be no more surprising than the fact that people like their iPhones. The second factor being easy to use because you are open source is a bit of mental speed bump: Easy to use open source? Well, maybe if you are a developer. The article states that the ease of use stemmed from its ease of modification. Now I can't comment on that since I am unfamiliar with the product and have never been involved in a hospital centric system.

However, open source and ease of modification? Yes, that fits. A successful open source project, by definition, must be relatively easy to modify: An interested developer should be able to jump in, modify the code and stand up a running test build in short order. Otherwise the project won't attract enough attention to survive. More importantly, I think a system that is easier to modify will leap ahead in functionality even if it starts out behind.

This is one of the reasons we've seen such useful build/test tools come out of the open source community e.g., ant/junit/maven etc.. All open source projects needs tools like these to succeed since they are critical situations where you cannot afford a dedicated buildmeister or QA organization e.g., you're a developer modifying the code to satisfy your needs.

Similarly a clean, modular, layered structure is going to be favored, codependencies (A depends upon B, B depends upon A) are going to be rejected, since they require understanding two pieces of code and their interrelationship to perform a successful modification.

Both of these issues can be more easily compensated for within a "closed source" shop, since revenue-generating projects employing full time personnel can invest the time and discipline to keep things working, even if the software has a few points of poor structure. In addition, if the points of 'bad architecture" are manageable there is little incentive to fix the problem, since it might easily cost more to fix than it's worth, given the costs of running a full-time development team.

One of the strongest examples I think we've seen of this is the EJB vs. Hibernate controversy with the resultant "conciliation" of EJB3. Hibernate, being open source, could give developers what they needed rather than what they "should want" & it won due to its simplicity and speed.

This argues for a bias towards an open source style, even in a closed source system. For example, can your consultants (internal or external) add/modify deep system functionality? Designing your system in a way that supports such modifications will make the architecture better and help keep the product fresh.

Again, why this works for hospital systems is beyond me, unless there has been an undocumented rash of coding by doctors and nurses.

iPhone: changing the way we think

2009-06-30T14:45:00.001-07:00

I'm struck by how the iPhone has changed the way we think about what can be done with software based devices assisting us as beings-in-the-world. I'm doing a Heidegger reference here because the iPhone is more than just ubiquitous computing: a device always at my side that could answer those important questions like:

Is there good coffee close by?

What's the weather going to be like later?

How old was Kennedy when he was elected?

Although it certainly is that, it has become a lot more, changing both the economics of software delivery and what it means for software to be delivered.

It's not just that there are a billion apps (or so it seems) in the app store, but the economics of iPhone software is such that a small gaming company can do a novel game e.g., tying rope around wooden blocks, get traction with it and make money. That didn't sound that amazing until I read an interview with the developers in Gamasutra that reminded me how hard it was to make money in computer games pre-iPhone. Not that it is easy now, but compared to the stories I heard when I attended a few Game Developer conferences earlier in the decade, it is trivial. Let's just say that the economics of doing a platform (XBox, Playstation, wii) or PC based game were daunting, to say the least, and the likelihood of getting paid for your game was minimal, even if the game was successful.

The core of the iPhone's difference is as a platform that is easy to use, location aware and ready-to-hand -- more like a hammer than a computer.

As a platform it is sufficiently distinct that it is also effecting the way we think about delivering healthcare. Looking at this list highlights core features that are "new," not "new" in the sense of being completely unheard of, but new in the sense of being practically available for use by the overwhelming bulk of the user community -- sort of like the difference between having a generator kit/knowing about electricity and having an electric grid that you can plug your device into.

As a user-assistant, the iPhone allows me fully exploit the affordances of my current location. I can see where I am on a map, look at overhead imagery of my current neighborhood to see if there is something that I want to photograph, and, if there is, use a small application to grab the geo coordinates so I can later tag the photos I took with my (non-GPS enabled) camera.

The end result is something that is always with you, knows who you are, knows where you are, has connectivity both up (3G/internet) and down (bluetooth to local devices) while providing a simple effective mechanism to easily add functionality in small increments.

I think this makes it the biggest game changer since the rollout of the internet to the general public. However, I also realize that this means that it is time to code up a small test application for the iPhone.

PS: I don't have any experience with the Google android platform or the Palm Pre; these observations may apply equally as well to them.

Linked Data

2009-06-01T18:37:00.001-07:00

Finally, thanks to a discussion with Eric Neumann a few weeks ago, I'm beginning to understand what Linked Data is all about. First a caveat -- although I credit Eric for helping me see how linked data fits into what I'm doing, the following interpretation is strictly my own as are errors of omission, commission or orthogonality, although I think my view is supported by the Design Issues document.

The short story is that linked data provides stable identifiers for stuff (a more abstract form of things). These stable identifiers then allow you to say things about this (particular) stuff without necessarily making a strong ontological commitment.

I like this. It provides for interoperability and integration. It does not provide any inference guarantees which is fine by be, and something that I have been advocating for a while. The Linked data site also has links to a number of datasets which publish stable identifiers for useful stuff. The site also gives examples of how to publish your own data.

Hopefully data.gov will provide its data in this form in the near future.

Wolfram Alpha

2009-05-17T18:42:00.001-07:00

Wolfram Alpha is supposed to be launching in the next few days and has been getting a lot of publicity. For background, here's a link to a short YouTube demo of Wolfram Alpha, and a NY Times article and Doug Lenat has a nice post on his impressions.

From what I can see (and I don't have access) even though it doesn't live up to some of the early hype, it achieves a very interesting result: it allows retrieval of general computable information using a simple natural language processing (NLP) interface.

This allows for analysis similar to that permitted by a data warehouse, but within different design space. The design goals of WolframAlpha, unlike those of a data warehouse, preclude prestructuring the data in marts to allow rapid querying of the data in relatively well defined ways. However similar to the mart/warehouse situation you must still provide a speedy response to the quantitative queries to prevent users from drifting away while waiting for an answer.

The question is how is this done? Rumors on the net indicate that the underlying data is an RDF triple store, which makes a lot of sense since RDF Triples constitute a vertical, model free storage approach. In operation, I imagine that the queries provide nice entry points for initiating a spreading-activation fan-out process on the graph. When the activations intersect you can proceed to roll back up to the initiation points suggested by the query, clustering in a bottom-up data-driven fashion along the way. The clustering also affords a natural way to structure the data for presentation to the user.

Although I'll admit that this is just an educated guess as to the mechanism, it does suggest an interesting set of technologies involving fast linking and roll up of data for ad-hoc queries without requiring a lot of effort to tune the data to a specific query.

Generating a set of vetted and annotated data is a different problem, but hopefully would not require a significantly greater level of effort than the ETL portion of current warehousing efforts.

Wolfram Alpha therefore constitutes another factor leading me to be more vertical in my storage designs. In the coming months, I'm hoping to run some benchmarks on production hardware/datasets so as to ground the practicality of this approach and then get permission to publish the results.

Update 18 May 2009: I did try Wolfram Alpha today and it failed on my first try "age distribution of England vs UK," not so much from any idiosyncrasies in parsing my query, but because it appears to be encoded with the identity "England == UK." This just goes to show how important it is to be spot-on with your identity information aka "synonym tables are easy, antonym tables on the other hand......aren't."

launchd

2009-05-06T17:08:00.001-07:00

I just upgraded to a new laptop (driven mostly by the need for more RAM -- hopefully 6G will be adequate for a couple of years). It got me thinking: even though it's great that the Mac will copy all of your old apps over effortlessly to your new machine, it also happily copies all your old unused cruft over to your new machine, and that's not so great.

So, in the spirit of good hygiene (and H1N1 preparedness), I decided to open up the console and look to see what I might find. I discovered that I had a couple of launchd jobs that referenced executables which didn't exist on my system any more e.g., carbon copy cloner.

I have been able to rid myself of all the launchd issues, by cleaning up the Launchdemons/launchagents under the Library folder but
I still haven't been able to rid myself of all of these

/Applications/Safari.app/Contents/MacOS/Safari[54428]: Warning: accessing obsolete X509Anchors.

This is even after searching the web a couple of times. I think the problem starts up after I open an article from NewsFire, but I'm not completely sure. This is definitely a space in which I believe correlation is not causality.

If anyone has any ideas on how to fix this, I'd appreciate it.

BTW it is really nice to have a built in tool like Console: it is simple and effective with just that little bit of extra functionality (string filtering) that makes all the difference in usability.

Java Concurrency

2009-04-20T15:12:00.001-07:00

A predictable side effect of having (way too) many years of experience in Java is that certain "new" features escape your notice. This is particularly true if the IDE's don't pressure you into changing your previously successful, and still functional patterns (the way they do with generics).

I realized this when reading Java Concurrency in Practice. It's a very good book -- I can't say it really opened my eyes on concurrency since I had done some work on multi-master VME based real-time systems years ago, but it is spot on, well written, and a nice refresh. In addition, it made me aware of the thread/concurrency capabilities available in newer versions of Java such as ThreadPoolExecutor

I recently built a file crawler/hash-calculator/storage system as part of my namedData work using an ArrayBlockingQueue and explicitly created threads. ThreadPoolExecutor appeared to allow an easier approach with cleaner shutdown/interrupt semantics.

Java tips has a clear example -- the primary change that I would make to this example is to size the thread pool based upon the number of processors available (on my laptop this returns the number of cores).

It took me less than an hour to make this change, test the code, etc. The final product is a lot cleaner, has better shutdown behavior, and even feels like it runs faster. Definitely the right way to go.

owl:sameAs is a very strong assertion

2009-04-06T17:53:00.001-07:00

There's been an interesting discussion on the public-semweb-lifesci mailing list with the subject "blog: semantic dissonance in uniprot" which, appropriately enough, was spurred by a blogpost entitled semantic dissonance in uniprot. This post talks about a uniprot entry which listed a Drosophila (fruit fly) protein sequence as having been isolated from "a young sporophyte contained within a seed."

The point being that although one doesn't find fruit fly genes in plants, following the owl:sameAs link leads directly to that conclusion. This generated a very long, fairly thoughtful and minimally flame based conversation on owl:sameAs and identity in general.

As the discussion progressed, the problem with associating identity across graphs (ontologies/systems of data developed by different organizations) was noted, e.g., (in pseudo annotation) mySystem:itemA owl:sameAs yourSystem:itemX, the issue being that the use of the terms is usually subtly (and often not so subtly) different between the two systems. This problem is especially apparent when making assertions about real objects which exist independently out in the world. For example: "gold" may have a property, but does the property adhere to a single molecule, or a group of gold molecules and if so what characterizes a group of the appropriate size? For example given:

A nanotechnology view of gold (still under development)

A semiconductor view of gold (probably reasonably well characterized)
A jewelry view of gold

what are the precise boundaries of their applicability? The issue doesn't arise in a system developed for nanotechnology, semiconductors, or jewelry. The problems surface only when these systems are linked together.

My thought is that the difficulty centers around the extreme power of owl:sameAs which indicates that things are identical in all contexts. However in the physical world not only is context everything, but context is also inherently incompletely specified.

In practice many of us heuristically treat identity in the physical world as operating as if identity means indistinguishable in this context, with the context being implicitly dependent upon the issue being considered. I would claim that this is the only reasonable way to proceed when reasoning in a practical manner about what is true about particular objects in the world (abstractions can obviously satisfy stronger conditions since they are abstractions -- with the context factored out to any level desired).

In the physical world, we cannot assure that even the ability to track a particular item with unlimited precision would allow us to make statements about that item which would hold through time. For example, although we might make assertions about a particular atom (#0x177FFEAA) of gold and its behavior, some if not all of the assertions may fail under unexpected conditions, e.g., after an event that alters the structure of the nucleus (nuclear collisions, extremely high temperatures etc.). Exhaustively specifying all of these conditions is impractical at best -- which is one of the reasons the phrase ceteris paribus has remained with us for so long.

In my own work, since I never worry about tracking individual atoms. I gravitate toward weak rather than strong assertions of identity, trying to be very attentive to context. This is very much in the spirit of the middle distance as developed in Brian Cantwell Smith's On The Origin of Objects. Smith's point is that our intuitions are well tuned to objects about our size that we interact with frequently. In data integration and architecture work (I had to get there eventually) it implies that integrating across fields that interact to some degree in the "world" is going to be more feasible than integrating across those that don't interact. The give and take of the practical interaction has allowed us to identify the particular features of each item that are important in context.

OSX Performance Analysis: Instruments

2009-03-23T18:03:00.001-07:00

I started working with OSX's Instruments performance analysis tool, partly out of curiosity and partly because I had just fixed a performance problem in an application using an ad hoc a priori analysis. It happened to solve the problem, but I have enough experience with performance issues to know that the a priori guess is often wrong.

Instruments is heavily related to dtrace and shares a lot of its core attributes. The key attributes are that it is low overhead and works with (almost) anything running on your systems (OSX apparently has the capability for some applications to turn off monitoring for security/DRM reasons).

There's a lot to like here: you can easily get it up and going on your system and it the analysis section is very user friendly:

Especially nice features include

Low overhead: the peak CPU usage I saw for the tool was ~ 16%
The ability to display exactly what is going on under the read head (the upside down triangle above the graph)
Being able to display parameters that you didn't think of turning on during the run. All parameters are captured. The selection only impacts the display -- a godsend for anyone who has had to rerun a test because they forgot to capture a parameter

That said, I couldn't get any particular instrument to focus only on the process specified. As you can see, all of the instruments capture all of the activity, even though they were set to focus on different processes. Additionally, the "default action" kept resetting whenever I dragged a new instrument onto the display.

It is still a very worthwhile tool, but if anyone has any tips as to how to get around these issues, I'd appreciate it.

Kindle

2009-03-04T18:26:00.001-08:00

It's a bit off topic, but I thought I'd point out how useful a Kindle can be for consulting. You can carry at least 500 reference books on it (and who needs more than 490 anyhow?). It is also very light and easy to read.

I do have a couple of qualms. It is a page oriented display (no scrolling), no touchscreen and has no spatial indexing e.g., the top side of the right page half way in, but other than that it's a win.

An important note on utility: O'Reilly e books can be read on the Kindle. The truly great thing about O'Reilly's e-books is that you get both the Kindle compatible mobiPocket files and the more aesthetically pleasing PDF files (for me, aesthetics matter--even in a SQL guide).

You can mix and match reading and reference between the formats depending upon your preference. Thankfully the files aren't copy protected. Thanks, O'Reilly, this is a very nice touch.

Seambay modifications to access Seam Annotations

2009-02-17T08:56:00.001-08:00

This post extends my last one about accessing Seam from the command line. Here I describe the transition from using EntityManager to using EntityHome.

The first thing I did was to create a new folder for the webSevice sources, which meant that I had to add this directory into the build.xml file and add all of the libraries into the compile path in NetBeans (both of which are obvious, but both of which I always forget to do).

The next was to make my action work similarly to an .xhtml page and interact with a home object rather than directly with the EntityManager
going from:

      
if (fileData == null) {
            fileData = new FileData();
            // various actions on fileData
            entityManager.persist(fileData);
        }

       
if (fileData == null) {
            fileDataHome.create();
            fileDataHome.persist(); //side effect of creating the defined instance
            fileData = fileDataHome.getDefinedInstance();
            // various actions on fileData
            fileDataHome.update();
        }

which also required adding these lines to components.xml

xmlns:transaction=”http://jboss.com/products/seam/transaction”   
....    
<transaction:ejb-transaction/>

None of which was particularly difficult and I was up and running in a hour or so.

Seam From a Command Line

2009-02-02T17:36:00.001-08:00

I recently wanted to access some seam derived functionality from a command line java program (something that I could run via cron). I ran into a few minor problems and thought I'd share their solutions.

The first problem was that seam annotations like @Logger won't work. I guess it isn’t that surprising, but the jboss seam annotations are unavailable to a command line program (at least not easily) since the portions of the framework that enables these annotations are designed to operate within a server.

This was disappointing. The @Logger annotation is really useful, but I couldn't come up with a way to get it going.

This pushed me into wanting to use web services as much as possible to take advantage of other annotations that I had built into my system, e.g., the ability to automatically stamp an object with time modified and time created to support temporal data operations.

I did not find the seam documentation about accessing seam web services particularly clear (especially when using netbeans) so I turned to the netbeans tutorial and was quickly up and running with the seambay example.

Note:

and

Data Deduplication

2009-01-05T18:10:00.001-08:00

IEEE Computer recently published a short survey on data deduplication. Conceptually deduplication is isomorphic to the named data approach I posted about a few weeks ago.

The vendors discussed include

ExaGrid

FalconStor

NEC DataRedux

Quantum DXi

Sepaton DeltaStor

Symantec

Since I'm not sure how long the pdf of the article will be available, I'm posting this as a follow up to my previous post.

My preference would be to see these capabilities built right into the internet/operating systems rather than separate utility servers.

Discussing Architecture

2008-12-23T18:20:00.001-08:00

If you're like me, you are forever grappling with finding the right format for discussing architecture with end users.

There are the standard architectural diagrams, but they really don't capture:

How the end users relate to the system
How all of the components are strung together to support the overall business processes
What artifacts (data, documents, etc.) are produced

I think there may finally be a "better mousetrap" available in ArchiMate which has a suitably small number of patterns/best practices that allow you to capture what's important in a large system at the appropriate level of detail. The most important aspect of ArchiMate is that it is a mixed mode modeling language with a distinctive, simple shape for each type of artifact.

ArchiMate allows for seven different types of things: Services, Processes, Organization, Products, Information, Infrastructure, Applications, and Functions.

As of yet I am not completely clear on the breakdown of these categories and to be honest I'm not sure that it really matters. ArchiMate is a communication tool. If it helps you communicate, it has served its purpose. If your way of breaking down the architecture is slightly different from their recommendations (which I strongly feel could use more examples) it may have no more impact than if your class structure for a domain is slightly different from someone else's. Don't get me wrong: I'm a big fan of standards when they have been properly vetted and heavily used, but until they hit that point it is important to be flexible in using them. Flexibility allows a standard to grow and cover a sufficient portion of the domain; otherwise it will wither from lack of use.

The presentation format that speaks to me the most is the layered diagram as shown on p 11(figure 12) of the Enterprise Architecture Development and Modelling paper. See below:

Here's my simplified take for a clinical trial system used by physicians and patients.

What I like about this format is that all of the elements on a particular layer are at the same level of abstraction and are easily placed in relationship to the other few things that are at that same abstraction level. Simultaneously, you can see what supports (and is supported by) a particular component. Each perspective (which appears on the same diagram) only requires concentrating on a small number things at a time and can easily be held in your short term memory.

One of the key things about the layers is that they distinguish externally available from internally consumed data interfaces -- especially highlighting those that cross abstraction boundaries. These external interfaces are distinguished from those that support multiple applications at the same level in the stack. Such "external" interfaces (which are internal to a particular level of abstraction) are more easily altered since they are more tightly coupled organizationally. This nicely foregrounds the implication that care should be taken in designing the external, level crossing APIs and lifecycles since changes to them will be harder to coordinate given the diversity of the interested parties.

Name that data

2008-12-08T17:28:00.001-08:00

There is an interesting trend that I feel has the potential to fundamentally shift the way we think about how data is used in networking and applications.

It involves attaching unique names to data, i.e., referring to the data by its SHA1 hash value. Unique identifiers allow a number of things. For example, they allow you to retrieve the data from the network without worrying where it resides on the network, as described in
A New Way to look at Networking in which Van Jacobson discusses breaking out the content of the page from the page itself.

At 42:47 in the talk there's the section on
Dissemination networking

in which
data is requested, by name using any and all means available (IP, VON tunnels, zeroconf addresses, multicast, proxies, etc),
Anything that hears the request and has a valid copy of the data can respond. The returned data is signed and optionally secured, so its integrity and association with the name name can be validated.

Rather than getting all of the data from a particular location e.g., as specified by a URL, we get the identifier for the content of the data and then let the network supply the data from the location "nearest" (using the appropriate distance metric) to its use.

A similar idea exists in ZFS' implementation of a copy-on-write transactional model:

ZFS uses a copy-on-write transactional object model. All block pointers within the filesystem contain a 256-bit checksum of the target block which is verified when the block is read. Blocks containing active data are never overwritten in place; instead, a new block is allocated, modified data is written to it, and then any metadata blocks referencing it are similarly read, reallocated, and written.

This design allows small changes in large files to be reflected by changing only the blocks that have been altered rather than by rewriting the whole file to a new location. This simplifies backup procedures, reduces R/W bandwidth requirements, etc. Part of what's significant here is that we're dealing with abstractions of the data, e.g., checksums rather than the data itself. If we use cryptographic hash functions rather than checksums the ideas become isomorphic aka, get me this block from wherever it is.

This has a number of potentially interesting applications depending on the granularity of the "named data". As a simple example, answering the obvious: "am I working with the copy of the file that was emailed to me last Friday, or an older version?" Even with coarse grain naming it would be possible to create mashups of music already on a users computer -- just transmitting offsets into and segment durations of existing content gets past DRM issues entirely (morally if not practically-- since much DRM protected media is encoded).

Uniquely naming the content is not just about networking, nor is it just about data but is really about the cross product of the two: the data and its location/retrieval. Data location and retrieval is most of what is involved in computing: unless data is being actively processed by a computational unit (in this case, I'm talking about the integer and floating point units on the chip) the rest of "computation" is about the retrieval and storage of data e.g., do I put this in a frame buffer, in the cloud or in a shredder?

Imagine an ecosystem of data that represents everything that you own/care about -- you could partition this data into multiple overlapping categories based on any number of attributes such as:

temporary: make no copies

pieces of a larger whole: applications would minimize the number of named datasets that they change when they perform updates e.g., editing out that first minute from your video won't change every block of the video.

number of independently survivable copies required, with what longevity?
coupling the data to geographic position: How closely should the data follow me as I move around the planet? Can it stay where I put it, or should it go where I go and be available for high bandwidth processing.

This are just my first pass ideas. This concept of divorcing the data off from its location and higher level structural organization opens up the potential for a whole new set of applications which provide enhanced user functionality by pushing a lot of these data management, replication and caching issues deep into the infrastructure.

Semantic Interoperability: from Mashups to Inference

2008-11-24T17:33:00.001-08:00

My last post looked at semantic interoperability from the standpoint of the CDISC BRIDG model. Thinking back on it, I found that writing it left me with more questions than I had before I started.

I have to admit to being unclear as to what is meant by “semantic interoperability,” since I have heard it used in a number of different ways depending upon the audience. (apparently I’m not the only one: the wikipedia entry on semantic interoperability has the caveat All or part of this article may be confusing or unclear.)

"Semantic interoperability" puts requirements on the data, on the models, and on the processes of using them. How we respond to those requirements implies different interpretations of what it means to be semantically interoperable.

I think that there are three basic ways of using data that are "semantically interoperable".

“hands-off” data integration between designated well curated systems--this is the way in which I think it is used most often.

“hands-off” data integration between any systems sharing common identifiers e.g., publish an interface and allow anyone to use it.

The dual use of integration with any published interface that provides the data that you're looking for - which I think is less common e.g., I'll use any map, or any book information service rather than Google/Amazon (or Yahoo/BN). I haven't seen this that often and I think it sounds a bit sketchy.

Using OWL reasoners etc. for inference across systems to generate new information.

The requirements around these things are pretty different, both in data quality and in the congruence of the requisite component models.

The first “hands-off” data integration between any systems sharing common identifiers doesn't really require any similarity of models other than around the key integration point(s). You need the name of the referent of the data, the name of the data item and the format of the returned data e.g., "The first president of the United States"; "date of birth" returned in ISO 8601 format. Of course the more points that you want to make referenceable between the systems, the more the models have to match e.g., your model of US presidents has to contain a way of dereferencing the person and that person's date of birth. The more independently developed and maintained your systems are, the more quickly you want to start using RDF to give you very stable identifiers for your referents.

If the systems are required to do some curation/analysis of the data, the exported models need to match more closely so that you can derive the correct metrics to perform the analysis and understand the relationships between individual data points. A good example of this comes from Nick Malik who points out

So, if you look in a database and you see a purchase order... has it been approved or not? The answer depends on the business unit that created it.

Your models can be in a number of different forms (UML, OWL, etc.) and be wildly divergent from the underlying reality, but if the delusion is shared you can achieve some synergy.

Inference of course requires (at least) a locally full up OWL ontology since that's the only modelling language that permits inference. Models also have to more closely resemble the shared "current best understanding" of reality (which is of course a moving target in a scientific domain) or the resulting inferences will be worthless, or at best amusing.

However, doing an ontology is a big deal (see The Joy of Ontology by Suzanna Lewis for a discussion). The increment of commitment that we're making here is decidedly non-trivial, especially if the domain that we are trying to model is of substantial size.

I think the clinical trial domain is a good example of substantial size. BRIDG took a long time to do, it is still undergoing revision and does not allow inference. I would argue that given the continued refinement of some of the base terms (sex and gender were recently updated), even if there was an ontology, hands-off inference is not something that lies in the near future, simply because the ground doesn't provide a sufficiently firm foundation.

Just for clarity -- this doesn't mean that turning loose an inferencing bot over a sufficiently sized test set would not yield interesting and perhaps even transformative results. It just means that the inferencing would be part of a web research project rather than a production operation.

I could be wrong on this (and gladly so), but I did live through AI winter an I can no longer utter the phrase "sufficiently smart compiler" without irony.

Semantic Interoperability: Adverse Events

2008-11-11T10:28:00.001-08:00

When reviewing the Bridg Release 2.0 Static Elements report.RTF I did look in a bit of detail at the adverse event model.

Here's a summary:

The AdverseEvent
class is decribed as having the following connections and attributes

Connections

Association link from class PerformedProductInvestigation

Association link from class Subject

Association link to class AEOutcomeAssessmentRelationship
Association link to class AECausalityAssessmentRelationship

AEActionTakenRelationship

Generalization link to class PerformedObservationResult

Association link from class PerformedProductInvestigation

Generalization of PerformedActivity adding an evaluationMethodCode attribute; PerformedActivity captures the duration of the activity

Association link from class Subject -- the clinical subject (An entity of interest, either biological or otherwise.)

Association link to class AEOutcomeAssessmentRelationship
links the AE to an observation
For example, recovered/resolved, recovering/resolving, not recovered/not resolved, recovered/resolved with
sequelae, fatal or unknown

Association link to class AECausalityAssessmentRelationship
links the ae to an observation.
For example, when an adverse event occurs, a physician may evaluate interventions that may have caused the
adverse event.

Association link to class AEActionTakenRelationship.
Specifies the link between an adverse event and the steps performed to address it.
For example, study dose reduced, protocol treatment change, etc.

Generalization link to class PerformedObservationResult
links all observations/protocol deviations etc together with a report.

The AE itself has the attributes:

gradeCode

severityCode
seriousnessCode

occurrencePatternCode
unexpectedReasonCode

expectedIndicator

highlightedIndicator
hospitalizationRequiredIndicator

onsetDate
resolutionDate

The end result is a structure that has a formal relationship that is well thought out, should cover all situations and permits systems to interoperate.

In my mind, this is not the same as assuring semantic interoperabillity. For semantic interoperability to really occur, the grade codes must be comparable across sites, hospitalization criteria must be identical (or at least commensurable) etc.. Achieving this comparability requires continuing education and harmonization efforts, constant feedback of metrics to practitioners etc.. It therefore represents a much higher bar.

This is not in any way a criticism of BRIDG. You need something like BRIDG -- a well vetted industry standard -- to be even be able to begin such an attempt. However, true semantic interoperability involves not only the structure of the data, but the data itself.

CDISC/BRIDG

2008-10-27T18:52:00.001-07:00

Last month I attended the Boston Area CDISC Users group meeting (BACUN). All of the presentations were interesting and useful. However, I found that the one by Lisa Chatterjee on BRIDG stood out as particularly informative.

The BRIDG Domain Analysis Model is a representation of protocol-driven biomedical/clinical research.

One of the goals of the effort is Semantic Interoperability - I don't think that this means that "following the model" guarantees semantic interoperability, but rather that BRIDG constitutes a starting point from which a semantically commensurable system can be built. The bridge team appears to view the model as a foundation for other more problem specific representations (CDISC/HL7 etc.). The idea being that if you can map BRIDG <-> HL7 and BRIDG <-> CSDISC the HL7 <-> CDISC mapping is (relatively) straightforward.

There is no question that BRIDG represents an excellent starting place for using data in an interoperable fashion.
All in all it shows a very inclusive approach
-- and a surprising openness to modifying the model to ameliorate difficulties encountered in use.

The core modeling language is UML and spreadsheets are used to track much of the mapping (there already is a draft version of a spreadsheet that maps the BRIDG R2.0 model to RIM2.18).

From the presentation:

I have to admit that I haven't examined the model in complete detail. However, from what I've seen almost everything that you need for the target domain is there and the level of abstraction feels right: low enough to be relatively easy to implement, but high enough so that you don't get wedged into a corner from the get-go.

I did look in a bit of detail at the adverse event model,which is represented in the Bridg Release 2.0 Static Elements report.RTF.

What we see in Figure 5 : View 4 - Adverse Event is ~ 90% of the complete domain model with a number of classes added in support of recording adverse events and tracking their eventual analysis/resolution.

Since adverse events raise some issues about semantic interoperabiltity that I want to talk about in detail I will cover them in my next post.

BTW on my mac, the only application that could open the .rtf file with the figures was open office 3. MS word 2004 elided the figures.

Seam: the ftl advantage

2008-10-13T20:24:00.001-07:00

I finished the first pass schema for the flexible drug discovery framework and pointed the most recent (2.02) GA release of Seam at the database.

My hope was that it would produce pages with the latest ajax-friendly table sorting headers -- sadly it didn't. In addition, this version retains the practice of generating pages that use labels and column headers wired to a particular language (English) rather than allowing the headers to be language dependent messages, even though the pages fully support localization (via language specific message_x.properties files in the resources directory).

I was in the process of fixing these issues manually via emacs using these instructions (I find that emacs makes it easier to select the files that I want to edit -- Netbeans picks up too many files and deselecting 70 or so "extra" files is too much work). While making these changes, I was discussing a new system with one of my clients and recommended that they consider an open source solution since they could change it (or hire somebody to change it) if at some point they encountered something that they didn't like. I thought this better than the alternative of waiting for a potentially unresponsive vendor to pay attention to their problem.

As was saying this, I thought to myself: "Seam is open source, maybe I should try to fix the code rather than edit the result." I'm usually hesitant to go down this path. Sometimes it works, but my a priori estimate is that the process of understanding the code structure, getting the build environment running etc. costs a day (or more) before anything productive comes out the other end.

However, as a proof of principle, I thought I should give it a try and see how it went.

I'm very happy to report that given a combination of good architecture and good tool selection on the part of the Seam team, making these modifications was almost trivial.

Let me explain why, and encourage you to "do this at home."
The core reason is that the seam generate-entities command operates in a number of phases, one of which generates the .xml and .xhtml files using freemarker template files (.ftl). This allows the required changes to be made without either looking at the java involved or developing any understanding of the calling structure, etc.

The .ftl files (in ./seam-gen/view/) are pretty much self-documenting (which is handy, given the level of documentation included in the files) and very easy to change -- errors thrown by the freemarker engine are clear and easy to work with.

A couple of (minor) caveats

The .java files in the src/action will need to be removed between runs of seam generate-entities.
I think it is a good idea to point the seam generator at a different target directory (specified by ./seam-gen/build.properties -- "workspace.home") while debugging so that you don't accidentally overwrite edits that you've already made (oddly I thought of this BEFORE I ran the generator)

In summary, if you find yourself making a lot of changes to seam generated files, change the .ftl files instead; it can be much, much easier.

Suddenly mysql won't start

2008-09-29T07:21:00.001-07:00

Suddenly is a bit of an exaggeration. This happened on my desktop -- I use mysql on my laptop almost daily, but only once every few months my desktop. They are similar environments: latest Mac OSX 10.5 patched intel based macs. The mysql on the desktop was transferred from my previous PowerPc based desktop, but has been used a few times since the transfer.

In any case, the normal startup action

 sudo /usr/local/mysql/bin/mysqld_safe

failed with the following output.

Starting mysqld daemon with databases from /usr/local/mysql/data /usr/local/mysql/bin/mysqld_safe: line 395: /usr/local/var/: Is a directory /usr/local/mysql/bin/mysqld_safe: line 401: /usr/local/var/: Is a directory STOPPING server from pid file /usr/local/mysql/data/rdf-8-Tower.local.pid tee: /usr/local/var/: Is a directory 080910 15:39:05 mysqld ended tee: /usr/local/var/: Is a directory

At this point I said to myself "well this hasn't been upgraded in a while, I should upgrade mysql" !!bad idea!!

The upgrade didn't solve the problem. After tracing through the script in more detail I did a

sudo ./my_print_defaults

which reminded me that

Default options are read from the following files in the given order: /etc/my.cnf /usr/local/mysql/etc/my.cnf ~/.my.cnf

/etc/my.cnf contained

[mysqld] log = /usr/local/var/mysqlLOG.log #If no specific storage engine/table type is defined in an SQL-Create statement the default type will be used. default-storage-engine=myisam max_allowed_packet = 16M #Enter a name for the error log file. Otherwise a default name will be used. log-error=/usr/local/var/ #Enter a name for the slow query log. Otherwise a default name will be used. log-slow-queries=/usr/local/var/

Which I changed to

[mysqld] log = /usr/local/var/mysqlLOG.log #If no specific storage engine/table type is defined in an SQL-Create statement the default type will be used. default-storage-engine=myisam max_allowed_packet = 16M #Enter a name for the error log file. Otherwise a default name will be used. log-error=/usr/local/mysql_ERROR_LOG.log #Enter a name for the slow query log. Otherwise a default name will be used. log-slow-queries=/usr/local/var/mysql_SLOW_l_LOG.log

 /usr/local/mysql/etc/my.cnf

didn't exist, nor did
~/.my.cnf

This allowed the db to start but I couldn't log in with any of the user accounts normally available (including root). It appears that using the mysql-5.0.67-osx10.5-x86.dmg file to update caused the db user information to get hosed.

Fortunately I'm pretty neurotic about backups and so I easily recovered just by copying /usr/local/* from my last backup.

NOTE: this backup was to a separate disk performed using SuperDuper -- TimeMachine isn't going to get files in /usr (which helps explain why the disk space used by TimeMachine is smaller than I expected).

Seam on Amazon EC2

2008-09-15T17:41:00.001-07:00

I just completed putting up a demo of my seam work on Amazon Web Services EC2 service. I primarily did this to ground my advocacy of EC2 as a good option for small biotechs that may need occasional bursts of compute power but have neither the cash to buy adequate servers for peak compute load nor the staff to maintain them.

I thought that putting up my jboss/seam/mysql demo would also be sufficiently non-trivial to give me a good feel of what it is like.

There are similarities between EC2 and other virtualization options (EC2 is based on XEN after all).

The core differences in my mind revolve around having S3 as a backing store. Since S3 is on Amazon's servers you need to pay more attention to security keys etc.

My recommendation is the go through the Getting Started Guide -- even to the point of saving a modified image. This will assure that you have the proper accounts set up both on EC2 and S3 and you have a bucket set up on S3 for storing your image.

I found it easiest to create a bucket using the Python examples -- even though I have never done much more than a "hello world" program in python (yes one that consists of

#!/usr/bin/env python
print "Hello World"

). The python code is the most self contained and required the fewest downloads of ancillary libraries in my environment (Apple OSX 10.5.4)

Building and saving an image takes a while -- I would recommend doing a reboot of your virtual machine to make sure that all of the changes take hold (boot processes start as designed etc.).

Not to overstate the obvious, but the image that you start with has a tremendous impact on the time it takes you to get up and running. I settled upon an image that already had mysql5, jboss 4.2.2 installed, and it made things much easier. In general I didn't feel that the images were particularly well documented. Not being that familiar with Fedora I thought that the difference bettween the Fedora-core-4 and fedora-core-8 was how many CPU cores they were optimized for, not the revision number. My initial foray with the fedora-core-4 image stopped when I realized that it had mysql4 rather than mysql 5.

All in all the experience wasn't too bad (I don't think that I could ever call one of these experiences "good" -- if it were good I would just be able to click a "do it" button that would do exactly what I wanted), and would have been much better if I hadn't "lost my keys".

TOGAF and Evaluating Architectures

2008-08-29T06:37:00.001-07:00

The old adage is that you always have an enterprise architecture even if you never designed one -- the point being to encourage an organization to spend the time to design one. This is all well and good, but given an ongoing enterprise, what's the best way to determine what enterprise architecture you have, where you want it to go and most importantly, how to get there.

For various reason I've started looking at this issue again and have just refamiliarized myself with TOGAF (The Open Group's Architecture Framework). I had forgotten how much I liked it: it is pragmatic, highly tailorable and focused on open cross-organizational solutions. I'm not going to do a detailed analysis of TOGAF vs. other frameworks -- it's not really my interest as I'm definitely in satisficing mode here. What follows is my (long) elevator pitch of the TOGAF take home message.

The top level graphic of the TOGAF process captures the flavor pretty well

The preliminary phase is key but very easy to overlook. TOGAF suggests that this phase consists of defining the overall objectives and scope

Define Objectives
- Assure that everyone who will be involved in or beneﬁt from this approach is committed to the success of the architectural process
- Define the architecture principles that will inform the constraints on any architecture work
- Define the ‘‘architecture footprint’’ for the organization — the people responsible for performing architecture work, where they are located, and their responsibilities

Define Scope and Assumptions
- The business units that are involved
- The level of detail to be deﬁned
- The speciﬁc architecture domains to be covered (Business, Data, Applications, Technology)
- The time horizon that should be addressed by the architecture.

What I find attractive about this whole approach is its focus on getting getting buy in from the key players in the organization, defining their roles and developing a shared set of expectations around what's going to be done as part of the architecture effort. The preliminary steps give the team some initial criteria for driving the architectural vision, but then TOGAF immediately requires them to ground it as supporting the needs of the business users. I find this grounding critical; often the business thinks that architecture efforts are worthless and often they are right because the architectural model hasn't been grounded in the business process. Note: in this case "business process" and "user's scientific process" are equivalent.

The key to the success of an architecture effort is to address current pain points as they will be reflected in the business processes that will be in place when the architecture rolls out. Sorry if the tense of the last sentence was a bit torqued. What I mean is that the architecture needs to hit a mark to support business operations as they will be in the future, not as they are now, and that some of the problems that are being experienced now will only be exacerbated by these planned changes.

The dialog with the Business Unit leader sounds something like

We're planning to do more collaborations in the future, but with our current collaborations we have a terrible time registering new users and tracking responses to our questions about the data. However, if we put an architecture in place which uses our new authentication mechanism that supports OpenId it will radically simplify the process of adding new users.

In addition, if we use vendor X's implementation of the Life Sciences Industry Architecture, queries will be automatically tracked.

Our ability to handle more collaborations on the back end is increased as our new system allows us to share extra capacity across multiple business units, thereby sharing the cost of reserve capacity to meet any unanticipated surges in demand.

Such a "political/operational" model for rolling out an architectural analysis implies that everyone who contributes to the effort should get something out of it (this is a goal, but the closer you can come to meeting the goal, the more self-organizing the system becomes).

As one proceeds around the TOGAF loop, you pick and choose what makes sense given the decisions made previously (which of course you are always free to revisit) analyzed to the level of depth that is appropriate.

Think of TOGAF as providing a (partial) checklist of processes to use and things to consider that help you reach the end state of
Boundaryless information flow (tm)

I think of it as being similar in spirit to the way the Software Engineering Institute's Risk Management Taxonomy provides a comprehensive checklist of things to consider when undertaking a project -- it keeps you from forgetting something that would be obvious in retrospect.

An aside:

My favorite quote from one of their pages:

Another company is developing a flight control system. During system integration testing the flight control system becomes unstable because processing of the control function is not quick enough during a specific maneuver sequence.

The instability of the system is not a risk since the event is a certainty - it is a problem.

Optimization: Premature and otherwise

2008-08-14T16:11:00.001-07:00

My post on structuring database tables made me think again about when (and how) to optimize one's code/design. The caveats about premature optimization are well known and well considered, as are the reasons for not following them slavishly.

In my mind the core questions involve "what are we trying to optimize" and "who cares".

My (obvious?) claim is that we should only spend time optimizing things that have impact upon the high level goals for the project. The reasons for performing an optimization should be articulated and evaluated in this framework. Driving optimizations by focusing on the top level goals sounds obvious but the tradeoffs are difficult to make in practice.

There are legitimate tensions about the proper framework relevant to the analysis. That is, it is all well and good to speak of "strategic business goals," etc. but if the product is unusable, the long term strategy doesn't matter. A strategic focus simply assures that long term impacts are also evaluated when an optimization is considered, e.g., an optimization that greatly increases execution efficiency may or many not be appropriate if it increases the complexity of product installation and set up.

For example, the "narrow table" approach that I've been advocating is designed to support deep changes in the business processes and science over the life the system, without necessitating deep changes in the data model. The "strategic horizon" for such a project is > 10 years (that is total system life of > 10 years) with the expectation that the fundamental data model reflected in the narrow tables will be relatively stable during that time. Even in a rapid prototyping environment with one or more iterations shipping each quarter, the essence of the data model should be fairly stable since fundamental changes to the data model induce data migration efforts which distract from product improvements.

The question is: are there inefficiencies caused by the narrow table approach that will make it overly difficult to achieve a usable product in the short term. My current intuition is to go with the narrow table approach and let either materialized views (which in my knowledge are most easily obtained in Oracle), special database jobs, or a data grid provide the optimizations

That said, one of my personal rules of optimization (based on some experience) is that your intuitions are almost always wrong -- if something is taking too long it is usually worthwhile to benchmark it even if you "know the cause of the problem" (unless the putative fix takes substantially less time than doing the benchmark). The corollary being that if you're worried about something taking too long, and think that you "should be OK" set up a test suite at scale to validate your intuitions going forward, so that you can monitor performance as development proceeds.

This principle holds even if the bottleneck is development time: you need to determine if the problem is with the language, the developer, the user/developer interaction, or the developer's manager (e.g., providing more interrupt driven activity than the developer can handle)? As always, measurement is the key.

richfaces:dataTable:column sortBy

2008-07-30T09:42:00.001-07:00

sortBy is a new richfaces capability that I mentioned previously. I've started to incorporate it into my system and what follows are some tips/notes/frustrations.

If the sorting glyphs appear, but nothing happens, the dataTable needs to be surrounded by < h: form> < / h:form> -- sorting doesn't work outside of a form context.

If the glyphs do not appear sortBy may not be being given a valid attribute for sorting. This may be the result of a simple typo e.g., sortBy="{competition.id}" rather than sortBy="#{competition.id}"

I had some very strange behavior in netbeans/seamonkey with this facility. For example, if I clicked on the sort glyph for id

I got

YES the header of the column changed to "comp_ID form ss" -- even though "comp_ID form ss" no longer appeared in the file.

It used to be there, but I had removed it and performed a "build" in NetBeans (rather than a "clean and build"). I'm not sure about the underlying cause of this, but in my mind the effect straddles the boundary between disconcerting and amusing. The bottom line is that if things start acting strangely do a "clean and build".

All in all sortBy is a real step forward: it works in ajax tabPanels and simplifies the xhtml code a great deal. Despite this warning, sorting has been working for me as expected.

Structuring Database Tables

2008-07-16T04:51:00.001-07:00

Continuing with the what should the storage actually look like line of my last post, I just read a paper entitled Storage and Querying of E-Commerce Data by Agrawal, Somani, and Xu. This paper discusses the trade offs between storing data in "wide" (up to 1K column) tables versus storing the data in sets of narrow (vertical) tables. The data under consideration consists of sparsely populated data sets (lots of null values), varying definitions of "sparse" are used to generate the results.

Their measurements clearly support the overall performance advantage of the narrow table approach despite the complexity of reassembling the data into a "wide" form, when (and if) it is required. Their work has been picked up a bit by the column database and semantic web crowds, but not to the extent that one would expect. I think that this reflects the fact that the datasets were fairly small (1000 cols, 20k rows).

A couple of notes about the results -- they achieved improved performance in the vertical representation despite the fact that they represented everything as an object, key, value triple, and built a translation layer to shield the user from the vertical representation. The triple store was built on DB2

Although these findings are both intriguing and encouraging (from the standpoint of wanting to break information up into its primal entities), I wonder about the scaling behavior of a system structured in this way as it radically increases the number of rows per table. After all, all systems have limits (e.g. MySql, Postgres, Oracle, SQLServer), and more importantly they have optimal operating regions, aka "sweet spots". A few years ago when the largest table in one of my systems reached 100 millions (wide) rows, running even simple queries against that table was painful (which I'm sure could have been alleviated with some clever tuning -- but it was neither the tallest pole in the tent, nor the squeakiest wheel on the cart).

My concern has to do with the risks of getting outside of the "sweet spot" of the systems upon which FDD is being constructed -- I remember back when one of my former employers switched database vendors (a non-trivial project to say the least). With vendor A, we were the customer with the largest DB in their installed base (aka outside the sweet spot). With vendor B, we were a "moderately large" installation, but certainly not in the top 100 (aka inside the sweet spot). The number of bugs which we encountered in Vendor B's database were substantially fewer. I assume that this was because the bugs had already been stumbled upon by the bleeding edge users and fixed by the vendor by the time we would have encountered them.

That to me is the prime reason to try to stay sweet spot. If you don't you're the one finding the new bugs and either fixing them, paying for them to be fixed, or hoping for the vendor to fix them (and developing a deep understanding of where you are on the vendor's list of priority customers).

More on this in my next post.

Temporal Data

2008-07-06T18:06:00.001-07:00

As part of fleshing out the design for the Flexible Drug Discovery (FDD) platform, I'm deciding upon the level of support for temporal data. The simplest decision, based upon the existing "webtwo" infrastructure, would be to have an insert_time and update_time attribute for each item. I think that these times are the bare minimum required to understand/debug system operation.

However, in the clinical domain I've become accustomed to thinking about the data in terms of what did we know when? so that it is possible to reconstruct the understanding of a trial at a given point in time. This obviously requires much more extensive tracking.

I recently came across an excellent book on the topic: Temporal Data and the Relational Model by Date, Darwin and Lorentzos. It presents a detailed analysis of the issues involved in working with temporal information using a refreshingly simple example consisting of a few tables of data about parts and their suppliers.

Many systems use begin and end dates for each row to track when the data has changed, supporting the type of use most relevant to clinical/scientific analysis. However, this technique does not support some interesting situations in the business domain. For example, p 166 of the book shows that given an item with the attributes: name, status, city answering simple questions such as "how long has a supplier been at that address", or "how long has a supplier had that name" requires a begin/end date for each attribute. Thinking through the implications of this issue results in refactoring the model into irreducible components (aka sixth normal form), as described on p 173.

As implied by the term sixth normal form, using the temporal behavior of the data as a design axis can have extensive implications e.g.,

splitting quantities out in a LIMS system
splitting out names (especially last names!) in a system that tracks employees, etc..

This implies that it is important to consider the temporal behavior of the data even if a temporal model is not planned for the system as it helps drive scenarios for evaluating the system's response to expected changes e.g., "high flux" items may require optimized interfaces, surface special reporting requirements etc..

Other noteworthy topics in the book include: merging intervals e.g., the two facts that attribute A had value 3 from t1-t3 and has value 3 from t3-now should be merged into a single fact.

There is also a discussion of the time-from/time-to in the persistent store, vs the time-from/time-to in the world, which although important in developing systems requirements, doesn't appear to require analysis different in character from what is conventionally performed. My view is that world and storage times are disjoint. In scientific systems there is rarely a reason to worry about world times -- other than referencing the date upon which an operation was performed.

Again, an interesting read, highly recommended (despite their frequent exhortations on how to read the book e.g., "are definitely meant to be read in sequence as written (p51)" or "note carefully" (carefully is used an inordinately large number of times in the text)).

As storage becomes cheaper, the downside of not having a temporal capability will more frequently exceed its implementation cost.

de novo project retrospective

2008-06-16T09:04:00.001-07:00

I’m doing the finishing touches on my “webtwo” application and thought I'd post a quick post mortem on the project.

The core set of functionality was straightforward:

Components/Capabilities

Users with system defined logins (not relying on database logins) and system specific roles

Competitions consisting of user submissions. Each submission can have multiple submission items consisting of text or images.

Everything can be tagged and commented upon.

Tags can be reused and applied to any object in the system.

Here are my observations

Ratings

My initial design had ratings as a separate class/table. I decided to push ratings down into both tag and user_text (the generic data type that holds comments) since, upon further consideration, I thought that the semantics of the term “rating” was different in each case. If the rating is attached to a comment, both the rating and the comment refer to some other item, and the comment explains the rating. On the other hand, if the rating refers to a tag, it reflects how strongly the item reflects the tag term.

Persistence

Even though tagging and commenting (adding user text) are distinct functionalities, much of the persistence operation is the same. I therefore packed persistence into a single class which persists tags and text in distinct methods. At some point I may factor this into three classes: one for the common persistence core and two for the specific tagging commenting functionality; but this single class is adequate for the current purpose.

Images

As mentioned previously, my initial design allowed fairly large scale image uploads with dynamic resizing for page display. I have since switched to a more conventional architecture in which standard image sizes are generated and stored upon upload. This is a big performance win, and also reduces storage requirements.

Seam

Seam is a platform that is evolving in an encouraging way -- when I wanted to add sorting to a table displayed within a ajax tab -- I could not see how to do it but upon further investigation it appears to be supported in an upcoming release.

e.g., in this view I can sort

while in this I cannot

Type X

For this system I had separate "type tables" for each type category e.g., user_text_type holds the allowable types of user_text such as comment, description etc. As I consider building larger systems with similar functionality, I plan to have an item_type table which holds the categories for every table that requires typing. This table would share a similar pattern to that used for user_text (comments, descriptions) and tags which holds the table_name and the id of the referenced object. This will simplify understanding the data model/code and facilitate building curation tools.

&Rest

One additional note about the seam framework: it sometimes feels like an &rest interface, since it is easy to bookmark access to a particular object etc.. However, it does not do so without “seeing past” its own request: parameters which are not specified for "pass through" in the .page.xml files are stripped out. It is easy enough to add them -- all that is required is to specify a

 < name="">parameterForPassthough"/>

in the .page.xml for each parameter that should be passed through. This annotation also needs to be in every intervening page.

I'm not particularly bothered by this, I'm just pointing it out. On the good side it forces standard parameter naming so that each page's logic can operate upon standard variable names (assuming all of the necessary parameters have already been incorporated into the .page.xml files) and it prevents the urls from becoming unwieldy. The downside is that all of the intervening pages do have to be modified if a new parameter is required. In balance it seems to be a reasonable decision.

Note: You may ask "when would this apply?". One example: you want to comment upon an item and return to it when the comment is complete. In this case the item/item_id would need to be passed thought the comment-editing-page to the comment-verification/completion-page so that the "done/comment-complete" button can return to the right location.

The only real regret that I have about the project is that I didn't know about the yahoo design patterns earlier.

Yahoo Design Patterns

2008-06-13T14:40:00.001-07:00

When visiting Yahoo's site for OmniGraffle stencils, I found that Yahoo also provides a nice set of design patterns.

These Web/Web2.0 patterns are clear, concise and have pointers to how they are used within Yahoo. Note: this doesn't constitute a strong endorsement on my part. My approach to patterns is to broadly survey existing patterns with the goal of settling on one, or worst case to modify/create one using the results of the survey.

BTW OmniGraffle is an amazing drawing program -- I've been using simple/mid-level drawing programs for a while and its interface is in a different league entirely. It does things that I haven't seen before, hadn't thought of previously, but are totally obvious to use and highly useful. If you're on a Mac you owe it to yourself to give it a spin.

debugging hibernate/mysql systems

2008-06-06T06:57:00.001-07:00

Just a quick note, on debugging hibernate/mysql systems.

In my ‘webtwo’ system I had the following constraint in the db:

        constraint fk_submission_item_image foreign key (image_id) references image_data(id) ON DELETE cascade,

However, image_data didn’t exist.

During operation it caused the error


18:34:56,009 WARN  [JDBCExceptionReporter] SQL Error: 
	1452, SQLState: 23000
18:34:56,009 ERROR [JDBCExceptionReporter] Cannot add 
	or update a child row: a foreign key constraint 
	fails (`webtwo_dev/submission_item`, CONSTRAINT 
	`fk_submission_item_image` FOREIGN KEY (`image_id`) 
	REFERENCES `image_data` (`id`) ON DELETE CASCADE)
18:34:56,009 ERROR [AbstractFlushingEventListener] 
	Could not synchronize database state with session
	org.hibernate.exception.ConstraintViolationException: 
	Could not execute JDBC batch update
        at org.hibernate.exception.SQLStateConverter.convert
		(SQLStateConverter.java:71)
        at org.hibernate.exception.JDBCExceptionHelper.convert
		(JDBCExceptionHelper.java:43)

I thought this indicated a problem with my Hibernate/java mapping since I’ve been working with the database for a few months without any errors being thrown by MySQL during db creation. I also thought that I had successfully populated all the tables.

In retrospect this wasn’t the case: I had not populated the tables and it is also not surprising that MySQL didn’t signal an error since I “SET FOREIGN_KEY_CHECKS = 0; “ during table creation -- so, my bad.

The moral is that any unpopulated table may be fundamentally misspecified.

This experience does serve to reinforce my heuristic to populate all the tables via an initial data population script at least for testing purposes. Realistically, this requires two load scripts: one to populate the data required for operation (ie., roles, menu values, etc.), and a second to do a test load to verify database integrity.

Modularity & Hygiene II

2008-05-29T06:24:00.000-07:00

A similar, but distinct situation involves the environment expected by a module when it is activiated:

What needs to be set up for the called function
What is expected to be unperturbed from a “pristine” environment -- and the definition of that pristine environment.

This places restrictions upon use e.g., the inability to initiate workflows within a page flow for seam.

These restrictions result from an implicit dependency upon the configuration of the calling environment. Since the appropriate configuration is assured if the module was called as planned, there is little checking to verify that the environmental assumptions have been met. Once the nested calling paradigm has been adopted, design choices are biased towards implicit configurations that cannot be easily set without perturbing the calling environment, since B (and A for that matter) can still reference the external environment.

I’ve found a useful way to think of this as being the difference between a linear and a nested calling environment: a “linear environment” would pass parameters in as a single object which contains required values while the nested approach sets up one or more global variables for access by the called functionality.

Linear

Nested

In the “linear” illustration nothing in the external environment is perturbed, side effects are minimal and any arguments could be copied, modified and passed onto the next module in a sanitary fashion (with the usual caveats around shared “stream-like” objects).

Admittedly there are some times when the nested approach makes the most sense, usually for “stream like” variables, e.g. an initialization step to read in a configuration file, initialize connections etc.. The problem arises when there is no way to spawn a new configuration (or initialize one if you’re called in a different context). In lieu of such an idealized situation, it would be useful, at a minimum, to be able to detect that you’re being called in the wrong context. When it is difficult for a module to decide if is being called in the correct context (or if it elides the context check), it is hard, if not impossible to provide easy to use modules.

Note: I think that the rails trick of overloading the const-missing exception handler is arguably in this space. The magic underlying this functionality wasn’t easy (for me) to find and it surfaced the capability in such a way that it couldn’t be used for my purposes, since it had no introspection capability and only covered the case of exceptions generated during system operation. Note: discovering this also helped answer one aspect of the environment that I had previously found opaque: “why does my new code seem to be loaded in some cases but not in others.” The answer being that the new code (class definitions etc.) was retrieved if the class had not been previously loaded. If it had been previously loaded the const-missing exception would not be generated and the class would not be reloaded.

At some level I have no problem with this implicit ‘environment’ structuring as long as there are ways to determine what environment you’re in and have the ability to spin up a new environment appropriately configured as necessary.

It’s also important to distinguish environment set-up and configuration using well defined files/apis and this is a perfectly reasonable and necessary practice to set up such things as database access, service locations etc in configuration files that are managed by an “environmental processor”

They are usually

One shots done at startup, or in response to a specific reconfiguration request.
accessed through a well defined API
‘stream-like’ in nature

Again these are issues that I’m experiencing as I spin up my application. I’m not claiming that their solution is either easy or practical, only that it is desirable.

Modularity & Hygiene

2008-05-14T10:08:00.000-07:00

This post (and the one to follow) discuss the issues of hygiene in coding libraries.
It is prompted by experiences that I’ve had lately in using (primarily xml/jsf) libraries and the pernicious errors that can be introduced by either mixing different libraries or not using them exactly as designed.

A little background: the term hygienic comes from the Lisp community. The short form definition: hygienic code is code that doesn’t have undue interactions with its calling environment.
A lack of hygiene manifests itself in a couple of ways:

Modules trip over each other: The “mix and match” promise of modularity, and widget sets is violated

modules work at one point in the development cycle but break after the addition of an apparently unrelated piece of code
mixing components from different widget sets requires much care and tweaking, if it can be done at all.

Modules have an implicit order in which they must be called; there is no well defined way to kick off either a new “top level process” or a child process that has its own (unshared) context.

Note: this turned into a pretty long post, so I will cover the second item in a subsequent post.

I’m not claiming that hygienic code is always necessary or even that “hygienic code” == “good code.” Situations vary, and in some cases e.g., device drivers, OS kernels being hygienic may not be worth it. Also as I describe below, I don’t think that completely hygienic systems are currently possible partly due to the limitations of xml.

However, in most situations, the more hygienic the better.

And now on to the discussion.

1 Modules trip over each other
The first issue concerns the problems encountered when modules end up tripping over each other, aka incorporating functionality from one set of module breaks existing functionality. I have seen this mostly in the javascript space (see this post on integrating UI widget sets in seam).

Although I’m working primarily in jboss/seam for this project, it is not a seam issue per se. If anything, the conventions seam uses for its generated code help to alleviate these issues.

As far as I can tell, this “tripping” arises from an inability to cleanly nest scope in the environment. For example, in xml it is hard to insulate oneself from what goes on in the xml around you as exemplified by the inability to nest comments in an xml file. This deficiency, coupled with the fact that many of the widget sets “compile to xml/html” has arguably had the side effect of diminishing concern for hygienic operation within the development community. Combine this with the silent failure aesthetics of JavaScript and you can produce results that are truly painful to debug.

The contrast with Lisp macros (a radically different idea from C macros Hall has a very nice page on the differences plus some nice simple examples of problems and how to avoid them) is striking. Lisp macros represent the most successful instantiation of code generating behavior that I know of. Lisp macros work because of Lisp’s ability to generate new variable names and then bind incoming values to them so they can be used freely. Achieving a similar result without language support for system-wide unique item naming is (very) hard.

The seam/richfaces framework does much to try to minimize this problem e.g., if you look at the source of the page that seam sends to the browser you will see a lot of html with the form id=”competition:tagtDecoration:j_id70” which is a nice try at preventing variable collisions etc.. However, without enforced namespace encapsulation or the use of a system-wide symbol generating facility (e.g., gensym) variable capture is still possible. I also have not been able to find documentation on when new bindings are generated etc. which probably means I’ll have to look at the source someday.

These issues usually occur more often in scripting languages (for the sake of argument I’m including xml, html, xhtml as scripting languages) rather than in compiled ones because compiled languages normally restrict these “global” environment accesses to compile time. Run time access in compiled languages to environment variables is more difficult, and inherently has to address multi-core/multi-processor/multi-systems issues. The end result is that the “external environment” is generally harder to get at in compiled languages and is relegated to “systems level” utilities.

I’m hoping that the next big thing in scripting languages revolves around scoping and environment giving one the ability to specify a particular type of environment for code to run in, set up a safe context for it to run in, etc.

Part II will be covered in my next post

Hibernate & Java /Hibernate Annotations

2008-04-26T10:20:00.000-07:00

Just a quick post on Hibernate ‘@Where’ annotation usage and a note on some odd behavior I’ve seen using Hibernate in Java (my environment NetBeans IDE 6.0 (Build 200711261600) Java: 1.5.0_13; Java HotSpot(TM) Client VM 1.5.0_13-119, System: Mac OS X version 10.5.2 running on i386; jboss: 4.2.2.GA; seam: 2.0.0.GA)

First, annotations:
I had a hard time finding a “pasteable” explanation of the use of @Where
Now that I’ve found how to do it, I thought I’d share.

@OneToMany(cascade = CascadeType.ALL, fetch = FetchType.LAZY, mappedBy = “itemId”)


@Where(clause = “item_table = ‘competition’”)
public Set getItemTagRatings() {
return this.itemTagRatings;
}

public void setItemTagRatings(Set<> itemTagRatings) {
this.itemTagRatings = itemTagRatings;
}

The effect of this is to set up an additional condition upon TagRatings, so that only those TagRatings that have the item_table value = ‘competition’ are retrieved (item_table is the name of the column in the db).

The second item is odd. I cannot determine the underlying cause, so I’m just sharing the symptom and a workaround

The loop below would only execute once (and occasionally I would get a javax.faces.FacesException with the message: “#{competitionHome.persist}: java.util.ConcurrentModificationException”)

The concurrent modification exception was not a particular surprise, but the silent (no visible exception) execution of the loop a single time befuddles me. If supressWarnings wasn’t a compile time operation that’s where I’d place the blame but.... (BTW I have no doubt that I'm doing something deeply wrong in the not working loop below, my goal is to highlight the relatively opaque effect of the error)

Note: I designate the loop as ‘working’ since the query string shown will not work as written, it required some changes to work since the #{newName} did not evaluate to the appropriate value.

The loop


for((String newName : newTagNames){
@SuppressWarnings(“unchecked”)
List tags = (List) em.createQuery(“select t from Tag t where lower(t.tagName) = #{newName}“).setMaxResults(pageSize).setFirstResult(page * pageSize).getResultList();

if ((tags == null) || (tags.size() == 0)) {
newTag = new Tag(newName, user);
em.persist(newTag);
tagAssist.useTag(newTag);
}
else{
tagAssist.useTag(tags.get(0));
}
}

Breaking the loop up into multiple loops worked. The ‘working’ loop(s)


for ( String newName : newTagNames) {
  ArrayList tags = null;
  try {
    Query q = em.createQuery(“select t from Tag t where lower(t.tagName) = #{newName} “);
    q.setMaxResults(pageSize).setFirstResult(page * pageSize);
    tags = (ArrayList) q.getResultList();
   } catch ( IllegalStateException e) {
      log.error(“error getting tags: “ + e);
      e.printStackTrace();
   } catch ( IllegalArgumentException e) {
      log.error(“error getting tags: “ + e);
      e.printStackTrace();
   }
   if ((tags == null) || (tags.size() == 0)) {
     Tag newTag = new Tag(newName, currentUser);
     newTags.add(newTag);
    }
  }
  for ( Tag newTag : newTags) {
    em.persist(newTag);
  }
  for ( Tag newTag : newTags) {
    tagAssist.useTag(newTag);
  }

jboss, seam, jbpm

2008-04-11T13:30:00.000-07:00

I’ve been working in the jboss/seam framework for a few months now and recently tried to add jbpm workflow functionality to a project. In attempting to do this, I’ve hit some rough patches that have caused me to place the jbpm parts of the project on hold as they were not “mission critical” to this prototype.

Note that these remarks are in the context of a seam 2.0/jboss 4.2x environment (jboss 4.2+ is required by seam 2+), running on OSX 10.5. The jboss/seam/jbpm combination appears to be the source of the problems, OSX 10.5 appears irrelevant.

Seam offers two ways to utilize jbpm in your applications: the first is pageflow; the second is what the documentation terms an overarching business process.

The seam pageflow model is, as one would expect, a way to map desired page to page transitions using an xml file. It is described as being similar to other pageflow definition languages such as spring web flow but built with a conversation model in mind. The idea is that conversations allow better support for random user navigation e.g., hitting the back button. I cannot comment on that claim, but having used pageflows a bit I’ve found them adequately expressive and useful.

The difficulties that I have had are centered around integrating an overarching business process into my seam application. These business processes may include asynchronous server and user processes executing over long periods of time, e.g., weeks. A simple example is the verification of user identity in a web facing system: A user requests an account on the system, and activates the account by responding to an email sent by the system.

The business process requires the system to send an email to the user following the initial request, validate the response or, in the absence of a response, retire the request/send another reminder

The first issue I had in incorporating this workflow is the classic “who’s on top” problem, since I found that pageflows could not initiate workflows.

My prototyping started with a pageflow, as pageflows are the natural frameworks for handling the request for a new user account. The pageflow structure makes it easy to verify that the requested username/password etc satisfy system requirements. If they don’t, the application can be configured to just sit on the request page and not proceed to subsequent pages.

For example, by enclosing the page redirect in a <> construct, the return of a null from the userHome.persist method keeps the application on the page (there is a protocol (not shown) that allows the application to post error messages on the page).


    <navigation from-action=?#{userHome.persist}?>                                                   
        <rule>                                                                                       
            <end-conversation/>                                                                      
            <redirect view-id=?/User.xhtml?/>                                                        
        </rule>                                                                                      
    </navigation>

However, as mentioned above, a pageflow cannot initiate a workflow, so it is necessary to enclose the pageflow within a workflow to achieve the desired result. This entails changing the preceding page so that it kicks off a jbpm workflow rather than simply going to the next page.

This is the point at which I realized that I would need tools for monitoring and debugging workflows. The jbpm console appears key to this. Disconcertingly, I could not get the console to run with jboss 4.2.2.GA. This appears to be a general problem. In my case, the instructions on the wiki enabled me to get my workflow initiated without it throwing exceptions but I still was unable to get the console working.

One suggested path was to build the console from source. However, I proved to be unable to get it to build from source with the level of effort I was willing to expend. After reading posts such as this, I concluded that building from source may not be the most productive experience.

The bottom line is that you’re in an odd space working with seam 2.0 -- it needs jboss 4.2X but some of the other redhat/jboss tools don’t yet support jboss 4.2. I’m a bit surprised that this is the case: seam 2.0 GA was released 2007-11-01 06:49 and jboss 4.2.2 was released 2007-05-11.

Since I expect this situation to eventually be rectified, and as I mentioned, workflow was not “mission critical” I decided to wait for a working console application to be posted on the jboss.org site.

In fairness I’m working from the community “minimally supported” version -- the situation could be radically different in the supported versions.

Taxonomies, Ontologies and the Semantic Web

2008-03-26T08:18:00.000-07:00

A couple of weeks ago I attended C-SHALS 2008 (Conference on Semantics in Healthcare and Life Sciences), one aspect of it that I found striking was the number of people who conflated taxonomies with ontologies -- my initial reaction was to want to post a remark about the confusion and highlight the distinctions (see this for a short set of descriptions of these and related terms).

I’ve instead come to view this conflation as reflecting the pragmatic bias of these systems: if the difference between taxonomies and ontologies isn’t apparent to you, the difference doesn’t matter for what you are trying to do (modulo the assumption that the speakers were competent, but that did appear to be the case). The implication is that such systems require no significant machine based inference across organizations. Significant inference, in this context, would involve something beyond the use of term matching to gather locally related terms/individuals (local vis a vis the terms being matched). Note: although I categorize this as being ‘non-signficant’ that’s only from the standpoint of inference -- these systems do cover most of the Business Intelligence/anlaysis use cases being implemented today.

As you might expect, given this characterization, these presentations involved the aggregation of data from multiple sites, using RDF or taxonomies such as Snomed to link data between sites. This is a good thing -- as I’ve mentioned a number of times having stable identifiers across systems is the key to integration. The system presented demonstrated that useful integration is possible even when the same term e.g., the same Snomed terms, have slightly different meanings in the different organizations. (see How Doctors Think for an anecdotal study of physicians classifying patients).

This is an interesting result: although fully vetted, 100% one-to-one mappings would obviously be preferable, in these systems the value of more data outweighs the penalty imposed by increased noise. Rough quick integration is proving more valuable than detailed integration requiring a thorough analysis of all systems used -- probably because the difference between ‘rough, quick’ and ‘thorough, slow’ is measured in months, if not years.

This is related to a discussion at the conference on the contrast between developing ‘problem specific’ ontologies vs. ‘general use’ ontologies. That is: does taking the time to ‘get it right’ add any value? This is roughly equivalent to the old AI scruffy vs. neat distinction.

Although I wouldn’t go so far as to claim that a general purpose ontology is impossible (at least in some limited domain), I am skeptical that it can be achieved. My concern centers around the fact that when you are constructing a general use ontology it is hard to know where to stop e.g., given a small molecule bioactive compound you should represent the formula and chirality, but what about the (possibly fractional) salt form? or the formulation? what about radioisotopes and their decay rates? subnuclear particles etc. I understand pragmatic stopping points for modeling these issues, but I don’t know how to determine principled ones.

It’s reassuring to see a number of researchers finding pragmatically useful parts of the semantic web, without the need for perfect definitions/ontologies. This, to me is the take-home message: there are a number of useful tools and techniques in the semantic web space, don’t be put off by the thought of merging ontologies and developing a grand unified theory of everything.

Porting a Ruby on Rails Application to jboss seam

2008-03-12T07:19:00.000-07:00

I did finally port my Ruby on Rails application to jboss seam.

The capsule summary is that it took longer than expected (not particularly unusual for software), looks better than it ever did but still needs some performance tuning.

Some specifics

image display
I went with the seam graphicImage tag (note xmlns:s=”http://jboss.com/products/seam/taglib)

< 's:graphicImage' value="”#{artwork.thumb}” rendered=”#{not empty artwork.thumb}” < 's:transformImageSize' height="”50”" maintainratio="”true”">
< '/s:graphicImage'>

which I found to be slow -- much slower than doing a normal html < width="”x”/"> tag. (update -- this is due to the use of the 'transformImageSize' tag -- rdf 24 March 2008)

In the RoR code I conditionally chose one of two versions of the image tag

< %= if(@artwork.width_inches > @artwork.height_inches)
then
image_string = “< id =""> @artwork.id) + “\” width=100/>”
else
image_string = “< id =""> @artwork.id) + “\” height=100/>”
end
image_string
%>

which rendered much faster.

The s:graphicImage tag doesn’t appear to be intended for rendering up to 20 images on a page -- my next revision will include the equivalent of the RoR code

data display
I was easily able to get data tables with nicely alternating row colors by adding the following to theme.css (I did it here, since the ‘alternating colors’ should vary with the theme)

.table-even { background-color: #ffffff; } .table-odd { background-color: #eeeeee; }

and then adding the following line to the *List.xhtml files

rowClasses=”table-even,table-odd”

I did have some minor problems getting it to work since I had overwritten a class that applied to each cell and had given it a background color. I forgot that this cell class would take precedence, but was able to figure it all out with Firebug, an indispensable tool!

seam generator
The generator provided an invaluable starting point and some really nice features e.g., it creates tables with sortable columns for the list view. The table defaults to displaying the ID of nested objects, but it was trivial to change it to display something more appropriate while maintaining the expected sort behavior.

ajax suggestions
My goal was to consistently work within the framework. This occasionally put me at a level of abstraction above which I was comfortable (the operational metaphor being “trying to do X while wearing thick gloves”). This caused me to have a much more difficult time doing a drop down suggestion menu than expected e.g.,

< id="”roleDecoration”" template="”layout/edit.xhtml”">
< name="”label”">role
< value="”#{peopleHome.instance.roles}”">mmediate=”true">

<>
< /h:selectOneMenu>
< /s:decorate>

since I was having a bit of a problem finding the exact ‘magic location’ for placing the “” tag relative to the tag, aka it all ‘just works’ if you have everything placed ‘just right’

Being at a higher level of abstraction also forced some patterns that I found difficult to work around. For example, in this trail system an Artwork object has a framed? attribute backed by a boolean. The behavior that I wanted in the listing ‘query by example’ code was to either

to return framed artworks if the framed? checkbox was checked

to return all artworks if the framed? checkbox is not checked.

However, I could neither come up with a way to have elements on the restriction list take multiple parameters nor to return different restriction lists depending upon the query

my notes at that point say:

if you do this
List restrictionList;
if ((this.artwork != null)
&& (this.artwork.getShipable() != null)
&& this.artwork.getShipable().booleanValue()) {
restrictionList = Arrays.asList(SHIPABLE_RESTRICTIONS); }
else { restrictionList = Arrays.asList(RESTRICTIONS); }

You break the transaction model

I was able to fix this by adding this line to the RESTRICTIONS
“artwork.framed in (true, #{ artworkList.artwork.framed})”
which obviously will not generalize beyond the boolean case.

hibernate

I found that the Hibernate documentation was very useful (when I took the time to read it in detail aka RTFM)
The expression language used is described reasonably well here

richfaces
When I moved to a new version of richfaces to get suggestion boxes working it broke other minor portions of the page layouts. Although not a big deal, I found it disconcerting. Building this rich functionality in the browser is cool and all, but it feels fragile and is causing me to think about trying out flex (or air, its latest incarnation)

unit testing
I used HTMLUnit since it tests the complete end-to-end interaction.
Although I appreciate the ability to do faster, more thorough testing via mocks, I found that they gave me yet another thing to configure and wouldn’t give me the full end-to-end functionality that I was looking for.

Summary
I think that jboss/seam will likely prove useful. I have one other application that I’m building as a precursor to the extensible discovery system My biggest area of concern is the ability to do a good UI in this space which might prompt me to investigate the air/flex framework(s) at some point in the near future.

Extensible System: Core Software Requirements

2008-02-26T06:17:00.000-08:00

In my last post I promised to detail the software requirements that an extensible system for discovery data shares with other Web 2.0 systems -- here they are:

Workflow Orientation: Support for a complete workflow beyond that which is offered by JSF navigation. This workflow must allow the orchestration of multiple events without requiring additional user interaction. Supported workflows may involve sequenced, conditional interaction with multiple back end systems (obviously existing on multiple platforms). For sanity and maintainability, the workflow language should be BPEL (or an slight extension) and should provide the ability to extend predicates and actions using a well designed and understood language (Java, C# etc.).

Integrateable (mashable) data: The data stored by the application (and the results of any analyses performed on the data) are available for repurposing in other applications. Repurposing should be supported in fine grained manner so as to put as few restrictions as possible upon its use.

This has two implications: the first is stable identifiers; the second is restful interfaces which allow data to be retrieved by referencing a static URL.
Note: Restful interfaces have a number of nice side effects: the seam *From trick mentioned below would be much more difficult without a restful interface. Additionally, the speed of rapid prototyping/development of web pages is greatly increased if one can directly access a ‘deep’ page without having to manually negotiate multiple precursor pages.

Event queues: Good workflow/system interaction is facilitated by message queues for guaranteed message delivery. Queuing systems also provide good interface points for logging and analysis tools.

Rules engines: In its most general form a rules engine is a piece of code that evaluates a set of antecedent-consequent pairs e.g., if antecedent then consequent. Given this abstract definition rules engines need to be distributed at a number of places within the product. I see four distinct areas each with its own role

1: Display within a page e.g., should the particular element be displayed corresponding to the ‘rendered’ predicate in JSF. Rules involve availability of data appropriate for display and authentication/authorization restrictions.

2: Predicates involving page flow (JSF, &REST, etc). Rules involve what page gets displayed next?
The jboss seam pages have a very nice convention in which the presence of a *From attribute/value pair allows an editing action, upon completion, to return to the page from which it was launched. Here is an example using peopleFrom

peopleFrom ? ‘PeopleList’ : peopleFrom}.xhtml”
id=”done”
value=”Done”/>

Which will return the the PeopleList page when the editing action has been completed.
3: Security rules for CRUD operations. Rules involve accessing and modifying data.
4: Back end BPEL operations e.g., if the request has been outstanding for more than a week then notify customer support. Rules involve the overall operation of the system.

Logging: Effective debugging of complex systems requires the ability to gather an integrated log for each activity in the chain of events that produces a given result; supporting this requirement in an operational setting requires that all relevant logs can be time-aligned and assembled into a single report for analysis

Monitoring and management: Business rules should be capable of being extended to monitoring system operation: server load, queue depth, latency etc. allowing the system to be ‘self monitoring.’ The use of a common tool permits the maximal number of people to understand its operation.

In addition, interfaces should be provided to allow information to be updated (recached) without bouncing the server. JMX is a reasonable example see also.

Security: Obviously any enterprise system must provide for some level of security minimally with LDAP support, hopefully with out of the box support for OpenID and SAFE. In practice, I would caution against making security/access overly fine grained since it must support people changing their roles in the organization, changes in business processes etc.. The more fine grained your access model the more thought is required to get it right and the greater the probability of getting it wrong.

I have personally found it useful to distinguish reading, writing, and editing data, opening up the reading and dissemination of the information while restricting writing and editing data to specific tools provided for specific stages of the process.

For example, given a standard lab workflow for data collection, analysis, upload and “publication” (to the persons requesting the tests and then the company at large): there is one tool for collecting, analyzing and uploading the data; there is a second set of tools for integrating and viewing the data in a larger context; and there may be a third set of tools for curating and editing data which has been found discrepant.

This rounds out the software requirements for a practical production system. Although these requirements appear (and are) extensive most, if not all, of them appear in a number of enterprise level toolkits. As I said at the beginning of this post: there are clear best practices. A Powerpoint that covers both these posts is available.

An Extensible System for Discovery Data

2008-02-11T12:09:00.000-08:00

I’ve been thinking about how to make discovery informatics tools significantly more flexible, extensible and perhaps even more maintainable than they currently are. I don’t think of this as being a software problem per se (at least as it involves any new thinking on my part). However imperfectly, the required underlying software capabilities already exist. That is, although we certainly need systems that are more configurable, workflow oriented and flexible in their ability to mix elements on a page (via mash ups etc), and most scientific software is unquestionably deficient in these areas, the state of the art in software development in 2008 affords a clear set of proven practices to satisfy these needs.

The key issue for scientific software is that the science which we are trying to support changes over time in two significant albeit essentially different ways.

The first is that the areas of the business that we need to support/integrate with expands over time e.g., starting at in-vitro testing and moving (in both directions) to support synthesis and in-vivo testing, perhaps eventually into clinical trials. It is inevitable that over time the business changes, the strategy changes, the organizational structure changes -- in any case, once solid (organizational) boundaries become permeable. These changes in the business have a strong impact on the design of scientific software adding anything from the need to support radioisotopes and formulations to tracking freeze/thaw cycles.

The second, and more interesting axis of change is that the science itself changes: new tests come on board that have a radically different cardinality

The nature of the data changes: test results start as point averages, evolve into time series and then become two dimensional vectors e.g., %INH, FLIPR, Imaging;
Our understanding of the biology changes e.g., one protein -> one gene;
Our ability to simplify the problem (a simplification that may have been unconscious) has been shattered e.g., genetic variations of targets, protein pathways, post translational modifications begin to impact the data which we are gathering today.

A common solution is to build the system to support the most complex case. However this doesn’t work if

We don’t know the most complex case.
The most complex case is a superset of all possible cases only one of which would ever occur in a particular system
The group currently being supported can’t supply the information that will enable the distinctions at a later date -- so the information collected is all the same.

In my mind the best way to solve this is to modularize the domain into its simplest building blocks and then build up the necessary complexity using these building blocks. I will admit that this is what we think we’ve been doing all these years, but I don’t think it’s true. I challenge you to look at the actual tables (or objects) in your systems and ask yourself if all of the columns (attributes) are strictly necessary in all (or even most) situations or do they (the columns/attributes) reflect the diffusion of business processes into our base design. My proposal is simply the following: for each building block we scale back the attributes for each entity to the absolutely necessary minimum, paying special attention to items which could potentially change the cardinality.

The temptation in doing this type of analysis is to start at the “most fundamental” level and work your way up. The problem I have with doing things that way is that I find the prospect of describing a mouse in terms of its constituent quarks to be both daunting and without obvious value. My current approach is therefore to start at a well grounded middle level (similar to the “middle distance” ) and stub out to one level beyond the current need.

The goal is to be able to support multiple overlapping hierarchieries so that in different situations we can classify assays by technology, gene, both or neither.

The diagrams below indicate the kind of modeling that I envision

The characteristics of such a “primal entity” include the absence of any bp (business process) foreign keys (direct references) to entities related solely as an aspect of the business process. These business process relationships are moved into relationship tables particular to the situation at hand and reflect the particular hierarchy of the business process currently being addressed. The foreign keys which are allowed under this no bp foreign key constraint involve items that must be present for any entity of this sort e.g., all synthesis batches must have components, methods of synthesis (even if ‘random’), an operator (even if ‘unknown’) and a units of measure in the frames that most of us work in, these can be taken as relatively fixed.

The advantage here is critical: your system, and more importantly your data, can survive in the face a significant unforeseen change in the scientific or business environment. The reason for this is straightforward: the model is now capable of supporting multiple conflicting hierarchies of relationships, so the introduction of a new, conflicting hierarchy doesn’t break your model. This works in a manner similar to they way in which an RDF export of your system can assist you in modeling its ontology, since it can support multiple, potentially conflicting ontologies.

My next post will focus on components of the software architecture.

RoR => Seam Part 1

2008-01-24T11:15:00.000-08:00

This is a short initial post about my first jboss seam project which is to move one of my small, but non-trivial Ruby-on-Rails applications to Seam.

A couple of preliminary notes:

I used seam-gen to generate the initial code/application structure. It gives you a good start -- the pages have a nice minimal and professional look to them and none of the default designs/layouts flows are crazy. All in all a nice job.

My environment Mac OSX 10.5.1, jboss 4.2.2 GA, seam 2.0.0 GA, mysql 5.0.45

I had a couple missteps/odd experiences that I wanted to share. They both revolved around trying to include some AJAX capabilities on a page

One page of the application is for the creation of shows which have titles, curators, institutions and artworks. The curators, institutions and artworks are in separate tables with the ‘shows’ table holding foreign keys. Title is a required field.

In AJAXifying the curator selection the onkeyup actionListener was not being called -- I was getting this error message in the background stream

14:07:00,243 INFO [lifecycle] WARNING: FacesMessage(s) have been enqueued, but may not have been displayed.sourceId=shows:aName Decoration:name[severity=(ERROR 2), summary=(value is required), detail=(value is required)]

The referenced aName section of the page was not involved in either end of the AJAX update. After struggling with this for a few days I realized that Name is a required field -- the framework was checking this requirement first and giving me an error. The combination of the error in validation + being in the middle of an ajax call resulted in neither the error being posted to the page nor the onkeyup action being called.

I worked around this by initializing the name on the page with a

#{empty showsHome.instance.name ? showsHome.instance.setName(“A New Show”) : showsHome.instance.name}.

I find the behavior odd and of course the workaround doesn’t cover the situation in which the user clears the title field and then jumps to the curator field.

Staying with the topic of AJAX support: keep in mind that the richfaces toolkit provides a lot of ajax functionality in a straightforward manner. For example, when looking for an ‘autocomplete’ functionality I mistakenly went down the path of trying to integrate a Yahoo UI component -- nothing against the YUI toolkit, but the exact functionality I was looking for was in richfaces & required a lot less work.

Using richfaces with seam isn’t as obvious as it should be since most of the examples that you’ll currently find are for Seam version 1.x prior to the richfaces mindmeld that occurred in Seam 2.0. Caveat: be sure to use richfaces 3.1.3+ since I found some of the most important rich:suggestionbox features broken in earlier versions

Hello Seam

2008-01-11T07:20:00.001-08:00

A note on getting a ‘hello world’ reverse engineered application up and running in jboss seam

For this test I am moving two applications to seam. The first is an existing Ruby on Rails application that I’m using as a reference and will be replicating/improving the RoR functionality. The second is a bare bones web 2.0 application for which I have only recently completed the table design and for which I have a minimal set of use cases. I plan to use this as a substrate for building workflow-based/“Web 2.0 enabled” applications in a number of areas including drug discovery

In preparation, I thought I’d take an infrastructure upgrade move to Leopard and Eclipse/Europa (+ for reasons that will be apparent Netbeans 6.0). This may not have been the best idea since I encountered some eclipse/Leopard incompatibilities that were eventually resolved using the technique described here.

The first step was to install jboss, seam (I initially went with the 1.2.1 GA version since the 2.0.0 GA had too many 0’s in it for me to be comfortable) and run the demos -- this was a very enjoyable process. The installation was easy and the demos were polished.

The next step was to begin reverse engineering. With Eclipse the best result was had by first using ‘seam new-project’ to create a bare project and then import the project into eclipse (using the file->new->project as recommended). I then performed a ‘seam generate-entities’ to get all the hibernate code in place and then do a refresh of the files. This was advantageous since it allowed me to disambiguate Eclipse project import problems (which I didn’t have) from the Hibernate mapping problems (which I did have).

In eclipse the only way I could find to get the project to run on the jboss app server was to right click->’jboss tools’->’add struts capabilities’ and then add it in from the project side right-click->’run as’->’open run dialog’. There’s probably some other way to do this but I couldn’t find it quickly.

I tried to open the project in NetBeans since the seam doc indicated that you could ‘just’ open the project up in NetBeans without any special actions. This was true, but NetBeans just did not appear to understand the file layout of the 1.2.1 seam-gen project, so I dropped that path (at least until I started using seam 2.0).

I did have some problems getting the reverse engineered project to run. After a bit (maybe more) of investigation I determined that the problems revolved around the fact that both projects had blobs (one has blobs of different sizes) going against a mysql back end.

Seam structures the project in such a way that the first page won’t come up unless hibernate can successfully map the database. The default hibernate mysql/blob mapping is ‘tinyblob’ which didn’t match my applications’ blob datatypes of blob and mediumblob. In addition, Hibernate won’t take the mysql data types and it is necessary to specify the appropriate lengths when I did this everything came up fine.

I have to admit that I was a bit thrown off by the ‘first page error’ since I didn’t expect that mapping to be invoked until I started to hit those pages. This might stem from the fact that there is a navigation bar on the top of the first page that takes you to pages and map to the db, or might be standard Hibernate behavior, I have to admit that I’m not familiar enough with Hibernate to be sure.

In any case, I was up and running and started performing some minimal customization.

Seam’s default behavior for displaying blobs is to show a hex string of the first few bytes - -not a bad default as it allows you to see that it is accessing binary data and the data is different for each item. However it’s not the final behavior that I wanted.

When I tried to add some images to the pages e.g., .
I ran into issues where there were warnings of the types
WARN [HtmlRenderKitImpl] Unsupported component-family/renderer-type: org.jboss.seam.ui.UIGraphicImage/javax.faces.Image
This appeared to be a known bug in seam 1.2 that was fixed in 1.3 -- but as near as I can tell 2.0 was release which followed 1.2.

Upgrading to 2.0 (including the attendant jboss upgrade) solved this issue but broke some of the Eclipse tools, specifically the jboss tools HTML editor was unable to find any of the @Name references(I’ve seen this noted on some of the jboss forums, so I’m not alone on this).

Since I hate to go forward with an IDE showing me a ton of warnings I decided to look at Netbeans since the 2.0 version of seam has different file structure from seam 1.x. Happily, this has made the file structure much more NetBeans friendly.

I think the only thing I had to change to get it working in NetBeans was to set the deploy project to ‘restart’. I’m now up and running and in the process of modifying things work work as I want.

One side note: I find that I’m liking NetBeans more than expected. It’s a nice tool, noteworthy is integration of NetBeans with subversion, it is trivial to revert to earlier versions in the repository. In addition to repository saves NetBeans graciously keeps track of all ‘local’ saves to the file system. This means that whenever I get a file to a point where I can’t figure out what’s going on anymore, I can easily recover -- a very nice touch, that I make use of all too often.

jboss Seam?

2007-12-22T10:45:00.001-08:00

Lately I have felt the need to get familiar with a new toolset for “industrial strength” applications. There are a couple of factors driving this:

1. The methods that I currently use to build Web 2.0 applications revolve around RoR. However, RoR doesn’t have a strong user community in my customer base (at least on deployed applications). In addition, I’m much more experienced with the Java stack and would be more confident that my proposed web 2.0 (and web 3.0) capabilities would work in production if I knew how to build them using a Java based/oriented toolkit.

2. In thinking about a ‘next generation’ flexible product for the pharmaceutical/discovery space (more on this in a future post) I feel that one of the key components is a good workflow engine. (see FootNote). For a workflow engine to be effective in a scientific environment it is important to be able to drop down into a standard language to perform any special case processing that might be required. This is more important in a scientific setting than in a general business application since a portion of the flow will likely depend upon an algorithmic analysis of the data -- using algorithms that have been newly defined to analyze this dataset. This requires a solid, well defined language suitable for general use, rather than an ad hoc scripting language possibly designed by someone without a background in designing computer languages. My early heuristic in this area was that if Guy Steele hadn’t written on it or had not been involved in it you shouldn’t use it. This heuristic doesn’t appear as useful as it once was e.g., I have heard good things about C#. However, if you read through some of the language specs that Steele’s been involved with, you see what I mean: functionality that is essentially Turing Complete, well thought out exception handling and a clear explanation of operation order, especially around object and class instantiation.

Given these core requirements an obvious candidate is the jboss seam toolkit. Seam has one other interesting feature: the concept of a conversation to handle user context. From the doc But suppose we suddenly discover a system requirement that says that a user is allowed to have multiple concurrent conversations, halfway through the development of the system.

Conversations appear to allow the support of users who want to have multiple browser windows open to look at different result sets, perform different activities etc.. This is definitely a ‘nice to have’ for scientific work -- I have found that users often want to look at and drill down on multiple data sets for comparison and analysis. A similar capability as developed in my group at Millennium by Vlado and David (two very talented guys) with their PageDataServlets framework. It is satisfying to note that PageDataServlets were developed back in 1999 and are still in use. Although it isn’t something that you would use in a new product (1999 was a while ago), it’s a testimony to their quality and depth that they are still in use and provide a capability that you see in very few web sites today.

Seam is open source and has some unusual characteristics for an open-source project:
1. Attractive Demos that work ‘out of the box’ (& I’m running an Intel Mac with Leopard, so that is saying something)
2. Extensive documentation

It also has the prerequisite active user and developer community aka it has enough momentum so that it won’t die soon

So as a test, I’m going to try to get a RoR project ported to seam and also develop a deNovo web 2.0 project in it -- more in my next post.

FootNote: I have been saying that workflow tools are important for a while -- my group shipped an workflow based application based upon BEA’s process integrator platform back around the turn of the century. The workflow space has taken much longer to mature than I expected. The standards groups appear to have split at one point an merged again. BPEL is starting to be a standard feature in product brochures, so I’m hoping that it is not premature to start to use it again.

Using Semantic Tools with real data

2007-06-20T12:27:00.000-07:00

In one of my previous posts I mentioned the question “what do you gain from having the instance data present and available for processing by semantic web tools?” The core idea is that this would simplify the detection of modeling errors and highlight domain misunderstandings, but I couldn’t be sure until I had tried it with data from a real system.

I’ve finally been able to review the results from one such system and have convinced myself that there it is useful.

A bit of context-- doing this in a way that I considered valid required data from of a system that I understood well. The best available system was an “art submissions” application that tracks available artwork and its attributes.

The data was extracted using my rdf_rails utility which generate RDF and OWL files from a RoR application

After trying this out for a bit, I have come to the conclusion there is utility here.

The utility can be best demonstrated with a simple example (all examples are in RacerPro, see my previous post)

(retrieve (?x ) (and (?x Artwork) (neg (?x shipable-art)) (neg (?x unshipable-art))))*

This query retrieves all artworks that are neither shippable nor unshippable.

Since the shippable-art & unshippable-art concepts were intended completely cover the Artworks category, any non-null result indicates either an error in domain modeling/understanding or data cleanliness, either of which should be detected prior to rolling out the model and the application that embodies it.

I admit that dichotomous coverings are a simple case but think that it clearly demonstrates the value of the approach.

While on the topic of semantic web tools it would be remiss of me to not mention the swtools spreadsheet at which is a handy and thorough catalog of available semantic web tools

Published by Michael K. Bergman

Notes:
* Note from the RacerPro documentation neg is a unary constructor, the negation as failure (NAF) negation. The argument is
a query body.

The concepts shippable-art and unshippable-art are defined as follows”

(define-concept shipable-art
(or (boolean= Artwork_framed #T)
(and
(some Artwork_weight_pounds (<> racer-internal%has-real-value 10) )
(some Artwork_width_inches (> racer-internal%has-real-value 10) )
(some Artwork_height_inches (> racer-internal%has-real-value 10) )
(some Artwork_depth_inches (> racer-internal%has-real-value 10) )
))

RacerPro

2007-06-18T11:50:00.000-07:00

I’ve been doing some ontology analysis using RacerPro. I decided to try out RacerPro primarily because my current modeling project uses numeric relationships to determine class membership. My quick scan of the available tools determined that RacerPro offered the best support in this space (any suggestions about other tools supporting such operations would be appreciated)

RacerPro supports the use of numerics in two ways.

The first is as a simple query e.g.,
(retrieve (?x) (?x (and (<= Artwork_width_inches 10) (<= Artwork_height_inches 10))))
;; aka retrieve every object for which the role (property) Artwork_width_inches < 010 and Artwork_height_inches <= 10

The second is as a class/concept

(define-concept can-ship
(and
(some Artwork_width_inches (<>
(some Artwork_height_inches (<>

Which is essentially the same only this time defining a concept rather than issuing a query.

It is fair to say that the racer-internal%has-real-value term did not leap out from the documentation. However the Racer technical support was both very accurate and extremely responsive, getting me up and running pretty quickly.

All in all, I’m happy with the product.

rdf_rails on rubyforge

2007-05-29T07:13:00.000-07:00

I just posted a project, rdf-rails, up on RubyForge that will take a RoR application and convert it to OWL/RDF (including all of the instance data)

This is very much an alpha level project -- I only had one RoR application available for testing.

If you’d like to give it a quick spin I’d appreciate any feedback.

I’m interested in two types of feedback

The first is the conventional software testing: bugs, missing features, inadequate documentation etc.

The second revolves around the utility of the function -- what do you gain from having the instance data present and available for processing by semantic web tools? My postulate is that this would allow a more thorough understanding of the data constituting the domain. An application could then be initially deployed with minimal constraints while the domain is still under exploration. Then, as experience is gained and the system has been populated with data, intuitions about the ontology of the domain could be validated against the data.

I haven’t fully convinced myself of this as yet. It still appears possible, but the compelling example hasn’t yet surfaced. I would like to hear any (positive or negative) experiences that you’ve had in this area.

Architecture and Spreadsheets

2007-05-02T16:37:00.000-07:00

I gave a presentation May 1st at BioIT World 2007 as part of a panel Interoperability and Standards: Progress towards an Industry Architecture in which I develop an example of the evolution and increasing scope of a spreadsheet to show the ramifications of software engineering principles, software architecture and eventually industry architecture upon bringing timely data to a cell in your spreadsheet.

Here’s a link to the PDF

RDF and RubyOnRails

2007-04-12T11:57:00.000-07:00

Haven’t posted in a while -- I’ve been working on a ruby utility that generates RDF and RDFS files from a Ruby On Rails application.

The goal is to enable the use of semantic tools to explore the underlying structure of your data, using the built in-capabilities of these tools to quickly discover contradictions between postulated structure and the actual data. e.g., “I think that every event only has a single location”

In this scenario, one uses the utility to build a base ontology (including instances) from the production system. An application could then be deployed with relatively loose constraints, with the constraints being refined after the application has had the opportunity to accumulate real data. This is especially useful where the character of the data is not well understood/controversial at the time of the initial deployment.

The project has a useful side benefit of allowing me to gain a better understanding of both Ruby and the RoR framework (see the note on class loading below)

The good news is that it was relatively straightforward to get .rdf and .rdfs files generated so that I could read them into Protege (which may not be the best tool for this, but it is one with which I’m familiar and has a substantial amount of documentation)

The bad news is that I couldn’t add any refinements to the an rdf/rdfs project in Protege -- it requires a ‘real ontology’ e.g., an .owl file (which is obvious in retrospect but.....)

I’m now in the process of generating the .owl file.

Class Loading: The classes in a Ruby On Rails application are not loaded when the console is started. ActiveRecord subclasses are loaded on demand from a hook added to the missing_const exception

The way it is handled is interesting so I’m reproducing it here
(from le: /usr/local/lib/ruby/gems/1.8/gems/activesupport-1.4.1/lib/active_support/dependencies.rb in my installation)

class Module #:nodoc:
# Rename the original handler so we can
# chain it to the new one
alias :rails_original_const_missing :const_missing

# Use const_missing to autoload associations
# so we don’t have to
# require_association when using single-table inheritance.
def const_missing(class_id)
Dependencies.load_missing_constant self, class_id
end

def unloadable(const_desc = self)
super(const_desc)
end

end

Microformats, RDF and Life Sciences

2007-02-05T09:58:00.000-08:00

I attended the microformats session at mashupcamp3 the week of Jan 17th which had an interesting sidebar about how microformats are not RDF.

For those unfamiliar with microformats the core reference appears to be http://microformats.org/. Microformats consist of ~10 specifications plus approximately the same number of draft specifications

The sidebar centered upon the fact that a lot of real work can get done with micro formats but some people particularly in the biology/life sciences domain are very heavily committed to RDF. The question arose as to why. There wasn’t enough time remaining (and probably not enough interest) to explore the why.

Looking at the adopted and proposed formats I am struck my a few things. The first is that they each capture a nice nugget of functionality: calendars, contact information, news feeds etc. The second is that they are designed to capture the common cases while avoiding the complexity of handling the uncommon situations, which I think is a good thing.

The nice side effect of this aesthetic is that you can be up and running quickly doing real work, exchanging information etc. The bad side effect is that if you need to do something more complicated the hooks don’t exist to allow you to describe what’s going on.

In general, the more unstructured you are willing to be, the easier it is to capture all of the information. The difficulty arises when you try to curate it or use it in another context. As an example, think of the initial attempt that often appears in a database design: a single table consisting of N text fields. It can work and can hold pretty much anything. Detecting duplicates and understanding the structure come later if at all. However, depending upon the scale and use of the information this may just be fine.

In my opinion, at the enterprise level we sometimes overemphasize the scale and structural integrity issues. Scale can be a big deal if you’re trying to achieve perfect reconciliation of information. If “good enough” is OK, large-scale integration can be achieved in practice with very unstructured data. A good example of this is Hype Machine http://hype.non-standard.net/ (which was demo’d at mashupcamp3) which mines music blogs effectively -- something that I would have thought impossible, given all the issues around spelling, new band names etc. It works partly because the problem is to find something rather than to find everything with complete accuracy.

At a deeper level what is more characteristic of the areas addressed by microformats is that we can develop a good understanding of what’s going on from our intuitions of how the world works and these intuitions should be able to cover a good number of the situations that we will actually encounter.

In life sciences this is simply not the case. Our intuitions are often wrong, in the clinical area the number of potential confounding factors is immense (e.g. http://gforge.nci.nih.gov/docman/view.php/53/2278/MedDRA_Source_Information.html
lists the MEDRA term count as 65872 ) so there is an understandable push to design formats that are extensible and can capture information in well defined and reusable ways.

However, microformats and mashups in general do raise the question “Is this a case of the best driving out the good?” Despite my bias towards ‘scalability’ and ‘enterprise solutions’, it is hard to argue with standing up an application in a few hours that provides some utility to end users (and some real data on use etc.). Even if requires some real work to migrate to a more scalable application when it comes online.

RDF vs Ontologies

2007-01-22T18:34:00.000-08:00

The way I look at it, RDF talks about what you have, Ontologies talk about what you can have. The combination of ontology and the data can then be fed into various reasoning engines to tease out the implications of your data.

This is pretty scary. Given my assumption that the science advances and the thinking about what you can have will change over time, incorporating inferred “facts” leaves one open to fundamental system instability.

My classic example in this is “sorry, we don’t really mean one protein per gene anymore.” The ontology and the implications drawn from inferencing upon the data are wrecked but identifiers for gene, protein, transcription etc are unaffected.

RDF at its most basic, gives you stable identifiers for what you have and allows the declaration of “stable” relationships between these objects. This allows you to communicate clearly about what you have and (possibly) easily version it when the time comes to change (see below). These statements should remain valid even if the ontology which they are thought to be embedded in changes radically.

Versioning

Thinking at the RDF triple level also allows a low overhead means of versioning your information in a manner analogous to the ZFS file system (http://www.opensolaris.org/os/community/zfs/) and its built in “copy-on-write” facility.

If I remember correctly this copy-on-write is performed at the disk block level rather than at the file level. This is thought to be the basis for Apple’s “time machine” capability (in the next release of the OS X). The system can just look back and determine the valid blocks at a particular time and reconstitute the file to appear as it did at that time.

A similar functionality could be made to work at the RDF triple level. Triples that changed would be “overwritten” but the old information would still be available with a timestamp of valid from/to dates. It is easy to overlay some provenance information on top (similar techniques are used in data warehouses to allow clear tracking of when information was updated/corrected).

This turns the fine grained structure of RDF into a feature that provides similar advantage to what is seen in these file systems: the incremental disk space required for versioning can be very small -- to keep multiple versions of a documents just requires the incremental diskspace ~= the size of the changes and is independent of the document size.

I have to admit a certain hesitance in saying “RDF is good” (aside from the rdf represented by my initials). I have not implemented an RDF based system, I plan to do one in the next month or so and will update.

Data Integration and Ontologies

2007-01-15T06:04:00.000-08:00

Ontologies

It is useful to think about three types of data integration

Type 1. Document level -- the user can determine what documents might have information of interest
Type 2. Term level -- the user can build reports using items from multiple documents/systems e.g., each cell in a spreadsheet can come from different systems.
Type 3. Inference level -- terms from one or more documents/systems can be combined to derive information (new terms) in the system being examined.

Both the functionality and ontological commitment increases from type 1 to type 3 systems.

The increasing level of ontological commitment as perceived by a user of the system appears as follows

Type 1: There is something here which may be meaningful,
Type 2: If something does exist it is meaningful
Type 3: The implications of a thing’s existence is meaningful

Rather than attempting to determine the costs of these systems a priori let’s look at some examples

The simplest Type 1system involves simply placing documents in a file system; with some attention to naming and structure this system allows the contents to be easily retrieved and accurately assessed. However anecdotal and personal experience shows that the retrievability of the information degrades over time and it does not scale beyond small collections of items. This degradation stems in part from the fact that these systems allow only a single axis of retrieval based upon the heuristics embedded in the (path)name of the files.

The next level up in complexity for Type 1 systems is the web and file/url tagging systems. Such systems continue to make few a priori claims for the utility of the retrieved items but the use of search engines and URL tagging allow for multiple axes of queries to be retrieved based upon either the algorithms embedded in the search engines or the tags and the sources of those tags. Local file system supporting tags allow the users to (eventually) retrieve their tag definitions either via introspection or an examination of other documents containing suspected tags.

Some of the terminology limitations of having free text tags are alleviated by the fact that the items being tagged are urls/files and therefore unique and retrievable. Retrieving and examining the tagged information allows one to assess the information content (retrieval power) of each tag in the context of the current search. Tagging has been getting a lot of traction on the web with sites such as del.icio.us http://del.icio.us/ , shadows http://shadows.com/ and flickr http://www.connotea.org/ appearing as popular web tools for gathering and sharing tags (social bookmarking). Similarly modern file systems allow tagging of files, directories and applications for images and other media content allow sorting and management of media files via tags

Type 2 Systems: Term level integration is the provence of what is commonly called enterprise integration, which allow reporting and integration of applications within the enterprise (Enterprise Application Integration -- EAI). In practice achieving integration requires stability of the term referents and their use in communication between systems. The stability is what might be termed “stability of use. Commonly the more general the use the more restricted the interface. This involves a conscious decision to “narrow” the functionality when moving from the internal data model to the published interface, often restricting the interface to data transfer objects with a limited number of attributes.

Examples include Service Oriented Architectures (SOA’s) with well defined semantics and ways of modifying them over time. Changes to public information require either explicit revision control and/or verification with all stakeholders that any changes will operate as expected. In general, the “wider” the interface the more frequently the verification is required, with the concomitant drag on system agility.

Type 3 Systems Semantic/Inference level integration: Allows inference of new data from existing/newly added information e.g., IF A AND B THEN C can be inferred. This can cascade into IF B AND C THEN D etc. This is a very strong ontological commitment that requires understanding the implications of the complete set of constraints and inferential mechanisms in the system. The payoff is substantial in that it becomes possible to infer a great deal from just a few additional pieces of information. This does however represent a significant “widening” of the interface, with potentially severe implications for system verification and evolution.

Practical Implications

In Type 1 systems imply no ontological fanout from a local commitment and so it is possible to spontaneously evolve the “definition in use” (“ in use” signifies that there is no requirement for an analytic definition) since the definitions are mostly manual or derived from manual definitions.

On the other hand, for change to occur in Type 2 and Type 3 systems, the implications of the change must be understood for downstream systems that rely upon this changed information as an integral part of an automated process.

In Type 2 systems there is a restricted ontological commitment which requires that changes be verified with systems that couple with the system being changed. The analysis is restricted since the change occurs through a restricted interface the analysis is similarly constrained. All else being equal the greater the functionality and use of the data, the greater the analysis that must be preformed.

Type 3 systems, with their ability to build inferences upon new data, have the largest analysis burden of any of these systems. This implies that they will be the least amenable to revision. There is a possiblity that the use of ontologies will assure that the implications of changes are a priori accounted for. Thus any changes consistent with the ontology will easily be integrated. The problem is that there is no historical precedent for developing stable systems of this type.

Aspects of a Platform Architecture: Part 2 - Evolution of a Platform

2007-01-10T06:42:00.000-08:00

Again, the goal is to allow a small number of applications, that share some core processes/entities to interact in a loose way and ship and evolve as independently as possible in the face of changing science, user needs, infrastructure and developer/application allocation.

In the last section, I talked about what a platform architecture looks like as a static entity. However the reason for having a platform architecture is the evolution of the application suite over time.

With stable identifiers and the architectural components discussed previously, I have found that middleware can be used as a tool for coherent platform development. Again, I’m basing everything on stable identifiers.Without stable identifiers everything is very hard, with them some things are possible, sometimes even easy. In the absence of stable identifiers it is hard for any common substrate to get a leverage point that allows a clear value add to all of the products under development.

My definition of Middleware is a bit broader that that in wikipedia

http://en.wikipedia.org/wiki/Middleware : Middleware is the enabling technology of Enterprise application integration. It describes a piece of software that connects two or more software applications so that they can exchange data.

For me middleware is also a place to incorporate the cross application business logic that allows users to interact with and see the data in a consistent manner. The middleware also shields custom applications from the hidden details of the database(s) or other persistent storage etc..

Some real life examples that I’ve seen in the life sciences include a situation in which the same substance may two different identifiers, the other is where the data is to be combined using a non-trivial algorithm e.g., geometric mean with outlier removal. In both cases the middleware served as the foundation for consistency since it was critical that all of the applications present the same information to users, and that there is only one implementation of the retrieval/calculation methods (for any non-trivial application the result given by two different implementations can skew over time).

Some of the considerations around method naming, signatures etc. are shared with library design and development. The best resource I know for addressing those considerations is Framework Design Guidelines: Conventions, Idioms, and Patterns for Reusable .NET Libraries
by Krzysztof Cwalina

A conventional diagram of such a system appears below

A more accurate diagram, given the goal of supporting rapid system evolution is

Where the red links show ad-hoc connections which to support rapid development. My preference is to have the middleware be the responsibility of a single person, as it is the key leverage point for the long term evolution of the architecture.

This person is given the time not only to evaluate architectural ideas that come in from other members of the team who may have implemented solutions that should be made available to others in a slightly generalized fashion but also to examine what’s going on in the industry as far as standards, toolkits etc that will help long term product evolution. In addition, for the ad-hoc connections and implementations to be capable of being moved to the middleware as described below, it is important for this person to have influence upon the design of these interfaces. I’ve found it uniformly tempting for the application owners to embed too much information into their interfaces so as to simplify their short term development, thereby hindering long term development.

Still what I’ve described so still sounds a bit static -- how does it play out over time?

Evolution proceeds as follows:

Even if we start with the Platonic "conventional diagram” shown above, it will quickly evolve into something along the lines of the more realistic version which shows that some ad-hoc connections that have evolved over time to give the individual applications the flexibility to meet their requirements.

The next release of the middleware (1.1) is picked up by the Ensemble Review application. In this case the release supports functionality that had required and “outside the box” access solution for the Ensemble Review and so its access can now occur through the middleware. Green arrows show functionality that has moved to the middleware with the release shown.

The arrow from the Small Group Drill Down application to the Ensemble Review app is shown as now going through the middleware since my practice was to ship the middleware as a labeled jar rather than a web service. Although this had the downside of increasing the footprint of each application it did allow the the interface to remain very transparent.

The next release of the middleware (1.2) is picked up by the New Data Requests application. We now have three versions of the middleware in production, each application has shipped independently, but they are all moving in the same direction and there has been no forking for feature support -- forking for bugs is of course possible.

and of course as shown below an application can pick up the latest version without requiring any improvements.

And then the cycle repeats. The only time that a “synchronized ship” (that is when all applications ship to production simultaneously with the same version of the middleware) is required is when there is an incompatible structural change to the shared data structures or a core business process/algorithm. At this point everyone picks up the same version of the middleware, the shared data store is migrated and extensive testing occurs.

The advantages of this sort of approach include:
The testing time for each application is reduced. An application need not pick up a version of the middleware if the new version doesn’t provide any functionality/bug fixed that it requires (an underlying assumption is that there is adequate regression test coverage to assure that the functionality required by the the application is not broken in the new release).

When an application needs the new functionality it upgrades to the current revision.
This also means that the application is not dependent upon the new middleware functionality is not governed by its timelines shipping

This has the additional benefit of allowing middleware releases to be more focussed on the need of a particular product, or to engage the product architect to support early testing of a feature that will help them.

Aspects of a Platform Architecture: Part 1

2007-01-10T04:54:00.000-08:00

How does one create a “reasonable” platform architecture? By which I mean, how does one put a system in place that allows a small number of (~ 10) related products to evolve and ship independently while building upon an infrastructure substrate to allow for common policies, consistent data access, and presentation.

First to be clear on the high level goals what is a Enterprise platform architecture
Multiple products

different developers, different use cases (users may overlap), same general business area

Multi-year timelines

changing developers, resourcing budgets, technologies, even the rate of change will all change.

Enterprise level

Multiple applications serving a particular group of areas within a business, which are different from issues involved in building an industry platform, Windows, Spring, Linux etc.

Common views upon the data

When the expectation around the data is the same, the result used/displayed should be the same, no matter how complex the processing is to derive it.
Ability to deviate when the expectation is unique or novel. Achieving the responsiveness required for iterative/agile techniques requires the ability to “special case” data access and processing methods. This allows these special cases to be well grounded before they are moved into the substrate (as appropriate).

Products should be normally able to ship independently
Products should be able to communicate between each other so that users can switch between applications (web based in this case) when desired with some minimal context being maintained.

Building this requires creation of a system consisting of a
Data Architecture

How do you identify what you’re looking for, where do you find it?

Application Roadmap

What application should own the functionality, and when will it be deployed?

Technology Architecture

What are the technologies that we’ve settled upon, how are we exploring new ones, how do we decide what to explore/who does it.

Functional Architecture

How are we structuring the functionality and identifying common components, how much of the implementation can we hide, what are the hiding requirements (timeliness etc.)

The first step, a Data Architecture foundation.
Select a small set of entities and assure that they have stable, anonymous identifiers. This is surprisingly difficult when building systems out of existing products in scientific domains.

I have been involved in frequent discussions with users who wish to embed domain information in the identifiers so that they can easily identify what they are holding in their hands without needing to go to a computer to find out what it is, or worst case wait for an application to be developed that will let them go to a computer to find out what it is -- a.k.a. every user’s worst nightmare. The best solution I’ve come up with is to print both when necessary.

This selection is also constrained by existing/off the shelf systems and what they can support. Selecting internal identifies from the core of these systems and building synonym tables is a reasonable compromise.

The issues of the remaining Data Architecture plus the other aspects of and Application Roadmap, Technology Architecture and Functional Architecture are important. However, given our multi product, multi-year, frequent revision goals, the issues are as much about building an infrastructure that supports a culture as much as building an architecture. Such a culture involves setting minimal contracts and some processes around their evolution more than instantiating any particular set. The core expectation is that the platform will outlive any applications and it is at the platform level that practices around technology adoption, quality practices, and user interaction should be set. Note: this is not meant to imply that the practices should be uniform across the applications, but the patterns of use categories vs development strategy should be set at the platform level.