January 13th, 2009

Lifestyle Listings Engine…stuff. PART 2

By Liam Dayan, GM Listings Group, CTO

Yesterday I left off with a promise to explain what an expert system is, and give some insight into how those play into Onboard Informatics approach to aggregating listings (and other) data. Ok…so as cliffhangers go it’s no Stewie Kills Lois, but I’m in this for the art not the ratings.

The first time the phrase “expert system” found its way into the dictionary (Merriam-Webster) was in 1977: computer software that attempts to mimic the reasoning of a human specialist. While there are many other concepts and an almost unconscionable amount of geekspeak jargon that go along with this concept, this definition holds up. Computers are stupid, standard software only mildly less so. In a perfect world I would have a human being working in constant attendance on an individual feed. Actually, make that a team of people all dedicated to one feed since the burnout rate would be staggering if you didn’t round-robin them.

So, acknowledging that human wisdom and expertise is the most reliable way to make good decisions, how do support a solution which scales? By attempting to codify that “wisdom”, i.e. domain-specific knowledge applied within a rules and/or model-based construct, into software. In any given domain there’s what’s generally known, and then there’s what true experts know–the real skinny.

Want the secret sauce? Distill the heuristic knowledge of a body of human experts into a knowledgebase (just think of “heuristics” as meaning “rules of thumb”). Now develop a robust system able to infer/derive new knowledge for decision-making from that knowledgebase, enabling appropriate feedback loops so that the system continuously improves over time. This amounts to a (tragically) long array of IF-THEN assessments which are dynamic in nature, growing ever longer as the system “learns”. We’re hardly the first to this particular party in a general way, but I don’t think it’s been done as significantly in RE before.

Some brief descriptions of what’s going on in this flowchart:

1) Controller: manages the occasionally complex interaction between some of the subsystems, with the heaviest load being the interaction between KnowledgeBase and Inference Engine. Those components make up the bulk of “intelligent” operation in the system and require significant management to maintain correct order of operation, etc. This piece also works as the “scratchpad”, allowing for fast ad hoc iterative assessment and conclusion.

2) KnowledgeBase: this is where the heuristic, IF-THEN rules, fuzzy logic, rules of thumb, expertise, wisdom, whatever you want to call it are stored. Don’t be too impressed, this is just a database, and a pretty simple one at that.

3) Inference Engine: this component is responsible for actually implementing the IF-THENrules, and can do so both sequentially and in parallel based on the need. Ok, you can be impressed with this one–it’s tough.

4) Knowledge Feedback Loop Engine: responsible for determining what information is fed back into the KnowledgeBase in order to refine the systems capabilities, most often (at this point) in partnership with human guides.

5) Human Experts: aka, wetware. These are folks whose brains are being drained (as we speak) into the KnowledgeBase–a relatively painless process. Occasionally they don’t even know they are participating. Obviously these people will be the first casualties of the war when the system inevitably decides it can serve real estate better than us. Heroes, all.

6) Report Generation: the primary way by which the Human Experts can see into the systems activities and provide feedback, correction, etc.–e.g. the system reports that image population in a particular feeds records has fallen below norm, but not critically. Only humans are capable of reaching the intuitive and experiential conclusion that the tolerance may need to be updated due to conditions beyond the systems ken.

“Listings” aggregator - means not the end

I think that might be a good place to conclude, by highlighting that humans are an intrinsic part of this system. We fold humans in all along the way because 1) we just don’t trust the computers that much–they’re dumb, and potentially evil (what…you never saw Terminator?); and 2) the system is in place, but as with any knowledge-based system it’s only as good as its knowledgebase is complete. I’d warrant ours is better than anyone else’s, but it’s too new to be complete and needs to be spot-checked by an actual for-real expert who repairs mistakes and reports them back into the system, thereby improving the knowledgebase and system. The upshot is our system may make mistakes from time to time, but they’ll be caught and repaired quickly, and shouldn’t happen twice.

That our approach uses an expert system puts it way ahead of the curve in the RE space, but I don’t want to overstate it. We’re still at the nascent stages of this. It should also be remembered that our goal isn’t to be an IDX, VOW, or even generic “listings” aggregator–that’s the means, not the end. The result we want to achieve is good enough information about the available properties in a locale that we can formulate effective search pathways, and help our customers create compelling user experiences.

Ultimately, our vision calls for the incorporation of a neural network to allow for some predictive analysis, advanced natural language processing, and some other pieces which are evolutionary by their very nature. “Better than anything I’ve ever seen in real estate” is a phrase I heard more than once at Inman this past week. That’s not a bad place to start, but we have a long way to go to bring this to where we think it can and should be.

So day one was basically a statement of the problem, day two a brief description of the “superstructure” used to solve that problem (trust me, that was brief). Tomorrow, I’ll post about some of the specific problems which occur in the aggregation of listings data, and try to give some insight into how our system handles those recurring issues.

- Liam

Tags: data, IDX, Lifestyle Listings Engine, listings feeds, Onboard Informatics, VOW.

December 19th, 2008

IDX, VOW, WTF? The techno-ninja perspective.

By Liam Dayan, GM Listings Group, CTO

I would have thought by now I’d be starting a blog post with the phrase “I get a lot of questions about VOW feeds”. Um…not so much, and I’m wondering why. I think it must be because folks are still trying figure out exactly what the real world implications of that NAR/DOJ agreement are.

I’m no lawyer (not bragging or anything, just saying), and maybe my good friend Das Hahn (The Hahnster? Robtastic?) will chime in if I say something completely wrong or idiotic (always a possibility), but if anyone’s interested in plowing through some inadvertently alliterative parenthetical prose about what a data-geek who’s been in the RE space for longer than he cares to remember thinks about this, here you go. If you want a much sharper legal mind’s take on this, check out Rob’s post. FYI, if you actually are interested in my thoughts on this, you should consider a hobby–I hear free running is a lot of fun, and great cardio.

So what the deally yo with VOW? Some background for the uninitiated (just skip this whole next paragraph if you’re generally familiar with VOW), and here I’m just going to just quote Brian Larson who knows what we in the business call a “crap-load” more about this than I do:

For purposes of the DOJ/NAR settlement, a VOW is:

A web site, or feature of a web site, operated by a Broker or for a Broker by another Person through which the Broker is capable of providing real estate brokerage services to consumer with whom the Broker has first established a Broker-consumer relationship (as defined by state law) where the consumer has the opportunity to search MLS data, subject to the Broker’s oversight, supervision, and accountability. (See Policy Section I.1.)

So, basically, any broker with an MLS participant relationship can serve up listing data online to someone they’ve got a “Broker-consumer relationship” with (more on that momentarily). Well…so what? Isn’t that just IDX?

Duck!

Nope, and the differences are critical and potentially very useful. It may look like a duck, but it quacks like a Hell’s Angel. IDX, Internet Data Exchange, is an agreement brokered by the MLS among its members to allow the limited display of limited listings data on each others’ sites–a marketing quid pro quo. The biggest of those limitations involve the number of attributes (data-geek-speak for “fields”) of which some MLSs only allow a handful in IDX feeds, the outright elimination of certain record types (e.g. most off-market data), restrictions on the display and enhancement of listing data ranging from the merely problematic to completely draconian, and–only in some cases, admittedly–the ability for brokers to opt-out of the program rendering the data-set incomplete.

This is in stark contrast to VOW feeds, which are meant to replicate the brick-and-mortar experience of eons past when listings books lived in every office and could be given to customer to take home and peruse. No really, actual books that were printed on something they called paper–yeah, I know. Those feeds are complete and whole, i.e. the majority of useful-to-the-consumer info about the listing is still there and (critical!) brokers cannot opt-out.

Additionally, these feeds are relatively unencumbered by a user-experience-destroying and ever-changing display policy. Meaning you can take the Christmas tree that is the listing and hang some truly kick-ass ornamentation on it, like really relevant contextual content, interesting search, maybe some tools to improve lead generation/distribution/management for brokers, etc.–even the potential for re-margining broker business in advantageous ways (hopefully we’ll hear more on that later from our resident broker biz expert, Dave Collins).

Say! I might know some folks who can help you with all that stuff. Sweet! Tree-trimming party at Onboard’s!

Before we start doing tequila shots, though, there are a couple of hiccups–some real, some merely perceived–that we need to address. Remember that “Broker-consumer relationship” thing? That’s the one folks seem most concerned with, including our own inestimable Rob Hahn. He sees a threat to brokers here–the prevailing theory being if you have to make people sign-up you kill a site’s “throughput”, providing an overwhelming competitive advantage to sites that don’t (i.e. Realtor.com, IDX-based sites, etc.). Me? I only see advantage, provided they can either negotiate some of the technical (not legal!) hurdles or partner with someone who can.

Between the time that I left eNeighborhoods and when Marc and Jon came and drafted me back into the fray, I spent two years as CTO of a pure leadgen company outside of RE. Reg-path management is somewhere between a science and an art, and I won’t claim it’s easy, but there are a variety of ways to overcome the objection of consumers to signing up, including some technology-based sleight of hand. Frankly, handled right registration can be converted from being a barrier into being a feature. Think velvet ropes and VIP rooms. Also, that consumer objection to signing up is just generally getting quieter and smaller every day as people’s use of social networking mediums that demand sign-up for participation grows. Add single sign-on mechanisms, either individual or one of the social media platform initiatives like OpenID, etc. and this becomes negligible.

And BTW? There’s nothing in the language of the VOW agreement or any other I know of (big caveat on that one) that precludes a broker from maintaining both of those feeds. That introduces some interesting hybrid UX/reg-path possibilities. This is a very solvable problem.

Another point Rob and I disagree on is whether brokers can be prohibited by their MLS from giving their listings to the Zillow’s of the world–he thinks they can be, I think brokers can do what they please with their listing and tell whoever says otherwise to pound sand. It should be noted that this point is pretty much all legal ground not technical, so Rob’s far more qualified than me to assess it. The only point I’ll make is that from a broker-site functional/operational perspective, I think his point is basically moot (and I want bonus points for using legal terminology!).

Even if his premise is borne out, the only “feature” really compromised is syndication, which was only ever engaged in by brokers out of desperation (though I would contend that more enlightened brokers shouldn’t feel threatened by alternative marketing channels). If brokers can get that traffic back from Zillow, et al, into their site (or potentially some other VOW-enabled broker site for the standard split which defines the central value proposition of the MLS to its members) by creating better destination sites, with SEO-friendly content, a better toolset, etc. as a result of having more robust listing data–that sounds like a win to me.

Good, bad...I'm the guy with the gun.

People in the space–brokers, agents, the MLSs, etc.–are going to make arguments that VOW feeds are good for this, bad for that, whatever. I feel the same way about this as when I hear arguments from developers about which language is “the best” for coding in, or from infrastructure folk debating the virtues of one OS over another. I tend towards the agnostic for abstract mechanisms of any stripe, and save my judgments for how appropriate a specific application of that mechanism is or how well it’s been implemented.

Long story short (too late!), smart management of this newly available mechanism could allow brokers to drive back the interlopers (their word, not mine) who are building businesses on the back of listings that brokers spend time and money farming. And they can do it by using that unshackled listing data to create a far more compelling user experience for the consumer, and ultimately a better buyer/home match and more ideal result all around. The natural traffic generated by (properly deployed and supported) listings comes home like the prodigal son, ready to courted and managed into leads by intelligent brokers, and handed off to caring and knowledgeable agents ready, willing and able to put families in their new homes. Happy brokers, happy agents, happy buyers, happy sellers.

Either way–VOW, IDX…I’m the guy with the listings.

- Liam Dayan

Tags: data, IDX, leadgen, listings, marketing, MLS, NAR/DOJ, Real Estate, techno-ninja, VOW.