scraping data from the web is difficult (but fun!) around the edges

Been working on a variety of data collection tools, and plan on continuing to do so. It's fun, I'm getting some things done, and it helps exercise the brain. Through all of these exercises:

Starting to realize, the hardest part about collecting data from the web... is not grabbing the data (Nokogiri makes that pretty easy)... it's making the data fit into the 'holes' you want it to fit in. Not referring to int, varchar, etc... not 'an array of hashes' either... that's super easy.. the ASCII/non-ascii character sets are a rather large pain. You have the web/html, Ruby (or Python or Java or Perl or PHP or whatever), MySQL. All have different constraints with text there and represent some characters differently. When collecting foreign and domestic names, as one example, it's especially apparent -- the tildes the oomlats the accents, etc. It's really a set of devilish details depending on the problem you're trying to solve. Yet another reason to master Regular Expressions, not to mention to grok the text representations of each system.

...am starting to believe I need to build an Adapter to bridge the gap between all of these character interfaces. In my Ruby Classes, make sure that inputs all funnel into what the database is expecting that can be represented properly. Web/Ruby/Database -- understand each other the way I (a human) can understand looking at names and interpreting. i.e. scraping Joakim BÄCKSTRÖM should equal Joakim BÄCKSTRÖM whether it's Joakim BackstrÖm or Joakim Backstrom or Joakim BACKSTROM or whatever -- and no matter which version i scrape on any site, they should all be equal, and should all lead back to PLAYERID = 623, for example, so all his data is collated and connected. And it also appears on the output side as one 'blessed' name.

Any thoughts on this issue? Any ideas of a different design pattern I could use besides Adapter?

Merry Christmas, Irish Optimists!

Irish annihilate Hawaii

What a phenomenal game for the Irish yesterday in the Hawaii Bowl. Clausen was perfect, completely accurate 401 yards through 2.5 quarters, 5 td passes. Armando Allen runs back a kick for a TD, Golden with 2 bomb TD catches, and one punt run back (called back for roughing, too bad).

The Irish looked dominant on both sides of the ball, and special teams. By far the best game they played all year, let alone the best game on the road. Over a Hawaii team that beat Fresno State and nearly beat Big East champion, Cincinnati.

Enjoy this sequence of pics from Armando Allen's ko runback... check out the block by Tate as he leads Armando through the hole in the wedge... awesome:

wedge develops and Golden starts through the hole...

line up Hawaii #18...


on his back!

Merb and Rails unite in Rails3!

I see this as great news for the Ruby and Rails and Merb communities... what are your thoughts?

Yehuda Katz post is a great rundown:

http://weblog.rubyonrails.org/2008/12/23/merb-gets-merged-into-rails-3/comments/24239#comment-24239

http://yehudakatz.com/2008/12/23/rails-and-merb-merge/

http://rubyonrails.org/merb

Google Finance - Feature Request

Dear Google Finance,

It is possible to include a music feature in your application, specifically in the My Portfolio section? I would like to play the music sound of a carnival carousel that goes up and down, up and down, up and down... make it have a downward trend though in the tone.

Weeeeeeeeeeeeeeeeeeeeeeeee!!!!

Sincerely,
Mark Holton
User

carl_spackler : Ruby to collect golf scores from web

Creating a new open source library hosted on Github, named 'carl_spackler'... it collects data on golf scores throughout the web, and produce a normalized form for each collection. Likely the output will be an array of Ostructs for starters.

Similar principle to OGWR, only for weekly tournament scores, not weekly golf rankings. This way anyone can collect both sets of data and use it as they so choose.

So lets say you want to grab all European Tour scores for the week, just dial up carl_spackler, and let it do your wet work for you.

Lots more to come on 'carl_spackler', 'OGWR', and others.

Carl Spackler, Greenskeeper, Bushwood CC

some good Ruby + Rails reads for the day

Threading in Ruby (and other dynamic languages with a GIL)
http://www.igvita.com/2008/11/13/concurrency-is-a-myth-in-ruby/

Rails deployment:
http://www.loudthinking.com/posts/30-myth-1-rails-is-hard-to-deploy

Nginx + memcached
http://www.igvita.com/2008/02/11/nginx-and-memcached-a-400-boost/

Ruby build for re-install of Leopard and install of RubyCocoa

I, like many others, had Mac OS X before Leopard came out. i.e. installed Leopard separately when it came out. Sometimes that causes some things to err out, because for one example, RubyCocoa was bundled with Leopard, and some gems depend on RubyCocoa depending on what you're doing. Plus it's nice to mess around with RubyCocoa.

Found a good post here related to manually building RubyCocoa for your /usr/local Ruby (http://stephencelis.com/archive/2008/11/the-definitive-leopard-ruby-build) . I modified it a little bit, because I want to run Ruby 1.8.6, not 1.8.7.

-bash3.2.17:>> curl -O ftp://ftp.ruby-lang.org/pub/ruby/1.8/ruby-1.8.6-p110.tar.gz

% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 4439k 100 4439k 0 0 50223 0 0:01:30 0:01:30 --:--:-- 41886

-bash3.2.17:>> tar xzf ruby-1.8.6-p110.tar.gz
-bash3.2.17:>> cd ruby-1.8.6-p110
-bash3.2.17:>> curl http://gist.github.com/22725.txt > ruby_thread_hooks.diff
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 3452 100 3452 0 0 7877 0 --:--:-- --:--:-- --:--:-- 58363

-bash3.2.17:>> patch -p0 < ruby_thread_hooks.diff

-bash3.2.17:>> ./configure --enable-shared --enable-pthread --enable-install-doc \
CFLAGS=-D_XOPEN_SOURCE=1

-bash3.2.17:>> make
-bash3.2.17:>> sudo make install

software mission statement

Was reading Signal vs Noise today, per the usual routine at lunch. ...they requested that folks describe them, to the layperson:
http://www.37signals.com/svn/posts/1371-describe-37signals-in-20-seconds-or-less

Here was my suggestion:
“We build sensible and usable customized web applications for humans… while contributing to open source software that helps people build, deploy and maintain web applications for themselves or their own business. In summary, we are entrepreneurial software humanitarians!”

That's my own philosophy on software, and the fellas at 37Signals seem to be of that mindset, after experiencing their products, reading their posts and book (GettingReal), listening to DHH, et al speak, etc.. Got me thinking that's a nice mission statement for individuals as well. So there it is.

It's nice to have the folks at 37Signals out there as an inspiration.

collect Golf's World Rankings on your own with new Ruby module!

...Official Golf World Ranking, OGWR...
wrote some Ruby code in a Module that grabs the official golf world rankings...
use the data as you wish in your own application...
released as open source...
 
code: http://github.com/holtonma/ogwr/tree/master/ogwr_scraper.rb
test: http://github.com/holtonma/ogwr/tree/master/test_ogwr_scraper.rb
 
enables anyone to take the active OGWR data and do with it what you wish
(E.g. store to a database each week, hold in your database to related to players on your website, etc).

...it's decent for a start, planning on continuing to make it more versatile and useful...

You can find the code on GitHub at the links above.

installing Nokogiri

Looking into using tenderlove's new library, nokogiri. Looks faster than Hpricot, according to the benchmarks he provides, the syntax also looks really nice.

hmmm.... I installed racc, and frex... getting an error when creating the Make file. I really want to try this out. If I figure it out, or talk to someone who knows, I'll post a reply. For now, here is the error I'm getting.


-bash3.2.17:holtonma:Fri Oct 31 12:09:37>> sudo gem install nokogiri
Building native extensions. This could take a while...
ERROR: Error installing nokogiri:
ERROR: Failed to build gem native extension.

rake RUBYARCHDIR=/opt/local/lib/ruby/gems/1.8/gems/nokogiri-1.0.1/lib RUBYLIBDIR=/opt/local/lib/ruby/gems/1.8/gems/nokogiri-1.0.1/lib
(in /opt/local/lib/ruby/gems/1.8/gems/nokogiri-1.0.1)
checking for xmlParseDoc() in -lxml2... yes
checking for xsltParseStylesheetDoc() in -lxslt... yes
checking for #include
... yes
checking for #include
... yes
checking for racc... yes
checking for frex... yes
creating Makefile
gcc -I. -I. -I/opt/local/lib/ruby/1.8/i686-darwin8.10.1 -I. -I/opt/local/include/libxml2 -I/opt/local/include -fno-common -O2 -fno-common -pipe -fno-common -g -DXP_UNIX -O3 -Wall -Wextra -Wcast-qual -Wwrite-strings -Wconversion -Wmissing-noreturn -Winline -c html_document.c
gcc -I. -I. -I/opt/local/lib/ruby/1.8/i686-darwin8.10.1 -I. -I/opt/local/include/libxml2 -I/opt/local/include -fno-common -O2 -fno-common -pipe -fno-common -g -DXP_UNIX -O3 -Wall -Wextra -Wcast-qual -Wwrite-strings -Wconversion -Wmissing-noreturn -Winline -c html_sax_parser.c
gcc -I. -I. -I/opt/local/lib/ruby/1.8/i686-darwin8.10.1 -I. -I/opt/local/include/libxml2 -I/opt/local/include -fno-common -O2 -fno-common -pipe -fno-common -g -DXP_UNIX -O3 -Wall -Wextra -Wcast-qual -Wwrite-strings -Wconversion -Wmissing-noreturn -Winline -c native.c
gcc -I. -I. -I/opt/local/lib/ruby/1.8/i686-darwin8.10.1 -I. -I/opt/local/include/libxml2 -I/opt/local/include -fno-common -O2 -fno-common -pipe -fno-common -g -DXP_UNIX -O3 -Wall -Wextra -Wcast-qual -Wwrite-strings -Wconversion -Wmissing-noreturn -Winline -c xml_cdata.c
gcc -I. -I. -I/opt/local/lib/ruby/1.8/i686-darwin8.10.1 -I. -I/opt/local/include/libxml2 -I/opt/local/include -fno-common -O2 -fno-common -pipe -fno-common -g -DXP_UNIX -O3 -Wall -Wextra -Wcast-qual -Wwrite-strings -Wconversion -Wmissing-noreturn -Winline -c xml_document.c
xml_document.c: In function ‘substitute_entities_set’:
xml_document.c:117: warning: unused parameter ‘klass’
xml_document.c: In function ‘load_external_subsets_set’:
xml_document.c:129: warning: unused parameter ‘klass’
gcc -I. -I. -I/opt/local/lib/ruby/1.8/i686-darwin8.10.1 -I. -I/opt/local/include/libxml2 -I/opt/local/include -fno-common -O2 -fno-common -pipe -fno-common -g -DXP_UNIX -O3 -Wall -Wextra -Wcast-qual -Wwrite-strings -Wconversion -Wmissing-noreturn -Winline -c xml_dtd.c
gcc -I. -I. -I/opt/local/lib/ruby/1.8/i686-darwin8.10.1 -I. -I/opt/local/include/libxml2 -I/opt/local/include -fno-common -O2 -fno-common -pipe -fno-common -g -DXP_UNIX -O3 -Wall -Wextra -Wcast-qual -Wwrite-strings -Wconversion -Wmissing-noreturn -Winline -c xml_node.c
xml_node.c: In function ‘new’:
xml_node.c:475: warning: unused parameter ‘klass’
xml_node.c: In function ‘new_from_str’:
xml_node.c:494: warning: unused parameter ‘klass’
gcc -I. -I. -I/opt/local/lib/ruby/1.8/i686-darwin8.10.1 -I. -I/opt/local/include/libxml2 -I/opt/local/include -fno-common -O2 -fno-common -pipe -fno-common -g -DXP_UNIX -O3 -Wall -Wextra -Wcast-qual -Wwrite-strings -Wconversion -Wmissing-noreturn -Winline -c xml_node_set.c
xml_node_set.c: In function ‘allocate’:
xml_node_set.c:106: warning: unused parameter ‘klass’
gcc -I. -I. -I/opt/local/lib/ruby/1.8/i686-darwin8.10.1 -I. -I/opt/local/include/libxml2 -I/opt/local/include -fno-common -O2 -fno-common -pipe -fno-common -g -DXP_UNIX -O3 -Wall -Wextra -Wcast-qual -Wwrite-strings -Wconversion -Wmissing-noreturn -Winline -c xml_reader.c
gcc -I. -I. -I/opt/local/lib/ruby/1.8/i686-darwin8.10.1 -I. -I/opt/local/include/libxml2 -I/opt/local/include -fno-common -O2 -fno-common -pipe -fno-common -g -DXP_UNIX -O3 -Wall -Wextra -Wcast-qual -Wwrite-strings -Wconversion -Wmissing-noreturn -Winline -c xml_sax_parser.c
gcc -I. -I. -I/opt/local/lib/ruby/1.8/i686-darwin8.10.1 -I. -I/opt/local/include/libxml2 -I/opt/local/include -fno-common -O2 -fno-common -pipe -fno-common -g -DXP_UNIX -O3 -Wall -Wextra -Wcast-qual -Wwrite-strings -Wconversion -Wmissing-noreturn -Winline -c xml_syntax_error.c
xml_syntax_error.c: In function ‘Nokogiri_error_handler’:
xml_syntax_error.c:162: warning: unused parameter ‘ctx’
gcc -I. -I. -I/opt/local/lib/ruby/1.8/i686-darwin8.10.1 -I. -I/opt/local/include/libxml2 -I/opt/local/include -fno-common -O2 -fno-common -pipe -fno-common -g -DXP_UNIX -O3 -Wall -Wextra -Wcast-qual -Wwrite-strings -Wconversion -Wmissing-noreturn -Winline -c xml_text.c
gcc -I. -I. -I/opt/local/lib/ruby/1.8/i686-darwin8.10.1 -I. -I/opt/local/include/libxml2 -I/opt/local/include -fno-common -O2 -fno-common -pipe -fno-common -g -DXP_UNIX -O3 -Wall -Wextra -Wcast-qual -Wwrite-strings -Wconversion -Wmissing-noreturn -Winline -c xml_xpath.c
gcc -I. -I. -I/opt/local/lib/ruby/1.8/i686-darwin8.10.1 -I. -I/opt/local/include/libxml2 -I/opt/local/include -fno-common -O2 -fno-common -pipe -fno-common -g -DXP_UNIX -O3 -Wall -Wextra -Wcast-qual -Wwrite-strings -Wconversion -Wmissing-noreturn -Winline -c xml_xpath_context.c
gcc -I. -I. -I/opt/local/lib/ruby/1.8/i686-darwin8.10.1 -I. -I/opt/local/include/libxml2 -I/opt/local/include -fno-common -O2 -fno-common -pipe -fno-common -g -DXP_UNIX -O3 -Wall -Wextra -Wcast-qual -Wwrite-strings -Wconversion -Wmissing-noreturn -Winline -c xslt_stylesheet.c
cc -dynamic -bundle -undefined suppress -flat_namespace -L/opt/local/lib -L"/opt/local/lib" -o native.bundle html_document.o html_sax_parser.o native.o xml_cdata.o xml_document.o xml_dtd.o xml_node.o xml_node_set.o xml_reader.o xml_sax_parser.o xml_syntax_error.o xml_text.o xml_xpath.o xml_xpath_context.o xslt_stylesheet.o -lruby -lxslt -lxml2 -lpthread -ldl -lobjc
ld: warning, duplicate dylib /opt/local/lib/libxml2.2.dylib
/opt/local/lib/ruby/gems/1.8/gems/hpricot-0.6.161/lib/hpricot/builder.rb:26: warning: `&' interpreted as argument prefix
Loaded suite -e
Started
..............................................................................................................................................................................................................................................
Finished in 2.551585 seconds.

238 tests, 701 assertions, 0 failures, 0 errors
/opt/local/lib/ruby/1.8/test/unit.rb:278: [BUG] Bus Error
ruby 1.8.6 (2007-03-13) [i686-darwin8.10.1]

rake aborted!
Command failed with status (): [/opt/local/bin/ruby -w -Ilib:ext:bin:test ...]

(See full trace by running task with --trace)


Gem files will remain installed in /opt/local/lib/ruby/gems/1.8/gems/nokogiri-1.0.1 for inspection.
Results logged to /opt/local/lib/ruby/gems/1.8/gems/nokogiri-1.0.1/gem_make.out

embedding a Gist in your blog

- Login to your Github account (free) https://github.com/
- go here: http://gist.github.com/ embedding a gist in your blog...
- add your code to the gist and click "paste"
- click embed, and embed the code in your blog (hopefully it is more useful than this code below!):

Open Source OGWR Scraper

Going to write some Ruby code to scrape the Official Golf World Ranking... going to release it as open source.

updated Rails to 2.1.2 along with various rubygems update

Info on Rails 2.1.2: http://weblog.rubyonrails.org/2008/10/23/rails-2-1-2-security-other-fixes

-bash3.2.17:holts:Tue Oct 28 09:23:10 ~ >> sudo gem update
Password: ********************

Updating installed gems
Updating ZenTest
Successfully installed ZenTest-3.11.0
Updating actionmailer
Successfully installed activesupport-2.1.2
Successfully installed actionpack-2.1.2
Successfully installed actionmailer-2.1.2
Updating activerecord
Successfully installed activerecord-2.1.2
Updating activeresource
Successfully installed activeresource-2.1.2
Updating hoe
Successfully installed rubyforge-1.0.1
Successfully installed hoe-1.8.2
Updating rails
Successfully installed rails-2.1.2
Updating rspec
Successfully installed rspec-1.1.11
Updating rubygems-update
Successfully installed rubygems-update-1.3.1

Ezra Zygmuntowicz Tech Talk : Ruby, Merb, Rubinius

This talk is great. Tons of information if you're interested in Ruby, Merb, Rubinius, etc.

"Engine Yard co-founder Ezra Zygmuntowicz gave a Tech Talk on Monday at Google. He covered some of the open-source projects we’re working on at Engine Yard, including Merb and Rubinius."

http://blog.engineyard.com/2008/10/24/ezra-gives-google-tech-talk-on-merb-and-rubinius

Sizzle

Browsers are getting faster (see: Google Chrome, see next release of Firefox) with JIT compliation, improved Javascript VM's, etc... but not only that, developers are getting better at writing codez and finding things in browsers... all of it makes the interwebs better: It's good to keep an eye on what is coming down the pike, as they say:

John Resig (jQuery author)... is working on Sizzle... which will become part of jQuery... check out the performance here:
http://jamesdonaghue.com/static/peppy/profile/slickspeed/#

Here's a link to Sizzle on GitHub: http://github.com/jeresig/sizzle/tree/master

Also, it was bounced around today on the Prototype mailing list that Prototype (traditionally slower than other mainstream libraries when traversing the DOM with CSS Selectors), is mulling over including Sizzle as well. Very nice developments -- check them out.

update Rails and gems on Joyent Accelerator

Was back on my Joyent Accelerator (been too long), and hadn't done an update in awhile. Needed to update Rails, among many other gems and was coming across this error whenever I tried updating any gems:

ERROR: While executing gem ... (Gem::GemNotFoundException)
could not find capistrano locally or in a repository

Realized I needed to install rubygems-update-1.2.0 from source:

$ sudo curl -O http://rubyforge.rubyuser.de/rubygems/rubygems-update-1.1.1.gem
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 418k 100 418k 0 0 146k 0 0:00:02 0:00:02 --:--:-- 165k

$ sudo curl -O http://files.rubyforge.vm.bytemark.co.uk/rubygems/rubygems-update-1.2.0.gem
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 246k 100 246k 0 0 147k 0 0:00:01 0:00:01 --:--:-- 183k

$ sudo gem install rubygems-update-1.1.1.gem
Successfully installed rubygems-update-1.1.1
1 gem installed
[admin@golfapp /home]$ sudo gem install rubygems-update-1.2.0.gem
Successfully installed rubygems-update-1.2.0
1 gem installed

$ sudo update_rubygems

$ sudo gem update rails
Updating installed gems
Updating rails
Successfully installed activesupport-2.1.1
Successfully installed activerecord-2.1.1
Successfully installed actionpack-2.1.1
Successfully installed actionmailer-2.1.1
Successfully installed activeresource-2.1.1
Successfully installed rails-2.1.1
Gems updated: activesupport, activerecord, actionpack, actionmailer, activeresource, rails

Rubygems 1.3.0 was out about 20 days ago (it's been awhile since I updated this Accelerator)...
http://blog.segment7.net/articles/2008/09/26/rubygems-1-3-0

$ sudo curl -O http://files.rubyforge.vm.bytemark.co.uk/rubygems/rubygems-update-1.3.0.gem
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 257k 100 257k 0 0 151k 0 0:00:01 0:00:01 --:--:-- 188k
[admin@golfapp /home]$ sudo gem install rubygems-update-1.3.0.gem
Successfully installed rubygems-update-1.3.0
1 gem installed
[admin@golfapp /home]$ sudo update_rubygems
Installing RubyGems 1.3.0...

After reading Eric's blog post, I should have just done a:
$ sudo gem update --system
$ sudo gem update

Live and learn. But RubyGems is updated now, and so are all my installed gems. Yayy. Need to deploy my app to this by Dec 1 hopefully.

Technology enhanced design : Bank of America ATM on Michigan Ave

I was in Chicago last week, visiting friends and family. It had been awhile since I had been back, great to see the folks, my brother and his family, and some close friends.

My wife had been to Chicago before, but somewhat briefly. So we spent a lot of time going to museums, a play at the Chicago Shakespeare theater (great venue!), the Hancock, some restaurants... and of course, some shopping on Chicago's famed Michigan Ave.

I had two very small checks in my wallet that I hadn't deposited for a couple weeks. We stopped into a Bank of America to withdraw some money and there was an ATM in there that was really impressive from a user experience perspective. It looked like most ATM's, but no need for envelopes to place your checks into. I took out the dog-eared and worn checks from my wallet, and slid one in on the prompt. Immediately a display of the check was on the screen, along with the Optical Character Recognition of the value of the check: $27.88. I confirmed the deposit amount via the touchscreen, and then deposited the second check. Once again, perfect. After depositing, the receipt that was disbursed also contained images of the checks as they had been scanned, along with the total deposit amount.

If you are in Chicago, near the Bank of America on Michigan Ave, give that ATM a try. :) Often times I find some implementations of technology to be cumbersome, or not yet mature enough to actually aid us -- whether it be the speed, the edge case errors, etc. But in this case, I hope to see these ATM's utilized throughout the country, as the user experience as enhanced by technology was an A+.

Syrin and Mudd ... are rewarded for this??

...thankful, in one respect, I don't own more than a marginal amount of Fannie/Freddie stock in the mutual funds...

...but boy, it sure worked out for Syrin and Mudd who will each exit with ~$15+ million each. We, the taxpayers, ultimately are paying that bill of course. ..$14 BILLION in losses the last 4 quarters... and it's far from over, as Paulson commits $200B from the Federal coffers to assuage the country's fears.

"As early as 2005, Fannie executives publicly expressed concerns about growing risks in the mortgage market In May of that year, Thomas Lund, a Fannie Mae executive vp, said that lenders should be concerned if borrowers straining to afford homes were given loans allowing for low payments in the early years but storing up much higher ones for later"

"$5 trillion of debt and mortgage-backed securities issued by Fannie and Freddie is owned by central banks and other investors world-wide."

While I mind chipping in to pick up the $200B with my N% in taxes, obviously we have no choice ... what I really mind, is these two "leaders" walking out of this with $15M cash after steering the ship into a glacial iceberg. Absolutely insulting, money out of our pocket, while the guilty gets rewarded with a golden parachute of 7 figures.

Where have all the leaders gone??

-Holts


http://online.wsj.com/article/SB122079276849707821.html

Dynamically typed languages are meritocracies

From Russ Olsen's "Design Patterns in Ruby":

"Statically typed languages are constantly asking about your parent or grandparent, or perhaps, in the case of Java-style interfaces, your aunts and uncles. In a statically typed language, an object's family tree matters deeply. Dynamically typed languages, by contrast, are meritocracies: They are concerned with which methods you have, rather than where those methods came from. Dynamically typed languages rarely ask about an object's ancestry; instead they simply say, "I don't care who you are related to, Mac. All I want to know is what you can do."

I love a good meritocracy.

InfoQ Virtualization article

Nice overview:
http://www.infoq.com/articles/virtualization-intro

DHH at "Startup School"

Knol and Semantic Web

http://knol.google.com/k

...reading between the lines on this for awhile... Google is seemingly going to move in the semantic direction... this is a step... by referencing a Knol (or a Wiki link or any link to a definition) in an RDF, you define that element in the HTML hierarchy with a real meaning "a shark is an aquatic creature, yada yada". Search for "shark" not only pulls up hyperlinked web content (current legacy which uses headers, urls, hyperlinks... to relate search terms to highest relevance), but pulls up all in the 'web' defined, so it pulls up shark, the creature that swims, not Shark the former wide receiver for the Irish Jeff Samardijza, nor The Shark Greg Norman, for instance... it's a way to relate words with different definitions in a semantic way, which is a common and big problem to solve on the web. It also enables new automated ways for apps to talk with other apps. The web has been missing this step towards progressing.

Google doesn't want to be beholden to Wikipedia... and why should they be, they have the resources and the intelligence to pull this off and move it forward.

... those ahead of the curve, start thinking about and programming your internetz and interwebs semantically...

http://www.w3.org/DesignIssues/LinkedData.html

Metaclasses and Ruby Inheritance Chain

This is a very large and involved topic. (great topic! it is part of what makes Ruby so powerful)
You can add new methods to a Class on the fly at anytime. Can also overwrite methods at any time. And, in Ruby, this truly means virtually any Class. Including classes like Array, String, etc.

Let's say we have

>  class Course 
>     attr_accessor :id, :name 
>     def initialize( id, name)
>       @id, @name= id, name
>     end
>  end

> c = Course.new( 1, "Augusta National" )
=> #<Course:0x81cfb94 @id=1, @name="Augusta National"...

> c.class
=> Course
> c.id
=> 1
> c.name
=> "Augusta National"


So in this case, a Course object, once initialized, will have a @id and a @course variable.  It can hold
any other variables at any time too...

 > c.instance_variable_set( "@par", 72 )
  => 72
 > c.par
  => MoMethodError: undefined method 'par' for #<Course..
It's holding that variable for par, but now we cannot get at it without a read method.  So let's say you
wanted to only have the reader/writer methods for par on that instance, not on the class
(just an example).  

Classes hold methods, objects do not.  Except in the case of metaclasses:  
> class << c
>   attr_accessor :par
>   def pretty_par
>     puts "par: #{@par}"
>   end
> end

Now you can do
 > c.par
=> 72
Also when you do:
 > c.methods
=> ["par", "id", "name", "pretty_par", ...] #=> these are referred to as the
                                        #=> object's "Singleton methods"

but when you show the methods for the Course class:
 > Course.methods
=> ["id", "name", ...] # i.e. no par, or pretty_par, because there is no method
                   # for these in the Course class, only the c metaclass
likewise:
 > c2 = Course.new(2, "Shinnecock Hills")
  => #<Course:0x91cfb94 @id=2, @name="Shinnecock Hills"

but it won't have access to any of the Singleton methods in c (i.e. neither "c2.par", nor "c2.pretty_par" method
calls will work.  They only work in the c metaclass).

You see, because when Ruby looks for a method, it first looks at the metaclass.
i.e. the metaclass intercepts the request, and checks for methods there first.

Then it goes to the object's class, then the SuperClass of that object, then any super's of that object and so on
...all the way back to the Object base class.

You can insert methods into the class, or into an object's metaclass at any level in the inheritance chain.
Very powerful.


One more beauty... if you attach a:
...
def method_missing
puts "there was no method that matched that"
end
...
you can intercept the call to a missing method (anywhere in the inheritance chain, the class, the metaclass, etc)
and do things like creating methods on the fly, programmatically.

It's called "monkey patching"  (some people call it "Duck punching", and there are other names).
Ruby is a great language to metaprogram with.  Rails uses method_missing often (along with const_missing) --
for instance, ActiveRecord uses method_missing to dynamically support the find_ methods.

That's a whole other interesting topic in itself to explore!

Bill Gates attempts to Master Kung Fu in retirement

Mr. Bill Gates, at his first Kung Fu lesson upon retiring from Microsoft.

Powerset : Ruby Front End

Great insights on Ruby's front end from Kevin Clark's blog:

http://glu.ttono.us/articles/2007/06/21/powerset-to-launch-front-end-on-ruby

"The simple fact is that Ruby wasn’t the source of Twitter’s woes. As it often happens with rapidly growing sites, they ran into architectural problems. Some design decisions don’t hurt until they reach a massive scale and at that point you have to rethink your approach. In an email he writes:

For us, it’s really about scaling horizontally - to that end, Rails and Ruby haven’t been stumbling blocks, compared to any other language or framework. The performance boosts associated with a “faster” language would give us a 10-20% improvement, but thanks to architectural changes that Ruby and Rails happily accommodated, Twitter is 10000% faster than it was in January

This is great news for Twitter, but even better for us because we don’t have the bottle necks that they’ve struggled with – databases, instant messaging servers, and regularly recycling cache systems – which makes scaling horizontally much much smoother. At that point, our scaling issue doesn’t concern Ruby. For a search engine, the front-end is largely just a templating system and the real work happens in the back when we process your query."

Microsoft buying Powerset?

...Rumor has it, that Powerset is being bought by MSFT... that can only mean they are trying to go semantic in their search quest... hopefully the great technical and semantic momentum with Powerset will carry on.

...Powerset is written in all OpenSource -- Ruby, Merb, Rails, god, Mongrel, Mootools, Memcache, Erlang, Fuzed*, YAWS, Hadoop... one of their developers is Kevin Clark, active Ruby/Rails developer: http://glu.ttono.us/articles/2008/05/12/holy-god-powerset-launches

More on Powerset:
http://searchengineland.com/080512-000100.php

had written about Powerset and Kevin Clark 9 days ago...
http://holtsblog.blogspot.com/2008/06/businessweek-article-on-search.html

Interesting to see how this plays out.

...great demo (and cross-browser)

e

...using Canvas... this guy created a photo-table-like system. Be sure to click the "show corners" and resize the images on the fly in the browser. Pretty slick (and thought provoking for you image-philes). Then you can export the image as a .jpg. "makes it easy to create highly customized compositions"... yeah yeah, that's cool and all, but beyond this implementation the demonstration of what 'can be done' in browser-native style is awesome. (amazing it works in ie6, with the 'excanvas.js', which is apparently a Google contribution to the community)

Have fun...
http://www.ernestdelgado.com/public-tests/canvasphoto/demo/canvas.html

http://developer.apple.com/documentation/AppleApplications/Reference/SafariJSRef/Classes/Canvas.html#//apple_ref/doc/uid/-CJBCECGI.html#//apple_ref/doc/uid/

http://developer.mozilla.org/en/docs/Canvas_tutorial

Anyone know of any good books or good tutorials on Canvas?? They seem to be few and far between.
? The books on are few and far between. Post a comment if you know please. Cheers-

Canvas compatibility table for the Canvas Photo Experiment:

Browser	Support
ie6+	Yes (+ excanvas.js)
ff2+, ff3+	Yes
safari2+	Yes
opera9+	Yes