I have long enjoyed using Open Source Software (OSS) as a developer for many good reasons. It is not even an issue of being "free", but of having community access to the code to learn, troubleshoot, and enhance the functionality. Support for OSS by the community is fast and accurate. As the source is scrtinized by both the white hats and the black hats, it is also make more secure from attacks and vulerabilities.

The greatness of a full OSS system is best seen by operating systems such as GNU/Linus and FreeBSD, databases like PostgreSQL, and a host of other powerful software that competes as well as or better than many commercial offerings.

We watched Linux for years, expecting it to overtabke commeercial operating systems like Windows. Each year, it seems just as far away from that goal as it did the previous year. Why is that, I wondered?

The limit of Open Source Software

Recently, Linus Torvolds (the "creator" of Linux) spoke of Abandoning the GNOME 3 GUI interface. Before that, the lastest major version of KDE 4, an alternative to GNOME on the Linux operating system, was received to a damp reception. Of course, these things happen with commercial operating systems too, like the blunders of Windows Vista. But within a year, Microsoft released a beta of Windows 7, aiming to fix the shortcomings of a new design.

Microsoft, and certainly Apple Computer have a large interest in making the increasingly complex software easy to use. Beyond operating systems, applications like MS Office, Photoshop, etc., need to maintain a user-friendly veneer on top of a complex software application.

Why does OSS fail? It does not have the same demand for interface design, ultimately commercial success. There are no interface designers creating a navifation structure across the operating system, no reaction to interface failures. They seem to stick to their plan, and blame the users for their own issues.

The OSS community creates and delivers powerful software and appeals to other developers and geeks for use on servers, but until they can compete for users on interface design, theye will always loose the desktop wars.

-- PS -  This post was written in September 2011.

Permalink | Leave a comment  »

Recently, NoSQL data stores have been getting a lot of attention, as an alternative to using a relational database. They allow more complex data structures to be stored and queried than you find in the table-row-column model.

The PostgreSQL relational database server has several data features that you would expect to be found only in the NoSQL data stores.

Documents (Text Datatype)

While not a data structure itself, the ‘text’ datatype is a “clob” (character large object) of an “unlimited length” and can be used to store documents. This can be useful for specialty formats like YAML, XML, JSON, and serialized data from programming languages. Some of these document types are supported as specialty data types; more on that later.

PostgreSQL has string functions and operators and matching operators including regular expressions and its “similar to” hybrid of the SQL “like” and regular expressions.

The real power of PostgreSQL here is the full text search (a successor to earlier Tsearch and Tsearch2 extentions) features to search these documents. Its full text search offers stemming (removing pluarizations and conjugations of a word) and weights to make searching your documents as easy as a search engine.

The GiST and GIN index types are used to speed up full text searches by indexing the content as a standard search engine would. A GIN index is faster to search than a GiST index, but slower to build or update; so GIN is better suited for static data and GiST for often-updated data.

Arrays

A PostgreSQL column can be created as an array of values, a table, or other variable-length multidimentional arrays. Append the [] brackets at the end of the datatype to define the column as an array of that datatype.

create table lists  (id serial primary key, items text[]);
create table tables (id serial primary key, items text[][]);

The array representation syntax can be either:

'{1,2,"Hello, there",word}'
array[1,2,3]

The first version, is how psql will print out the array and how it will be returned to your program, as a string of comma-separated values. PostgreSQL does support different delimiters. The second version may only contain integers.

Access the elements of the array using the column_name[index] syntax. The first item in the array is at index 1, not 0. The split[start:end] syntax returns the array slice between the two given indexes, and is returned in the ‘{start,middle,end}’ syntax.

Also, PostgreSQL provides an intarray module with functions and operator for working with arrays of non-null integers.

To increase performance for lookup of array values, create a GIN/GiST index on the array column

create index lists_index on lists using gin (items);

Name-Value Pairs

The hstore datatype provides a column defined as a set of name-value pairs. This feature is not in core PostgreSQL, but is delivered as an extenstion in the “contrib” directory of the PostgreSQL distribution. The feature may already be compiled into your database, but may need to be enabled in the databases you need it.

create extension hstore;
create table catalog (id serial, specifications hstore);

NOTE: If you wish to have the hstore installed in all future databases you create, install it into the “template1” database.

The syntax for specifying the dictionary is like this:

'cores=>1, "graphics card=>"xyz graphics"'

So note that double-quotes have to be used for non-simple names and values, or when the contain special characters. Access done as follows:

select specifications->'graphics card' from catalog where specifications->['cores']=1;

To insert/replace or delete a key:

UPDATE tab SET h = h || ('c' => '3');
UPDATE tab SET h = delete(h, 'k1');

To increase performance for lookup of names or values, create a GIN/GiST index on the array column

create index catalog_specifications on catalog using gin (specifications);

XML

PostgreSQL comes with a specialized datatype for XML documents, including functions to parse, alter, and traverse the structure of the document.

CREATE TABLE test (a xml, b xml);
SELECT xmlelement(name test, xmlattributes(a, b)) FROM test;
SELECT xpath('/my:a/text()', 'test',
         ARRAY[ARRAY['my', 'http://example.com']]);

JSON

Soon, PostgreSQL will release a JSON datatype to handle JSON documents as it handles XML. There has been work done through a Google Summer of Code project in 2010, but the PostgreSQL team needs more time to merge it into the distribution.

Custom Data Types

PostgreSQL can be extended to handle new user-defined datatypes that can handle new types of data. Existing specialty type include:

* inet - Holds an IPv4 or IPv6 internet address with CIDR functions and operators
* money - hold curreny amounts with a fixed precision
* enum - holds a static, ordered set of values
* PostGIS - holds positions of geographic information systems

Caveats

With all these wonderful types, you still get the ACID complienace you expect from a relational database.

However the downside is that the datastructures are updated as an entire column and row in your table. It does not treat each element in the data structure atomically. As a result, the database is not appropriate to hold very large data structures where you intend to do frequent updates.

These extenstions are best used as a tool to encode and access more specific information in your row, rather than as a Data struture store like Redis and Riak, or document store like CouchDB or MongoDB.

Permalink | Leave a comment  »

Understanding how Resque works

Resque is a fast, lightweight, and powerful message queuing system used to run Ruby jobs asynchronously (or in the background) from your on-line software for scalability and response. I needed to integrate software written in different languages and environments for processing, and this is my understanding of the implementation.

How Queuing with Redis works

Resque’s real power comes with the Redis “NoSQL” Key-Value store. While most other Key-Value stores use strings as keys and values, Redis can use hashes, lists, set, and sorted sets as values, and operate on them atomically. Resque leans on the Redis list datatype, with each queue name as a key, and a list as the value.

Jobs are en-queued (the Redis RPUSH command to push onto the right side of the list) on the list, and workers de-queue a job (LPOP to pop off the left side of the list) to process it. As these operations are atomic, queuers and workers do not have to worry about locking and synchronizing access. Data structures are not nested in Redis, and each element of the list (or set, hash, etc.) must be a string.

Redis is a very fast, in-memory dataset, and can persist to disk (configurable by time or number of operations), or save operations to a log file for recovery after a re-start, and supports master-slave replication.

Redis does not use SQL to inspect its data, instead having its own command set to read and process the keys. It provides a command-line interface, redis-cli, to interactively view and manipulate the dataset. Here is a simple way to operate on a list in the CLI:

$ redis-cli
redis> rpush mylist "hello, redis"  # <= Adds the value to the right side of the list/queue
(integer) 1

redis> keys mylist*                 # <= Returns the matching key names
1) "mylist"

redis> type mylist                  # <= Returns the datatype of the value of this key
list

redis> lrange mylist 0 10           # <= Returns a elements 0 through 10 from the list/queue
1) "hello, redis"

redis> llen mylist                  # <= Returns the number of elements in the list/queue
(integer) 1

redis> lpop mylist                  # <= Pops the leftmost element from the list/queue
"hello, redis"

redis> lrange mylist 0 10
(empty list or set)

How Queuing with Resque works

Resque stores a job queue in a redis list named “resque:queue:name”, and each element is the list is a hash serialized as a JSON string. Redis also has its own management structures, including a “failed” job list.

$ redis-cli
redis> keys * 
1) "resque:stat:processed"          # <= Number of jobss successfully processed
2) "resque:failed"                  # <= This is the failed job list (not a queue)
3) "resque:queue:myqueue"           # <= This is your work queue!
4) "resque:queues"                  # <= The "Set" of work queues
5) "resque:stat:failed"             # <= The number of failed jobs
6) "resque:workers"                 # <= Set of workers
7) "resque:worker:host.example.com:79163:myqueue:started" # <= Count of jobs processed by worker
8) "resque:processed:host.example.com:79163:myqueue:started" # <= Timestamp of worker start

redis> get resque:stat:processed    # <= Returns the count of processed jobs 
"9"

redis> smembers resque:queues       # <= Prints the members of the set of queues
1) "myqueue"

redis> smembers resque:workers      # <= Prints the set of workers
1) "host.example.com:79163:myqueue"

Resque namespaces its data within redis with the “resque:” prefix, so it can be shared with other users.

Designed to work with Ruby on Rails, Resque jobs are submitted and processed like the following boilerplate:

class MyModel
  @queue = :myqueue                 # <= jobs will be placed in this queue name

  # call to queue processing in Resque until later
  def defer(*args)
     Resque.enqueue(MyModel, self.id, *args)
  end

  # Resque calls this method with the additional arguments. Must be named #process
  def self.process(id,*args)
    model = MyModel.find(id)
    # Do something here, raise an exception to send job to failure list
    raise "Oh Noes!" if failed?
  end
end

This does not serialize an object to the queue, instead it saved the (ActiveRecord) model name and record id which is re-instantiated from the database later. The additional arguments are saved in an array to call later. To keep the operation light, do not pass a lot of data to the job. Instead pass references to other records, files, etc.

Each job in Resque is a hash serialized as a JSON string (remember data structures can not be nested in Redis) of the format:

{"class":"MyModel", "args":[123, "arg1", "arg2", ...]}

When the job is popped from the queue, Resque instantiates the ActiveRecord object and calls its process method, passing the additional parameters. Functionally, the worker code behaves something like this (simplified):

klass, args = Rescue.reserve(queue_name)
model = klass.process(*args)

If processing raises an exception, the job and relevant information is placed on the failed list of the JSON format (as a string):

{ "failed_at":"2011/08/22 15:55:16 EDT",
  "payload":{"class":"MyModel","args":[123,"arg1","arg2"]},
  "exception":"NameError",
  "error":"uninitialized constant SalsaJob",
  "backtrace":[...],
  "worker":"host.example.com:56870:myqueue",
  "queue":"myqueue",
  "retried_at":"2011/08/22 16:07:50" }

A failed job can be retried (only once though) through the web interface started with the resque-web command.

Using Resque without Rails

Resque runs out of the box on Ruby on Rails. If you have a ruby application not in Rails, you can still run the Resque workers with the rake command by adding

require 'resque/tasks'

to your Rakefile.

Calling external systems with Resque

There are ports of Resque to other languages such as python, C, Java, .NET, node, PHP and Clojure. If your external system is written in one of these languages, then you can start workers listening to their queues. Since you are not talking to a ruby class with arguments, you can set up a placeholder class with the proper queue name. This will allow Resque plugins to fire on enqueue. (I assume the other libraries work the same way as the original, though some of the languages are not object-oriented—I have not verified them.)

class ExternalClass
  @queue = :external_class
end

Rescue.enqueue(ExternalClass, *args)

That class does not have to implement process() since that will be called in the real class.

If you need to call an external system to perform the task, either that system can be written to accept Resque-style queuing requests (hash of “class” and “args”), or you can push the expeted format directly to the queue

Resque.redis.rpush("queue:#{queue_name}", args.to_json)

The format does not have to be json, but has to be a string of a format the external system expects. You can not use the Resque workers

Calling the Ruby Resque from an external system

Maybe your external system needs to trigger a job to run on your Ruby Resque system, but can does not have a Resque implementation. You can drop your work (as a JSON hash of “class” and “args”) on the raw Redis list/queue yourself from the Redis library or the command line

redis-cli rpush "resque:queue:myqueue" '{"class":"MyModel","args":["arg1"]}'

Epilogue

I am new to Redis and Resque and wanted to dig into the Redis data structures used by Resque, and learned it more in depth while writing this. After understanding how it all fits together, I can now write some integration code! I hope you found this useful, and not too incorrect.

Permalink | Leave a comment  »

I just discovered this little surprise working with an app on Rails 3.0.0.rc. I have a class

# app/models/account.rb
require 'Resolv'
class Account
  def verify_email
     mx = Resolv::DNS.open { |dns| dns.getresources(domain, Resolv::DNS::Resource::IN::MX) }
  end
end

This would work fine. The first time only. After that, it though an exception

  NameError: uninitialized constant Account::Resolv

Huh?

Running in the development environment, every time the web server reloads the class, or after the reload! command, it would remove the symbol and not require the library again. Or something like that.

What should you do?

Do not use the require statements in your rails class files. I moved all my require statements at the top of my config/application.rb file and all was better!

Permalink | Leave a comment  »

The default 2.3.5 Rails logger does not print a Timestamp, or other useful background information. Also, you need to know how to change the log path when deploying in production environments when you don't want your logs mixed in with your app. Most Unix systems put the logs in /var/log where they can be monitored for size and content and rotated.

The first parameter is the location of the log. If you do not need a customized logger, initiate the standard ActiveSupport::BufferedLogger instead. The second parameter is the severity level: DEBUG, INFO, WARN, ERROR, or FATAL constants as shown above.

config.logger = MyLogger.new( "/var/logsrails.log", MyLogger::Severity::INFO)

Another problem I had was getting this to work. The config.logger line was not having any effect on my app I started developing on an earlier 2.x release then upgrading to 2.3.5. I managed to get it working by using a fresh config/environment.rb file from a 2.3.5 project.

Permalink | Leave a comment  »

Color Picker for a 256-Color XTerm (xterm, konsole, iTerm). Here is a simple Perl script that prints out this lovely chart. It reminds me of my early years on my first 1982 IBM PC with the Color Card and Monitor. I am using these colors for tweaking vim colorscheme data.

perl -e ‘foreach $i (0..255) {printf(“\e[38;5;$i”.”m%03d\e[0m “,$i); }’

Permalink | Leave a comment  »

(How to change a service user account home directory in Apple OS X 10.6 Snow Leopard)

I’m sure I did something differently than expected. After upgrading to Snow Leopard, my macports weren’t working. After installing the macports for snow leopard, it told me to do a `port upgrade outdated` but that didn’t work because of the variants. I added the —enforce-variants option but had other issues, I don’t recall which now. So I decided to just re-install the ports from scratch, and it worked very well. Until I tried to start postgres.

The postgres user runs the database, and I had been using 8.3 under Leopard, and now using 8.4. Whenever I tried to initdb or do any postgres command, it complained that the home directory for the postgres user, the old 8.3 directory, was not found. Where was this setting, and how can I change it?

My first stop was at ole /etc/passwd. A note at the top reminded me that it is unused except in single-user mode. The service is now provided by “Open Directory”. Oh joy, LDAP. There are no standard *NIX commands to add and modify the users, except on the Desktop GUI (what’s this called?), which wasn’t the answer. (Actually an old apple tutorial to install postgres says to create a log-in user to run your database instead of a system account.)

A friend suggested something called “netinfo”. A little google-fu led me to this post

http://forums.macosxhints.com/showthread.php?p=552033

which showed me some commands! The `dscl` (Directory Services Command Line) was the one I wanted. I read the man page and switched my brain into LDAP mode temporarily. Then I issued this command:

dscl localhost change /Local/Default/Users/postgres NFSHomeDirectory /opt/local/var/db/postgresql83 /opt/local/var/db/postgresql84

and then tried to be the postgres user

sudo -u postgres -i

And no errors! Then I was able to initialize the database and start it

sudo -u postgres /opt/local/lib/postgresql84/bin/initdb -D /opt/local/var/db/postgresql84/defaultdb -E UTF8
sudo -u postgres /opt/local/lib/postgresql84/bin/pg_ctl -D /opt/local/var/db/postgresql84/defaultdb start

I hope you find this post if you are in a similar situation, and it will help you resolve it.

Meow!

Permalink | Leave a comment  »

Watch out! I was innocently loading a YAML file in Ruby created by another system

value: 012

The Ruby YAML library interprets the 012 as octal, as it would in Ruby source code (and as Perl would do). To have this interpreted as string, you must quote the value.

value: “012”

In case you didn’t know, any number starting with a leading zero in C, Ruby, Perl, or Python, is assumed to be octal. “Of course!”, you say, “Everyone thinks and codes in octal”. The only time I use octal is using the unix permissions on the “chmod” command—but now there are modern alternatives to that.

Hexadecimal numbers start with “0x” such as “0x12”. That is not a hard mistake to find as its not a valid integer. But innocent leading-zero numbers can cause syntax errors if the number contains a 9 (non-octal digit) such as “09”, or even floating point numbers  like “012.345”.

You can test this in “irb”…

$ irb
» 07
=> 7
» 012
=> 10
» 0x12
=> 18

» 019
SyntaxError: compile error…

» 012.345

SyntaxError: compile error…

So this is error-prone and annoying in these languages.

As it turns out, this behavior is part of the YAML specification, but I do not like it!  I doubt many people expect that behavior. Data encoding should not take these “short cuts”, as YAML files can be created by non-programmers who have never heard of octal and expect a leading-zero to be harmless.

I’m finding the rule of thumb in YAML is to use quoted values when possible.

Permalink | Leave a comment  »

Install Ubuntu 9.04 Virtual Server With Passenger and Rails Application

There have been a few post and code I referenced to set up my server. See gist

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91

#!/bin/bash
 
# Update the system first
sudo ntpdate ntp.ubuntu.com
sudo apt-get update 
sudo apt-get upgrade

# Ubuntu stuff
sudo apt-get -y install git-core openssh-server openssh-client build-essential wget ntp-simple
# be aware: vim vim-ruby installed x crap :(

# Setup ruby, apache, and necessary dev libs
sudo apt-get -y install ruby rdoc irb libyaml-ruby libzlib-ruby ri libopenssl-ruby ruby1.8-dev  libopenssl-ruby
 
# Ruby Enterprise Edition?
REE=ruby-enterprise_1.8.6-20090421_i386.deb
wget http://rubyforge.org/frs/download.php/55510/$REE
sudo dpkg -i $REE
rm $REE
sudo ln -s /opt/ruby-enterprise/bin/* /usr/local/bin/
 
# rubygems and rails
#wget "http://rubyforge.org/frs/download.php/45905/rubygems-1.3.1.tgz"
#tar -xvzf rubygems-1.3.1.tgz
#rm rubygems-1.3.1.tgz
#cd rubygems-1.3.1
#sudo ruby setup.rb
#cd ..
#rm -r rubygems-1.3.1
#sudo ln -sf /usr/bin/gem1.8 /usr/bin/gem
sudo apt-get -y install rubygems -y
sudo gem update --system
sudo gem sources -a http://gems.github.com
sudo gem install rails --no-rdoc --no-ri
 
# database libs
#sudo apt-get install mysql-server mysql-client
#sudo apt-get install libmysql-ruby libmysqlclient15-dev
#sudo gem install mysql --no-rdoc --no-ri

# postgresql https://help.ubuntu.com/9.04/serverguide/C/postgresql.html
sudo apt-get -y install postgresql libpq-dev
sudo gem install postgres --no-rdoc --no-ri

# Apache2 / passenger
#sudo apt-get -y install apache2 apache2-mpm-prefork apache2-prefork-dev
sudo apt-get -y install apache2-mpm-prefork libapr1-dev apache2-prefork-dev
sudo gem install passenger --no-rdoc --no-ri
# Ruby-MRI: sudo passenger-install-apache2-module
# Ruby-EE:
sudo /opt/ruby-enterprise/bin/passenger-install-apache2-module
sudo echo "LoadModule passenger_module /usr/lib/ruby/gems/1.8/gems/passenger-2.2.1/ext/apache2/mod_passenger.so
PassengerRoot /usr/lib/ruby/gems/1.8/gems/passenger-2.2.1
PassengerRuby /usr/bin/ruby1.8" > /etc/apache2/mods-available/passenger.load
sudo a2enmod passenger
sudo a2enmod ssl
sudo a2enmod rewrite
sudo /etc/init.d/apache2 force-reload
 
# other required gems
sudo apt-get -y install libxml2 libxml2-dev 
sudo gem install rake rspec rspec-rails ruby-debug capistrano libxml-ruby fastercsv --no-rdoc --no-ri
sudo gem install mislav-will_paginate --no-rdoc --no-ri

# setup backup scripts
#wget http://s3.amazonaws.com/ServEdge_pub/s3sync/s3sync.tar.gz
#tar -xzvf s3sync.tar.gz
#chmod 700 -R s3sync
#mkdir -p ~/s3backups
#sudo apt-get install git-core
sudo ssh-keygen

# Install an app
sudo mkdir /var/app
sudo chown allen:allen /var/app
cd /var/app
git clone ~/k23
sudo chown www-data:www-data /var/app/k23/config/environment.rb
sudo echo "<VirtualHost *:80>
    ServerName k23.us
    DocumentRoot /var/app/k23/public
</VirtualHost>" >> /etc/apache2/sites-available/k23.us
sudo a2ensite k23.us
sudo /etc/init.d/apache2 reload

############################################################################
echo "1) add ssh keys to your github account and git clone the rest of your sensitive scripts"
echo "2) edit /etc/postgresql/8.3/main/postgresql.conf to enable listen_addresses = 'localhost,127.0.0.1'"
echo "3) edit /etc/postgresql/8.3/main/pg_hba.conf change md5 to trust in line: host all all 127.0.0.1/32 md5"
echo "4) sudo /etc/init.d/postgresql-8.3 restart"
echo "5) Check passenger version in /etc/apache2/mods-available/passenger.load"

I’ll assume we are started from a post-install state of the Server, with the proper networking, etc. in good working order. Here are the basic command line I entered.

First, let’s get the system up to date

sudo ntpdate ntp.ubuntu.com
sudo apt-get update
sudo apt-get upgrade
sudo ssh-keygen

Now to install the basic needs for the server

sudo apt-get -y install git-core openssh-server openssh-client build-essential \
wget ntp-simple

This sets up the MRI 1.8.7 Ruby interpreter. I think you can skip this step if you want to run the Phusion Ruby Enterprise Edition instead (see next).

sudo apt-get -y install ruby rdoc irb libyaml-ruby libzlib-ruby ri libopenssl-ruby \
ruby1.8-dev libopenssl-ruby

Install Phusion Ruby Enterprise Edition. Check that page for the current version. This installs executables into /opt/ruby-enterprise/bin, so you should either add that to your path or link those command into your path. I did the link, and since Ubuntu ruby installs into /usr/bin, and my default PATH puts /usr/local/bin before that, I simply created links for each command into that directory. This may not be the best way, but this was a proof of concept run.

wget http://rubyforge.org/frs/download.php/55510/ruby-enterprise_1.8.6-20090421_i386.deb
sudo dpkg ruby-enterprise_1.8.6-20090421_i386.deb
rm ruby-enterprise_1.8.6-20090421_i386.deb
sudo ln -s /opt/ruby-enterprise/bin/* /usr/local/bin/

Next, we need to install Ruby Gems, the heart of ruby library package management. You can install via the ubuntu ‘apt-get install rubygems’, but the version you install doesn’t like to upgrade itself. It is best to do it from the rubygems distrubution package. Download the current release from the rubygems download page. Since a lot of gems are released though github these days, add that source to the gems sources.

wget “http://rubyforge.org/frs/download.php/45905/rubygems-1.3.1.tgz”
tar -xvzf rubygems-1.3.1.tgz
rm rubygems-1.3.1.tgz
cd rubygems-1.3.1
sudo ruby setup.rb
cd ..
rm -r rubygems-1.3.1
sudo gem sources -a http://gems.github.com

Install PostgreSQL. Its my database of choice, fast, robust, friendly (…except for replication). Set it up to accept connections from this server. After that, create the user accounts you require of your applications. Consult https://help.ubuntu.com/9.04/serverguide/C/postgresql.html for more details.

sudo apt-get -y install postgresql libpq-dev
sudo vi /etc/postgresql/8.3/main/postgresql.conf
enable this setting:
listen_addresses = ‘localhost,127.0.0.1’”
sudo vi /etc/postgresql/8.3/main/pg_hba.conf
Change md5 to trust in line:
host all all 127.0.0.1/32 trust
sudo /etc/init.d/postgresql-8.3 restart
sudo -u postgres psql template1
ALTER USER postgres with encrypted password ‘your_password’;
create user appuser createdb createuser;

Alternatively, you can run MySQL. It’s also a nice database. You may want to verify this elsewhere to get everything up and running.

sudo apt-get install mysql-server mysql-client
sudo apt-get install libmysql-ruby libmysqlclient15-dev
sudo gem install mysql —no-rdoc —no-ri

Now to install the Apache2 web server and the Phusion Passenger (modrails) module. It instructs you to add some apache configurations, but this location is slightly different. Verify the passenger version numbering if using the “echo” command below.

sudo apt-get -y install apache2-mpm-prefork libapr1-dev apache2-prefork-dev
sudo gem install passenger —no-rdoc —no-ri
sudo /opt/ruby-enterprise/bin/passenger-install-apache2-module
sudo echo “LoadModule passenger_module /usr/lib/ruby/gems/1.8/gems/passenger-2.2.1/ext/apache2/mod_passenger.so
PassengerRoot /usr/lib/ruby/gems/1.8/gems/passenger-2.2.1
PassengerRuby /usr/bin/ruby1.8” > /etc/apache2/mods-available/passenger.load
sudo a2enmod passenger
sudo a2enmod ssl
sudo a2enmod rewrite
sudo /etc/init.d/apache2 force-reload

Now we are on the road to deployment. Let’s install Rails and some other basic gems.

sudo apt-get -y install libxml2 libxml2-dev
sudo gem install rails rake rspec rspec-rails ruby-debug capistrano libxml-ruby \ fastercsv —no-rdoc —no-ri
sudo gem install mislav-will_paginate —no-rdoc —no-ri

Say we decide to host our app out of /var/app and set up up ready to go. I’m not sure about the chown command below, but saw it elsewhere and it didn’t hurt.

sudo mkdir /var/app
sudo chown allen:allen /var/app
cd /var/app
git clone git://github.com/account/appname.git
sudo chown www-data:www-data /var/app/k23/config/environment.rb

Chances are, you will deploy with a yummy capistrano or vlad recipe. That out out of the scope at this time, so we’ll create the database and test to make sure te connection is right. Play with your app and fix settings before going on..

cd /var/app/appname
cp config/database.yml.example config/database.yml
rake db:create RAILS_ENV=production
rake db:migrate RAILS_ENV=production
script/console production

Now we install the app through Passenger.

sudo echo “<VirtualHost *:80>
ServerName appname.com
DocumentRoot /var/app/appname/public
</VirtualHost>” » /etc/apache2/sites-available/appname.com
sudo a2ensite appname.com
sudo /etc/init.d/apache2 reload

If we were successful, we should be able to load that site (assuming DNS is pointing here already).

Load http://appname.com on your workstation web browser.

I hope this helped you as much as it did me. I apologize for errors that crept into this script while converting it to a blog entry.

Permalink | Leave a comment  »

I just had this crazy idea. If you already read the title, then you already know! ;-)

Twitter is crazy cool. You can tweet as much as you want (I think) and can create gigabytes of useless personal trivia… or you can store somthing kinda cool in it… DATA! As a result, you can get a free Small-Document-Oriented database you can access from the cloud.

Sure, this is nuts, but its fun to consider, right?

First, each “table” is a twitter “account”, call it “app.table” or whatever. Each “tweet” to the table is a record. Ok, twitter posts can only be 140 characters—a limit I otherwise appreciate. We can store the record as any parsable key/value store.

Perhaps we could use JSON:{“ircEvent”: “PRIVMSG”, “method”: “newURI”, “regex”: “^http://.*”}

Or a Ruby Hash: {:key=>”value”, …}

Or Python Dictionary: {‘jack’: 4098, ‘sape’: 4139}

Insert the data by tweeting the record to the table. Now you are storing it. You can’t update it, only supercede a record.  How would we access it? Feeds, api, searches, and mashups.My twitter-fu on this isn’t so strong as I never had a need for it before.

But still… could be interesting, right? And you could even SHARE data by posting it to public databases like this. For sanity, you would validate and post this through a gateway service of your own devising.

I wonder now much this could be like CouchdB, SimpleDB, or the Google App Engine Datastore this is?

Perhaps this could be work exploring. Maybe I could even write ActiveRecord (Ruby on Rails) bindings to emulate this. Someday.

I’ll keep you posted…

Permalink | Leave a comment  »

In PostgreSQL (8.2), we can create a view to access a join of two tables.

create view join_view as SELECT p.id, p.group, c.parent_id, c.status FROM child c JOIN parent p ON c.parent_id = p.id ;

Now we can do

select * from join_view where group = 1;

However, if we try to update the view, we can not.

update join_view set status=4 where group=1;
ERROR:  cannot update a view
HINT:  You need an unconditional ON UPDATE DO INSTEAD rule.

So we need to create this rule to execute this trigger-like command

create rule join_view_update as on update to join_view do instead (update child set status=NEW.status where id=NEW.id and parent_id=NEW.parent_id );

Now we can execute the update!

update join_view set status=4 where group=1;
UPDATE 1
select * from join_view where group = 1;

And see the results!

Note: I changed internal conditions and naming, and hope I have no inadvertently mis-typed something. Apologies in advance if this had a simple syntax error.

Permalink | Leave a comment  »

The join executes a large query and returned potentially duplicate records for a one-to-many association. After 2.1, the query is broken down and eager-loaded using an additional query per association, passing the set of id‘s to load, and avoiding the duplicate rows.

The new method eliminates duplicates, but can incur more database overhead. If you are loading a very large set of records (more than a “page”), you may need to “force” the join or use find_by_sql instead.

When you specify a “table.column” syntax within a

:conditions=>["child.name=?", name]

or

:order=>'child.name'

then ActiveRecord will build the older, full query with the join because you are referencing columns from another table to build. This will cause the duplicate rows to reappear.

Whenever you reference a column from another table in a condition or order clause, ALWAYS use the table name to prefix the column, even if it not ambiguous among the tables involved. Otherwise the query will not be executed as a join and you will receive an SQL error referencing the “missing” column.

You can “force” a join by adding a reference to the other tables in your :conditions or :options parameters, even if the test or sort is irrelevant.

Permalink | Leave a comment  »

The Merb application framework is getting close to a 1.0 release. Merb takes many of the ideas of Ruby on Rails and does them one better. RoR was designed for developers and applications, but has not been the best to run in production. Merb was written from the ground up with performance and scalability. But as a result, its like travelling abroad in that things are similar but not exactly the same as you are used to developing with RoR.

The platform of choice of most rubyists these days is OS X, with its fine unixy-goodness and groovy Textmate editor. Ubuntu Linux provides most of these features for a fraction of the price! Also, since these applications are generally deployed on the Linux OS, it makes sense to know how to get things running on that platform as well. Let’s roll up our sleeves and get to work.

I used Justin’s post as a starting point on what to do. Open up a terminal window and start.

Get the Git

First, we install the git version control system which merb uses. Note the current Debian git package has a minor bug requiring the /etc/mailname file. Create that file with your hostname in it for now.

sudo apt-get install git-corevi /etc/mailname

Gems

I assume you have been using Ruby before and have a modern Ruby and RubyGems installed. If not, go do that now. I’ll wait.

Justin recommends having these gems. All seem a good idea, but mongrel erubus and rspec seem more necessary if you are trying to cut down. Merb requires rack, and a ORM like Rails’ ActiveRecord, Sequel, or Datamapper, which it looks like it prefers. Merb is built upon the Rack web server abstraction library.

Like Merb, Datamapper is thread-safe to allow multiple requests being handled. However, it is not yet fully developed like ActiveRecord and may not be as production-ready. If you are porting an application from Rails, you may want to continue using ActiveRecord.

sudo gem install mongrel json json_pure erubis mime-types rspec hpricot mocha rubigen haml markaby mailfactory Ruby2Ruby -ysudo gem install racksudo gem install datamapper -y

Install Merb

Merb is available in gem form, but they may not be as up to date as we would like. So let’s git them from the git repository. If you are having problems, try the http:// protocol instead of the git:// protocol. Merb in divided up into merb-core, merb-more, and merb-plugins in order to let you craft the server with just the parts you need. We need to go into each repository and “install” it in your gems.

mkdir trymerbcd trymerbgit clone git://github.com/wycats/merb-core.gitgit clone git://github.com/wycats/merb-more.gitgit clone git://github.com/wycats/merb-plugins.gitcd merb-coresudo rake installcd ../merb-moresudo rake install# has link_to, etccd ../merge-assets sudo rake installcd ../../merb-plugins/merb_helpers/sudo rake installcd ../merb_activerecord/sudo rake installcd ../merb_datamapper/sudo rake installcd ../merb_rspec/sudo rake install

Whew! That’s going to get old fast. The upcoming gems will be better.

Merb with PostgreSQL

PostgreSQL is my RDBMS of choice. MySQL would work similarly. Ubuntu needs your PostgreSQL header files, so we load them first.

sudo apt-get install libpq-devsudo gem install do_postgres

Now let’s go roll an app!

The MyBlog Merb Application

To show off our merby goodness, we will build the “hello world” of ruby web applications, the blog.

cd trymerbmerb-gen myblogcd myblogcreatedb myblog_development

The createdb command is for postgresql, please substitute the create database for your db of choice.

### Uncomment for DataMapper ORM  use_orm :datamapper  use_test :rspec  ### Add your other dependencies here  dependencies "merb_helpers", "merb-assets"

---  # This is a sample database file for the DataMapper ORM  :development: &defaults    :adapter: postgresql    :database: myblog_development    :username: the_user    :password: secrets    :host: localhost  :test:    <<: *defaults    :database: myblog_test  :production:    <<: *defaults    :database: myblog_production

Merbful Authentication

Optionally, we add in authentication, the “Restful Authentication” plugin port to merb. Read me about here

cd trymerbgit clone git://github.com/hassox/restful-authentication.gitcd restful-authenticationgit checkout -b merbful_authentication origin/merbful_authenticationsudo rake installcd trymerb/myblogmerb-gen authenticated user_model session_controller

Our post resource

merb-gen resource post title:string content:text created_at:datetimerake dm:db:automigrate

# RESTful routes  r.resources :posts # Change this for your home page to be available at /  r.match('/').to(:controller => 'posts', :action =>'index')

Starting Merb

cd trymerb/myblogmerb

Note that this is where some of the differences between Rails and Merb shows up. Merb has its own routing using the url() helper. Read more about it here .

url(:posts)       #=> "/posts" url(:new_post) #=> "/posts/new" @post = Post[1]url(:post, @post) #=> "/posts/1"

<h1>My Blog</h1><% @posts.each do |n| %>  <h2><%= n.title %></h2>  <div2><%= n.content %></div><% end %><div>  <%= link_to "Write new", url(:new_post) %></div>

Edit the app/views/posts/new.html.erb and add in the compose form. Feel free to embellish your HTML. I added the hidden field to remember how to do it if I need to figure it out again.

As you see, the field control helpers are different, but self-explanatory. They were actually the hardest part to figure out since they needed to be installed from the proper repository and enabled in the init.rb file. The source code also shows more of what you need to do.

<h1>Write a new post</h1><%= error_messages_for :post %><% form_for(@post, :action=>url(:posts), :method=>"post") do %>  <%= hidden_field :name=>:wave, :value=>'hello' %>  <%= text_control :title, :label=>"Title", :size=>50,     :maxlength=>250, :style=>"font-weight:bold;" %>  <br /><%= text_area_control :content, :label=>"Rant", :rows=>15, :cols=>50 %>  <br /><%= submit_button  "Save" %><% end %>

That should do it! Enjoy your new merb blog app.

More Things with merb

The Merb console

merb -iirb(main):001:0> merb.show_routes

Debugger

merb -D

Merb on Thin

sudo gem install thinmerb -a thin

Disclaimer

This was my first experiment using Merb on Ubuntu, and I am thrilled that it works so well already. This is not quite a recipe for starting from scratch. Some instructions may be convoluted, unnnecessary, copied here incorrectly, or just plain wrong.

Merb is the exciting new direction for developing ruby applications. Ezra is addressing the shortcomings we have in developing and deploying robust applications in ruby. The Rubinius and Merb projects are doing a lot to make this happen.

Permalink | Leave a comment  »

I am giving a talk this evening at the Philly On Rails meeting on an introduction to email and sending and receiving email with a Ruby on Rails application.

I am referencing the slides and a sample application in case anyone needs to reference them!

Permalink | Leave a comment  »

What is email exactly? It is quite different from the view we see in our MUA, (Thunderbird, gmail, mutt, pine, etc.). Let’s review the basic of email mechanics, shall we?

Email consists on an Envelope and the Message Data. The envelope is

• Return Path: The email address to return the message if undeliverable
• Recipients: The email address(es) of the recipients of the message

The Message Data is the meat of the mail,

• RFC822 Headers such as Subject, From, To, Date, etc. Headers can span multiple lines by starting the subsequent line with a white-space. A blank line signals the end of the header and the start of the body follows. Non-standard header names begin with the “X-” prefix, used to denote experimental features. These usually have meaning only to the software that placed them there.
• Body, text, html, etc. Using the MIME standard, the body can be made to include alternate versions of the message based on markup, and also attach files to the message.
• File Attachments. Usually, binary files are encoded in Base64 no non-printable characters. This increases the file size by about 33% when encoded in the message.

Note that the From and To headers in the headers are really informational. It is the envelope recipient list that determines who it was from.

Also, since the From header can be virtually anything, it is easy to spoof the sender or even the full message. Even the recipient return path can be bogus. Therefore, email is not secure as such (except for PGP/GPG signings), and should not be trusted outright.

Since each email server that passes the message along prepends a Received header to the top of the message, you can generally trace its path to see if it came from a source server that is a proper MTA for the sender. Of course, you can only trust the received headers from servers as far back as you trust; they can also be tampered with.

To: you@example.comFrom: me@example.comDate: 20 Jul 2007 09:21:51 -0700Message-ID: <521.1090340511@example.com>Subject: Simple Email MessageLorem ipsum dolor sit amet, consectetuer adipiscing elit. Maecenasultrices sem sed urna accumsan cursus.

Content-Type and MIME Types

When you want to create more complex email messages, say with alternative content or attachments, you need to construct your message using MIME containers and body parts.

• Content-Type: Use this header in your email message to identify the MIME type of your content
• text/plain is the default MIME type of email. This is viewable by all mail clients.
• text/html denotes an HTML formatted body or part. This is only viewable in GUI-based clients that support HTML.
• multipart/alternative is a MIME container that holds the text, HTML or other versions of the main message. Only one of these (the best one it can show) is viewable by the user’s mail client.
• multipart/alternative is a MIME container used for attaching files to the message body. The first part is the body part (which can also be another container), and the rest are attachments.
• multipart/related is a MIME container that wraps included graphics referenced from an HTML body. These graphics are shown “in place”, such as a logo in the letterhead, instead of being seen as attachments.
• The boundary parameter of the Content-Type header is used to provide a unique identifier to define the start and end of the body parts within a MIME container. Lines starting with two hyphens followed by the boundary value is the split point in the message. The final boundary line is the two hyphens, boundary value, and two more hyphens.
• image/png is a PNG graphic file, also a image/gif or image/jpeg could be used.
• application/pdf is a pdf attachment, which could be any application-defined file
• Stir to combine…

Each MIME body part (attachment, container, or message version) itself has a small MIME header set to indicate its content-type, encoding, and other information.

Here is an example message that is composed of a text and HTML body alternatives, with an image attachment called out from the HTML version, plus another image as a regular attachment. The structure of the MIME parts is

multipart/mixed (Holds the body part plus attachments)    multipart/alternative (groups the different version of the message body)        text/plain        multipart/related (groups the HTML part with images it references)            text/html            image/jpeg    image/png (attachment)

Here is how this looks in the email message.

To: you@example.comFrom: me@example.comDate: 20 Jul 2007 09:21:51 -0700Message-ID: <521.1090340511@example.com>Subject: Complex Email MessageMIME-Version: 1.0Content-Type: multipart/mixed;        boundary="mm001" This is a multi-part message in MIME format.--mm001Content-Type: multipart-alternative; boundary=mb001--ma001Content-Type: text/plainThis is the plain text body--ma001Content-Type: multipart-related; boundary="mr001" --mr001Content-Type: text/htmlContent-Transfer-Encoding: quoted-printableThis is the <em>HTML</em> body<IMG=20SRC=3D"No%20AttachName"=20alt=3D"Picture=20(Metafile)">--mr001Content-Type: image/jpeg; name="logo.jpg" Content-Transfer-Encoding: base64Content-Description: Picture (Metafile)Content-Location: No%20AttachNameQk2ewgIAAAAAADYAAAAoAAAAJAIAAG4AAAABABgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA////....--mr001-- --ma001----mm001Content-Type: image/png;     name="elroy-jetson.png" Content-Transfer-Encoding: base64Content-Disposition: attachment;    filename="elroy-jetson.png" R0lGODlhMgAvAPcAAAAAAJQAAPfOjP//////////////////////////////////////////////.....--mm001--

Content-Transfer-Encoding

Note the use of quoted-printable in the above HTML segment. Quoted-printable encoding escapes special characters with an equal symbol (=) followed by the 2-character hexadecimal ASCII representation of the character value. For example, any equal symbols in the body are replaced with ”=3D”, where 3D is the hexadecimal representation of the equal symbol in the ASCII collating sequence.

Web browsers do something similar when sending special characters in the URL, but using a percent (%) symbol as the escape symbol.

Quoted-printable also wraps text so lines do not become too long. An equal symbol at the end of the line (=\n) indicates the line is wrapped. Email standards define the maximum length of a line to be 77 (?) characters, but since this is not a hard limit, most email software is flexible about this limit.

Binary files are usually encoded in Base64. The Base64 method maps every 6 bits to a printable character. Ruby has a Base64 helper class

require "base64" enc   = Base64.encode64('Send reinforcements') # -> "U2VuZCByZWluZm9yY2VtZW50cw==\n" plain = Base64.decode64(enc)  # -> "Send reinforcements"

SMTP: How Email is Tranferred

Email is delivered via SMTP, Simple Mail Transport Protocol. This is a simple state-machine which accepts email through a “command line” interface, usually over port 25. Open a telnet connection to any MX(mail exchanger) host on port 25 to try your hand at delivering a mail manually.

• Return Path (MAIL FROM)
• Recipients: (RCPT TO)
• Message: (DATA), which is ended by a single period on the last line.

220 example.com mailfront ESMTPMAIL FROM: <me@yahoo.com>250 2.1.0 Sender accepted.RCPT TO: <you@example.org>250 OKDATA354 End your message with a period on a line by itself.Subject: Hello thereFrom: Me <me@yahoo.com>I just love SMTP!.250 2.6.0 Accepted message qp 16590 bytes 226QUIT221 2.0.0 Good bye.

Permalink | Leave a comment  »