girders blog

Color Picker for a 256-Color XTerm (xterm, konsole, iTerm). Here...

Wed, 16 Sep 2009 18:17:09 -0400

Color Picker for a 256-Color XTerm (xterm, konsole, iTerm). Here is a simple Perl script that prints out this lovely chart. It reminds me of my early years on my first 1982 IBM PC with the Color Card and Monitor. I am using these colors for tweaking vim colorscheme data.

perl -e ‘foreach $i (0..255) {printf(“\e[38;5;$i”.”m%03d\e[0m “,$i); }’

Changing postgres user info on OS X 10.6

Sat, 12 Sep 2009 16:11:00 -0400

(How to change a service user account home directory in Apple OS X 10.6 Snow Leopard)

I’m sure I did something differently than expected. After upgrading to Snow Leopard, my macports weren’t working. After installing the macports for snow leopard, it told me to do a `port upgrade outdated` but that didn’t work because of the variants. I added the —enforce-variants option but had other issues, I don’t recall which now. So I decided to just re-install the ports from scratch, and it worked very well. Until I tried to start postgres.

The postgres user runs the database, and I had been using 8.3 under Leopard, and now using 8.4. Whenever I tried to initdb or do any postgres command, it complained that the home directory for the postgres user, the old 8.3 directory, was not found. Where was this setting, and how can I change it?

My first stop was at ole /etc/passwd. A note at the top reminded me that it is unused except in single-user mode. The service is now provided by “Open Directory”. Oh joy, LDAP. There are no standard *NIX commands to add and modify the users, except on the Desktop GUI (what’s this called?), which wasn’t the answer. (Actually an old apple tutorial to install postgres says to create a log-in user to run your database instead of a system account.)

A friend suggested something called “netinfo”. A little google-fu led me to this post

http://forums.macosxhints.com/showthread.php?p=552033

which showed me some commands! The `dscl` (Directory Services Command Line) was the one I wanted. I read the man page and switched my brain into LDAP mode temporarily. Then I issued this command:

dscl localhost change /Local/Default/Users/postgres NFSHomeDirectory /opt/local/var/db/postgresql83 /opt/local/var/db/postgresql84

and then tried to be the postgres user

sudo -u postgres -i

And no errors! Then I was able to initialize the database and start it

sudo -u postgres /opt/local/lib/postgresql84/bin/initdb -D /opt/local/var/db/postgresql84/defaultdb -E UTF8
sudo -u postgres /opt/local/lib/postgresql84/bin/pg_ctl -D /opt/local/var/db/postgresql84/defaultdb start

I hope you find this post if you are in a similar situation, and it will help you resolve it.

Meow!

Ruby YAML loads leading-zero numbers as Octal

Thu, 13 Aug 2009 17:29:59 -0400

Watch out! I was innocently loading a YAML file in Ruby created by another system

value: 012

The Ruby YAML library interprets the 012 as octal, as it would in Ruby source code (and as Perl would do). To have this interpreted as string, you must quote the value.

value: “012”

In case you didn’t know, any number starting with a leading zero in C, Ruby, Perl, or Python, is assumed to be octal. “Of course!”, you say, “Everyone thinks and codes in octal”. The only time I use octal is using the unix permissions on the “chmod” command—but now there are modern alternatives to that.

Hexadecimal numbers start with “0x” such as “0x12”. That is not a hard mistake to find as its not a valid integer. But innocent leading-zero numbers can cause syntax errors if the number contains a 9 (non-octal digit) such as “09”, or even floating point numbers like “012.345”.

You can test this in “irb”…

$ irb
» 07
=> 7
» 012
=> 10
» 0x12
=> 18

» 019
SyntaxError: compile error…

» 012.345

SyntaxError: compile error…

So this is error-prone and annoying in these languages.

As it turns out, this behavior is part of the YAML specification, but I do not like it! I doubt many people expect that behavior. Data encoding should not take these “short cuts”, as YAML files can be created by non-programmers who have never heard of octal and expect a leading-zero to be harmless.

I’m finding the rule of thumb in YAML is to use quoted values when possible.

Install Ubuntu 9.04 Virtual Server With Passenger and Rails Application

Wed, 13 May 2009 15:39:00 -0400

Install Ubuntu 9.04 Virtual Server With Passenger and Rails Application

There have been a few post and code I referenced to set up my server. See gist http://gist.github.com/111244 for a raw install script, which you must adapt to your needs.

I’ll assume we are started from a post-install state of the Server, with the proper networking, etc. in good working order. Here are the basic command line I entered.

First, let’s get the system up to date

sudo ntpdate ntp.ubuntu.com
sudo apt-get update
sudo apt-get upgrade
sudo ssh-keygen

Now to install the basic needs for the server

sudo apt-get -y install git-core openssh-server openssh-client build-essential \
wget ntp-simple

This sets up the MRI 1.8.7 Ruby interpreter. I think you can skip this step if you want to run the Phusion Ruby Enterprise Edition instead (see next).

sudo apt-get -y install ruby rdoc irb libyaml-ruby libzlib-ruby ri libopenssl-ruby \
ruby1.8-dev libopenssl-ruby

Install Phusion Ruby Enterprise Edition. Check that page for the current version. This installs executables into /opt/ruby-enterprise/bin, so you should either add that to your path or link those command into your path. I did the link, and since Ubuntu ruby installs into /usr/bin, and my default PATH puts /usr/local/bin before that, I simply created links for each command into that directory. This may not be the best way, but this was a proof of concept run.

wget http://rubyforge.org/frs/download.php/55510/ruby-enterprise_1.8.6-20090421_i386.deb
sudo dpkg ruby-enterprise_1.8.6-20090421_i386.deb
rm ruby-enterprise_1.8.6-20090421_i386.deb
sudo ln -s /opt/ruby-enterprise/bin/* /usr/local/bin/

Next, we need to install Ruby Gems, the heart of ruby library package management. You can install via the ubuntu ‘apt-get install rubygems’, but the version you install doesn’t like to upgrade itself. It is best to do it from the rubygems distrubution package. Download the current release from the rubygems download page. Since a lot of gems are released though github these days, add that source to the gems sources.

wget “http://rubyforge.org/frs/download.php/45905/rubygems-1.3.1.tgz”
tar -xvzf rubygems-1.3.1.tgz
rm rubygems-1.3.1.tgz
cd rubygems-1.3.1
sudo ruby setup.rb
cd ..
rm -r rubygems-1.3.1
sudo gem sources -a http://gems.github.com

Install PostgreSQL. Its my database of choice, fast, robust, friendly (…except for replication). Set it up to accept connections from this server. After that, create the user accounts you require of your applications. Consult https://help.ubuntu.com/9.04/serverguide/C/postgresql.html for more details.

sudo apt-get -y install postgresql libpq-dev
sudo vi /etc/postgresql/8.3/main/postgresql.conf
enable this setting:
listen_addresses = ‘localhost,127.0.0.1’”
sudo vi /etc/postgresql/8.3/main/pg_hba.conf
Change md5 to trust in line:
host all all 127.0.0.1/32 trust
sudo /etc/init.d/postgresql-8.3 restart
sudo -u postgres psql template1
ALTER USER postgres with encrypted password ‘your_password’;
create user appuser createdb createuser;

Alternatively, you can run MySQL. It’s also a nice database. You may want to verify this elsewhere to get everything up and running.

sudo apt-get install mysql-server mysql-client
sudo apt-get install libmysql-ruby libmysqlclient15-dev
sudo gem install mysql —no-rdoc —no-ri

Now to install the Apache2 web server and the Phusion Passenger (modrails) module. It instructs you to add some apache configurations, but this location is slightly different. Verify the passenger version numbering if using the “echo” command below.

sudo apt-get -y install apache2-mpm-prefork libapr1-dev apache2-prefork-dev
sudo gem install passenger —no-rdoc —no-ri
sudo /opt/ruby-enterprise/bin/passenger-install-apache2-module
sudo echo “LoadModule passenger_module /usr/lib/ruby/gems/1.8/gems/passenger-2.2.1/ext/apache2/mod_passenger.so
PassengerRoot /usr/lib/ruby/gems/1.8/gems/passenger-2.2.1
PassengerRuby /usr/bin/ruby1.8” > /etc/apache2/mods-available/passenger.load
sudo a2enmod passenger
sudo a2enmod ssl
sudo a2enmod rewrite
sudo /etc/init.d/apache2 force-reload

Now we are on the road to deployment. Let’s install Rails and some other basic gems.

sudo apt-get -y install libxml2 libxml2-dev
sudo gem install rails rake rspec rspec-rails ruby-debug capistrano libxml-ruby \ fastercsv —no-rdoc —no-ri
sudo gem install mislav-will_paginate —no-rdoc —no-ri

Say we decide to host our app out of /var/app and set up up ready to go. I’m not sure about the chown command below, but saw it elsewhere and it didn’t hurt.

sudo mkdir /var/app
sudo chown allen:allen /var/app
cd /var/app
git clone git://github.com/account/appname.git
sudo chown www-data:www-data /var/app/k23/config/environment.rb

Chances are, you will deploy with a yummy capistrano or vlad recipe. That out out of the scope at this time, so we’ll create the database and test to make sure te connection is right. Play with your app and fix settings before going on..

cd /var/app/appname
cp config/database.yml.example config/database.yml
rake db:create RAILS_ENV=production
rake db:migrate RAILS_ENV=production
script/console production

Now we install the app through Passenger.

sudo echo “
ServerName appname.com
DocumentRoot /var/app/appname/public
” » /etc/apache2/sites-available/appname.com
sudo a2ensite appname.com
sudo /etc/init.d/apache2 reload

If we were successful, we should be able to load that site (assuming DNS is pointing here already).

Load http://appname.com on your workstation web browser.

I hope this helped you as much as it did me. I apologize for errors that crept into this script while converting it to a blog entry.

twitterdb - Using twitter as a database

Sat, 21 Mar 2009 21:28:13 -0400

I just had this crazy idea. If you already read the title, then you already know! ;-)

Twitter is crazy cool. You can tweet as much as you want (I think) and can create gigabytes of useless personal trivia… or you can store somthing kinda cool in it… DATA! As a result, you can get a free Small-Document-Oriented database you can access from the cloud.

Sure, this is nuts, but its fun to consider, right?

First, each “table” is a twitter “account”, call it “app.table” or whatever. Each “tweet” to the table is a record. Ok, twitter posts can only be 140 characters—a limit I otherwise appreciate. We can store the record as any parsable key/value store.

Perhaps we could use JSON:{“ircEvent”: “PRIVMSG”, “method”: “newURI”, “regex”: “^http://.*”}

Or a Ruby Hash: {:key=>”value”, …}

Or Python Dictionary: {‘jack’: 4098, ‘sape’: 4139}

Insert the data by tweeting the record to the table. Now you are storing it. You can’t update it, only supercede a record. How would we access it? Feeds, api, searches, and mashups.My twitter-fu on this isn’t so strong as I never had a need for it before.

But still… could be interesting, right? And you could even SHARE data by posting it to public databases like this. For sanity, you would validate and post this through a gateway service of your own devising.

I wonder now much this could be like CouchdB, SimpleDB, or the Google App Engine Datastore this is?

Perhaps this could be work exploring. Maybe I could even write ActiveRecord (Ruby on Rails) bindings to emulate this. Someday.

I’ll keep you posted…

Updating a PostgreSQL view

Tue, 06 Jan 2009 10:56:19 -0500

In PostgreSQL (8.2), we can create a view to access a join of two tables.

create view join_view as SELECT p.id, p.group, c.parent_id, c.status FROM child c JOIN parent p ON c.parent_id = p.id ;

Now we can do

select * from join_view where group = 1;

However, if we try to update the view, we can not.

update join_view set status=4 where group=1;
ERROR: cannot update a view
HINT: You need an unconditional ON UPDATE DO INSTEAD rule.

So we need to create this rule to execute this trigger-like command

create rule join_view_update as on update to join_view do instead (update child set status=NEW.status where id=NEW.id and parent_id=NEW.parent_id );

Now we can execute the update!

update join_view set status=4 where group=1;
UPDATE 1
select * from join_view where group = 1;

And see the results!

Note: I changed internal conditions and naming, and hope I have no inadvertently mis-typed something. Apologies in advance if this had a simple syntax error.

ActiveRecord#find using :include=>association no longer necessarily joins the association

Thu, 30 Oct 2008 11:01:33 -0400

Before Rails 2.1, adding an :include=>[:association] in your find method caused ActiveRecord to generate SQL using a join. Since 2.1, it MAY NOT execute as a join.

The join executes a large query and returned potentially duplicate records for a one-to-many association. After 2.1, the query is broken down and eager-loaded using an additional query per association, passing the set of id‘s to load, and avoiding the duplicate rows.

The new method eliminates duplicates, but can incur more database overhead. If you are loading a very large set of records (more than a “page”), you may need to “force” the join or use find_by_sql instead.

When you specify a “table.column” syntax within a

 :conditions=>["child.name=?", name]

 :order=>'child.name'

then ActiveRecord will build the older, full query with the join because you are referencing columns from another table to build. This will cause the duplicate rows to reappear.

Whenever you reference a column from another table in a condition or order clause, ALWAYS use the table name to prefix the column, even if it not ambiguous among the tables involved. Otherwise the query will not be executed as a join and you will receive an SQL error referencing the “missing” column.

You can “force” a join by adding a reference to the other tables in your :conditions or :options parameters, even if the test or sort is irrelevant.

Merb 0.9 on Ubuntu Linux 7.10

Sun, 02 Mar 2008 00:00:00 -0500

The Merb application framework is getting close to a 1.0 release. Merb takes many of the ideas of Ruby on Rails and does them one better. RoR was designed for developers and applications, but has not been the best to run in production. Merb was written from the ground up with performance and scalability. But as a result, its like travelling abroad in that things are similar but not exactly the same as you are used to developing with RoR.

The platform of choice of most rubyists these days is OS X, with its fine unixy-goodness and groovy Textmate editor. Ubuntu Linux provides most of these features for a fraction of the price! Also, since these applications are generally deployed on the Linux OS, it makes sense to know how to get things running on that platform as well. Let’s roll up our sleeves and get to work.

I used Justin’s post as a starting point on what to do. Open up a terminal window and start.

Get the Git

First, we install the git version control system which merb uses. Note the current Debian git package has a minor bug requiring the /etc/mailname file. Create that file with your hostname in it for now.

sudo apt-get install git-core
vi /etc/mailname

Gems

I assume you have been using Ruby before and have a modern Ruby and RubyGems installed. If not, go do that now. I’ll wait.

Justin recommends having these gems. All seem a good idea, but mongrel erubus and rspec seem more necessary if you are trying to cut down. Merb requires rack, and a ORM like Rails’ ActiveRecord, Sequel, or Datamapper, which it looks like it prefers. Merb is built upon the Rack web server abstraction library.

Like Merb, Datamapper is thread-safe to allow multiple requests being handled. However, it is not yet fully developed like ActiveRecord and may not be as production-ready. If you are porting an application from Rails, you may want to continue using ActiveRecord.

sudo gem install mongrel json json_pure erubis mime-types rspec hpricot mocha rubigen haml markaby mailfactory Ruby2Ruby -y
sudo gem install rack
sudo gem install datamapper -y

Install Merb

Merb is available in gem form, but they may not be as up to date as we would like. So let’s git them from the git repository. If you are having problems, try the http:// protocol instead of the git:// protocol. Merb in divided up into merb-core, merb-more, and merb-plugins in order to let you craft the server with just the parts you need. We need to go into each repository and “install” it in your gems.

mkdir trymerb
cd trymerb

git clone git://github.com/wycats/merb-core.git
git clone git://github.com/wycats/merb-more.git
git clone git://github.com/wycats/merb-plugins.git

cd merb-core
sudo rake install

cd ../merb-more
sudo rake install
# has link_to, etc
cd ../merge-assets 
sudo rake install

cd ../../merb-plugins/merb_helpers/
sudo rake install
cd ../merb_activerecord/
sudo rake install
cd ../merb_datamapper/
sudo rake install
cd ../merb_rspec/
sudo rake install

Whew! That’s going to get old fast. The upcoming gems will be better.

Merb with PostgreSQL

PostgreSQL is my RDBMS of choice. MySQL would work similarly. Ubuntu needs your PostgreSQL header files, so we load them first.

sudo apt-get install libpq-dev
sudo gem install do_postgres

Now let’s go roll an app!

The MyBlog Merb Application

To show off our merby goodness, we will build the “hello world” of ruby web applications, the blog.

cd trymerb
merb-gen myblog
cd myblog
createdb myblog_development

The createdb command is for postgresql, please substitute the create database for your db of choice.

Edit config/init.rb file and enable the following configurations.

  ### Uncomment for DataMapper ORM
  use_orm :datamapper

  use_test :rspec

  ### Add your other dependencies here
  dependencies "merb_helpers", "merb-assets"

Now create your /config/database.yml

  ---
  # This is a sample database file for the DataMapper ORM
  :development: &defaults
    :adapter: postgresql
    :database: myblog_development
    :username: the_user
    :password: secrets
    :host: localhost

  :test:
    <<: *defaults
    :database: myblog_test

  :production:
    <<: *defaults
    :database: myblog_production

Merbful Authentication

Optionally, we add in authentication, the “Restful Authentication” plugin port to merb. Read me about here

cd trymerb
git clone git://github.com/hassox/restful-authentication.git
cd restful-authentication
git checkout -b merbful_authentication origin/merbful_authentication
sudo rake install

cd trymerb/myblog
merb-gen authenticated user_model session_controller

Our post resource

Restful merb resource that is. This creates the controller, model, and view. If you are using ActiveRecord, I would also expect a migration. For Datamapper, the columns are defined in the model. Use rake to create the tables from the model. (For Postgres, use the datetime type instead of the timestamp type you would use in Rails.)

merb-gen resource post title:string content:text created_at:datetime
rake dm:db:automigrate

Edit config/router.rb to enable these actions:

  # RESTful routes
  r.resources :posts

 # Change this for your home page to be available at /
  r.match('/').to(:controller => 'posts', :action =>'index')

Starting Merb

Open a new terminal window and start merb from your app directory

cd trymerb/myblog
merb

Now point your brower to http://localhost:4000/ and you should be able to see it. The resource generator does not yet build the ERB files out like Rails’ scaffolding. So we have to write that ourselves.

Note that this is where some of the differences between Rails and Merb shows up. Merb has its own routing using the url() helper. Read more about it here .

url(:posts)       #=> "/posts" 
url(:new_post) #=> "/posts/new" 

@post = Post[1]
url(:post, @post) #=> "/posts/1"

Edit the app/views/posts/index.html.erb and add in a simple post listing.

My Blog


<% @posts.each do |n| %>
  <%= n.title %>

  <%= n.content %>
<% end %>


  <%= link_to "Write new", url(:new_post) %>

Edit the app/views/posts/new.html.erb and add in the compose form. Feel free to embellish your HTML. I added the hidden field to remember how to do it if I need to figure it out again.

As you see, the field control helpers are different, but self-explanatory. They were actually the hardest part to figure out since they needed to be installed from the proper repository and enabled in the init.rb file. The source code also shows more of what you need to do.

Write a new post

<%= error_messages_for :post %>

<% form_for(@post, :action=>url(:posts), :method=>"post") do %>
  <%= hidden_field :name=>:wave, :value=>'hello' %>
  <%= text_control :title, :label=>"Title", :size=>50, 
    :maxlength=>250, :style=>"font-weight:bold;" %>
  
<%= text_area_control :content, :label=>"Rant", :rows=>15, :cols=>50 %>
  
<%= submit_button  "Save" %>
<% end %>

That should do it! Enjoy your new merb blog app.

More Things with merb

The Merb console

The merb object has some things worth exploring, like the app object in Rails.

merb -i
irb(main):001:0> merb.show_routes

Debugger

Like with Rails 2.0, call the “debugger” function where you want a breakpoint. and start merb like this:

merb -D

Merb on Thin

You can run the Thin web server with Merb.

sudo gem install thin
merb -a thin

Disclaimer

This was my first experiment using Merb on Ubuntu, and I am thrilled that it works so well already. This is not quite a recipe for starting from scratch. Some instructions may be convoluted, unnnecessary, copied here incorrectly, or just plain wrong.

Merb is the exciting new direction for developing ruby applications. Ezra is addressing the shortcomings we have in developing and deploying robust applications in ruby. The Rubinius and Merb projects are doing a lot to make this happen.

Email on Rails

Mon, 06 Aug 2007 00:00:00 -0400

I am giving a talk this evening at the Philly On Rails meeting on an introduction to email and sending and receiving email with a Ruby on Rails application.

I am referencing the slides and a sample application in case anyone needs to reference them!

Email Basics

Sun, 05 Aug 2007 00:00:00 -0400

What is email exactly? It is quite different from the view we see in our MUA, (Thunderbird, gmail, mutt, pine, etc.). Let’s review the basic of email mechanics, shall we?

Email consists on an Envelope and the Message Data. The envelope is

Return Path: The email address to return the message if undeliverable
Recipients: The email address(es) of the recipients of the message

The Message Data is the meat of the mail,

RFC822 Headers such as Subject, From, To, Date, etc. Headers can span multiple lines by starting the subsequent line with a white-space. A blank line signals the end of the header and the start of the body follows. Non-standard header names begin with the “X-” prefix, used to denote experimental features. These usually have meaning only to the software that placed them there.
Body, text, html, etc. Using the MIME standard, the body can be made to include alternate versions of the message based on markup, and also attach files to the message.
File Attachments. Usually, binary files are encoded in Base64 no non-printable characters. This increases the file size by about 33% when encoded in the message.

Note that the From and To headers in the headers are really informational. It is the envelope recipient list that determines who it was from.

Also, since the From header can be virtually anything, it is easy to spoof the sender or even the full message. Even the recipient return path can be bogus. Therefore, email is not secure as such (except for PGP/GPG signings), and should not be trusted outright.

Since each email server that passes the message along prepends a Received header to the top of the message, you can generally trace its path to see if it came from a source server that is a proper MTA for the sender. Of course, you can only trust the received headers from servers as far back as you trust; they can also be tampered with.


To: you@example.com
From: me@example.com
Date: 20 Jul 2007 09:21:51 -0700
Message-ID: <521.1090340511@example.com>
Subject: Simple Email Message

Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Maecenas
ultrices sem sed urna accumsan cursus.

Content-Type and MIME Types

When you want to create more complex email messages, say with alternative content or attachments, you need to construct your message using MIME containers and body parts.

Content-Type: Use this header in your email message to identify the MIME type of your content
text/plain is the default MIME type of email. This is viewable by all mail clients.
text/html denotes an HTML formatted body or part. This is only viewable in GUI-based clients that support HTML.
multipart/alternative is a MIME container that holds the text, HTML or other versions of the main message. Only one of these (the best one it can show) is viewable by the user’s mail client.
multipart/alternative is a MIME container used for attaching files to the message body. The first part is the body part (which can also be another container), and the rest are attachments.
multipart/related is a MIME container that wraps included graphics referenced from an HTML body. These graphics are shown “in place”, such as a logo in the letterhead, instead of being seen as attachments.
The boundary parameter of the Content-Type header is used to provide a unique identifier to define the start and end of the body parts within a MIME container. Lines starting with two hyphens followed by the boundary value is the split point in the message. The final boundary line is the two hyphens, boundary value, and two more hyphens.
image/png is a PNG graphic file, also a image/gif or image/jpeg could be used.
application/pdf is a pdf attachment, which could be any application-defined file
Stir to combine…

Each MIME body part (attachment, container, or message version) itself has a small MIME header set to indicate its content-type, encoding, and other information.

Here is an example message that is composed of a text and HTML body alternatives, with an image attachment called out from the HTML version, plus another image as a regular attachment. The structure of the MIME parts is


multipart/mixed (Holds the body part plus attachments)
    multipart/alternative (groups the different version of the message body)
        text/plain
        multipart/related (groups the HTML part with images it references)
            text/html
            image/jpeg
    image/png (attachment)

Here is how this looks in the email message.


To: you@example.com
From: me@example.com
Date: 20 Jul 2007 09:21:51 -0700
Message-ID: <521.1090340511@example.com>
Subject: Complex Email Message
MIME-Version: 1.0
Content-Type: multipart/mixed;
        boundary="mm001" 

This is a multi-part message in MIME format.

--mm001
Content-Type: multipart-alternative; boundary=mb001

--ma001
Content-Type: text/plain

This is the plain text body

--ma001
Content-Type: multipart-related; boundary="mr001" 

--mr001
Content-Type: text/html
Content-Transfer-Encoding: quoted-printable

This is the HTML body


--mr001
Content-Type: image/jpeg; name="logo.jpg" 
Content-Transfer-Encoding: base64
Content-Description: Picture (Metafile)
Content-Location: No%20AttachName

Qk2ewgIAAAAAADYAAAAoAAAAJAIAAG4AAAABABgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA////
....

--mr001-- 

--ma001--

--mm001
Content-Type: image/png; 
    name="elroy-jetson.png" 
Content-Transfer-Encoding: base64
Content-Disposition: attachment;
    filename="elroy-jetson.png" 

R0lGODlhMgAvAPcAAAAAAJQAAPfOjP//////////////////////////////////////////////
.....

--mm001--

Content-Transfer-Encoding

Note the use of quoted-printable in the above HTML segment. Quoted-printable encoding escapes special characters with an equal symbol (=) followed by the 2-character hexadecimal ASCII representation of the character value. For example, any equal symbols in the body are replaced with ”=3D”, where 3D is the hexadecimal representation of the equal symbol in the ASCII collating sequence.

Web browsers do something similar when sending special characters in the URL, but using a percent (%) symbol as the escape symbol.

Quoted-printable also wraps text so lines do not become too long. An equal symbol at the end of the line (=\n) indicates the line is wrapped. Email standards define the maximum length of a line to be 77 (?) characters, but since this is not a hard limit, most email software is flexible about this limit.

Binary files are usually encoded in Base64. The Base64 method maps every 6 bits to a printable character. Ruby has a Base64 helper class


require "base64" 
enc   = Base64.encode64('Send reinforcements') # -> "U2VuZCByZWluZm9yY2VtZW50cw==\n" 
plain = Base64.decode64(enc)  # -> "Send reinforcements"

SMTP: How Email is Tranferred

Email is delivered via SMTP, Simple Mail Transport Protocol. This is a simple state-machine which accepts email through a “command line” interface, usually over port 25. Open a telnet connection to any MX(mail exchanger) host on port 25 to try your hand at delivering a mail manually.

Here you can really see that the email envelope is powerful, it requires 3 part of the email message:

Return Path (MAIL FROM)
Recipients: (RCPT TO)
Message: (DATA), which is ended by a single period on the last line.


220 example.com mailfront ESMTP
MAIL FROM: 
250 2.1.0 Sender accepted.
RCPT TO: 
250 OK
DATA
354 End your message with a period on a line by itself.
Subject: Hello there
From: Me 

I just love SMTP!
.
250 2.6.0 Accepted message qp 16590 bytes 226
QUIT
221 2.0.0 Good bye.