Das Bityard

Web Frameworks: When Love is Gone

2011-12-15T00:36:00.000-05:00

Django and I were not meant to be.

For the last few months I've been trying to throw together a quick little site for my wife's photography hobby. Normally I reach for CodeIgniter (a PHP web framework) when I want to bang together a web site where an existing CMS doesn't quite fit the bill. CodeIgniter simply takes care of all the mundane stuff that you normally have to worry about when developing a web site or application. To use it, you simply extract the CodeIgniter tarball, perform some minor configuration, and point Apache at it. From there, you just add your models, views, controllers, templates, and static files. You can utilize any of the plentiful CodeIgniter helper libraries and classes if you happen to need or want them.

But where I work, Python is more or less the programming language of choice for anything that isn't primarily shell work (bash) or high-performance (C). Since I haven't used Python extensively for Serious Work, I thought I would give a Python web framework a try for this new project. Django came highly recommended, so I went with it. One thing I fell in love with immediately was Django's built-in customizable admin interface based around the app's models. Man, that's slick.

I followed the rather lengthy tutorial in the beginning and was encouraged. At the outset, it looked like Django was going to be a lot like CodeIgniter only "Pythonic" and with many more built-in features. But as dug into it, I found that the authors were explaining less as they went along, which forced me to wade through the Django documentation to get an idea of what was really going on. Now, the Django docs aren't bad per se but to me they read more like system specs than a teaching tool. I'm big on thorough, simple explanation, with plenty of examples. Maybe I'm just a bit thicker than your average Python hacker.

Anyway, tonight I drew the line. Django does not fit the bill for my fly-by-night one-off projects. At this point, I feel I've spent almost as much time reading (and re-reading) Django documentation as I would have in implementing the whole thing in PHP with CodeIgniter from the beginning. And that includes referencing the docs. The last straw was when I figured out a quirk of Django's static file handling by trial and error.

I spent about five hours trying to troubleshoot why the wrong CSS file was being loaded from a static files directory. I pored over docs. Googled here and there. Questioned my own sanity. A wild and almost random stab in the dark finally solved it, but the fact that it took so damn long and that the unexpected caveat doesn't seem to be mentioned anywhere in the docs (or at least, anywhere I thought to look) spoke volumes to me. On top of this, I just had a conversation with a good friend of mine a couple days ago about how highly I recommended CodeIgniter as a starting point for getting most any small- to mid-size app off the ground quickly.

Maybe Django is great for web developers crafting large production sites and who have all the time in the world to read docs and figure out the minutia of their intricate framework. But I don't, so it looks like Django and I will be parting ways, at least for now.

Banned from the U.S.A.

2011-11-15T22:50:00.002-05:00

Pogo (Nick Bertke) is one of my favorite independent electronic music artists. He's a young, extremely talented DJ who is travelling around the world to do tours and remix movies and culture. His fans alone foot the bill for travels. Here's one video he did while in Johannesburg, South Africa:

I just found out that while attempting to travel to the U.S., Pogo was detained for not having all the proper paperwork. Rather than attempt to remedy their goof, his travel agency cut their ties to him and refused to speak to either him or U.S. customs on his behalf. Pogo sat in a New York jail for three weeks before finally being deported and being informed that he was barred from attempting to enter the country for 10 years.

In light of this situation, I'd like to cordially extend my middle finger to the United States Customs and Border protection and Department of Homeland Security.

Symmetric Encryption with PyCrypto, Part 2

2010-10-05T23:06:00.004-04:00

This is the second part of my introductory series on cryptography in Python and it covers the AES block cipher with lots of actual code this time. Please read the first part if you're new to this stuff. I consider these articles to be works in progress so don't be surprised if you notice something change between visits. I appreciate feedback and suggestions!

AES Encryption

There are many encryption algorithms that can be used for secure cryptography, but we'll settle on AES (the Advanced Encryption Standard) in this tutorial. If you have your heart set on implementing something else, fear not: everything you'll learn here applies to other commonly-used symmetric block ciphers as well. The only real differences will be their internal implementations, and the range of key and block sizes that the ciphers accept.

AES is an open encryption standard set forth by the U.S. government in 2002 and is based on a set of ciphers called Rijndael. Due to its importance, transparency, and longevity, AES is widely used around the world and has received much scrutiny within the cryptographic community. If properly implemented, it is considered unbreakable by all currently known techniques.

All AES ciphers have a block size of 128 bits (16 bytes). A “block” is the string of plaintext data that is fed into the cipher, along with the key, for which a corresponding ciphertext block is returned. The three AES ciphers are AES-128, AES-192, and AES-256. The numbers indicate the key size that each cipher utilizes (16 bytes, 24 bytes, and 32 bytes respectively).

Illustration 1: How AES Works

The illustration above is a simplified example of how AES works. To encrypt, the cipher takes two inputs: a key and plaintext. It uses the key to encrypt the plaintext and yields the resulting ciphertext. (For you purists out there, it should be noted that the ciphertext given in this illustration is only an example. It was not actually generated by AES. The ciphertext generated by AES is binary and cannot normally be printed in a readable fashion directly.)

As their name implies, block ciphers are designed to work on data in terms of discrete blocks. The AES block size is 128 bits (16 bytes), so programs which perform AES encryption feed plaintext (or ciphertext, for decryption) to the cipher in 16-byte chunks. We illustrate this in the above example by breaking the message and corresponding ciphertext into seven 16-byte lines.

You might have noticed that the last line in the plaintext is technically too short. The last block of the message consists of only two characters followed by what programmers call “padding” to fill out the remaining length of a fixed-size string. Block ciphers depend on their input being a specific known size; the algorithm won't work otherwise. To satisfy the requirement that every block be exactly 16 bytes in AES, we use the uppercase letter 'X' in this example. In many cases, this is sufficient. Anyone who decrypts this ciphertext and reads the resulting message will probably realize that the long string of 'X' isn't intended to be part of the message. But this doesn't necessarily work all the time, since there may be legitimate cases where the final character 'X' is an important part of the message. Fortunately, there is a much better solution to dealing with padding that we will discuss in the next article.

A Practical Example

As with any decent cryptographic system, AES is sufficiently complex that you wouldn't want to try whipping up your own implementation to use for important data unless you have a few years of hardcore crypto experience under your belt. (And probably not even then.) Fortunately, we're using Python, which has a number of quality cryptographic modules available. Since the standard C Python distribution doesn't come with modules for symmetric block ciphers, we'll use the Python Cryptography Toolkit, also known as PyCrypto.

PyCrypto can be fetched and built from pycrypto.org, but many Linux distributions carry it in their software repositories as well; look for and install the 'python-crypto' package on Debian, Ubuntu, and Fedora. There is no official Windows or OS X binary distribution of PyCrypto, but this article describes how to install Python and PyCrypto on Windows and OS X.

Once you have PyCrypto installed, we can start off with a simple example:

# Example 1
 
from Crypto.Cipher import AES
 
key = 'mysecretpassword'
plaintext = 'Secret Message A'
 
encobj = AES.new(key, AES.MODE_ECB)
ciphertext = encobj.encrypt(plaintext)
 
# Resulting ciphertext in hex
print ciphertext.encode('hex')

Example 1: Basic encryption

This script creates a ciphering object using a key and a feedback mode (ECB, which we'll discuss later). It then feeds the plaintext into that object with the encrypt() method which returns a ciphertext string. Since the resulting ciphertext is a binary string, we encode it in hex before printing it to the screen. When the script is run, it prints:

e8da47acc08bc751745ef8fbff44e107

Not very interesting, is it? That's exactly the point. You can change the key and the plaintext (keeping in mind the key sizes and block size of AES) and note how the ciphertext comes out entirely different every time. You might have noticed that the length of hex-encoded ciphertext is 32 characters. By encoding it in hex, we've doubled the size because each binary character is represented by two hex characters, thus the actual ciphertext is only 16 characters. This is–not coincidentally–the block size of AES.

Let's see how the reverse operation works:

# Example 2
 
from Crypto.Cipher import AES
import binascii
 
key = 'mysecretpassword'
ciphertext = binascii.unhexlify('e8da47acc08bc751745ef8fbff44e107')
 
decobj = AES.new(key, AES.MODE_ECB)
plaintext = decobj.decrypt(ciphertext)
 
# Resulting plaintext
print plaintext

Example 2: Simple decryption

This script performs decryption on the ciphertext that was generated from the script in Example 1. The binascii module is used only to convert the hex representation of our ciphertext back into binary before it is fed into the decrypt() method of the ciphering object.

A Weakness Uncovered

Symmetric block ciphers, powerful though they are, have an inherent weakness.
When you encrypt a block of data, you typically do so with the expectation that eventually you'd like to retrieve that data again. Thus, the algorithm has to be deterministic. That is, for a given key and plaintext, a specific ciphertext will be generated each time. Which poses a problem: suppose you have blocks within the plaintext that are identical. This isn't terribly hard to imagine. A spreadsheet often has multiple cells containing the same data. An image almost always has multiple pixels of the same color value. If you encrypt files like these with a block cipher and do nothing else, an attacker could see a pattern in the ciphertext and use it to gain crucial insight into the nature of the plaintext. Take these two images for example:

Illustration 2: Patterns in ciphertext

The plaintext image on the left is Tux, the Linux mascot. The image on the right is not actually an encrypted file, but is the result of running each pixel color value through a block cipher (with a key unknown to us). The image is then recreated with the encrypted color values. The image is technically encrypted with a strong cipher, but patterns in the plaintext data result in easily-discernible patterns in the ciphertext. This is a violation of the principle of non-uniformity we talked about in Section 1. Encrypting the data did little to hide the essence of the original file.
Here's a demonstration in code:

# Example 3
 
from Crypto.Cipher import AES
 
key = 'mysecretpassword'
plaintext1 = 'Secret Message A'
plaintext2 = 'Secret Message B'
plaintext3 = 'Secret Message C'
plaintext4 = 'Secret Message A'
 
encobj = AES.new(key, AES.MODE_ECB)
ciphertext1 = encobj.encrypt(plaintext1)
ciphertext2 = encobj.encrypt(plaintext2)
ciphertext3 = encobj.encrypt(plaintext3)
ciphertext4 = encobj.encrypt(plaintext4)
 
# Resulting ciphertext in hex
print ciphertext1.encode('hex')
print ciphertext2.encode('hex')
print ciphertext3.encode('hex')
print ciphertext4.encode('hex')

Example 3: Demonstrating the ECB mode of operation
This prints:

e8da47acc08bc751745ef8fbff44e107
49b1f4635c9a53c4f7538ba3b4452b32
0d750ffff1ab846b838a6a304860372e
e8da47acc08bc751745ef8fbff44e107

You can see that the ciphertext for plaintext1 and plaintext4 are identical. In real-world applications, ciphers are usually paired with a method to eliminate these redundancies in the resulting ciphertext. These methods are called modes of operation. In all of the examples so far, we used the Electronic Code Book (ECB) mode when creating the AES cipher object:

encobj = AES.new(key, AES.MODE_ECB)

ECB is just the cryptographers' formal way of saying, “nothing has been done to eliminate obvious patterns in the ciphertext.” PyCrypto supports another mode of operation, Cipher Block Chaining (CBC). To eliminate redundancies in the resulting ciphertext, CBC incorporates the ciphertext of the previous block with that of the next so that the entire volume of ciphertext appears to be pseudo-random. When encrypting the first block, a random value called the initialization vector (IV) is XORed with the plaintext. The result is encrypted with the provided key. Before the second block is encrypted, its plaintext is XORed with the ciphertext of the previous block. And so it goes until all of the blocks in the file or message have been encrypted in this manner. Here's an illustration (the circled plus sign indicates the XOR operation):

Illustration 3: Cipher Block Chaining

To use CBC in PyCrypto, all we have to do is supply an IV and change the mode:

# Example 4
 
from Crypto.Cipher import AES
import os
 
key = 'mysecretpassword'
iv = os.urandom(16)
 
# Output the initialization vector
print 'IV: ' + iv.encode('hex')
 
plaintext1 = 'Secret Message A'
plaintext2 = 'Secret Message B'
plaintext3 = 'Secret Message C'
plaintext4 = 'Secret Message A'
 
encobj = AES.new(key, AES.MODE_CBC, iv)
ciphertext1 = encobj.encrypt(plaintext1)
ciphertext2 = encobj.encrypt(plaintext2)
ciphertext3 = encobj.encrypt(plaintext3)
ciphertext4 = encobj.encrypt(plaintext4)
 
# Resulting ciphertext in hex
print 'A:  ' + ciphertext1.encode('hex')
print 'B:  ' + ciphertext2.encode('hex')
print 'C:  ' + ciphertext3.encode('hex')
print 'A:  ' + ciphertext4.encode('hex')

Example 4: Demonstrating the CBC mode of operation

If we run this example, we get something like:

IV: 1c531872a9b7a71bfc060b22c459136b
A:  ed7a68aa4a7c316047b6c6493b2d03d9
B:  5177729609a9bbd0500f8ea2614c7906
C:  46b19bace80917f655846aa697f201fb
A:  0e298e3b3072e8c595637d2e5feb16ec

The first thing to note here is that the ciphertext for message A no longer repeats. This is good! This keeps repeating plaintext blocks from forming patterns in the ciphertext. In fact, on every invocation of the script, all the ciphertext blocks change completely because a different IV is generated at random each time, even though the key and plaintext messages stay exactly the same.

You may have also noticed that we now have 5 blocks of ciphertext for 4 blocks of plaintext. The reason is simple: we added an extra block to the output by including an IV. The IV is generated at random and passed into the ciphering object. The IV has to be known at the time of decryption in order to produce the expected plaintext, so it is often prepended to the ciphertext. Knowledge of the IV itself cannot help an attacker break the encryption, so including it as generated poses no risk.

If you've understood everything presented so far, you should have a good idea as to what a decryption script for the above data looks like. If you're in the mood to do some homework, now is a great time to write one.

Stay tuned for Part 3 where we'll be discussing keys and key strengthening techniques!

The image of Tux in Illustration 2 is copyrighted by Larry Ewing and was created with The GIMP.

Playing with localStorage

2010-02-02T22:07:00.007-05:00

This seems to be something of a well-kept secret. It appears that almost all newer web browsers support various types of client storage without the use of cookies. The web storage specification (part of HTML5) describes new facilities that web authors can use to store arbitrary data on the client using a simple key/value store.

The best part: instead of being limited to 4K of data total as with cookies, each website can store megabytes of data on the client. (Implementations vary, but the draft specification recommends somewhere in the area of 5MB per origin.) This ultimately means that we'll start seeing a lot of web applications (E.g., Google Docs) storing data locally for offline access. The specification describes (so far) session storage and local storage.

Two other kinds of local storage were once proposed, but eventually dropped: global storage and database storage. Global storage is like session and local storage, except that the website could scope data along the domain name heirarchy. (See Mozilla's explanation for details.) Database storage is essentially an SQL engine inside the browser. The only browsers that support it are those based on recent versions of WebKit, namely Chrome and Safari.

Back to local storage, however. These are the browsers that I successfully tested with support for local storage:

Chrome 4.0.302.2
Firefox 3.5, 3.6
Safari for iPhone/iPod Touch 3.1.2
Internet Explorer 8

While these browsers are not the majority of the ones currently in use, they or their descendants will be soon. Users are finally starting to realize that it's in their best interest to stay reasonably up to date on their software and some large web properties are beginning to drop support entirely for crusty old ancient browsers (ahem). It's safe to say that local storage will be a big part of future web apps.

So, here's how it works. Items are stored as simple key/value pairs. Programmers know this as a hash table. The object in Javascript to manipulate the storage is called, unsurprisingly, localStorage. This object has the following methods:

getItem(key) - retrieve a value via the key
setItem(key, value) - add or change a key/value pair
removeItem(key) - remove an item from storage
key(index) - retrieve the key via its index
clear() - empty the local storage

And one property:

length - the number of items in local storage

The above should be mostly self-explanatory. You can get the key for a specific item by specifying its index number but note that since this is a hash table, the index-to-key mappings are not static. You can never presume that a particular index belongs to a certain key. The key() method is primarily meant to be used with the length property so that you can iterate over each item in storage.

Pretty simple, eh? This tiny feature adds a load of capability and, together with the rest of HTML5, will finally start to turn the web browser into a proper applications platform, even if it never will be a particularly efficient one.

A quick note on privacy and security: each fully-qualified domain name that you visit has its own storage. For example, yahoo.com and google.com can never share storage. And as far as I understand, the policy is the same for subdomains as well, so mail.google.com and code.google.com can't access each others' storage either.

I put together a demo that lets you directly play around with localStorage in your browser. Take a peek at the source code, save it, steal it, whatever. Everything except the jQuery library is contained within the one HTML file.

HOWTO: Backup an Entire Hard Disk to a Smaller One

2010-01-28T17:31:00.003-05:00

In my book, one of the best ways to back up an entire disk is to simply dump an image of it to external media. For instance, let's say you have an OS installed on your workstation and you want to try a new Linux distribution. You could repartition the drive, mess with the bootloader, and install the new OS next to the old one. Then, if you don't like it, nuke the new partition, resize the old one, and fix up the MBR to point back to the old OS. This is certainly a common scenario, but it carries with it the substantial possibility that something will go wrong. You could lose your main OS and have to start over from scratch. Or you might end up spending hours trying to recover your old OS that you didn't properly back up before starting the procedure. Because hey, you're a Linux superstar by now, who needs backups anyway?

It's much simpler to just boot from a live CD and copy the entire disk image (partition table and all) to a file on external media with dd(1). When done with your experimentations, simply copy it back. This way you don't have to muck about with partitions, permissions, metadata, bootloaders, or anything. It takes a bit of time, but it is exceedingly simple and much less risky than other backup and restore methods. Here's an example. Assume that /dev/sda is the main disk in the machine that you're backing up and /dev/sdb1 is a partition on external storage.

mkdir /mnt/sdb1
mount /dev/sdb1 /mnt/sdb1
dd if=/dev/sda of=/mnt/sdb1/sda.img

This works great, but one obvious drawback is that the free space on sdb1 must be greater than (or by freakish coincidence, exactly equal to) the total size of /dev/sda. Unless, perhaps, we compress it. Then the minimum free space requirement on /dev/sdb1 becomes much lower. In effect, required space is equal to the amount of actual data on /dev/sda after it's been compressed. For example, if /dev/sda is 80GB in size, but the df(1) command reports that only about 10GB of it is in use, it's fair to guess that a compressed image of the disk should end up somewhere under 10GB in size.

So, there's no theoretical reason this couldn't work. But we have to recognize that the free space must compress well in order to get this kind of efficiency. Except in very specific circumstances, the "free" space on a hard disk is not actually empty. When you delete a file, the data is still there; the OS simply removes the references to the file. This data will be included in a dumped image of the disk and it may or may not compress well.

The solution is to fill up the free space with zeros. When it comes down to it, there's nothing that compresses better than a whole lot of nothing. For best results, this should be done on all file systems in addition to the swap partition.

The Backup Procedure

Mount the main disk and fill up the remaining free space on all partitions with a file containing all zeros. (/dev/sda1 is the only data partition in this example.)

mkdir /mnt/sda1
mount /dev/sda1 /mnt/sda1
dd if=/dev/zero of=/mnt/sda1/zero

Wait until you get the error "No space left on device", then remove the zero file on each partition.

rm /mnt/sda1/zero

Umount the partition.

umount /mnt/sda1

Zero the swap partition. (Shown as /dev/sda5 here.)

dd if=/dev/zero of=/dev/sda5

Reinitialize the swap partition. (Otherwise it won't be detected and enabled on boot after restore.)

mkswap /dev/sda5

Mount the backup partition.

mkdir /mnt/sdb1
mount /dev/sdb1 /mnt/sdb1

Copy the entire disk to an image file on the backup drive (/dev/sdb1), piping it through gzip for compression.

dd if=/dev/sda | gzip - > /mnt/sdb1/backup-sda.gz

Umount the backup device, reboot, and you're free to do whatever you like to your main disk, knowing that you have a good backup on hand.

The Restore Procedure

Restoring is even easier than backing up. Once you're up and running on the live CD:

mkdir /mnt/sdb1
mount /dev/sdb1 /mnt/sdb1
gunzip -c /mnt/sdb1/backup-sda.gz | dd of=/dev/sda

Unmount the backup drive, reboot, and you should have your original OS back just as it was before the backup.

Symmetric Encryption with PyCrypto, Part I

2010-01-24T01:53:00.007-05:00

This is a multi-part series on the basics of implementing symmetric cryptography in Python with PyCrypto. By the end of the series, we'll have a program that takes a password plus a file and outputs the encrypted version of the file. (And of course, the reverse.) This first part is mainly intended for the novice. Those already well-versed in encryption or don't care for a refresher should consider skipping to Part 2. Also, I'll attempt to explain concepts as best I can, but you'll want to have a decent grasp of Python in order to make full use of the tutorial.

Before we dive right in, a disclaimer: I am writing these articles in the hope that they will be educational and useful to others like myself and to solidify the concepts in my own mind. Crypto is hard to do right. Please don't use these articles as a replacement for a professional security consultant.

The Goal

We are going to write a Python program that will take an unencrypted file, prompt for a password, and spit out an encrypted version of the file. It will, of course, also be quite capable of the reverse: read an encrypted file, prompt for a password, and give you the unencrypted file back. This is called symmetric encryption because one key allows you to perform both the encryption and the decryption operations.

Before we can talk semi-intelligently about crypto, we need to know a few basic definitions. More than almost any other technical field, crypto is rife with metaphors. The jargon, therefore, is generally pretty easy to comprehend.

key: The piece of information that allows you to either encrypt or decrypt your data. Although it's tempting to think of a crypto key as being similar to a physical key, a slightly better real-world analogy is that of the combination to a combination lock. A combination is a random-looking series of numbers which can be memorized, transferred, and transformed easily, rather like an encryption key.

plaintext: The information that you want to keep hidden, in its unencrypted form. The information does not actually have to be text, it's just the term that cryptographers use. The plaintext can be any data at all: a picture, a spreadsheet, or even a whole hard disk. The synonym "message" is sometimes used as well, especially when dealing with situations where encrypted data is being passed from one party or computer to another.

ciphertext: The information in encrypted form. If the encryption software is written correctly and all of the proper procedures are followed, the ciphertext is completely unreadable and unbreakable without the key. Although it's generally considered good security practice to reasonably protect your ciphertext from third parties, the theory goes that you could print it in a national newspaper if you wanted to and nobody would be able to decode it to plaintext without knowing the correct key.

cipher: The algorithm that converts plaintext to ciphertext and vice-versa. We won't go into any details about how the internals of cipher algorithms work, as it would be a very technical subject well beyond the scope of this introduction. Fortunately, ciphers tend to be simple enough to use that a high-level overview of the basic concepts is sufficient to make full use of them. An analogy might be that you don't have to know how a compression algorithm works in order to "zip" a file.

A First Example

To visualize how these parts work together, consider this example code for an encryption operation:


plaintext = 'I say, mater, cabbage crates coming over the briny.'
key = 'ocelot'
ciphertext = encrypt(plaintext, key)
# The variable 'ciphertext' now contains the string: 
# 'Qng5tnrf97OG4pHooBCT96aSykSsdAiHb92RPXQPVDfdAKapuX4'

And the reverse operation, decryption:


ciphertext = 'Qng5tnrf97OG4pHooBCT96aSykSsdAiHb92RPXQPVDfdAKapuX4'
key = 'ocelot'
plaintext = decrypt(ciphertext, key)
# The variable 'plaintext' now contains the string:
# 'I say, mater, cabbage crates coming over the briny.'

Note that the key is the same in both operations. That's what makes this symmetric encryption. If we were to leave the first listing alone but change the key to "caribou" in the second listing, the decrypt() function would return either gibberish or nothing at all, depending on the implementation.

Doing Encryption Right

In order to stand up against a focused attack, the encryption system as a whole must be well designed and well implemented. There are certain properties of each part of the encryption system that--when put together--make it secure. We'll briefly mention some of these properties here.

Full Disclosure

The complete workings of a cryptographic system should be open, public, and available for analysis by expert cryptographers. Many ciphers have been invented, and many have been subverted due to various design flaws. A cipher is deemed secure by experts only after it has withstood sufficient scrutiny from the public and experts in the field of cryptography. A programmer implementing an encryption system should never rely on some hidden feature or obfuscation for the security of the system. Even if the system is never intended to be released publicly, it should be designed as if it will be. The only component that should ever be treated as truly secret is the user's encryption key itself.

Non-Uniformity

The ciphertext produced by a cipher should be completely indistinguishable from random data. If the ciphertext had any observable pattern at all, an adversary could theoretically use that information to make a guess about the nature of the plaintext or use whatever is known about the pattern to more easily crack the cipher. The reverse is not necessarily true, however: random-looking cipher output is not by itself any indication of a secure cipher.

Another closely-related principle is a small change in the either the plaintext or the key should cause a dramatic change in the ciphertext as whole. If either input is off by a even single bit, the ciphertext should come out completely different.

Key Strength

The key should be as secure as possible. In many implementations, users will choose a password which is then converted into a key. The problem with passwords is that they can be easy to guess. If an adversary wants access to your encrypted data and you have an easily guessable key, they don't need millions of computers brute forcing the password for millions of years, they can just try a few hundred thousand obvious keys until they find the one that unlocks your data. Using a weak password nullifies any benefit obtained from using encryption. In fact, it can make things worse on the whole, because it could lull you into a false sense of security.

Key-strengthening techniques have been devised to counter these types of brute-force attacks. They are no replacement for a strong password, but they make it harder to pre-compute the keys from a password list. We'll discuss these techniques later.

Key Length

An encryption key has to be large enough so that simply trying every possible combination of bits would be an insurmountable task. Imagine, again, the key as the combination to a lock on a safe. The fewer numbers in the combination, the easier it is to find the right one to open the safe simply by incrementing the combination for each try. In cryptography, a small key can be easily found with the same method. For example, an attacker trying to find an 8-bit key (one byte) only has try to all 256 (2^8) possible different combinations.

For each bit that is added to the key, the number of combinations to try is doubled. So, a 9-bit key takes 512 guesses, a 10-bit key, 1024 guesses, and so on. The standard in symmetric cryptography is 128 bits. If you enter 2 to the power of 128 into a calculator, you'll see an extremely large number as the result. Experts have theorized that trying to sequentially search for a key of this size would take more than 10 trillion years to exhaust the key space. So if an attacker were able to set up a device to brute-force a 128-bit key at the moment the universe was created, they will have had only about a 1 in 1000 chance of finding it by now. (Source.)

Barring any unforeseen fantastic breakthroughs in mathematics or physics, 128 bits is more than plenty for a good long time. Many ciphers support key sizes of up to 256 bits to make them stronger against unforeseen attacks. The extra 128 bits provide an enormous margin of safety in case weaknesses are later discovered in the ciphers or (more likely) their implementations.

Cipher Strength

These days, modern encryption ciphers tend to be very well engineered. They are invented by people with a firm grasp of the mathematical theories behind cryptography and they tend to be peer-reviewed, openly discussed, and publicly available. Researchers are constantly on the lookout for ways to break popular ciphers or at least compute them with far less effort than it would take to brute-force them with ordinary means. Those that withstand such scrutiny over time become trusted.

History is littered with ciphers that were once thought to be secure but have since been proven to have serious flaws. Some symmetric ciphers that are considered secure (as far as we know) are AES (a.k.a. Rijndael), Blowfish, Serpent, and Twofish.

Stay Tuned

In Part 2, we cover AES, write some code to test PyCrypto functionality, and discuss modes of operation.

The State of Solid State

2009-10-25T21:18:00.004-04:00

So, this guy does some speculative research and comes to the conclusion that mechanical hard disks will still be the dominant secondary storage technology for computers in 2020. Somehow, I'm a bit skeptical.

If you want to see what's going to happen in regard to mechanical vs solid state hard disks, you don't need a crystal ball. Just look at the transition from CRT to LCD displays. It wasn't so long ago that LCD monitors were horribly expensive and that fact (combined with their other drawbacks) made them an unattractive option for most people. I can recall many, many people saying that they would never give up their enormous, power-hungry, failure-prone CRT displays. Now, you can't even buy a CRT computer monitor because LCD quality caught up and surpassed CRTs for most purposes while price plummeted. The same will happen with mechanical disks and SSDs. Maybe it'll happen faster, maybe slower, but it will happen.

Keep in mind also which company this "prediction" is coming from: Seagate lived a long and prosperous career engineering and manufacturing mechanical hard disks. They are a huge company whose entire operation is based around the concept of shipping hunks of metal with rotating platters inside. Since an SSD is just a bunch of memory chips duct-taped together, the memory companies (Transcend, Crucial, Corsair, Samsung, etc) were the first ones with SSDs on the market. The SSD thing likely hit Seagate by surprise and they can see that their run won't last long.

It's not too late for them to start transitioning to manufacturing memory chips, but doing so would be brutal for many reasons. To start with, their decades of mechanical drive development experience, manufacturing facilities, engineers, trade secrets, R&D, etc are mostly about to be worthless. If they start selling this stuff off now while it's still fairly valuable, shareholders are going to do a huge "WTF?" and walk off. Second, the memory companies have a few years head start. Even if Seagate could enter the market and compete with them, the company would be leaving their position as a market leader to be a market newcomer, taking cues from everyone else. (Cue the sound of their last few shareholders stomping out.)

Basically, unless Seagate can buy up a few of the leading memory companies making SSDs right now, they're screwed. Until that happens, all they can do right now is appease their shareholders and put their executives up on stage to have them parrot the lie that their business is going to be viable for a good long time yet. Oh, and frivolously sue all the SSD manufacturers on broad patent infringement grounds.

Sonic 3 Ice Cap Zone Act 1 on Korg EMX-1

2009-10-12T21:49:00.002-04:00

For fans of a particular blue hedgehog.

x0xb0x #3

2009-08-12T01:57:00.006-04:00

Just finished up x0xb0x #3. All of the basic functionality seems to work and it sounds just like a 303. Well, better than a 303 in my book but I guess I'm probably biased.

They gray knobs were special-ordered from Korg Europe. They were made for the Electribe ESX-1 and EMX-1 but they happen to be perfect for the x0xb0x. I could have ordered the large switch knobs too, but I would have had to hack the encoder shafts even more than I already did.

The top panel is a standard Adafruit x0xb0x clear acrylic panel. To make it awesome, I just sprayed the underside with black paint. I wanted to use vinyl instead, but the paint turned out to be so much easier. Looks great, the only downside is that the panel came with some scratches from the store. Oh, and it's a friggen fingerprint magnet. Next time, I may take a drill bit and scrape out the LED holes a bit so that they light up the laser-engraved letters on the panel.

I used a clear I/O panel this time, so the innards can be seen without taking the x0x apart. I think I'll go back to metal unless I can devise an easy way to get an LED or two in there for show. All I have left to do on this one is tune it, perform a full function check, and hack in a power switch and then it's eBay time!

I posted the innards of this x0x in a previous post here and did a writeup on x0xb0x #2 here. I hope to put together a video of me rocking the x0x sometime this week or next.

x0x #3 Almost ready!

2009-08-11T01:22:00.004-04:00

I'm almost done with x0xb0x #3.

This one is being quite the problem child, actually. Two note LEDs flat-out didn't work, so those had to be replaced. The TEMPO LED had to be replaced because I scratched it to hell and back. The TEMPO encoder was replaced because the first one didn't work. (Talk about a wild goose chase there.) The USB chip wasn't talking to the microcontroller, which was fixed by cleaning up and reflowing the teeny-tiny SMD pins of the FT232 chip. The MIDI-in port still isn't working as I write this, despite the fact that I've already swapped the resistor that I assumed was causing the trouble.

On the upshot, I did a nifty thing with the control panel faceplate, so this one should look fully bad-arsed once it's all working and put together. Pics will be forthcoming for sure. I was hoping to have the unit done and sold in time to buy myself an EMX-1 for my birthday, but it doesn't look like that's going to happen at this point.

On not coming in last

2009-05-28T22:06:00.005-04:00

Not content with making it as difficult as humanly possible for a person to enter back into his or her own country, now the U.S. Customs and Border Patrol want to scan and store the fingerprint profile of every person exiting the border, even U.S. citizens.

And incredibly, they can't even state why this would be a good security measure:

Michael Hardin, a senior policy analyst with the US-Visit Program at the United States Department of Homeland Security told a Biometrics Institute conference today that the DHS will use the data from the trial to "inform us as to where to take [exit screening] next."

Seriously. He actually said that the only reason they're implementing it to begin with is because they want to see how much they can get away with. If I didn't know any better, I'd say that they only seem to be interested in keeping tabs on citizens' whereabouts...

"We are trying to ensure we know more about who came and who left," he said.

C'mon, Michael! You're not even trying!

You know, there were other countries that pulled stunts like this. Ones that ended up having such a tight grip on their borders and citizens that travel and free movement was effectively impossible. Countries that, if I am not mistaken, we either went to war with or fought against in other ways because we did not want to see the world becoming that kind of place. We did not want America to be the last bastion of freedom. I guess it proves that you have to be careful what you wish for. An anonymous poster on Slashdot said this:

You know, I'm a Canadian, and ten years ago, I would have voted to join the US. I felt that Americans recognised the value of their freedoms and that they had, and would fight to keep, a more free society than just about anywhere else on Earth. Today, I won't even travel there. It reminds me of all those B movies just after WW2 "Achtung! Show me your papers". How could y'all have just let this happen?

x0x #2

2009-03-25T03:09:00.003-04:00

To make up for the rather non-trivial amount of money it cost to build my x0xb0x, I decided to make another and sell it on eBay. I've been working on it since January (often while I should have been studying) but finally got it done after about 2 months of work. This is the result:

The primary differences between this x0x and the "stock" ladyada edition are mostly aesthetic. I chose to paint the case a darker shade of red, which turned out better than expected with the silver vinyl overlay. The overlay itself is sweet, but the alignment of the holes left much to be desired. I had to trim all of the LED holes with a knife after applying the overlay so the vinyl didn't end up sticking to the LEDs. That alone took several episodes of Bullshit to fully accomplish. The end result is reasonable as long as you don't look too closely.

The knobs are the same as those on ladyada's parts list, but grey instead of black, and with a red pointer instead of white. If I had it to do over, I would have gotten black knobs with a white pointer. The grey doesn't really stand out enough and the red is actually more of a pink.

On the inside, I used individual wires instead of ribbon cable for the jumpers. This might have worked better with a smaller gauge wire. (I should have used 24 or 26, but 22 is all I have on hand right now.) Also, I used connectors instead of soldering the wires straight to the board. This ended up being a life-saver but introduced a few problems as well. For starters, the connectors are hideously expensive (about $25 for the whole lot, I think). Also, the 12-pin row on the mainboard ends up standing too tall and hits the bottom of the case when assembled.

Had no problems at all during the construction of this x0x and only found one silly but glaring issue the day before I shipped it to the buyer: The MIDI input and outputs weren't working at all. After an hour or so of troubleshooting with the oscilloscope, I finally figured out that I had soldered in six resistors of entirely the wrong value. Replaced them and it worked like a champ!

I'm a little sad now that it's gone, but building the second one was every bit as fun as the first.

kick-ass song time

2009-03-10T02:02:00.009-04:00

I'm an avid fan of music. Electronic music in particular, with only a few exceptions. I try to keep only freely-redistributable muzak in my playlists. Not only is it more affordable, but I believe music turns out better when the person composing it is doing so out of passion rather than profit. Nowhere is this more true than in the demoscene, where geekery intersects artistry.

One song I've been grooving to lately is Ride the Lightning by keith303. His website seems to be down as I post this, but check out his artist pages on Virb and Last.fm. He's German! Always a plus in my book. Ride the Lightning is a demo song that comes with Renoise, the most ass-kicking-est mashup between an old-school tracker and modern digital audio workstation that you'll ever see this decade.

Artist: Keith303

MP3 download: Ride the Lightning

Renoise Song download: Ride the Lightning

getting wired

2009-02-09T16:41:00.002-05:00

A recent Slashdot article reminded me of my first attempts at trying to communicate with the outside world via computer. Growing up, I lived in a relatively remote area with no one I could talk to about computers or swap tips and programs with. But I learned as much as I could through magazines and books and eventually I learned that it was possible (with the right equipment) to connect your computer to a phone line and have it "talk" to another computer.

It wasn't until we got a Packard Hell 16MHz 386 that I was able to try this out for myself. It happened to come with a 2400 bps modem pre-installed. And if you know me, you'd know that there was no way I was going to let that thing go unplayed with. One boring day, I ran a long phone line over to the computer, plugged it in, and used HyperTerminal (or something like it) to connect to a BBS on the other side of the state. I can't remember exactly how I got the number, but on this BBS I found that you could download programs, leave messages on the electronic forums, play games, and even talk with people in real time. I was floored. I had to share it with someone.

Me: Mom! I'm talking to someone on the computer!
Mom: What? How?
Me: I connected it to the phone line and dialed up someone else's computer. Now I'm talking to them.
Mom: Oh cool!
Mom: ...
Mom: Wait, is that long distance?

In my area, pretty much every phone call was long distance and it wasn't cheap. Although we had a CompuServe account for some time, it wasn't until our local phone company added a local dial-up number for Internet access that I was able to spend huge amounts of time online.

Now, a mere 15-20 years later, it's rare to not have instant and constant access to the largest computer network in the world. I can't even imagine what kind of access the next couple of decades will bring.

OpenSSH: The Poor Man's SOCKS Proxy

2009-01-21T01:17:00.006-05:00

Just when I think I know everything I need to know about OpenSSH, I end up learning something new and tremendously useful. Today, that would be the -D argument.

Many times I have been stuck on an "untrusted" Internet connection and need to log in (insecurely) to a certain site. My university, for example, uses a system that has no way of logging in via HTTPS, nor does it secure the traffic to and from the browser. I have moderate faith that the folks at my ISP aren't snooping my traffic (since I know the company pretty well and used to work with them), so I don't have a huge problem logging into their site at home. I also have a colocated server at the web hosting company I work for, so I know the layout of their network even better and trust them not to snoop or interfere with my traffic. But when I'm on the road connected to some dodgy insecure hotel wifi, I acquire no small amount of anxiety over the fact that anyone with a packet sniffer can get access to all of my personal and academic details.

For the past few years, I've had this plan to get OpenVPN set up for my network and laptop so that I can always have a secure connection to my home and colocated server. And for the past few years, I've kept putting it off. While OpenVPN is easier to use than many other VPN solutions I could name, it's still at least a good hour of my time getting all the settings right and testing it out.

I was already aware of OpenSSH's -L option which simply forwards a local port through an SSH tunnel to a port on the remote machine. Very handy when you want to connect surely to a site hosted on that server and happen to have a shell account on it. But to do much more than that ranges from the complex to impossible. This is where -D comes in.

The -D arg tells OpenSSH to be a SOCKS proxy. So you simply log in to the endpoint via SSH with the -D arg like:

ssh -D 1234 user@host.example.com

And then tell your web browser to use a SOCKS v5 proxy on localhost at the specified port and bingo, you have a secure connection to your endpoint. In fact, any application with SOCKS support can have its traffic routed through the SSH tunnel via SOCKS. Firefox supports SOCKS just fine, Opera doesn't. Konqueror is supposed to, but judging from the Google responses I got, support might be a little flaky.

The final test was whether I'd be able to use this newfangled (to me) proxy method on my Nokia N800, a device that I browse and email with quite often whilst traveling. Obviously OpenSSH has to be installed as it doesn't come with the firmware. And the N800's web browser, MicroB, uses the Gecko engine. The UI has no widgets for entering a SOCKS proxy, but you can set the preferences manually with about:config:

network.proxy.socks localhost
network.proxy.socks_port 1234
network.proxy.type 1

The result? Portable proxy surfing!

Korg Electribe EMX-1

2009-01-18T19:12:00.003-05:00

Next on my list of completely bad-ass gadgets to acquire in the distant future:

(This is not me playing by the way, it's the user known as Denkitribe on YouTube and his beats generally rock.)

My Progeny

2008-12-16T02:39:00.001-05:00

muddy thinking

2008-11-25T22:27:00.004-05:00

3,000 Riot to Protest Crackdown on Illegal Amazon Logging

You know you've had your head stuck inside a computer for too long when, after reading the headline above, you start wondering why:

1. How long had Amazon.com been keeping these logs?
2. In which country is excessive logging of customer data illegal?
3. How much did Jeff Bezos have to pay these protesters to incite a riot?

But it turns out of course that it's a rainforest article. (Insert "couldn't see the forest for the trees" pun here.)

We be x0xing

2008-11-15T18:46:00.008-05:00

It occurred to me today that I hadn't put up many pictures yet of my x0xb0x whilst under construction. It further occurred to me I had not taken many pictures of my x0xb0x whilst under construction either. Besides the one a few posts down, this is it:

Here the power supply is done (the components on the smaller PCB to the right) as well as the VCO and VCA sections of the mainboard. And also a few IC sockets are in because I got bored one night and wanted to solder but didn't have all the parts yet.

Here's a photo of the finished product:

It simply sounds marvelous and is a joy to tinker with. I used diffused blue LEDs (which are actually hard to get a hold of cheaply) and metal knobs. The knobs ended up being the wrong size though, so they sort of wobble a little when you turn them. (The center of the pot is not the same as the center of the knob.)

I also happened across a picture of the oscilloscope I bought on eBay. Four channels, 100MHz, all awesome. It's the exact same model that I used in the Air Force to repair autopilot systems so I'm already familiar with its operation. Usually, buying something like an oscilloscope on eBay is extremely risky. An oscilloscope is a precision instrument with a million different things that can go wrong and quite a few bits inside that have to be calibrated every so often. On top of that, most of the test equipment for sale on eBay comes from auctions a.k.a., refuse.

I got lucky, though: $300 (free shipping) and there's not a single thing wrong with it. I like gambles that work out in my favor.

Overcontroller

2008-11-02T00:19:00.010-04:00

Not only is this my first x0xb0x track, it's the first full piece of "music" I've composed. Evar.

The main sound is a x0xb0x connected to a Zoom 506 bass guitar effects pedal with some pretty wild settings. The x0x provided MIDI sync to ReBirth, a software synthesizer for Windows which donated a second 303 to the track as well as authentic 808 and 909 drum machine sounds. ReBirth performs admirably on Linux with the help of Wine, even the MIDI stuff.

This was recorded "live" in a single go, although there was some rehearsal. (And obviously the patterns were not programmed on the fly.) It's also kinda noisy since I've yet to perfect the art of recording things from the audio-in of a sound card. The drum machine effects themselves are known to be cheesy (808 cowbell!) since I was just trying to get something that sounded somewhat catchy without getting bogged down in minutiae. Overall, I'm quite happy with how it turned out.

Here are links to it in two formats: OGG and MP3. The OGG one sounds better but you might only have an MP3 player handy.

Linux terminal speed benchmarks

2008-10-27T14:54:00.005-04:00

In system administration, you spend a lot of time typing into and reading back information from a terminal. Although all terminals pretty much do the same thing, they can differ somewhat in their UI features or which desktop they were designed to be integrated into.

A few years back I was doing a lot of compiling (Gentoo, FreeBSD) and I felt that a good deal of that time was spent just waiting for the terminal to print the enormous amount of compiler cruft to the screen. So I did some quick benchmarks. I don't remember the exact results of those benchmarks nor if I actually made a decision based on them but I clearly remember that results were interesting.

The topic of terminal speed came up at work today so I set out to replicate the experiment. Creating a benchmark like this is harder than it sounds because every time a single a character is printed in a graphical terminal, code is being run in the Linux kernel, numerous places in X, the video card driver, the command shell (bash), and the application running the benchmark itself and even the raw performance of the video card itself can come into play. To design the perfect graphical terminal benchmark, you'd need deep knowledge of how all of those work and carefully craft the benchmark so as to maximize the "stress" on the graphical terminal code while minimizing "stress" on the other components of the system.

However, I'm far too lazy for all that.

So I just catted a Linux kernel changelog to the screen. Each benchmark was run four times times sequentially and the time averaged among the last three trials. (The first is a dry run to ensure that the file is cached in memory.)


Terminal        time cat ChangeLog-2.6.23
-----------------------------------------
xfce4-terminal  11.109
gnome-terminal  11.022
terminator      10.878
xterm           7.320
konsole         3.191
rxvt            2.983

I was rather expecting rxvt to win since it's widely regarded as the minimalist terminal, but Konsole was a surprise. It beats even xterm by a large margin. Like KDE, Konsole is almost certainly written in C++, widely regarded as slower than C which is what makes these results pretty interesting. It's also noteworthy that the xfce4 terminal is right on par with the Gnome terminal when XFCE is supposed to be more lightweight than Gnome. (And probably is, overall.) Based on these figures, one could speculate that terminator, xfce4-terminal, and gnome-terminal are all based on similar code or libraries.

And finally, just in case you skipped the part above where I said how poorly this "benchmark" was really constructed, I want to emphasize it again: This benchmark is completely unscientific. This is how these terminals did on my computer. You may get a different (even perhaps contradictory) set of results if you run them on your computer. Nevertheless, I'm fairly confident that the results here are representative of what most people will see.

Ghetto-sistor

2008-10-24T02:12:00.006-04:00

This, my friends, is what is called a "ghetto-sistor."

It's what you get when you need a 1K ohm resistor, but don't have a new one handy, and instead have to settle for ripping one out of an old telephone before you realize that one leg is going to be too short so you solder a piece of solid scrap wire onto it so it's the right length.

Ghetto-sistor. GET IT?

Sheesh, you have no sense of humor.

It's Alive!

2008-10-24T01:26:00.003-04:00

And by god, it is blinky.

Well, heck dang.

2008-10-13T21:14:00.001-04:00

Seeing as everyone else in the known universe has a blog now, I figured it was high time I jumped on the bandwagon. Technically, I was blogging before it was cool, but ye olde archive machine doesn't have too many of my pages from way back when.

Because I'm picky, I'd like to write my own backend for the site and host it on my own server as I don't exactly trust teh Googles. But writing that is going to take a small eternity, most likely, so this will have to do for now.