Display an image from stdout [ubuntu]

To send bitcoin from my mobile phone to a plaintext address on my laptop, I like to use QR codes. At first I used websites like qrstuff.com to generate the QR codes, but leaving the command line just to generate and display an image felt like too much of a hassle to me. Then I started using the feh and qrencode utilities in Ubuntu’s package repositories, but that involved creating an image file on my hard drive and then deleting it afterward, which felt messy. So I found a quick and easy way to generate and display a QR code from the command line without creating an intermediate file.

The setup:

sudo apt-get install qrencode giblib-dev libimlib2 libcurl4-openssl-dev libpng-dev libX11-dev libXinerama-dev

git clone https://github.com/derf/feh.git

cd feh

make

sudo make install

To use the new qrencode and feh combo to, for example, display the QR code for Andreas Antonopoulos’ Dorian Nakamoto assistance fund:

qrencode 1Dorian4RoXcnBv9hnQ4Y2C1an6NJ4UrjX -o - | feh -

In the example above, the -o option on qrencode specifies where the generated image should be output, and passing - to that option specifies that it should be sent to stdout. Likewise, the - passed to feh specifies that feh should read the image from stdin.

Finally, because I am extraordinarily lazy, I created a function in my ~/.bashrc file to save my future self a few extra keystrokes:

function qrcode () {
    qrencode $@ -o - | feh -;
}

Which turns the previous example into the slightly more succinct:

qrcode 1Dorian4RoXcnBv9hnQ4Y2C1an6NJ4UrjX

Things coming

Since my last post, I’ve decided that I really ought to do as much as possible with other people’s code. It seems to me that “Don’t Repeat Other People” ought to be considered almost as important as “Don’t Repeat Yourself.”

The fact is that WordPress has been a quite serviceable platform for me so far, and I don’t really want to expend a bunch of effort writing a whole new blogging platform. I don’t have much reason for doing so, to be honest, and I will probably be of more use to more people if I build my new platform using other people’s components as much as possible

At present, I’m of the mind to set up some sort of pub-sub server and to use that to loosely join a bunch of different software platforms into a symbiotic ecosystem of parts.

Things to come

I’m planning on overhauling this website and moving to a custom platform. Fortune.js would be perfect for the kind of minimalism I’m looking for at the moment, but we’ll see what I settle on. I find the Indie Web very intriguing at the moment.

De-obfuscating a botnet infection

Last night I was working on a PHP website for a client when I stumbled upon this line of code whitespaced way off the screen on the first line of a few of our files: https://gist.github.com/ryepdx/5016290

That looked pretty suspicious to me, so I googled “$GLOBALS['QQOO']” and the only result that came up was this Pastebin: http://pastebin.com/71nwAsj6

Definitely didn’t like the look of that, so I grepped the rest of our files and found the same code embedded in the same way on four more files. I removed it from all of them, re-uploaded the cleaned files, and got to work figuring out what this code was doing.

I copied the code, commented out the call to “$Ill11I1lI” at the end of “function Q0QQOOQO,” and echoed out $Ill11I1lI. This gave me “preg_replace.” A little research revealed that the “/e” flag in preg_replace makes preg_replace automatically eval the result. (This seems like a spectacularly bad design decision to me, but that’s just my opinion.)

So I echoed “Q0QQOOQO(710, 2563)” to see what code they were trying to make preg_replace run. I got back: https://gist.github.com/ryepdx/5016310

More preg_replace eval’ing, so I commented out the call to “$Q00Q0QOQQ” and echoed IlI11lll(721, 2563). I got back a pretty sizable string of code, which I formatted and then took the liberty of renaming what was pretty obviously a “get string” function to “get_string.” That gave me this code: https://gist.github.com/ryepdx/5015935

I then did a search for all calls to “get_string” and isolated them to one line each, then used a little regex-fu to generate this PHP file: https://gist.github.com/ryepdx/5015963

And then I used the output from that PHP script to create a Python script (I realize I could have just used PHP, but my Python-fu is stronger) to find and replace all the calls to “get_string” with their actual returned values: https://gist.github.com/ryepdx/5015978

That got me a file I could actually read somewhat. I started working my way down the file, renaming variables like “$QO0OQO” and “$IIl11I” to things like “$curl” and “$curl_init.” After working at it like a crossword puzzle (“What’s a name for the third parameter in fsockopen?”) for around half an hour, I was finally treated to its final, naked, legible form (which I then filled with helpful comments): https://gist.github.com/ryepdx/5016252

Basically this code allows the botnet owners to execute arbitrary PHP code on the infected server. Not a very friendly piece of code!

TL;DR: Found obfuscated botnet code on a client’s server and spent about an hour and a half making it readable: https://gist.github.com/ryepdx/5016252

Building a web crawler for extracting GPS data from JPGs (part 3)

At this point things are going to pick up a bit. I wrote a Python class to handle reading GPS data from a JPEG using what I learned from walking through the file with the EXIF spec and the Fetid Cascade example at hand. I’ve posted the Python class on GitHub as part of the gps-exif-webcrawler repository I started for this project. I haven’t had as much time to spend honing my Python skills as I used to have, so there are probably a few things about it that could stand some tightening up. (If you see anything, please drop me a line. I’m always interested in improving the quality of my code.)

Really though, you need to look at the class in that GitHub repository. I’ll sum it up for you here, but I’m not about to reproduce and explain it line-by-line. That would be tedious for both of us.

The class was written with a bufferless stream reader in mind. It never requests data at a position the stream reader has already passed. The only small difficulty with that approach was the fact that all IFD tags are 10 bytes long, with 4 bytes allocated for the tag’s value. If the tag’s value is longer than 4 bytes, it stores an offset pointing to the beginning of the tag’s actual value in the 4 bytes it’s allocated. The offset can be anywhere in the APP1 layer after the IFD tags have ended. I got past that by keeping a dictionary of tag metadata tuples keyed by offset and then sorting it by its keys. (Like I said, it was a small difficulty.)

The only thing that bothers me about that approach is my use of tuples. I may change those to simple objects or dictionaries sometime today. I already had one incident in development where I got the values out of order while unpacking them and I imagine that will keep happening if this code changes at all in the future.

There was also a problem with the fact that ASCII values are stored as null-terminated strings, with the null included in the tag’s value count. Each character in the string is considered a single value. “Foo” would have a value count of 4, since there’s a null at the end. While that is perfect for C-based EXIF readers, it meant that each of the strings I read in with Python looked ["l", "i", "k", "e", " ", "t", "h", "i", "s", ".", "\x00"]

So ASCII characters were a definite special case and had to be handled differently from everything else, which detracted from the code’s generic-ness. But sometimes these things can’t be avoided, I guess.

There are also alot of “magic values” in the code. Realistically, I may or may not change that in the future. It’s probably something I’ll give a go at some point because I find it embarassing.

The last thing that really bothers me is the length of the read_gps function. Minus the pydoc at the beginning, it’s 132 lines long. Most of that is caused by the fact that I only wanted one function in the class modifying the stream. I figured it would make the code a lot more maintainable if there weren’t five or six methods all changing the stream reader’s position. I’m not sure if I made the right trade-off or not.

Despite these shortcomings, I did comment the code heavily, include pydoc-style summaries of each function, and include a simple unit test along with a JPEG I constructed for testing. Hopefully those will be enough to help you gain an even deeper understanding of EXIF data and help you along in your own projects!

Next up: putting together a lightweight HTTP stream reader and constructing a simple webcrawler to grab JPEG URLs from some source, feed them into our ExifGPSReader, and save the resulting values.

(To be continued…)

Building a web crawler for extracting GPS data from JPGs (part 2)

Recap:

>>> f = open("gps_exif.jpg", "rb")
>>> f.seek(20)
>>> f.read(10)
'\xff\xe1#\x86Exif\x00\x00'

 Big-endian, little-endian:

The endianness of the JPEG is determined by the four bytes following the APP1 opening headers. ‘II\x2a\x00′ stands for Intel, whose CPUs are little-endian, and ‘MM\x00\x2a’ stands for Motorola, whose 680×0 CPUs are big-endian. (Source)

>>> f.read(4)
'MM\x00*'

Awesome, we have a big-endian file on our hands. So the higher-order byte will come first in this encoding. Concretely, this means that the number 256 will be encoded as ‘\x01\x00′ instead of ‘\x00\x10′ (as it would be in little-endian encoding).

“So how do I deal with endianness,” you ask? Here are a couple functions I totally copied from an example script on fetidcascade.com:

  def s2n_motorola(self, str):
    x = 0
    for c in str:
      x = (x << 8) | ord(c)
    return x

  def s2n_intel(self, str):
    x = 0
    y = 0
    for c in str:
      x = x | (ord(c) << y)
      y = y + 8
    return x

These will translate the hex values for you depending on whether your file is big-endian or little-endian.

All the offsets for the IFDs will be in terms of bytes from the end of the null-terminated “Exif” string we saw earlier. For purposes of our exploration, we’ll want to keep track of where this point is. Otherwise we won’t be able to ‘seek’ to the appropriate positions. (An alternative would be to read the rest of the APP1 segment into a string and simply use array index notation to grab the values we need, but I’m writing this with an eye to processing values as they are streamed so we can download only the data we need and no more.)

Since we just read four bytes past the end of the “Exif” string, we’ll need to back up by four bytes to get our start position.

>>> f.tell()
34L
>>> f.seek(30)
>>> start = f.tell()
>>> start
30L

Now we skip the four “endianness” bytes and get the four bytes that follow. This is the offset from the end of the “Exif” string at which the fields begin.

>>> f.seek(start+4)
>>> f.read(4)
'\x00\x00\x00\x08'

So…

>>> f.seek(start+8)

…puts us at the beginning of the IFDs. Almost. Read the next two bytes to get the total number of IFDs the JPEG contains and then you will be up against the fields themselves.

>>> f.read(2)
'\x00\x0b'

Our JPEG contains 11 IFDs.

Alright, before we dive into getting values from these fields, it’s important to note that, much like directories, IFDs can be recursive. That is to say, IFDs can contain more IFDs. That 11 we read up there is just the number of top-level IFDs. Within those IFDs, there may be more IFDs, just as a directory can contain more directories.

Each IFD begins with a two-byte code indicating what kind of IFD it is. (IFDs that contain other IFDs start with the code ‘\x87\x69′, FYI.) So we read the next two bytes to find out what type of IFD comes first:

>>> f.read(2)
 '\x01\x0f'

Reading from the handy-dandy EXIF spec PDF, I see that this is the IFD field corresponding to “Make.” (I imagine this field holds the make of the camera that took the picture.)

The next two bytes contain a code corresponding to the IFD’s datatype:

>>> f.read(2)
 '\x00\x02'

Okay, so looking at the spec I see that this corresponds to ASCII data. I’ve reproduced the datatype table below for your convenience.

1 = BYTE: An 8-bit unsigned integer.
2 = ASCII: An 8-bit byte containing one 7-bit ASCII code. The final byte is terminated with NULL.
3 = SHORT: A 16-bit (2-byte) unsigned integer.
4 = LONG: A 32-bit (4-byte) unsigned integer.
5 = RATIONAL: Two LONGs. The first LONG is the numerator and the second LONG expresses the denominator.
7 = UNDEFINED: An 8-bit byte that can take any value depending on the field definition.
9 = SLONG: A 32-bit (4-byte) signed integer (2′s complement notation).
10 = SRATIONAL: Two SLONGs. The first SLONG is the numerator and the second SLONG is the denominator.

The next four bytes indicate how many values of that type comprise this IFD.

>>> f.read(4)
'\x00\x00\x00\x06'

So it looks like the Make IFD is comprised of six one-byte values. Since this is greater than four bytes, the next four bytes will give us the offset where the values begin. If this IFD’s values were four bytes or less, the values would be stored here. (Note that values less than four bytes are right-padded, so if the IFD’s value was 0x0a 0×34 and two bytes long, it would be recorded as ‘\x0a\x34\x00\x00′.)

>>> f.read(4)
'\x00\x00\x00\x92'

So we go to that offset and get the six bytes there…

>>> f.seek(start+0x92)
>>> f.read(6)
'Apple\x00'

This picture was taken with an Apple device!

Okay, so obviously we know now how to, in general, find our EXIF fields and get their values. You can refer to the spec for more information if you need it.

Now that we know how to grab EXIF data on the fly from an image, we can start building our webcrawler. But that’ll have to wait until next time.

(To be continued…)

Building a web crawler for extracting GPS data from JPGs (part 1)

Motivation:

A few weeks ago I attended a talk about the ways our mobile devices can compromise our privacy without us realizing it. One of the things mentioned was the GPS data automatically embedded in the EXIF data of pictures taken with GPS-enabled devices such as smartphones.

I had heard about this already, but on this occasion it entered my head that it might be fun to crawl the web for pictures containing GPS and time data and build a database of those pictures. With such a database, I could build an animated map showing the pictures being taken over the course of any given timeframe. In other words, I could choose to display a month of pictures over the course of half an hour and watch as all the pictures taken that month pop up over the places they were taken on the map, fading after a few seconds to keep the map from getting too crowded. Or I could just have all the pictures taken a certain day pinned on the map, available for me to browse and examine, giving me a general feeling for that day as seen through the eyes of strangers. Or I could do the same without any time constraints and see a place as it has been photographed by many different people. Not much practical value, I suppose, but I find a certain beauty in it.

To build such a crawler, I would need to examine a lot of JPGs as quickly as possible. All the EXIF libraries I found required opening the whole image first, which seemed at cross-purposes to my particular need. The GPS data is usually contained within the first few hundred bytes of a file, so why should I have to download the entirety of a multi-megabyte file only to find out that there never was any GPS data in it at all?

So I decided to roll my own in Python, with some thought to porting it to C if I ended up wanting even more speed. (Or if I ended up wanting to brush up on my C skills.)

Caveat:

The file I’m working with is a JFIF JPEG. These are identical to “regular” JPEGs except for the addition of an APP0 layer between the SOI and APP1 layers. The APP0 layer begins with the 5-byte, null-terminated “JFIF” string.

File Structure:

Rather than working with the official EXIF 2.3 specification, I decided to work with a slightly more readable PDF of the EXIF 2.2 specification supplied by EXIF.org. I also found the Syntax and Structure section of the JPEG Wikipedia article to be very useful.

The segments we are concerned with are presented in order of their appearance.

  1. SOI – Start of Image. 2 bytes long, offset 0. Indicates what type of JPEG this is.
  2. APPn – Application-specific data. Variable length and offset. Marked by the pair of bytes FF En. The 2 byte marker is then followed by 2 more bytes indicating the length of the segment. These 2 bytes include themselves in the length, so if you have just read 16 as the segment length, reading the next 14 bytes will read out the rest of the segment.

EXIF data is contained within the APP1 and APP2 segments. The data we’re after, the GPS data, is in APP1. Data within the EXIF segments are broken up into Image File Directories (IFDs) that contain image meta-data. The pointer to the beginning of the GPS IFD is marked with 0×8825.

Walking Through a JPEG:

Here’s a transcript from a Python shell session demonstrating what I’m talking about:

>>> # First, we import the os module. This will be useful to us later.
... import os
>>> # First we open the JPEG for reading as a binary file.
... f = open('gps_exif.jpg', 'rb')
>>> f.read(2)
'\xff\xd8'
>>> # ^ The SOI marker
>>> f.read(2)
'\xff\xe0'
>>> # ^ The APP0 marker
>>> f.read(2)
'\x00\x10'
>>> # ^ This means the APP0 segment is 16 bytes long,
... # including the 2 bytes we just read.
... # So we skip ahead by 14 bytes to skip the APP0 segment...
... f.seek(14, os.SEEK_CUR)
>>> f.read(2)
'\xff\xe1'
>>> # ^ The APP1 marker we were looking for!
... 
>>> f.read(2)
'#\x86'
>>> # ^ The length of the APP1 segment.
... # Note that the ASCII code for # is 0x23. When reading a file like this, sometimes
... # Python prints the ASCII character corresponding to the hex value we're looking
... # at. So the length of the APP1 segment is 0x23 0x86, or 9094, bytes.
... f.tell()
24L
>>> # Since we're currently at position 24, and since we are already 2 bytes
... # into the APP1 segment, this means that the SOI, APP0, and APP1 segments
... # comprise the first 9116 bytes of the file. (9094 + 24 - 2 = 9116)
... # If we were feeling lazy, we could actually just pass those bytes on to an
... # EXIF library and it should have no problem parsing our data for us.
...
>>> f.seek(9116)
>>> f.read(2)
'\xff\xdb'
>>> # ^ This marks the beginning of a quantization table.
... # Demonstrates that we've done our math correctly and reached the end of the
... # APP1 segment.

Finding the GPS data in the APP1 segment:

As noted in the section above, the most developer-efficient way to proceed at this point would be to just take the bytes from the beginning of the file through the end of the APP1 segment and pass them on to an EXIF library. This would save us quite a bit of developer time. We could even spawn a thread to take care of this parsing and start downloading the next file immediately to improve efficiency. But for the sake of learning, and because it would be even more efficient to stop downloading the APP1 header as soon as we get the data we’re interested in, we’re going to press on and figure out how to recognize and extract the GPS data as soon as we get it.

But I have work due in the classes I’m taking, so that will have to wait for now.

(To be continued…)

Setting up Python 2.7 and PyPI (pip) on an iPod Touch

I just wanted to harvest tweets over a period of about a month or two, but I didn’t want to pay for all that processor time on an EC2 instance. I tried running my Python script on my shared hosting account, but the admins kept killing the ‘screen’ containing it. I would have used my desktop computer, but I can’t sleep with it on. And my laptop couldn’t provide the uptime I needed.

First thought: Raspberry Pi. But getting one looked like a rather lengthy process, and I didn’t want to have to wait weeks or months to proceed with my project.

I happened to have a 2nd generation iPod Touch running iOS 4.2.1 that was just collecting dust. After trying and failing to get Linux on it, after wrangling with Darwin and reading walkthroughs, I’m ready to share the steps that eventually got me there. Hopefully they’ll help you.

How I did it (minus the dead ends and false starts)

  1. Downloaded the latest version of Redsn0w.
  2. Downloaded the iPod Touch 2G 4.2.1 firmware. (Thanks to k1ttyrain for the file.)
  3. Ran Redsn0w and went through the jailbreak process.
  4. Used Cydia to install OpenSSH, wget, BerkeleyDB, and SQLite 3. (I also installed screen so I could run my script and leave it running after ending my SSH session, but you don’t have to.)
  5. SSHed into my iPod.
  6. Changed my root password with passwd.
  7. wget http://yangapp.googlecode.com/files/python_2.7.2-5_iphoneos-arm.deb
  8. dpkg -i python_2.7.2-5_iphoneos-arm.deb
  9. wget --no-check-certificate https://github.com/pypa/virtualenv/raw/develop/virtualenv.py
  10. python virtualenv.py env

At that point I used the copy of pip in the env/bin directory that virtualenv created to install the modules I needed and used the copy of python in the same env/bin directory to run my script in a screen I detached. Now I’ve got a quiet little iPod Touch plugged into the wall, slurping down tweets and saving them into date-stamped files!

Edit: Added mention of SQLite 3 and BerkeleyDB in step four.

Prolog in Python (pt. 2)

Note: I’ve uploaded a basic barebones project based on this series to a GitHub repository for your convenience.

Last time we figured out how to use SWI-Prolog routines in Python. Now we will learn how to connect SWI-Prolog to MySQL and use it to extend Prolog’s built-in database.

Connecting and Disconnecting

The odbc_connect predicate will connect you to your MySQL database. Here I encapsulate it in another predicate for ease of use throughout the rest of my code.

connect :- odbc_connect('mysql.your_database_name', _,
[ user(your_username),
alias(database_symbol),
password(your_password),
open(once)
]).

Obviously you will need to fill in your own database’s name and your own username and password. Also, replace database_symbol with the name of the global symbol you want to bind your database connection to.

The following predicate will disconnect you from your database:

% Disconnects from the database.
disconnect :- odbc_disconnect(database_symbol).

Running Queries

Now that we can open and close a connection to the database, we can start passing raw queries to it and getting back values. Here’s an example of pulling data from a row lifted directly from the GitHub project mentioned above. Assume that we used db for our database_symbol.

% Extracts the values from a row in the Value table.
parse_value(Row, Thing1, Price, Thing2, Inferred) :- row(Thing1, Price, Thing2, Inferred) = Row.

% Finds the value of Thing1 in terms of Thing2.
generic_value(Thing1, Price, Thing2, Inferred) :-
connect,
odbc_prepare(db, 'select * from value where ((thing1 = ? and thing2 = ?) or (thing2 = ? and thing1 = ?)) and inferred = ?', [default, default, default, default, integer], Statement, [fetch(fetch), types([atom, float, atom, integer])]),
odbc_execute(Statement, [Thing1, Thing2, Thing2, Thing1, Inferred]),
odbc_fetch(Statement, Row, next),
parse_value(Row, _, Price, _, _),
disconnect.

The first predicate, parse_value, does a little bit of pattern matching to split the returned row out into its constituent columns.

The second predicate, generic_value, does a few things:

1. It opens the database connection.

2. It creates a prepared query with odbc_prepare. There are a lot of parameters there, so I’ll explain them a bit.

  • First is global symbol we defined in our connect predicate. In this case I used db, though you can use whatever you replaced database_symbol with.
  • Second is the actual SQL query with ? marking where we want our query parameters to go.
  • Third is a list of types for the values we’ll be binding to our query parameters.
  • Fourth is the symbol we want to bind the prepared statement to. In this case I bound it to Statement.
  • Fifth is an optional parameter containing a list of options for our query. fetch describes how I want the results to be returned: by default Prolog will grab all the rows returned. By passing in fetch(fetch), I specify that I want only the first row to be returned. types contains a list of types that I want the returned columns to be automatically coerced to. By default Prolog will try to guess.

3. It executes the prepared statement with odbc_execute, which takes a prepared statement and a list of values.

4. It fetches the returned row with odbc_fetch, which  takes a prepared statement, binds the returned row to a specified symbol (in this case I used Row), and takes a parameter specifying which row should be returned. In this case I specified that the “next” row should be returned.

5. It binds the second column returned to Price, discarding the values in the other columns.

6. It closes the database connection.

Simple, no? Inserting and deleting data are similarly easy. I’ll let you figure out these examples for yourself. If you have questions, feel free to leave a comment.

% Sets the value of Thing1 in terms of Thing2.
% If inferred is not specified, defaults to true.
set_value(Thing1, Price, Thing2) :- set_value(Thing1, Price, Thing2, 1).
set_value(Thing1, RPrice, Thing2, Inferred) :-
Price is float(RPrice),
connect,
odbc_prepare(db, 'insert into value(thing1, price, thing2, inferred) values (?, ?, ?, ?)', [default, float > decimal, default, integer], Statement),
odbc_execute(Statement, [Thing1, Price, Thing2, Inferred]),
disconnect.

% Clears all inferred values from the database.
clear_inferrences :-
connect,
odbc_query(db, 'delete from value where inferred'),
disconnect.

Another note: Did you notice that sneaky float > decimal action up there? All that means is that the decimal value that Prolog will provide when the statement is executed corresponds to a float value in the MySQL table.