The Costs of Thread Synchronization

Written 2012-12-04

Recently I implemented a set of simple programs that use Monte-Carlo simulations to estimate the value of PI. I ran it on a few different machines to see the cost of different thread syncronization methods on these architectures.

The Algorithm

This algorithm approximates the value of pi with a Monte-Carlo simulation. Effectively, this works by pseudo-randomly placing points in a domain, evaluating them, and tallying results. In our case, the domain is the set of x-y pairs from (0,0) to (1,1). The evaluation function checks if the distance from the point to (0,0) is greater than 1. The tallying consists of two counters - points farther than 1 away from (0.0), and points closer than 1 to (0,0).

The final approximation is the ratio of points in the circle to total points, and must be scaled by 4 to match a simulation of a full circle and square centered on (0,0). Since we can approximate the ratio of areas via the Monte-Carlo method, we solve for PI and approximate PI via the adjusted area ratios.

The Synchronization Options

Binary Semaphore/Mutex

This is a standard mutex operation - only a single thread at a time can control it, and any thread who tries to reserve it when it is already reserved will block until it is available. These are fairly heavyweight structures that usually involve contacting the operating system for resolution when multiple threads want the mutex.

Atomic Operations

Atomic operations have the ability to read/modify/write all in one instruction, or the ability to rollback an operation that failed to acheive atomicity, and report it to the user. In this way, two threads can share a single variable in memory, and each thread can modify it with atomic operations. However, this still doesn't scale to large numbers of threads due to cache-line sharing and coherency-tracking.

Thread-based storage in a shared area

This mode allocates all thread-control structures and counters in a linear block of memory. Each thread has its own value to increment, but because they're near each other in memory, there may be some contention for cache-lines. Had I though about it earlier, I'd've placed all the counters in one linear block and the thread-control structures in another to try and illustrate the effect.

Thread-based storage local to each thread

This mode uses a variable unique to each thread-function. This is nearly the same as above, but effectively places the variables in areas that will not be vulnerable to cache-line sharing.

The Machines

Each plot shows execution time(Y) in seconds versus to the number of threads used(X). The number of computations is fixed at 1 Million, and the work is divided as evenly as possible between the threads. Additionally, each run was run a single time, so the graphs are somewhat noisy.

Intel CoreDuo-T2300 running Linux 3.2

The first box is my old CoreDuo powered Thinkpad T60 laptop. Semaphores are pretty unscalable. Atomics scale better, but are still surprisingly expensive. Thread-local options work much better, and with a good number of threads, it is possible to beat the single-threaded performance(untrue for atomics and semaphores).

Intel Core2Duo-T7600 running Linux 3.2

The next box is my current Core2Duo powered Thinkpad T60P laptop. The results are similar to the CoreDuo results, but the Core2Duo is quite a bit faster(as expected).

Intel dual Xeon 5650 running Linux 3.2

This box is my webserver and VM host. It has two six-core chips and hyperthreading, so it is expected that the runtime of scalable implementations of the benchmark will decrease in execution time up to or around using 24 threads. However, I couldn't take down the minecraft VMs that are also on this box, so I won't put as much faith in these numbers. Semaphores were quite slow and leveled off well before we reached 24 threads. Atomics scaled better here than on the above two intel chips, which I find suprising - I expected that if any threads ended up on the second chip, this would be much slower. However, accumulating the results local to each thread was the definite winner.

Genpfault pointed out I should play with numactl on this box to enforce splitting over sockets or locality to a single socket.

Texas Instruments MicroSPARC running OpenBSD 5.1

The next graph is from my 50MHz MicroSPARC running OpenBSD. This one is interesting because it performs surprisingly well. However, since Compare-and-Swap was only implemented with SPARCv9, and MicroSPARC is a SPARCv8, there were no atomic operations to test. Additionally, since this is a single-core machine, flat curves are expected. Lastly, the performance for the semaphore mode isn't bad when using only one thread, so uncontended semaphores aren't too expensive on this box, but contended ones ( thread counts 2-100 ) are terrible.

Intel Atom N455 running OpenBSD 5.2

This pair of graphs is from Ax0n'a single-core, hyperthreaded netbook running OpenBSD 5.2. The first graph should seem familiar - it seems OpenBSD semaphores are expensive. I'm not sure what the dip around 7 threads is; it probably warrants running the benchmark a few more times.

Next is the same data with the semaphore plot removed, as it made the scale awful.

Interestingly, once you discount outliers, atomic operations on this platform appear to be no more expensive than using independent threads. I suspect this is due to the fact that the N455 is really a single-core CPU - with only a single L1 cache, there won't be any cacheline-sharing.

IBM PowerPC Quad-G5 running OSX 10.5

This pair of graphs is from CCCKC's Mac-Pro. Similar to OpenBSD, semaphore performance is slow and skews the scale so that you can't see anything else.

Here's the same plot but with semaphores removed and scale reset.

As expected, atomic operations are slower than independent threads on this platform, but similarly so to other multi-core devices. Atomics are still much faster than semaphores.

Conclusions

Atomic operations aren't a magic bullet

On the platforms that support them, atomic operations can be much better than semaphores. However, even less thread communication scales even better.

Semaphore cost can vary widely over different platforms

At first I thought that OpenBSD/SPARC32 just had slow semaphores. However, after running the same benchmark on OSX/PPC-G5 and OpenBSD/Intel, it seems that Linux just has really cheap semaphores. Additionally, on all platforms, the semaphore benchmark ran best with only a single thread. Lastly, on every platform, the semaphore benchmark was the worst of the benchmarks.

Often times, a single-thread can do pretty well

On CoreDuo and Core2Duo a single thread beat any number of threads using atomics or semaphores. On Xeon, a single thread was equivalent to all cores when using atomics, and was better than using semaphores.

Costs of synchronization should be mitigatable

Costs should be mitigatable by doing more work between synchronization or communication with other threads. The work being done by this benchmark is quite simple. It consists of two calls to rand_r(), computing distance-squared, a comparison, and an action based on that comparison. As the complexity of the work done goes up, the ratio of the time spent synchronizing vs time spend working should go down quite a bit.

Source Code

Source code is available at https://github.com/rsaxvc/monte-carlo-benchmark. Slight modifications were needed for OSX, whose atomic builtins are named differently than GCC's default ones, as well as OpenBSD, which needed changes for the path for bash.

A New, Future-Proofable Archive Format

Written 2012-11-30

I've been dealing with different archive formats recently, and I'm a little bummed that none of them do everything I want. So I've been kicking around the idea of an archive format that is scalable to future algorithms, can be optimized at pack-time for what is stored, and can be updated with more advanced decompression algorithms without needing to update the client. What I've come up with so far is embedding a small subset of a programming language into the archive itself.

Examples:

Text files can be compressed with the same LZxx algorithms today

WAV files can be compressed as FLAC files, and decompressed on the fly as needed

Bitmaps can be compressed any number of ways(PNG/JP2/...)

One hurdle is making the decompression algorithm small enough for this to be effective. That said, for small files, a smaller but less efficient decompressor may be included to keep the total filesize still small.

Future proofing is intrinsic - when an archive is packed, it contains the algorithms used to compress it, so any new compression algorithms will work fine on old implementations.

Security concerns - each archive contains one or more programs running on the user's machine. This is where exposing only a subset of normal OS bindings comes in - the program gets access to read the compressed data block, ability to read/free memory, the ability to write to the output buffer, and of course entry/exit points for open/close/read/seek.

Blagr

Written 2012-09-30

This is a simple blogging platform I wrote to replace my dependence on Movable Type. I suppose it will require me to actually learn some css/html.

Previously I tried updating my Movable Type install, but that wouldn't work.

So I tried installing the new version and uploading the backup to it. But that didn't work.

Next I tried a clean install of the old version, uploading the backup, and upgrading. But that didn't work

Finally I looked at the documentation online, but it was plastered with spam comments.

I really don't like being unable to get my data back in a nice parsable format.

So that's how this started. Blagr uses a simple plain-text format, one file per post. Blagr really isn't a blog or blog-engine by itself - it is only a transfer format. You can see the source to all my posts in the blog/input directory.

The format is quite simple - there are two parts, header and body, as well as a seperator between them: --- , that's three dashes in a row on one line. The header itself is a list of key-value pairs. Each line in the header has the key, then a colon, then the value. So you want to tag a post with pickles? Add Tag:Pickles to the header. The post body is XHMTL

The blog you are reading right now is statically compiled by a small python program that only knows how to read .blagr files, sort them, and output html.

Advantages of Reader/Writer locks on small single-core systems.

Written 2012-09-23

Most commonly, reader/writer locks are applied to larger systems where it can decrease contention when there are many threads trying to read. In my case there were actually very few threads, but a reader/writer lock still proved advantageous. Let us examine what can happen when two threads need to read, using both a standard lock, and a reader/writer lock.

Premise: Thread A is high-priority, thread B is low-priority, and the OS provides hard-priority-scheduling. Only these two threads will run in the period of time we're interested in.

Here's what could happen using a single standard lock.

An event occurs, causing thread B to run.
Thread B takes the lock and begins reading
An event occurs, causing thread A to run.
A context-switch occurs due to thread-priority. Thread A is now running.
Thread A attempts to read but is blocked.
A context-switch occurs, allowing thread B to release the lock.
A context-switch occurs due to thread-priority. Thread A is now running.
Thread A takes the lock, reads, unlocks, and blocks elsewhere.
A Context switch occurs due to only one ready-to-run thread.
Thread B continues until it blocks elsewhere.

But now, lets look at what happens with a reader/writer lock instead, with similar event timing.

An event occurs, causing thread B to run.
Thread B takes the lock and begins reading
An event occurs, causing thread A to run.
A context-switch occurs due to thread-priority. Thread A is now running.
Thread A also takes the lock, reads, and releases the lock. It then blocks elsewhere.
Thread B finishes reading, releases the lock, and blocks elsewhere.

First we notice that using the reader/writer lock avoids a pair of unceccesary context-switches. This is nice, but the advantage is heavily architecture/OS dependent.

Even assuming the context-switches are cheap, there's another advantage to the reader/writer lock in this situation. The advantage is system latency - by using the reader/writer lock instead of a simple lock, high-priority threads no longer have to wait for low-priority thread to release the lock. The ability to allow multiple threads to enter the critical section is the same reason it is often chosen in high-contention environments, but it also usefully provides reduced latency when experiencing any contention.

Hooking ARM IRQs with FIQs

Written 2012-09-02

I stumbled on to an odd little trick recently. On the ARM architecture, there are two types of interrupts- Fast Interrupts(FIQ) and standard/slow interrupts(IRQ). I was writing an FIQ, but needed some way to test it. I figured I would re-use an existing external IRQ line, but I didn't really want to prevent the interrupt from reaching the OS. Luckily for me, it turns out it is possible to install an FIQ that is triggered by an existing IRQ, without breaking the behaviour of the existing IRQ.

On the processor I was using, there is a bit-vector that controls if an interrupt is actually an FIQ. If the interrupt is marked as an FIQ interrupt, the execution will jump FIQ vector. It is important to note that you can still keep an interrupt vector in place.

Most non-nested interrupts look something like this:

ACK interrupt

Do Something

SUBS PC, R14, #4 ;Return from interrupt/fiq and swap back the PSR(Program Status Register)

My FIQ looks something like this:

Do Something

Clear FIQ bit

SUBS PC, R14, #4

You'll note I didn't ACK the interrupt. That's because the slow interrupt will still run. When the external stimulus occurs, the CPU stops current processing(or may block a little while if interrupts are disabled), and executes the FIQ. The FIQ then marks the interrupts as not an FIQ. The FIQ keeps running until it returns. When it returns, the interrupt controller catches that there is still an outstanding interrupt, and then runs the standard interrupts handler. Because we cleared the FIQ bit in the FIQ, we should probably re-enable it after the IRQ completes.

I think there are several useful things you could do with this. You could measure interrupt delay between the event and the IRQ response by installing an FIQ that reads a timer, and reading it again in the OS after the IRQ completes. You could count the number of IRQ events in the system overall. You could test adding latency to existing IRQs.

Kraken-II: A small virtual machine.

Written 2012-08-23

Long ago, in the age of state-collegiate education, I had a class with Ted McCracken. On the first day, he announced we would be designing a computer. Specifically, a small 8-bit RISC machine. By some strange chance one of the best teachers at the school was having his course reviewed by the state. This terrible maelstrom yielded my semester of CpE315. We would cover 8086, 80386, brush on PowerPC, serial busses, parallel busses, memory and timing, segmented memory, virtual memory, pipelining, and would touch on nearly everything I would come to understand years later about electronics from diode-transistor logic to cache architecture on multicore devices.

The machine was done piecemeal - each student was given a few parts of the design, and the basic design was completed. The next step of the course was programming the system. Dr McCracken graded by stepping through the programs on paper. I would often make a simple error when translating the assembly to machine code. Eventually, I wrote a small virtual-machine and assembler so I could test my work before I submitted it. I added instructions as I needed them.

Years later, and a few OS-installs too, I've lost the code to the original 'Kraken' architecture emulator. So this is a new simple architecture based on many of the things I learned in CpE315, with more years of experience. I've also started on an assembler that uses a grammar parser instead of getline as the main text processing construct.

https://github.com/rsaxvc/Kraken-II

libgraphoscope: a project in need of a newer name

Written 2012-05-03

I'll prefix this with; when I wrote it, I needed a name meaning graphing on an oscilloscope. Later I found it had been taken for some time. This all started when I saw this video:

So, I wrote a little library that can render out to the audio-card. Now I just need to write a sweet demo like the above video.

https://github.com/rsaxvc/libgraphoscope

It may require a little tuning in graphoscope_config.h for you particular soundcard.

Atari Flashback Controller Pinout and Frame

Written 2012-01-07

The previously disassembled Atari Flashback Controller has 5 important pins inside. When looking straight into the connector::

OUT(Orange,Top row, near right)
SCK(White,Top row, near left)
DO(Yellow,Top row, center)
GND(Black,Bottom row, near left
5V(Red,Bottom row, near right)
and 4 grey lines, two of which appear to be open, two are connected to each other, none of which hit the connector.

The data comes out of the OUT pin, and the bits are clocked by the SCK pin. The bitword(OUT pin), from 1 at t=0, goes:

Left Button
Right Button
Select
Pause
Up Stick
Down Stick
Left Stick
Right Stick

These are active low, meaning pushing the button makes the corresponding bit go from a one to a zero. I'd like to maybe make an arduino library for Pickin Duino

Atari Flashback Controller Teardown

Written 2012-01-03

Recently, I disassembled an Atari Flashback controller to see if it could be easily interfaced to an arduino for Pickin' Duino.

Once the case is open, you can easily see the side buttons are implemented with membrane switches, as is the joystick, and as are the front switches.

The board appears to be using some sort of clocked serial interface, probably SPI related.

I was hoping that the 9-pin interface was just a switch array, but there's a IC under an epoxy blob. It should be possible to learn the protocol with an oscilloscope.

Game:Pickin' Duino

Written 2012-01-02

Rachel and Jestin, of CCCKC fame, pointed me toward a library named arduino tv-out, which is a software modulator for NTSC and PAL video signals. Rachel wrote a game long ago, called Pickin' Sticks. Pickin' Duino is an implementation of Pickin Sticks on Arduino. The game itself is pretty simple - there is one player, and one other item on the screen, a bundle of sticks. Move your character to the sticks to gain a point. Oh no, more sticks appeared. Eventually I set the score to 'WIN' at some point when you've been playing far too long.

Tada! Also, I didn't have a controller to interface to the arduino, so instead I wrote a simple screen-saver like guidance system. Here's a long exposure:

And that's about it. Eventually I intend to add a real controller, but for now I'm going to use two potentiometers.

Project: pyGNU

Written 2012-01-02

pyGNU is a reimplementation of the GNU coreutils in Python. Python is a fun language, and happens to have a pretty nice OS abstraction package, named 'os'. When combined with the 'sys' package, most coreutils are pretty straightforward to implement in Python.

I thought this would be a simple way to learn a little more Python. But I've come to an interesting realization about my programming over time. When I was young, to solve a problem I'd choose a simple language, like basic on my Apple II. As I became older, I would do more projects in the language I knew best, C/C++. Now, I usually pick the language that involves the least amount of typing.

Python's for-iterator syntax, sets, lists, and preexisting packages make it a joy to write. Many of the pyGNU programs aren't quite done yet, but all the programs are usable. If you know python, feel free to add a few more.

Retraction: BSidesKC Samsung Presentation

Written 2012-01-02

On October 26, 2011, at BSidesKC, I gave a talk on how to remotely exploit Verizon and Sprint femtocells. It turns out I was wrong. My attack was based on some services running locally on the device. Early in my work with the device, I wrote a small script to help me boot the units. This included taking over the bootloader and setting up the kernel properly, as well as mounting filesystems and clearing iptables. The service I used to gain control of the device appears to be entirely unused, but is still running on the devices. However, iptables had blocked it entirely. Since I only probed my units over the VPN, I had note noticed that my testing environment changed the units I was examining. I apologize for crying wolf.

Exploring HP's FAT Filesystem

Written 2011-10-22

When high end HP printers receive a document, they may spool it to a hard disk temporarily. These drives use HP's own implementation of a FAT filesystem. The vendor string is "HpFAT1.0". I've identified the directory entry structure, and a few members inside of it. It goes a little something like this:

typedef packed struct

{

char filename[100];

uint16 some_interesting_value;

uint8 always_null_so_far_a[15];

uint8 attribyte;

uint8 always_null_so_far_b[3];

uint8 always_0x88_so_far;

uint16 start_of_file_in_clusters;

uint32 filesize;

};

I think the filename can be up to 100 characters, including period and extension. The attributes appear to be the same as FAT16, as do the last six bytes of the entry.

I have an example filesystem pulled from a printer, and can retrieve some files, but don't know what they mean. There are some 'MDT' files, 'MNF' files, and a '.DB' file. I should write a parser, and either extend the Linux kernel module for msdos to handle HPFAT, or copy and modify it into a new driver.

Some Numbers:

Deletion marker: 0xE5

Bytes/sector: 512

Sectors/cluster: 63

Reserved clusters: 13

Copies of FAT: 1

Max Dirents: 64

Type:0xF8-fixed disk

Sectors/FAT: 0xFB

Sectors/Track: 0x3F

Heads: 0x10

Hidden sector count: 0x0

555 Timer Schematics and Math

Written 2011-10-12

This is a followup on building a time-lapse shutter controller, now that I have time to draw schematics and write math

There are several stages to the design:

Power Supply
Power Usage
Frequency Range
Output Coupling

Power Supply

I wanted to drive the device with AA batteries, but since the 555 timer I used requires a minimum of 4.5 volts and maximum of 15, that would require four rechargeable AA batteries. Instead I decided to go with a 9-volt cell, as a single battery can drive the circuit until it is nearly empty. However, this does have implications for the rest of the circuit. In particular, the output stage must accept being driven by voltage in the range of 8 to 3 volts, as the 555 may differ between 1.5 and 2 volts from a high on the output pin to the input voltage, but if everything is designed to handle this, then I don't need to regulate the power supply

Power Usage

Since this circuit is driven by a transistor, there should be little power usage in the output stage, but the 555 timer can be designed to use too much power. To avoid this, I went with a higher-quality capacitor with lower leakage current, and large-valued potentiometers.

Frequency Range

The frequency range and duty cycle are controlled by ratios of resistances to capacitance C2. The formula for time on is TimeOn = [C] * [R1] * ln(2). The formula for time off is usually TimeOff = [C2] * [R2] * ln(2), but also note that I added R3 in series with R2 so that the total resistance there can never drop below the speed my camera can take photos. My complete formula for off would be [C2] * ([R2]+[R3]) * ln(2). I would suggest picking a physically small, high quality(low leakage) capacitor, in the range at least 10uF, and the pick potentiometers to meet the range of timing you want to hit.

Output Coupling

My circuit is tuned to efficiently drive my Canon 20d. Using a meter, the remote port was always about 10 milliamps shorted, and 3.3 volts open. Since the 555 outputs a voltage driven square wave, an NPN transistor can be added to convert a to a DC switch. I'd have to check the particular transistor used, but anything handling 15milliamps should do fine. Mine had a DC gain of 30->90, so I did all work assuming worst case hfe gain of 30. Keep in mind that the power supply section states a 4.5 volt supply is acceptable with 2.5 to 3 volts output from the 555. Also note that the NPN transistor will have some DC voltage drop over it, in my case 0.7 volts. The simplest method is to compute all calculations assuming smallest voltage to the 555, and pick the [R4] to match to ensure the transistor can trigger the camera, then go back and make sure the transistor can also deal with the current driven from the 555 when the 555 is fully powered and puts 8 volts out toward the transistor.

Picking Parts

555 timer - there are several different kinds, including high power, high efficiency, rail-to-rail voltage output, and CMOS. Mine was nothing special, and I think any will do. You do not need a high-speed one for camera work
Main capacitor C2 - this guy controls the range of frequencies the circuit will operate at. Also low-leakage is nice if you have it. Electrolytics can be sensitive to reverse voltage, so be careful about that
On timing potentiometer R1 - this resistor controls the exposure time, so make sure to pick one, when combined with C2, that does what you need.
Off timing potentiometer R2 - this resistor controls the time between exposures, so make sure to pick one, when combined with C2, that does what you need.
555 capacitor C1 - this is specific to your 555, and some list a range. 5pF to 22pF worked here for me, although 10pF was the 'ideal'. Some 555s do not need one.
diode D1 - just a standard diode, this application is low voltage; almost any will do. This part allows you to run the exposure(on time) less than the off time.
Transistor Q1 - most small signal NPN transistors will work. Needs to be able to sink the current from the camera remote port, which isn't much. I would pick higher gain over lower to try and save a little power, but I got mine out of a spare transistor bucket.
DC Battery V1 - I added a 9-volt dry cell connector, but forgot a power switch
power switch - not pictured, but do add this.
optional LEDs - I added an LED driven from the 555 output so I could see the delay before shooting. Also could add power LED.

Robot Cupcakes

Written 2011-10-11

What's this? A robot cupcake at work?

Oh no an entire army!

a Time-Lapse Shutter Release

Written 2011-10-09

I plan to post again with the schematics and the math, but today I finished:

This circuit connects to all my other remote shutter peripherals. I used a 555 timer to generate the timing, and two potentiometers - one for on-time/exposure, one for off-time. Add the times together to see the times between shots. I also used a transistor, as the 555 generates an active-high pulse, and the Canon releases use a current-short to take photos.

Inside the LinkStar ViaSat Satellite Modem

Written 2011-10-01

Cleaning out CCCKC, I found a Satellite modem.

Ironic, as this was found inside a cave. The model number is CL0006910-01. The manual states that the IP address is 10.X.Y.Z, where X,Y, and Z are the decimal equivalents of the last three octets of the MAC address. This yields only a telnet port for setting up the unit. Inside you will find a port labelled JTAG, a port labelled EMU, and an RJ-11 port labelled CONSOLE. The console is a 9600-8N1 serial port running full RS232 voltages. vMost logic inside appears to be 3.3 volt, although there is a 5 volt regulator on board as well. If we had two, it would be neat to try and make a point-to-point link.

In the serial console, early enough on boot, if you press enter 3 times it will drop you to a bootloader that can be used to reset the unit to default settings, change console ports (I wouldn't recommend it unless you find the other port), set the MAC address, and read/write the flash and ram. To reset the unit, run 'zapp', then 'boot', and now the root password is blank. when logged in as root, many more commands are available. In both the OS and bootloader, run '?' to get a list of commands. Be aware, if you type a single letter, it may match it to the first available command.

More photos are up: http://www.flickr.com/photos/40925843@N03/sets/72157627794846750

More Remote Shutters

Written 2011-09-29

While unpacking CCCKC at the new Hammerspace, I found some unused switches soon to be discarded. This is a Kodak projector controller.

And I think this was a projector screen up/down controller, but it has good mechanical action.

Of course, they all use the 1/8 inch headphone plug I've converted everything to use

a Universal Canon Remote Shutter

Written 2011-09-25

A bit ago, I purchased two remote shutter releases for my Canon 20d. The plan was to keep one, and convert the other into a time-lapse controller by removing the remote and adding a 555-timer circuit. This is all well and good, but I still couldn't attach either to my old EOS 350 Rebel. After thinking about it some, I've added 1/8 inch stereo jacks to all my remote release gear. This way I can swap cameras, control multiple cameras with a headphone splitter, and when I have time to build the time-lapse rig, I just have to add a 1/8 inch plug to it and it will work with everything else.

I did consider going with 2.5mm (headset jacks/plugs) connectors, as that is what older Canon SLRs used for remote releases, but one thing troubled me - long 2.5mm stereo extension cables are hard to find while any electronics store will have a 1/8th inch stereo headphone extension. Instead I bought a headphone->headset adapter for the old Rebel.

Poking around an Ademco RapidEye Multi

Written 2011-09-25

The Ademco RapidEye Multi is a small CCTV DVR, serving the same purpose as the Pelco. As it turns out, this hardware is quite different from Pelco's. The Ademco uses a Pentium-I desktop, wrapped with a custom housing and interface boards.

The unit is also running Wind River VxWorks. Since we all know a Pentium cannot simultaneously compress 8 different video streams, this card contains the extra horsepower. It also has a small wire running to the motherboard's reset line.

The unit also has a modem and two serial ports. If you attach a cable to the second port, you can interrupt the boot sequence to change settings like the IP address. Use 8-N-1, 9600 baud. You'll need to press enter quickly to block the boot-up sequence. Here's an example capture:

AVAILABLE COMMANDS    DESCRIPTION
?                     This command list
q                     Quit; Start Application; Run OS shell on console
cls                   Clear Screen
ver                   Display Product Version
show                  Show all parameter values
=        Set LAN parameter, e.g.,
                       'ip = 172.25.2.1'
                       'gateway = 172.25.100.1'
                       'netmask = 255.255.0.0'
                      Set PPP parameter, e.g.,
                       'local = 172.26.2.1'
                       'host = 172.26.200.1'
                       'comport = port1'
                        (options are 'none', 'port1', 'port2', or 'internal')
                       'timeout = 60'
                        (timeout may be between 0 and 999 seconds)
                       'baudrate = 9600'
                        (options are 9600, 19200, 38400, 57600, 115200)
                       'modem prefix = at'
                       'modem init = z'
                       'modem dial = D'
recover               Recover system to the previous version.

Enter a command, or '?' for help:
ver

Rapid Eye MULTI
Configuration Shell
Version 3.2

Copyright 1999-2000, Ademco

Enter a command, or '?' for help:
show

Current LAN Settings:
  ip       = 10.174.8.64
  gateway  = 10.174.8.1
  netmask  = 255.255.254.0
Current PPP Settings:
  local    = 172.26.2.1
  host     = 172.26.200.1
  comport  = internal
  timeout  = 60
  baudrate = 115200
  modem prefix = AT
  modem init   = Z
  modem dial   = D


Enter a command, or '?' for help:
q

Configuration Shell Stopped...
Attached TCP/IP interface to eeE

I also imaged the disk. The disk is split into two partitions. The first is a 2GB FAT containing VxWorks. The second is a partition listed as 'free space', but actually holds the video data. Here's the file list:

.
./streadir.dat
./STARTUP
./VERSION
./eventdup.dat
./JAV308.O
./system.log
./JAV306.O
./VXWORKS
./EVENTDIR.DAT
./TEMP
./KALATEL.O
./blockdir.dat
./PPPWAIT
./DAU
./storage.cfg
./streadup.dat
./alarmdir.dat
./PPPINIT
./PELCO_P.O
./MANUF
./serialid
./alarmdup.dat
./eventcol.dat
./SECURITY
./BOOTROM.SYS
./MTC
./MTC/SECURITY
./MTC/USR.INI
./MTC/LUCENT.INI
./MTC/config
./MTC/cfgdup
./GPOSTATE
./DELTA.O
./storage.csv
./CONTEXT
./PELCO_D.O
./DSH2
./config
./DSHUNLD
./cfgdup

More photos are available here: http://www.flickr.com/photos/40925843@N03/sets/72157627615155249/

PPPRS launches Kickstarter

Written 2011-09-25

The Powerwheels Racing Series (PPPRS) has launched a kickstarter. They're raising funds with the goal of adding new races to the circuit, more obstacles, and a new timing/scoring system. I signed up for a great big banner.

Poking around a Pelco CCTV DVR

Written 2011-09-24

Some time ago our hackerspace was given a Pelco CCTV DVR. We used it in the cave for a while, but nobody was monitoring it. The plan was to write some open-source software to interconnect with it so we could use it as an 8-port ethernet video receiver. While moving to the new space, I took apart the unit so we could take some photos, probe some lines, and image the disk.

Here it is. The Hard Disk area is obscured, but is a single channel IDE disk. A second disk comes with the 16 port model.

A 5 Volt Serial Port is available on the middle board, although I did not have my adapter

JTAG is available on the upper board

I pulled the disk to see how the video was stored. The disk has a single ext3 partition. All settings are stored in a file named dx3100.ini - clear text passwords too. Also, you can configure camera names and the like through the file, which is probably easier than typing them in on the device while it is running. Also, alarms and network settings are in the file. There's a log directory, and an old_log directory, both of which were empty on our unit. The video is stored in files, about 100MB or so each, in a folder with the date of the recording. However, the folder sturcture, /disstech/diss/<4digityear><2digitmonth><2digitday>/*.dsf matches digiop DVR recording systems, so we may be able to play footage using their tools, some of which are available here. I've placed one dsf file, dx3100.ini file, and a file list to into this project. There were also some INX files, but I have yet to take a look at them.

Das Motorsportspiel Liga

Written 2011-09-23

I took a few shots at the races of http://www.dasmotorsportspielliga.com/. This is a racing league dedicated to the dice and board game, Das Motorsportspiel. The game is a lot of fun. Currently in the league we each race two different cars, and the league runs a series of races, both originals from the game and prints created and shared online.

More Shots Here

3rd Party Canon Remote Shutter Teardown

Written 2011-09-22

So, we all know you can shove a screwdriver into a Canon 20d to make it take pictures. Eventually I want to build a 555 timer based time-lapse controller. However, I ned a cable with Canon's connector on the end. Originals from Canon cost cash money, but full remotes from China cost ~$4. So I bought 2 and took one apart.

Pull the two screws. One is hidden under the sticker

TaDa! The switch consists of three pieces of metal. I've also cut the wires for later use.

So, eventually I want to attach a timer to that extra wire. It will consist of a 555 timer and some pots so that I can adjust the timings.

Kansas City Mini Maker Faire: Parkville photos are up

Written 2011-09-16

http://www.flickr.com/photos/40925843@N03/sets/72157627680840028/with/6152239392/

Building an Office Bike Rack for $30.

Written 2011-09-15

So, one day I found some bike racks in part of the office I seldom frequent. There were three following a similar design. A right triangle frame extruded along a wall with some pegs to hold the tires.

I tried to take the best aspects from each. One had a good, strong frame. Another used wooden dowels to hold the wheels. The last fit two sizes of bike in the same slot.

To start, I used 5 sections of 8 foot 2x2 lumber. Three of these go directly into the frame, so pick the straightest ones. Next you'll need 4 20 inch sections, 4 14 inch sections, and the remaining lumber makes 4 pieces cut as trapezoids to form the hypotenuse. If you're careful you can get all of those from the last 2 pieces of 2x2.

Assemble as such:

After building the frame, you'll need some short dowels to hold the bikes. I built a jig on a radial arm saw to help me cut faster:

And here I am in action:

Once the dowels are sliced, screw them to the frame. I started with a pattern of six so that along one rail it goes dowel - space for a mountain bike - dowel - space for a road bike - dowel. Eventually I intend to attach more of the dowels to fit different sizes of tire, as well as fill in all the spaces on the rack as it currently only fits three bikes.

Bill of Materials:

5x$1.25 8ft x 2in x 2in lumber

1x$10 box of screws(enough for 3 bike racks)

3x$4 dowel rod

some furniture coasters

some velcro(only the toothy side)

RSAXVC.NET: Now a real website

Written 2011-08-29

As you can see, I have at least one visitor from every state in the union. This had been a little game I was playing for myself with google analytics, until I realized I could set the map start and end dates, and that I had already won.

DefCon, with Penn and Teller

Written 2011-08-28

As a preface, I usually don't post out-of-order from the order my pictures are taken in. That said I'm running behind.

At the last night of DefCon, some fellow I didn't really know at all that we were talking with said he had half-price tickets to the Penn and Teller show. It turns out that he didn't, but we were able to strike a deal with the ticket seller, because the guy I didn't know had his 60d Canon, I had a 20d Canon, and the ticket seller was also a photographer.

The show was pretty slick, and better in person than I expected. One of the people they took onstage was a young lady from toool.

After the show, Penn and Teller stuck around for photos. Well, here we all are:

AT&T 2Wire DSL Modem Teardown

Written 2011-08-27

Some time ago, I took apart an AT&T 2Wire DSL modem. The design was similar although simpler than the AT&T 2Wire VDSL/UVerse unit that had an administrative serial console

You can find more photos here.

Interestingly, The 3800 UVerse device, this unit, and the next generation UVerse device all seem to have the same edge connector built into the PCB. I wonder what that does.

Bike Camping at Middle Creek Lake and Park

Written 2011-08-13

Some time ago I heard Noah of kc-bike talking about going bike camping. The day before, I decided to go.

Luckily, my work has a small cache of bikes for testing electronics, and I wanted to try out my Garmin Montana with their cadence sensors. The bike of choice was a rockhopper hardtail-mountain bike. Normally I use my old red Raleigh, which was made sometime in the 1970s, and I have been riding for the last 10ish years. However, the Raleigh is a little old to trust farther than I can walk out.

The night before I built a Montana bicycle mount. I started by cutting some threaded rod and combining it with hex nuts to make custom-sized bolts. Next I bandsawed some aluminum strip down and punched two holes. These strips wrap around the handlebar, while the custom bolts drop through a Garmin-brand mount to hold the handlebars via the aluminum. Overall it worked well, although I was planning to mount it on the center of the handlebars.

The morning started with brunch at Perkins at 135th with Noah. From there we headed south then east to the arboretum. This made the first 10 miles

The way out was slow, and I stayed near the rear with Noah - he had tons of gear including solar panels and lead acid batteries. However, he was charging a phone and could run other electronics. I had a few issues going out. First, I wish I had carried more water, although we were able to stop and fill up at a Sonic seven or so miles from camp. I wish I had gotten a cargo mount of some type. Although I carried only a backpack, it sure would've been nice to have that mounted to the bike frame instead of me. One thing I had neglected before heading out for the 30 miles of road was to fill up the mountain bike tires. The previous employee must've been out in the dirt, as they were filled nice and low, making extra rolling resistance for me as we covered the series of small hills between miles 10 and 30 to camp.

Camping was awesome, like all camping always is. I wanted to take a hammock setup, but Noah warned me that there were very few hammock-quality trees, so I opted for a bike-tarp shelter. This is when you take a bike, flip it wheelside up, anchor it somehow, and then attach a lean-to to the bike. This is very useful in areas where there are no trees. Noah made a similar shelter, with the main differences being that he stayed perpendicular to his bike, and I stayed parallel to mine. After the small rainstorm we had, I've concluded that his design is more waterproof, as my design relies on the bike being at least as long as the rider, although I didn't mind the rain on my feet, as it was Kansas hot all night.

Of other note, a large storm developed North of us, and fired a beautiful lightning show up and down from the ground. A little rain fell, but the cooling was nice

After a tasty breakfast, we all headed back.

In Louisburg, KS I filled my tires to the max. After reaching Louisburg the group split - the main pack headed for the gravel; Noah and I stuck to the flat lands. Later, Noah and I found our own gravel, and after a bit, ended up meeting the main group again - Gene had another flat.

Overall, it was a great bike trip with new friends, lots of being outside, new experiences, and lots of open road

boot-animator: part2: Command-Line flags

Written 2011-07-25

tonight I added command-line flags to boot-animator. Of note:

framerate
frameskip
frameseek
loop
width
height
numframes

This makes the tool much more useful for generating android boot images, as well as taught me how to use libpopt, which handles interfacing the command flags for me.

Here is the current code: https://github.com/rsaxvc/boot-animator

Gaining root on Samsung FemtoCells

Written 2011-07-17

To summarize the steps needed to take over a femtocell:

Build a serial cable
Find your baud rate: SCS-2U01 is 115200:8N1, the Sprint Airave and SCS-24UC4 are 57600:8N1
Retrieve the bootloader key from their source code drop by diffing their u-boot with mainline u-boot (hint, it is "sys\r")
log into u-boot by using above key - it is time-limited different amounts on different devices
go read the u-boot manual so I don't have to read the nitty-gritty
copy the u-boot settings by logging your console and running `printenv`
quickly turn off the hardware watchdog with `setenv watchdog_off 1`. Otherwise this is a great bugger.
root the device by adding `init=/bin/sh` to $ramboot
boot the device with `onandboot`
Almost there: "sh-3.00#" At this point you have root, but very little is mounted(/etc/shadow is present though)
insmod the mdoc/tffs.ko and mdoc/tffsbd.ko from the /lib/modules/<yourkernelhere>/
`mount /dev/tffsa1 /mnt/mdoc` - brings up the main disk
`mount /proc` `mount /sys`
`mkdir /mnt/mdoc`
`mkdir /mnt/mdoc/RFS`
`mdoc_cp RFAB /mnt/mdoc/RFS/append_rfs.tgz 900000`
`cd /mnt/mdoc/RFS`
`tar xzfm append_rfs.tgz`
copy the created directories over / as needed.
bring up GPS with `insmod /mnt/mdoc/RFS/PATH_MODE/lib/modules/GpsCtrlDev.ko`
You can now fiddle around with the device. Keep in mind that many of the read/write areas will actually store results in RAM for now, but you may get a 'No space left on device' error if you write too much.

Interesting bits:

The web interface (only available to verizon) is running thttpd, and stores a hash in /udata/htdocs/.htpasswd
The cgi web interface is a set of ARM binaries in /udata/htdocs/cgi-bin/
Shadow passwords are used, but you're root anyhow
Component versions are in /udata/image.ver and /udata/oper.ver
iked-v.conf contains some very interesting settings, like 'set local psk XXXXXX'

This is written as a summary of work on the SCS-24UC4, although it should apply directly to the SCS-2U01 and Airave.

This was a collaboration by RSAXVC and Doug.

boot-animator: convert videos to android boot animations

Written 2011-07-17

Tonight I wrote a simple tool to read avi/mpeg/other video files and convert them to bootanimation.zip files for android. It depends on opencv to do the heavy lifting, the command-line zip tool does the zipping, and libpopt will do the switch handling. Right now it generates a bootanimation.zip with the same framerate and resolution as the original video, but I plan to add command-line switches for resolution, framerate, and a few others.

https://github.com/rsaxvc/boot-animator

CCCKC: PianoBar Player

Written 2011-07-10

Long night here at the cave, but Phillip Dorr and I built a PianoBar player. PianoBar is a command-line player for Pandora, written by Lars Braun. The unit is an old thin client sporting 533MHz Via C3 and 256MB PC133. Compile options make a big difference for this little guy. GCC does have a Via C3 CPU model, but doesn't do instruction scheduling for it. We found that with -O3 -march=pentium-mmx we were able to reach only 30% CPU usage, whereas the default -O2 took 40%ish. I suspect this is because pentium-mmx has an instruction scheduling model that matches the C3 better than no scheduling + 3dNow instructions.

Here's what it looks like:

Currently the box lies out of the way, with only a keyboard hanging out.

We hope to create a new controller by using an arduino that can appear as a USB keyboard, like the UNO.

Taking apart a PQI IDE Flash Card

Written 2011-07-10

A while back I ended up with a bunch of Via C3-based thin clients. Each one came with a 16MB IDE flash card designed by PQI Industrial. Today I took one apart.

Use a screwdriver to pry the locks back. You'll need two screwdrivers to get it to happen at the same time, or you can break one and hold the other with the driver.

After pulling back the device, you can see a standard flash TSSOP.

On the back is the TDK TSSOP<->IDE controller.

In particular, this device uses a GBDriverRA2e, which is an older design.

I'm curious if I can pick up some cheap thumbdrives to make my own internal SSDs. However, the number of computers I own with IDE slots is quickly dwindling, although these may find themselves in some small embedded x86 projects.

Building a Coin Sorter

Written 2011-07-09

A while back I had the idea of building a device that would optically sort coins. I've always been fascinated by the gravity-fed machines that sort different sizes of coins into bins. I want to take it further. I would like to be able to sort my coins by date into different tubes not only by size, but also color, year, and maybe quality.

The first step for this, is to build a test rig that can hold a camera and different coins:

Once the computer+camera can make a repeatable good decision, I'll build a system to move the coins into different storage containers, but for now I'll settle for detecting different coins.

Building a Garmin Montana Motorcycle Mount

Written 2011-07-09

I guess I can post this now. A while back I built a custom receiver to attach the Garmin Montana Rugged Mount to my Honda Magna. This product is a harness for the Montana GPS that fits the RAM bracket mounting system. However, most of their mounts involve some sort of arm mechanism, and I had a much better location on my bike. The Montana mounts just below the instrument cluster, almost like it belongs there. I used four strips of aluminum, two on each side, to hold the mount to the handlebars. I need to find some stainless steel bolts, as the hardware store didn't have the right ones and they're showing a little rust. The swap will be a good winter project.

How to Visit 4 HackerSpaces and have an Awesome Weekend

Written 2011-07-04

Start in Chicago. I handled this by booking it from Olathe, KS to Chicago, IL on a late Friday night to avoid traffic. This gave me an early start in Chicago.
Visit Pumping Station One. Watch Sacha make guitar picks from and old motherboard after removing all the chips. Leave a note on how to control the cricut with open source software. Buy cologne from Sacha's company.
Drive to Madison, WI.
Have a sandwich and salad at the Green Owl Cafe. Top notch.
Watch Rhythm and Booms:a fireworks, laserlight, and music show. FM radio transmits the music to crowds of people all over the city. By far the best firework show I've ever seen. I sat with well over a thousand people on a hillside watching the reflections in Lake Mendota.
Crash with Chris and Heather.
Visit Sector67, Madison's HackerSpace
Go sailing at Hoofer's with Chris, Heather, Will, and friends.
Start driving to Davenport, stop in Dubuque.
Find a perfectly good steam-powered dredge ship and river museum.
Photograph a catfish and a sturgeon
Drive to Davenport, IA.
Visit Quad-Cities Colab This is a hackerspace built next to a technical school. Photograph making a computer-controlled lighting setup. They used a vacuum former to build housings for the lights.
Drive to Des Moines, IA
Realize that Ray doesn't live in Des Moines, he lives in Urbandale, IA. Photograph Des Moines Tron building instead.
Crash at Ray's.
Go see three of the five largest towers in Iowa (they're all more than 1/3 mile tall), and photograph two of them: WOI and KCCI. Tower Elkhart was in the distance.
Go see Ames, IA MakerSpace. Check on the space's brew and figure out how to calibrate 3 lens CRT projector.
Find a bar Ray saw on T.V., which is closed, and instead eat at Mullet's.
Pass by Rocket Transfer Station
Visit BeerCrazy, which was closed, but it looks awesome. Combination specialty beer store and brewshop. Ray gets ingredients here
Drive home to Olathe, KS
Go blog about it.

Learning about HammerSpace's Andover Controls Systems

Written 2011-06-20

The best I can Andover Controls is a company that made facility automation systems. Connect multiple buildings together, and control access, security, fire alarms and controls, power system, HVAC, and more from a central location. The system installed at HammerSpace controlled at least access, power, and HVAC. They used a mixture of RS-485 and ethernet to control the system. Since HammerSpace used to be a network trunk center for Cingular, the system is far overspec'd for the current purpose. We'd like to control the network, but do not have any software belonging to it. Thus the saga begins.

First, the Andover Controls systems installation is modular, and consists of the following components:

Two units provide I/O, an DX 800i and an LCX810. The 800i is an input-only device, although there are relay solder points. The LCX810 has 8 input and 8 outputs for controlling HVAC systems.

An ACX780 provides access control interfaces to prox readers. I assume a clocked serial like most prox and card readers.

i2 867 units are the most complicated thermostats I've ever seen. Dave ended up putting a thermometer on one since it doesn't have a display.

A CX9200 provides an ethernet management interface to the rest of the system, or at least it used to. It also has four serial ports, and one of them is attached to a touch-screen controller with a broken display. Because we lacked the needed 25-pin serial connector, we went for ethernet first(I left my serial adapters at home, as I didn't know I'd be fiddling around with anything fun). After rebooting the system, it sent out some diagnostic packets and an ARP request.

So of course, I set my laptop to appear as that IP address...

Hey look, SNMP traffic. I've peppered out the community string, but at least now we're getting somewhere. Next we NMAPed the IP address to find two open TCP ports: 33440 and 33456. 33440 appears to have a simple ascii text client, but everything I've entered emits the same error message and closes the connection. 33456 emits some binary data and closes the connection. I need to look for a MIB to explain the SNMP trap.

Next to investigate are the serial ports. There's one designated to be attached to a modem, and another attached to a touch screen. Hopefully the modem port has a text-UI we can use. That is, after finding the serial port settings.

Also, I've posted a few datasheets here:http://rsaxvc.net/datasheets/hammerspace/

Have you seen these flowers?

Written 2011-06-11

I found an image of flowers. I'm not sure I have the byte-alignment correct, but I found these flowers inside a memory device in my Android GTablet.

Anyhow, if you set GIMP to open a raw file that is 1024 pixels wide, supply it with /dev/mtd6 from a ViewSonic GTablet, and scroll on down to about 3302456, you just might find these flowers. I'm also running cyanogenmod7, and I wonder if it makes a difference.

Porting Debian to the G-Tablet

Written 2011-06-11

One day, I bought this sweet GTablet. I also bought a keyboard case with kickstand. I used it for a month or so exclusively, and I've decided about the only think I need but can't do is develop software. So here's an idea, port a real OS like Debian. I've started fiddling around with the device through the ADB shell. Currently I'm running CyanogenMod, an aftermarket Android version.

So, lets talk about how android starts up:

Board-specific bootup ( turn on RAM, set up DRAM timings, ...)
Bootloader
Linux Kernel
Kernel runs first process ( init )
Android version of init goes through /init.rc script
/init.rc launches zygote
zygote runs bootanimation
google's android environment comes up

So what can we take from here? First, killall zygote will restart the android environment, without restarting the linux environment below it. Second, it looks like the best way to get Debian going may be to build an image, then burn it through the bootloader ( APX mode for Nvidia). I had originally hoped to override the android system at init, then bring up debian around it, and that is still possible, but I suspect I'm better off going Debian from scratch.

Note: This is not installing debian in a chroot, as many have done before. This is trying to install debian on the 16GB internal flash disk.

A Walk in the Park

Written 2011-06-09

I head down the walking path, lunchbox in hand.

An elderly indian couple crosses paths with me.
We smile and greet each other.
I have spoken to them a few times before.

Soon I am at the turn.
I found this place about a year ago.
An abandoned park bench, with no path, overlooking the lake.

I sweep away the leaves and sit down.

Almost a year ago I fixed this bench.
I measured and cut the lumber.
I tied it into a bundle, and carried it to the bench.
The ranger asked why i was carrying it.
He told me he had never been to my bench.
The next day I took the bench apart.
The next day I put it together.
The last day I stained it.

As i open my lunch bag, I spot a moth stuck in the velcro.
I carefully remove her.
She flutters away.

As I bite into the cheese and sausage, and listen to the crunch of the crackers, I am reminded of home.
My father showed me this place long ago, above a different lake.

I drink my 2.1 ounce yoghurt drink.

There is something in the brush.
It is a small dog.
He barks at me.
I offer some cheese and sausage.
He quiets down and I pet him.

Soon we both hear his owners calling for him.
He looks at me, the runs back through the brush.
He will not betrsy my citadel.

A goose squawks as it lands in the lake.
Another goose squawks back.
The first goose leaves.

I toss the last of the sausage into the lake.
Ths same catfish comes and snaps it up before decending down again.
I finish the cheese.

I wonder if anyone was here before me, maintaining the bench.
Perhaps feeding the same catfish, with the remains of simple meals.

I wonder who built it, so many years ago.

And i wonder, who i am.

BreezeCom BreezeAccess AP with Special Guests: Charles Huber and Doug Kelly

Written 2011-04-21

Today we took apart a BreezeCom BreezeAccess AccessPoint.

Charles is excited to begin disassembling the device

The electronics come on two boards, connected with a .1 pitch header

Here's what each side of both boards looks like

The serial console settings are conveniently exposed on the outside of the device on a 3x1 .1 pitch locking connector. The settings are 9600 baud, 8N1. Once on the serial console, Doug explored many options and menus. Some are only accessible by 'authorized users'.

This documentation might match the device we took apart

breezeCom.txt is a serial capture of talking to the device

Litronic Argus 2202 PCMCIA Teardown

Written 2011-04-21

Quite some time ago, I took apart a Litronic Argus 2202 Parallel to PCMCIA adapter. However, since I didn't write the name down of the device, I couldn't post about it. Now I have a picture of the manual too.

The Front

The Back

Seems like a simple little board. More photos are up on flickr

Is it really easier to write the second time?

Written 2011-04-19

Talking with a friend today, the concept that a program is easier to write the second time around came up. I think in general, this is true. However, most times it falls into one of two reasons.

You didn't really have a good grasp of the solution before you started
Your programming language doesn't allow clear concise expression of ideas

In particular, we were discussing my opencaching rss generator. I lost it when my virtual host crashed as I hadn't added /opencaching to my backups. The RSS generator was written in PHP. #1 definitely applied - I didn't know what the finished product looked like, but it would be something like construct query / parse GPX (later switched to JSON)/convert to RSS. However, by using PHP, #2 made up for #1 - things like json_decode() and curl made the implementation short and simple.

RHex Feature: Fast Editing

Written 2011-04-09

When I talk about fast editing, I mean responsive editing of very large files. Previously, I have edited multi-megabyte images and multi-gigabyte filesystem images. Of editors that support inserting bytes in the middle of a file, editor responsiveness has been difficult to say the least.

Under as many circumstances as possible, writes must not block. To manage this, I'm working on a small I/O library. Reads will be memory-mapped back to the file, but writes will be buffered for later writes. To accomplish this, the file is represented by a list of places to get the data. The list contains two different types of entries - part of a file can be memory mapped to the backing store and part of a file can be saved only in ram, waiting to be flushed to disk. On the start of a write, the file will be expanded to reserve the space that will be written soon. The only blocking case remaining is when the system is low on memory, a library thread may be waited upon to free up a large write buffer.

RHex Feature: File Identification

Written 2011-04-08

I want RHex to be able to look at the current cursor position, and report possible file types assuming that byte is the beginning of a file. To do this, I'm going to use libmagic. libmagic is a library for recognizing magic numbers, much like those often placed at the start of a file as a special marker. For example, FF D8 marks often the start of a JPEG file. When working with different types of archives and binaries the ability to quickly guess at the content of a file embedded within is a very useful feature for a hexeditor.

So I tried to write a simple implementation of the unix command 'file'. I used libmagic, and it only took about 15 lines.


#include <stdio.h>
#include <magic.h>
int main( int numArgs, char * args[] )
{
magic_t m;

m = magic_open( MAGIC_NONE );
magic_load( m, NULL );

printf( "%s", magic_file( m, args[ 1 ] ) );
magic_close( m );
}

Do note not to free the memory. I suspect libfile loads the tables into memory for the time between magic_load() and magic_close(). Also, there is a function, magic_buffer( magic_t, const void * bufptr, size_t bytecnt ) that operates on a byte-array.

Introducing... RHex

Written 2011-04-05

What is RHex? Richard's Hexeditor - currently is a list of features I want in a hexeditor. Eventually, I would like to implement it, as it seems every hex editor is missing something I need. Currently I use hexedit and codewright as hexeditors. Here's what I want:

Side-by-side text and hex display (almost all editors have this)
text regex(most have this)
binary regex(some have this)
replace section with different size(few have this)
asynchronous editing(If I insert a byte into a 7 gig file, I don't want to wait for the rest of the file to be re-written)
libmagic support(none I know of do this, also I want a mode where each byte is a possible start of magic)
scriptable file decoder(If I'm editing a JPEG, I want some color highlighted text for different areas)
big file support( >4gig, but I'm not opposed to the 32 bit mode only supporting 2gig files or smaller)
variable sized groupings ( If I'm looking at a structure that consists of uint32s, I want the editor to reflect that), and...
offset groupings ( If I'm looking at a structure that consists of uint32s, but at an address not divisible by 4, I want the editor to reflect that )
reverse-endian groupings( If I'm looking at a structure that consists of uint32s, I want the ability to have the editor reverse the groups of bytes for user i/o)

To implement this, I'd like to design it as a library for file editing. This file would be responsible for memory management and either threading, asio, or nonblocking i/o. Hopefully this way you could make a gui or console version without too much work. However, the lowest level has a lot of bookkeeping to do to allow asynchronous editing. I might want two modes, a 'safe mode' where the editor is writing a new version of the file and a 'live mode' where the editor is actively editing the file directly, probably through mmap(). Either way, it has to manage outstanding changes so that the editor is always presented with whatever it wrote into the hex library, and the library is trying to get it synced to disk quickly.

Tough stuff, but manageable. If only I had time to write it.

Notes on Throttle Cable Replacement

Written 2011-03-31

You might not notice it at first, and I certainly didn't, but there's a very easy mistake to be made when installing a new throttle cable.

The throttle cable's collar is keyed, and this is important. However, since the screw is on the bottom of the handlebar it is easy to miss. Also, since another plate holds the collar down, it is hard to see misalignment. Also, depending on the degree of misalignment, the throttle may work, but be a little sticky

Of course, the receiver is keyed too, but most of it is hidden by the second screw and plate that holds the cable's collar in.

All Done. New cable, new screws, and new plate

More Samsung SCS-2U01 GPS logs

Written 2011-03-22

Today, I took a friend driving. The purpose? To drive up and down an east-west street to see how the gps output changes. The friend's purpose? To hold a probe onto some serial lines as I tackled one of the bumpiest roads in Missouri. And boy does it change. There's new sentences available and other things I don't yet understand. Anyhow, serialgps.31stSt.cap.lzma

One little, two little, three little endians

Written 2011-03-13

I've done a little work on two different endians. Currently, I've missed out on PDP-endian. Again, the question came up, which one is best, so I'll give it a go at explaining the three and their differences. Endianness, is the pattern of storing a multi-byte word into multiple bytes of memory. In particular, the bytes are often stored most-significant or least significant byte first. You might wonder why anyone would store objects backwards (indeed, little-endian is the only 'language' I know where the 'sentences' are written in ascending order, but the words are written in reverse order). However, there are some distinct advantages to storing words in little-endian order(union alignment and addition/subtraction/multiplication pipelining), as well as there are advantages to storing words in big-endian order( natural treatment of multi-byte objects, easier comparison ). Please keep in mind that with modern processors, some of these reasons end up not meaning nearly as much as they used to.

Union Alignment

Lets take the following example:

union{

uint8_t one;

uint16_t two;

uint32_t four;

}sometype;

With little endian, because all the objects are written backwards, the union members one, two, and four can be located at the same address in memory, which can really simplify things at an assembly level. With big endian, two has to be stored 2 bytes past the beginning of the union, and one has to be stored 3 bytes past the beginning of the union. *

Addition/Multiplication/Subtraction Pipelining

By storing bytes in little-endian order, something neat happens**. Think back to multi-digit arithmetic(except division). When performing an operation, you always start with the least-significant digit. This is because the less significant digits can only overflow into more significant digits, so when one byte is loaded in, enough information to begin computation is already available.

Natural Treatment of Object Byte Order

I'm going to call big-endian as natural byte order, because by keeping words in big-endian, things like memcmp() can get a little more efficient. For example, when comparing large objects with memcmp() on a big-endian system, it becomes possible to aggregate compares to full word sizes. This can be significantly more efficient than comparing per byte. (Potentially, you can implement something similar with little-endian, but it incurs an overhead when there is a difference in the two areas of memory, or a per word overhead)***. Also, most network protocols are written in big-endian. This usually isn't much overhead compared to say, generating the data to ship over the network, but for small applications it can build up.

Comparison and Sign information

By storing the most significant information first in memory, operations requiring sign information or comparison can be implemented more easily(**). Because the sign information is stored earlier in the word, a big-endian compare can complete as soon as there is a difference in the word. This becomes even more important when dealing with objects larger than the native word size.

PDP-endian

I haven't mentioned PDP-endian, because it is such an oddball. Worst of both worlds it seems. As a 16-bit architecture, 16-bit words are stored in little-endian format, but compilers on PDP-11s would store 32-bit words as two consecutive 16-bit little endian words. This makes it extra-confusing. Potentially compilers could be updated to store words in little-endian order, but this architecture is almost entirely dead, and all the other software running on it expects PDP-endian already.

*Depending on architecture, the alignment might move things differently, but this scenario is pretty common

**on systems with memory buses narrower than the cpu word size (this actually happens a lot on embedded stuff (8088, some ARM and some MIPS )

***On systems like ARMv7, you can swap endianness in a single instruction. On little-endian systems, you can still use word compares to speed up memory access, but when the words compared are different, you then have to switch the endianness and figure out the right memcmp return.

Samsung SCS-2U01 GPS Logs and Possible Tamper Switch

Written 2011-03-12

First off, there might be a tamper switch located under the GPS shield. Note the bottom-left corner of the shield connection. There is a small island of metal that isn't connected with the rest of the shield area. Normally this wouldn't be good for shielding, and if you test with a multimeter, the shield does make a good connection to this point. I've added a copper wire to mine just in case.

Now on to the serial logging. The GPS processor here is an STR711. The datasheet is available from ST here: http://www.st.com/stonline/books/pdf/docs/10350.pdf. The STR711 is an ARM7TDMI with 4 UARTs, SPI, I2C, internal flash, USB, CAN, and internal RAM. 3 of the UARTs didn't appear to have anything on them, but UART0 (2 pins clockwise from the orientation dot) are communicating at 115200 baud. On powerup, the GPS emits:

Here's what I'm guessing is a packet from gps to board cpu with some satellite stats:

#DBRM 14 66040 1 145 75 13 &05

#RMCH 0 0 0 0 0 0 0000 0 0 0 0 0 0 0 0 0000 0 0 0000 0 0 0 0 &AA

#RMCH 1 29 0 0 13 0 7000 0 0 0 0 3 32 0 91697 2D4E 483655680 -14531 AD07 61 0 0 -99 &02

#RMCH 2 0 0 0 0 0 0000 0 0 0 0 0 0 0 0 0000 0 0 0000 0 0 0 0 &AC

#RMCH 3 29 27 0 13 32 7012 1 0 0 0 5 540 0 92042 4886 -421134336 10427 0A4C 16 0 0 -2 &F2

#RMCH 4 29 27 0 11 32 7012 1 0 0 0 5 539 0 174817 B1AA -471597056 6117 9433 36 0 0 -2 &1E

#RMCH 5 29 27 0 0 32 7012 1 0 0 0 5 516 0 174878 3F8B -14876672 5187 3946 36 0 0 -5 &06

#RMCH 6 29 27 0 13 16 7012 1 0 0 0 5 444 0 257759 1780 479068160 6095 26D8 15 0 0 0 &F5

#RMCH 7 29 27 0 13 32 7012 1 0 0 0 5 702 0 91942 3F55 32702464 170 C629 16 0 0 -3 &C2

#RMCH 8 29 27 0 12 16 7012 1 0 0 0 5 443 0 257469 C727 25231360 -14966 0E2B 55 0 0 0 &0F

#RMCH 9 0 0 0 0 0 0000 0 0 0 0 0 0 0 0 0000 0 0 0000 0 0 0 0 &B3

#RMCH 10 29 27 0 12 32 7012 1 0 0 0 5 558 0 174451 4D28 433192960 -12729 E500 36 0 0 -3 &2B

#RMCH 11 29 0 0 0 0 7000 0 0 0 0 3 555 44 610216 6202 -473300992 65600 22E5 89 0 0 -99 &03

#RMCH 12 29 0 0 0 0 7000 0 0 0 0 3 555 44 610221 0E28 -22347776 -2170 A3C4 89 0 0 -99 &09

#RMCH 13 29 0 0 0 0 7000 0 0 0 0 3 555 44 610225 BA72 428605440 -64071 3958 89 0 0 -99 &1F

#DBTT 66040 1 0 0 0 1855 510 0490 0081 &CE

#INFO 0 Ham1A_Rom5 510 E510P052 Rel Sep 16 2009 10:36:54 MP07 TYNDALL 16368 RTC &AF

#INFO 1 66 16 2 0 2 0 0 61 0 0 0 0 0 0 0 1 &CC

My Favorite 64-bit Porting Guide

Written 2011-02-16

By way of an almost unrelated Google search, I re-found my favorite 64-bit porting guide. http://www.ibm.com/developerworks/library/l-port64.html. The article takes you through a few different 64-bit systems, new alignment/packing issues, and how to do what you used to do (storing pointers in uints) in a more portable way(uintptr_t). I've had to refer to this a few times now when porting the boomerang decompiler to 64-bit linux. Boomerang is a particularly fun case because sometimes you need to store a pointer in a uint. Do you use uintptr_t? Sometimes. You can only safely use it to store a native pointer. Because potentially you could add disassembly for a 64 bit architecture, but build on a 32 bit machine, you still need an extra-big type capable of holding the biggest pointer you could get. Luckily for us, a 64-bit datatype covers most pointers. Sadly for AS/400 users, this effectively means that boomerang won't be able to decompile your stuff without a major overhaul.

There's also a small typo referencing the 'fred function' (fread)

Sprint / Samsung AIRAVE SCS-26UC Console Captures

Written 2011-02-13

The Sprint/Samsung AIRAVE has a serial port in the HDMI socket, just like like the Verizon/Samsung SCS-26UC4 and SCS-2U01. The baud rate is 57600, just like the SCS-26UC4. It appears to be running linux 2.6.10 on an Omap 1710, with U-Boot 1.1.1. To stop U-Boot autoboot, enter sys<enter> within 1 second after the prompt. The device includes 64MB ram, TFFS, mdoc, and thttpd. Here's the log: airave.cap

Libcutter now draws G-Code

Written 2011-02-13

Thanks to Ray S. libcutter now supports some basic G-Codes. Ray wrote a g-code interpreter in python (sites.google.com/site/drbobbobswebsite/cricut-gcode-interpreter ), which I then ported into C++, and libcutter. This is my first PCB drawing, which is a reprap PCB that Ray sent me. G-code support helps open up many more possibilities, as it is a common standard for machine control. Currently, the interpreter only supports absolute positions, and Z of 0 or 1, to pick up or put down the tool.

Taking Apart the Symbol AP4131

Written 2011-02-11

Today I take apart a Symbol AP4131.

Symbol AP4131

Bottom

PCB + Wireless card extracted

PCMCIA Wireless Card

PowerPC CPU

Opening the 3Com WL306 / 3Com Access Point 8000

Written 2011-02-11

Today I document the teardown of a 3Com Access Point 8000, an older device.

A blast from the past

Only one screw to open

Opened, with RF Shields removed

A wild 4 pin header appeared. The CPU is an Intersil ARM940t Prism chipset, 8MB RAM, and some flash

The rest of the 2.4 gear

Inside the Adtran TSU100

Written 2011-02-10

Opening up the Adtran TSU100

The unit

Top Board

Bottom Board

CPU is an Intel 8031

LCD+input Submodule

Another FPGA + Board art

Verizon / Samsung SCS-2U01 Port Scan

Written 2011-02-05

Starting Nmap 5.00 ( http://nmap.org ) at 2011-02-05 19:34 CST

Interesting ports on ws166 (172.30.3.166):

Not shown: 996 closed ports

PORT STATE SERVICE

22/tcp filtered ssh

23/tcp open telnet

80/tcp open http

9100/tcp open jetdirect

Verizon / Samsung 3G SCS-2U01 Serial Console Captures

Written 2011-02-05

The SCS-26UC4 console cable also connects to the Samsung SCS-2U01 3G Verizon Router.

The bootup process goes something like this:

OneNAND Bootloader - ONBL2
ShadowUBL
U-Boot 1.1.1 waits 5 seconds for user input
U-Boot 1.1.1 engages Watchdog with 64 second timeout
U-Boot loads a 1.1MB kernel and 6.6MB RFS image
MonteVista linux boots with a 2.6.18 kernel

OtherNotes:

The built-in ethernet mac is a Faraday FTMAC110
"Machine: Faraday A360"
32K Instruction Cache, 32K Data Cache
224MB usable ram
thttpd is still present
"MonteVista(R) Linux(R) Professional Edition 5.0.0"
"The board type [0x61:12] vid[1]"
generates SSH keys on startup

Full log available here: samsung3g.cap

Update on Hulu for Android

Written 2011-02-04

The key to running Hulu on Android is getting a User-Agent string and Flash Version String that match. Previously I posted how to change your flash version and how to change your flash version on Android. However, you need to match your User-Agent and Flash String. That is to say, Adobe Flash "MAC 10.Sandwich" failed rather quickly.

By default, Android supports 3 User-Agents: Intel Mac Desktop, Android, and iPhone. By changing my user-agent to appear as an Intel Mac Desktop running 10.5.7, I was able to use Hulu for a little while, but it stopped working within a day. I suspect that this is due to the fact I have no idea what flash runs on 10.5.7, making it easy to blacklist. If anyone knows a flash version for 10.5.7 that runs Hulu, let me know, as then you could run Hulu without a full firmware update.

My current solution is to use CyanogenMod, which supports a User-Agent of IE6. I copied the Android Version from my desktop, "WIN 10,1,103,20" to libflashplayer.so on my phone. Since doing this, Hulu has run, albeit slowly.

What's the downside to this little hack? Since the User-Agent and Flash Version are both for desktops, every website thinks you are running a full desktop with plenty of RAM. Each flash library that contains mobile optimizations won't use them, because it thinks it is running on a desktop. Also, Hulu renders in full quality and full bandwidth.

The results? 1-2 FPS, and sound that actually doesn't skip. I'm running a fairly sketchy overclock to 800MHz. However, I was able to test this on a 1GHz T-Mobile G2, which performed much better, and since Flash uses 40-50 threads while playing, I hope it scales well to multiple cores just like it does on desktops.

Sprint / Samsung AIRAVE Photos Up

Written 2011-02-04

http://www.flickr.com/photos/40925843@N03/tags/airave/

Samsung / Verizon SCS-2U01 GPS Captures

Written 2011-02-03

So today, we were able to get some serial captures from the GPS chipset on the Samsung SCS--2U01. The protocol appears to be GloNav API, running at 115200, 8N1. The chipset on the SCS-2U01 is built directly on the PCB, unlike the SCS-26UC4. In addition, the GPS firmware version is printed, 2.07.11.

Here are some logs:

gpslogs.scs-2u01.tar.lzma

You stay away from here!

Written 2011-02-03

http://www.syracuse.com/news/index.ssf/2011/02/att_vs_verizon_iphone_-_a_comp.html

My first cell phone was a 3G Motorola RAZR on AT&T. It provided fast upload and download, and I used it as a wireless modem while traveling. Sometimes I could reach a megabit each way.

Then Apple launched the iPhone 3G. Soon the cells were full of iPhone users, taking all the bandwidth. Even in the Kansas City area, bandwidth was less than 100kbps, and often less than 56k.

So I moved to Verizon, who again provided fast 3G everywhere I needed it. Until now. You stay away Apple, I've seen this before.

You know you've been using Google Reader too long...

Written 2011-01-31

...when you press J to skip a Hulu commercial.

Google Analytics License Plate Game

Written 2011-01-30

Did you ever play the license plate game as a child? On a long trip, for each state's license plate you haven't seen, write the state down down, and try to see how many you can get. With Google Analytics, you can play the game with your visitors. As you might imagine, this is a low traffic blog, but here's how far I am:

Note Kansas, which is where I live. Note Missouri, which is where my hackerspace is.

SCS-2U01 Teardown Photos Up

Written 2011-01-29

A teardown of the Samsung SCS-2U01 Femtocell is up on flickr: http://www.flickr.com/photos/40925843@N03/tags/scs2u01/.

Hulu Flash Update

Written 2011-01-29

Sometime in the last 8 hours, flash has stopped playing Hulu on my phone. I'll look into why this is happening now.

Also, it appears that the browser can use a very large amount of RAM while playing Hulu. Without closing all other applications, the flash player would run out of memory and drop to tap-to-play mode.

Running Hulu on Android

Written 2011-01-29

***Don't try this at home. Don't blame me if it breaks your phone. It didn't break my phone though.***

Recently, I posted how to change your flash version. I looked into this because I was curious how the web applications like Hulu could detect that your device was an Android client instead of a real web browser. Initially, I looked into packet dumps that I compared between my Droid phone and my laptop. The user-agent strings can be changed to match, but some web apps can still detect they are running on a phone. The key is in the Flash version string. This string, accessible through the Adobe Flash API, is built into the library. So to make a phone look like a real browser, you both need a user-agent string from a real browser, and a flash version from a real browser.

What flash version should we use? Since the 'Desktop' user-agent option emulates an Intel Mac running 10.5, "MAC 10,1,103,19" is a valid OS X flash version. (You can find this by asking a friend with a Mac to go check their flash version.)

If you have a hexeditor on your phone, you can do this without ADB. I used ADB because I haven't yet installed my favorite hexeditor, Pixel's Hexedit, on my phone.

Close all browsers ( use a task killer )

adb pull /data/data/com.adobe.flashplayer/lib/libflashplayer.so /tmp/libflashplayer.so

cp /tmp/libflashplayer.so /<backup-path>/libflashplayer.android.orig.so

Use hexeditor to change flash version to "MAC 10,1,103,19"

adb push /tmp/libflashplayer.so /data/data/com.adobe/flashplayer/lib/libflashplayer.so

At this point, I cleared all caches, cookies, and history from my browser and rebooted my phone. Finally, and before going to Hulu.com, set your user-agent to desktop mode, which acts like an Intel Mac.

And then...

The sound is clean, but the videos are pretty laggy. I'll have to try this on a friend's phone to see if a newer, perhaps 1+GHz CPU, will improve things.

Opening and Getting Console on AT&T U-Verse 3800HGV-B

Written 2011-01-28

How to open your 3800HGV-B: First push in the plastic tabs on each side of the case and prop with a plastic knife. There are two tabs per side.

Halfway there.

Here's the top case now removed.

This is the bottom case and PCB.

What an interesting little 4-pin header. Could it be Power, Ground, TX, and RX?

It is indeed serial, running 3.3Volts@38400Baud. Pinout from top/rightmost pin should be power, serial from unit to pc, serial from pc to unit, ground. Power and ground might be swapped.

These next shots are some console output. Sadly I forgot to enable logging, so I only have pictures.

Changing your Google Chrome Flash Version

Written 2011-01-26

Sometimes it is useful to change your Flash version. Here's a simple trick to do that and see how apps behave.

Steps:

Get the current flash version from http://kb2.adobe.com/cps/155/tn_15507.html
grep for your current os + version in %USERPATH%\AppData\Local\Google\Chrome\Application\ . You'll find it in a subdirectory in gcswf32.dll
Hexedit and replace the os + version number. Because it is a NULL-terminated string, don't use any more characters than it already has.

...and tada!

Linux uses the prefix LNX and Android uses AND.

eZ430-F2013 Development Tool

Written 2011-01-26

Normally I take apart all new toys, but this one has only 5 parts, and is designed to come apart easily. It costs only $20, available from TI here. Previously, there was a giveaway for up to 3 units, although you had to wait up to a month for shipping and handling.

Packaging: I'm a big fan of the DVD case and double thick DVD case form factors. This comes in a single DVD case:

Looks a lot like a thumbdrive, which is a great form factor for something like this:

Here's the important part

It breaks apart into two pieces: the USB to RSR232/RS485 with TUSB3410, and the MSP430 itself:

For some perspective, this is sitting on an American Lincoln Penny

This looks to be a fun toy, for two reasons. One, is the MSP430 microcontroller. Two, is the USB to Serial Adapter. The serial adapter is a TUSB3410, which also holds an 8052 microcontroller with 16K of RAM, 10K of ROM, UART with DMA, dual 16-bit timers, IRDA encoder/decoder, and also I2C.

Well this is interesting

Written 2011-01-21

I photographed my teardown of my first SCS-26UC4. However, after bricking the second one, Doug Kelly, Kevin, and I disassembled it. Funny enough, the components weren't quite the same. Working on some information from Laboratory B, we re-examined the GPS port, to see if the pins could be scoped once it had a signal. More interestingly, the GPS chipset is different from the first unit we took apart.

This first one is a SigNav:

This next one is a Trimble:

Since the device was bricked, it probably doesn't need any signal shield...

And it uses an STMicroelectronics GPS Baseband processor.

Cool. There are datasheets for one of the Trimble units here. In all likelihood, the device is a Copernicus I/II, but it might also be a Condor. SparkFun sells a Copernicus for $45. A re-check of the serial i/o voltages should confirm which model. Potentially, there are three possible GPS protocols: NMEA(A text based standard), TSIP(Trimble Standard), and TAIP(Another Trimble Standard). This seems to explain it well enough. Hopefully we're one step closer to GPS in the cave.

My SCS-26UC4 is bricked

Written 2011-01-21

As a note, I bricked my unit. Inside U-Boot, in the MDOC interface, I picked a menu item Update/Verify. To note, this isn't a menu: it updates, and then it verifies. Since I hadn't loaded anything explicitly, it loaded garbage to the preboot environment (IPL or Initial Program Load). This happened a while ago, and hadn't posted about it. This is actually my second unit, as I bought one from the Verizon Store, then returned it as I found one $100 cheaper. Anything from here on out is static analysis, or analysis of the device without the CPU running.

OpenCaching.com RSS converter Online

Written 2011-01-16

It may not be much to look at, but my opencaching to rss converter will let you periodically search a small area for particular types of geocaches as they are posted on OpenCaching.com. There are two files to make it happen: index.html and rss.php, although rss.php is completely capable of running by itself.

Index.html prompts the user to set up a filter. This filter is then encoded into a URL passed to rss.php. RSS.php builds up an OpenCaching.com JSON query, and transforms the result into RSS. To install the RSS into your favorite reader, simply fill in the form click submit, and copy the current URL into your RSS reader.

The Original Red Vs. Blue

Written 2011-01-15

10 years later, I had almost forgotten this game: Tribes 2. Tribes 2 was fast combat, huge levels, jetpacks, and CTF. In short, the good things many games today lack.

http://www.penny-arcade.com/comic/2001/4/30/

DeWalt DC9096 Battery Pack Teardown

Written 2011-01-11

DeWalt makes some nice tools. Today, I disassembled a DC9096 Nickel-Cadmium battery. I was intrigued by the fact that there only appears to be space for 14 sub-c cells in the battery, but 15 cells are needed to reach 18 volts. Also, I wanted to examine the possibility of replacing the NiCad cells with NiMH cells, which can last much longer.

Remove these six screws

It should come out fairly easily, but some wiggling may be required

It doesn't add up. 14*1.2 = 16.8, not 18 Volts. There must be another cell

What's this?

Here it is. There didn't seem to be a good way to remove the other cells, so I cut them, and pulled the ribbon wire from the 15th cell. These appear to be welded on to the cell four times at each connection.

Pinouts:

Front pin (left in this image) goes to the white line in the pack.
Center in goes to the thermocouple ( not pictured )
Rear pin (right in this image) goes to the 15th battery cell.

Rooting the Astaro AP-30

Written 2011-01-07

Previously, I took apart an Astaro AP-30, and found the serial output line. Today, I found the input of the serial port, and rooted the device.

Starting from the square pin on the board,

Pin 1 - Power?
Pin 2 - Ground
Pin 3 - Serial from device to console
Pin 4 - Serial from console to device

In the picture, I used a 100Ohm resistor, because when you aren't certain it is the console, it pays to be cautious.

From my earlier post, you can see that the device doesn't have an OS installed; it boots from TFTP. It appears to do this every time, so potentially, you might be able to upload a new config to one by power-cycling it while running your own TFTP server.

. Press enter to activate console you say? After guessing the right serial line ( right next to the other), it is my old friend, OpenWRT! Here is another log. Also, we can see the list of OpenWRT packages here. Of note are the RT2860 kernel module and libpcap packages. So the chipset is a Ralink RT2860 with MIPS24K CPU. This puts it in a class similar to Buffalo WHR-G300N and One common chipset that provides both RT2860 and a MIPS24K is Ralink's own RT3050 chipset.

Uname: Linux OpenWrt 2.6.21 #1 Thu Nov 4 14:30:02 CET 2010 mips unknown

Samsung SCS-26UC4 Web Interface Password

Written 2011-01-03

The password for the web interface is stored in /udata/htdocs/.htpasswd in the form username:traditionalDESHash.

Digilent BASYS2 FPGA

Written 2010-12-24

I purchased a Digilent BASYS2 FPGA to continue my efforts with OISC. It includes four seven-segment displays, USB 2.0, VGA, four momentary switches, eight switches, and a PS/2 port.

The built-in demo shows color gradients(RGB332), a counter on the seven-segment displays, and results when buttons are pressed. The included loadable demo shows reading PS/2 scancodes. Possible uses include a USB terminal, by programming USB to act as a baudless serial port, and using the PS/2 and VGA for user I/O.

The chip is a Xilinx Spartan 3E-250, although a 100,000 gate version is available for slightly less. Sadly, the development tools are over 4GB, and hosted on a slow server. Digilent did include a nice utility for installing new FPGA configurations, but I cannot get it to work with Windows 7 x64. Windows XP 32, Ubuntu 64, Debian 64, and Ubuntu 32 worked fine.

Packaging is cool. The whole system fits in a multi-DVD case, and Digilent includes a free stretchy/clicky USB cable.

http://www.flickr.com/photos/40925843@N03/sets/72157625662412978/

Focus Enhancements TView Gold VGA->NTSC/PAL Converter Teardown

Written 2010-12-23

Longest title ever.

Anyhow, Thor dropped off a TView VGA->NTSC/PAL converter at the space, so I took it apart. http://www.flickr.com/photos/40925843@N03/sets/72157625537176771/

There is a spot for a header, but I don't know what it does. Someone wants to use it for the intended purpose, so I probably won't do any more with it.

Astaro AP30 Teardown

Written 2010-12-23

I was given access to an Astaro AP30 access point.

So I took it apart.

Also, the serial port appears to speak 56K @ 3.3Volts.http://www.flickr.com/photos/40925843@N03/5287383838/

Good grief. The antennas are glued down. At least they used a standard coax connector, U.FL.

Also, the CPU has a heatsink! A heatsink for a microcontroller.

Here's what the serial port says on powerup:

============================================

ASIC 3052_MP2 (Port5<->None)

Product Name: EAP9550

SDRAM CAS = 3(d1835272)

============================================

Please choose the operation:

1: Load system code to SDRAM via TFTP.

You choosed trigger_button--->0x1

http://www.flickr.com/photos/40925843@N03/sets/72157625536971053/with/5287362528/

My very own OBJ Viewer

Written 2010-12-20

The OBJ file format is a simple container for 3D geometry.

http://en.wikipedia.org/wiki/Obj

I needed a viewer for it, so I wrote one. So far it supports 3 element face-vertex form.Potentially, each face can have an unlimited number of vertices, but 3 serves my purposes. Also, there are several other items that can be stored, like texture and normal information that are not yet implemented, but it serves my purposes. Source code is available here: http://rsaxvc.net/code/obj3dviewer

This Post is for the UNIX Users

Written 2010-12-12

I am not alone.

How to stop AutoBoot on the Samsung SCS-26UC4

Written 2010-12-11

After downloading the Samsung source code, you may try to find differences between the stock sources, and the ones Samsung provides. This results in a list of the changes Samsung had to make to the stock sources to get their product going. After you build a serial cable for the femtocell, you also might've noticed the message 'Hit any key to stop autoboot', tried hitting a key, and the boot proceeded anyhow. Here's why.

Performing a difference between the u-boot 1.1.1 source from Samsung and the stock source from sourceforge.net, reveals a few key differences:

MDOC filesystem support. This is a type of flash + smart flash controller.
OMAP1710 support( http://linux.omap.com/pub/documentation/omap_1710v1.4.txt, read the u-boot section )
NFS boot timeout went from 10 seconds (stock) to 60 seconds(Samsung)
This tidbit in the autoboot/console code:

If you know C, this is pretty obvious. You have to type 's', 'y', 's', Enter, but after that, the unit drops into a standard u-boot bootloader menu. You have to be quick though; there's a 1 second timeout. Be careful in this menu system - you can rather easily brick your unit to the point of needing a JTAG cable to fix it. And without the 1710's datasheet, you won't get very far.

Where to get your Samsung Femtocell Source Code

Written 2010-12-11

If you're the proud owner of a Samsung SCS-26UC4, but the link for GPL'd code in the manual doesn't work, where do you go?

http://www.samsung.com/global/business/telecomm/opensource/femtocell.html

The source code must have moved since the manuals were printed, but included is all of the kernel, all of the boot loader (u-boot), cracklib, some of the GPS driver, and some other tools.

Mounting ISOs in HP-UX

Written 2010-12-08

First, su to root, then kick off some PFS(Portable File System) daemons into the shell background:
`nohup pfs_mountd&`
`nohup pfsd &`
Finally, mount the image with:
`pfs_mount -x unix /full/path/to/iso.iso /full/path/to/mount/point
The "-x unix" makes the mount so that filenames don't contain extra ISO9660 information and makes all filenames lowercase.
For more information, `man pfs_mount`

SCS-26UC4 Teardown Photos up

Written 2010-12-06

http://www.flickr.com/photos/40925843@N03/tags/scs26uc4/

Sometimes 64 bits just aren't enough.

Written 2010-12-05

Often times in programming, fixed point numbers are used. For example, a 32 bit representation of angle(0:1) using fixed point provides much greater accuracy than the same 32 bits used for a floating point representation. The resolution of the circumference of the earth in 32 bit fixed point is about a centimeter, but 10 feet or so in 32 bit floating point.

But what if we expand this same idea? The approximate diameter of the known universe is about 156 billion light years. There are 18446744073709551616 unique 64 bit integers. Thus, the resolution of a 64 bit scaled representation would be about 80,000 kilometers, or the radius of Jupiter. That is to say that the known universe is about 2^63 Jupiters wide.

Attaching a Console Cable to the Samsung/Verizon SCS-26UC4

Written 2010-12-03

You may have noticed that the SCS-26UC4 has what appears to be an HDMI port. However, if you have a meter, you may have also noticed that the HDMI port doesn't follow the HDMI specification - it has a 3.3 volt power line instead of a 5 volt line. Curiouser and curiouser, many of the other pins are not connected properly either for an HDMI port. It turns out, that this HDMI connector is actually a 3.3 volt asynchronous Linux console port.

Assuming you used a standard HDMI cable from Wal-Mart and a 3.3Volt FTDI serial adapter, make the following connections:

HDMI bare copper ground to FTDI Black(Ground)
HDMI White to FTDI Yellow (RX to computer)
HDMI Orange to FTDI Orange(TX from computer)

Now configure your terminal to 57600 baud, no flow control, and tada:

samsung_boot.log

Freespace 2 and Prime Factorization

Written 2010-11-30

Some time ago, I wrote a patch for the open-source continuation of the Freespace 2 project (FS2_open). Previously, each ship could only select between firing one set of primary weapons, or all sets of primary weapons. Firing individual weapons in a set was decided by the ship's data files.

However, games with more complex ships needed a different way to control weapon firing sequences. In particular, Fate of the Galaxy, a fan-made StarWars space shooter, needed a way for pilots to pick how many versus how quickly each bank should fire. As situations change, it may become necessary for pilots to reconfigure primary weapons to fire in parallel, for more of a shotgun effect instead of sequentially firing each weapon in the bank, which leads to a much more even power draw. The X-Wing is a perfect example of this - when attacking faster craft like interceptors, quad lasers are much more useful. But when attacking swarms of small asteroids, waiting for all four lasers to recharge is a detriment.

Because FS2_open is a game engine instead of a game, the same code needs to control the firing sequence of each ship. But how does a ship decide what firing sequences to expose to the pilot? The simplest answer is factorization. For example, a ship with four primary weapons in a bank may fire arrangements of 1x4, 2x2, or 4x1. A ship with six can fire 1x6, 2x3, 3x2, or 6x1. Over a long enough time period, the power draw should be consistent, so for each arrangement, delay the firing time by the number of shots, and it will average out evenly.

Finally, there is the question of how to factor these numbers in flight. The simplest solution would be a linear search from 1 to int(sqrt(n)). This could be done at mission load time for each model of ship... or it could be done at compile time with a look-up table and an assumption about the maximum number of firepoints per bank, which is the solution I implemented.

New Neighbors

Written 2010-11-23

I've moved RSAXVC.NET hosting to Santrex. The provide cheap monthly rates on OpenVZ machines. I'm a big fan of OpenVZ.

Here's my new neighborhood:

jetbooster.net (208.53.181.10)
soon.jetbooster.net (208.53.181.11)
lakebob.net (208.53.181.12)
manabless.net (208.53.181.13)
neoblink.net (208.53.181.14)
blueribbon.fusionhosting.us (208.53.181.82)
0v1.us (208.53.181.83)
stealth-ip.info (208.53.181.84)
stealth-ip.us (208.53.181.85)
gajas-boas.e.na.tuxSP.pt (208.53.181.98)
nao.sabes-nao.mexas.tuxSP.pt (208.53.181.99)
big.world.tuxSP.pt (208.53.181.100)
divulga-te.na.tuxSP.pt (208.53.181.101)
nao.sejas.forreta.tuxSP.pt (208.53.181.102)
xxx-twins.com (208.53.181.106)
algol.galacticcore.net (208.53.181.108)
capella.galacticcore.net (208.53.181.109)
euromarqueauto.co.uk (208.53.181.110)
www.xionweb.com (208.53.181.134)
vps.santrex.net (208.53.181.144)
sysnix.blacklistedhosting.com (208.53.181.147)
Xtronic.Net (208.53.181.148)
Krashed.us (208.53.181.149)
pasti.keren.la (208.53.181.155)
pasti.cakep.la (208.53.181.182)
crew.cybercafe.la (208.53.181.183)
acyied.com (208.53.181.198)
Shell.CrewByroe.Net (208.53.181.210)

libcutter/Cricut SVG Support

Written 2010-10-07

My latest update to libcutter implements basic SVG support.

https://github.com/vangdfang/libcutter

There are a few things missing, but ovals, arcs, circles, rectangles, and Bezier all work. In fact, everything I've drawn in InkScape has worked so far. The SVG interpreter is based on the early libsvg. Sadly, libsvg was merged into Cairo, and is now the libsvg-cairo. libsvg-cairo is much less sharable with other projects than the old libsvg.

Building FS2_Open on OpenSolaris

Written 2010-09-30

Issues

Solaris Lua does not have a pkg-config script
fs2_open does not have a build script option for OpenSolaris
Sun make is awful
dependencies
Optimized PowerPC code leaks into OpenSolaris x86 builds.
The word 'sun' is a reserved word on Solaris
the INTEL_INT macros are broken on everything but OSX, Windows, and Linux/intel.
Solaris/X/SDL has a bug if LANG is set, SDL will crash with an XError and a segfault.
PKG_CHECK_MODULES doesn't work very well on solaris

Dependencies - install with the 'Package Manager'

SUNWgmake
gcc-432
gcc-runtime-432
SUNWlibtheora
SUNWogg-vorbis
SUNWpng
SUNWjpg
SUNWlibsdl
SUNWaconf
SUNWgnu-automake-110
SUNWlua
OpenAL

Lua PKGConfig -here's my /usr/lib/pkgconfig/lua5.1.pc

prefix=/usr
major_version=5.1
version=5.1.4
#prefix=/usr
#major_version=5.1
#version=5.1.0
lib_name=lua${major_version}
libdir=${prefix}/lib
includedir=${prefix}/include
#
# The following are intended to be used via "pkg-config -variable".
# The location of the libtool library. This is used when linking to the Lua
# library via libtool.
libtool_lib=${libdir}/lib${lib_name}.la
# Install paths for Lua modules. For example, if a package wants to install
# Lua source modules to the /usr/local tree, call pkg-config with
# "-define-variable=prefix=/usr/local" and "-variable=INSTALL_LMOD".
INSTALL_LMOD=${prefix}/share/lua/${major_version}
INSTALL_CMOD=${prefix}/lib/lua/${major_version}
Name: Lua
Description: Lua language engine
Version: ${version}
Requires:
Libs: -L${libdir} -l${lib_name}
Libs.private: -lm
Cflags: -I${includedir}/${lib_name}

Patch 1 - code/globalinc/pstypes.h

remove the section dealing with INTEL_INT, INTEL_FLOAT, stwbrx...
replace with
- #define INTEL_INT( x ) ( x )
- #define INTEL_FLOAT( x ) ( *x )
- #define INTEL_SHORT( x ) ( x )

Patch 2 - code/fs2netd/tcp_socket.cpp and code/network/chat_api.cpp

under #include <sys/ioctl.h>
add #include <sys/filio.h>

Patch 3 - code/starfield/starfield.h

On Solaris, sun is a reserved word.
replace int stars_add_sun_entry(starfield_list_entry *sun);
- with int stars_add_sun_entry(starfield_list_entry *sun_ptr);
replace int stars_get_num_entries(bool sun, bool bitmap_count);
- with int stars_get_num_entries(bool is_a_sun, bool bitmap_count);
replace void stars_mark_instance_unused(int index, bool sun);
- with void stars_mark_instance_unused(int index, bool is_a_sun);
replace const char *stars_get_name_from_instance(int index, bool sun);
- with const char *stars_get_name_from_instance(int index, bool is_a_sun);
replace const char *stars_get_name_FRED(int index, bool sun);
- with const char *stars_get_name_FRED(int index, bool is_a_sun);
replace void stars_delete_entry_FRED(int index, bool sun);
- with void stars_delete_entry_FRED(int index, bool is_a_sun);
replace void stars_modify_entry_FRED(int index, const char *name, starfield_list_entry *sbi_new, bool sun);
- with void stars_modify_entry_FRED(int index, const char *name, starfield_list_entry *sbi_new, bool is_a_sun);

Patch 4 - code/starfield/starfield.cpp

On Solaris, sun is a reserved word.
in int stars_add_sun_entry(starfield_list_entry *sun);
- replace every variable sun with sun_ptr
in the following functions, replace all instances of the variable sun with is_a_sun
- int stars_get_num_entries(bool sun, bool bitmap_count);
- void stars_mark_instance_unused(int index, bool sun);
- const char *stars_get_name_from_instance(int index, bool sun);
- const char *stars_get_name_FRED(int index, bool sun);
- void stars_delete_entry_FRED(int index, bool sun);
- void stars_modify_entry_FRED(int index, const char *name, starfield_list_entry *sbi_new, bool sun);

Patch 5 - code/starfield/supernova.cpp

On Solaris, sun is a reserved word.
in void supernova_get_eye(vec3d *eye_pos, matrix *eye_orient);
- replace every variable sun with sun_vec
- optionally replace every sun_temp with sun_vec_temp

Patch 6 - configure.ac

Solaris is an unrecognized platform
after fs2_os_osx="no", add fs2_os_solaris="no"
Copy the stanza for *-*-darwin*)
and edit to look like:
- *-*-solaris*)
- # Solaris
- echo "Using Solaris 10 defines (for $host_os)"
- fs2_os_solaris="yes"
- ;;
Also, after the stanza starting with elif test "$fs2_os_osx" = "yes" ; then
add:
- elif test "$fs2_os_solaris" = "yes" ; then
- FS2_CXXFLAGS="$FS2_CXXFLAGS -I/usr/include/SDL"
- FS2_LDFLAGS="$FS2_LDFLAGS -lpng -lpng12 -ljpeg -lpthread -logg -lSDL -ltheora -lopenal -lsocket -lvorbisfile -lvorbis -lnsl -lGL -lGLU"
- AC_DEFINE([SCP_UNIX])
- AC_DEFINE([NO_DIRECT3D])
- ## don't need the CFLAGS here if recent SDL is used
Next, replace AM_CONDITIONAL(FS2_OS_UNIX, test "$fs2_os_unix" = "yes" || test "$fs2_os_osx" = "yes") with AM_CONDITIONAL(FS2_OS_UNIX, test "$fs2_os_unix" = "yes" || test "$fs2_os_osx" = "yes" || test "$fs2_os_solaris" = "yes")

Building

run ./autogen.sh or ./autogen.sh -enable-debug
edit configure and remove all calls+params to PKG_CHECK_MODULES, because you already installed all the prereqs...right?
run ./configure again
run make
executable will be in code/fs2_open_r for release and code/fs2_open_d for debug

Launching

copy binary to game data directory
run unset LANG - This sorta fixes a solaris/SDL/X bug
run ./fs2_open_r or ./fs2_open_d

Building OpenAL on Solaris

Written 2010-09-19

Get an OpenAL software implementation from here: http://kcat.strangesoft.net/openal.html, then downloads, then latest release (openal-soft-1.12.854.tar.bz2 worked great for me)
run 'cd openal-soft' ( or into directory from latest release)
run 'cmake .' if it complains about not finding nanosleep, remove the offending lines from CMakeLists.txt
run 'make'
run 'make install'
At this point, you need to manually copy the files to their locations. I used /usr/lib, /usr/include, and /usr/bin, because that's where fs2_open expected to find them.
run 'cd sfw_stage'
run 'sudo cp -r include/ /usr/include/'
run 'sudo cp lib/libopenal* /usr/lib/'
run 'sudo cp lib/pkgconfig/openal.pc /usr/lib/pkgconfig/'

Cricut USB Protocol Reversed

Written 2010-07-21

Some folks up at CCCKC have figured out how to talk to the Cricut over USB. This paves the way to drive the Cricut with a laptop to cut puzzles, engine gaskets, PCBs, and much more. Before this software, crafters were limited to the shapes they could purchase from ProvoCraft.

http://built-to-spec.com/cricutwiki

https://github.com/vangdfang/libcutter

Cricuts can be upgraded with better knife tools to cut puzzle materials. This same knife can cut gasket material. Also, a small PCB Etch Resistant marker can be installed in the Cricut, and draw on copper clad board.

NA-FS, Not Actually a File System

Written 2010-02-24

I've worked with several embedded filesystems, from jffs2, to FAT12/16/32, to hard coded tables of filenames and their containing data. What I would like to propose, is not a filesystem, per-se, but an interface.

I understand that no one filesystem fits every problem. If you need to be able to open a file quickly, FAT won't do it. If you need a tiny driver, don't go with Reiser. If you need flash wear-leveling, your choices are pretty limited to start with. If it only exists in RAM, who wants to bother with a filesystem anyways? My point is that the diversity in these systems is necessary.

However, interfacing with so many filesystems is a nightmare. Even Linux drops filesystem support quite regularly. How can we improve this? By offloading the burden of filesystem support to the device itself. By necessity, the device already has a driver, and this method adds new capabilities.

Real World Example One: A media player acting as USB mass storage. After loading new MP3s, a media player needs to open each file and scan it for meta data such as the band, length, and volume profile. If the media player were the storage back-end directly, every time another device sends it a file, it parses the file as part of the storage operation, preventing a lengthy power-on.

Real World Example Two: Utilizing a system like NA-FS, intelligent memory cards could be swapped with any device supporting NA-FS, without actually changing the backing store. For example, if SD was designed with NA-FS as the interface, then this SD/SDHC/SDXC monkey business would have never been an issue. Small cards could use a compact filesystem like vFAT16, and larger devices could use whatever they needed, from exFAT to ZFS-based RAID.

Technical Example: In Linux, there are filesystems like /proc, which are generated at file-access time. NA-FS would support this intrinsically, as actual file i/o is passed down to the device OS.

Technical Example: Compression - as an intelligent storage device, the bytes on disk don't need to match the bytes at the transaction layer. A smart enough device could compress plain-text documents or use PNG compression for uncompressed images.

Downsides:

Currently, the media controllers in memory devices are quite limited. Often, they only handle flash wear-leveling, and perhaps block-atomicity.
No current implementation/standard, just a though I had
How would features not standard across all filesystems be handled? ACLs come to mind.
Some features, such as automatic image compressions, would be computationally expensive to implement, for a memory card.

Upsides:

No device would have to enter 'Mass Storage Mode'. Devices could keep functioning normally, and even use NA-FS to transfer information, much like /proc on linux.
No device would need to support more than two filesystems: NA-FS, and the backing store of choice.
A filesystem need never worry about a poor fs implementation on a device corrupting it.

Toshiba T135D

Written 2010-02-13

Now that I've had this machine for a few weeks, I'm pretty happy with it. The battery lasts several hours when running Visual Studio 2008.

The good:

Automatically underclocked CPUs.
Discrete video card (shared ram though)
Virtualization and 64 bit support
Good screen quality, and LED backlight.
Built-in accelerometer.

The bad:

The frame isn't actually square, meaning that when I set it on a table, the front right corner doesn't touch.
The frame is pretty flimsy - when setting on above mentioned table, the weight of my palms is enough to bend it down.
The keyboard isn't flat. Near the 'Y' key, the keyboard has a bulge of 2-3 keys in any direction.
The keyboard is squishy. Even after making a full keystroke, the keyboard continues to depress.
The screen is very light, but now the screen is so light the individual keystrokes cause it to bounce back and forth.
Linux support, as of Ubuntu 9.04, doesn't support either of the networking interfaces.
This thing gets pretty hot, and the fan runs quite a bit.

Although the line 'Toshiba Media Controller. Copy protection technology, if any, associated with the content may prevent or limit sharing of content.', does worry me, it's nothing compared to the USB whitelists/blacklists of the HP Mini 311.

My Toshiba T135D Came in the Mail Today

Written 2010-01-20

After rethinking the requirements of a laptop. I've come up with the following:

Real video card (shared memory is fine, just no Intel GMA )
CPU Power equivalent to Atom 330 (Dual Core 1.6 Ghz Included )
CPU Virtualization Support (Turion Neo has it, Atom 330 doesn't)
CPU 64 bit processing (Turion Neo has it, Atom N280 doesn't)
Built In SD Slot or SSD Drive ( SD Slot)
Hardline Gigabit-Ethernet or N Wifi ( Both )

First impressions:

Large deadzone between left and right click. The mouse is a single bar, like a see-saw, with microswitches on each side.
Solitaire is far cooler than Windows XP and previous editions.
Internet Explorer comes only with Google Toolbar, which is far better than HP's version.
The screen is bright, better resolution than my old 13.3 inch Macbook.
Smaller and lighter than the Mcbook, but bigger than the HP 311.
The 'Toshiba HDD Protection' system often pops up while the machine is sitting still informing me that a vibration was detected and that the hard drive head was parked to protect it.
Windows UAC is pretty annoying.
There's a PrintScreen key!
Paint has been reworked.
No weird USB Whitelist junk like HP uses either.
There's a built-in watt meter application. This should just be a simple power controller query, but when combined with a graph, it is pretty neat. The machine idles around 3-4 watts.

I never thought I would buy a new AMD Processor, ever since Intel's Core 2 Duo stompfest, but on this small market, low speed high capability CPUs, AMD really has something. Combined with the fact that Intel has attached a chipset directly to the CPU on their newer units, I just can't see usable 3D graphics with the new Intel Atoms. The whole thing's pretty slick.

On a side note, I managed to never use Windows Vista, except for one time I used it to launch Firefox.

My HP 311 went back in the mail today!

Written 2009-12-28

Well, it could've been a nice machine except for:

Suspend would reboot the machine.
USB Whitelist - Only HP Approved devices can be used. This prevents my pl2303 usb serial adapter from working, as well as any wireless 3g modems.
Nvidia Ion LE - The Mini 311 is marketed with Nvidia Ion, but comes with the less capable Ion LE.

So back it went to HP. HP does not refund the original shipping (to you), but in this case it all worked out because they offered free shipping right before Christmas.

My HP 311 came in the mail today!

Written 2009-12-24

First Impressions:

First power-on: Expect to spend about half an hour with the machine rebooting and installing software. It's a shame HP didn't think of doing this somewhere else. Maybe they could've done it in the factory, or maybe they could've done it as an OS image. About a minute after finally booting into Windows, the machine went into suspend. After it woke up, it died. Now it seems to be working fine.

Screen: Screen seems pretty bright and quite clear. Resolution is 1366x768, which isn't bad for a screen that size. For comparison, my macbook is a 13.3 inch 1280x800 screen. Slightly more pixels in a smaller screen is a good thing. Also, since the screen is LED backlit, there shouldn't be any inverter problems like the macbook was plagued with ( 5 inverters have tried and failed, Apple keeps feeding the machine new ones ). I haven't had a chance to take the 311 outside in direct sunlight yet, but I have high hopes.

Keyboard: The keys are nicely sized, but there's little border between the keys themselves. This leads to often losing where your fingers are located. Also, the blind-person nubs are real short and easy to miss.

Trackpad: I like the trackpad. The trackpad is large for this size of laptop, it is smooth, and the built-in gesture support is great. However, it would be nicer if the trackpad weren't at the same height as the buttons. This often leads me to graze the trackpad when clicking, making the cursor fly off to the corner of the screen, although I'm sure I will get used to it.

Bloat-ware: Expect a few columns of icons on the desktop, about 7 or so in the tray. HP was even so nice as to include another Internet Explorer Toolbar!!! Can you guess what it does? It does an MSN/BING search, just like the built-in search bar.

Networking: There are three wireless options available for this machine: Wifi, Bluetooth, and 3G. I went with Wifi and Bluetooth, planning on tethering through my phone while away. I went with the upgraded A/B/G/N Wifi card. It's a Broadcom, but seems to work fine. However, HP installed a generic wireless card manager ( It handles Bluetooth, Wifi, and presumably 3G ), and so Wndows fights with this thing constantly. Once you kill off one of them the Wifi works great. Bluetooth tethering was painless. I did notice that HP opted to cut the ethernet port down to 10/100 instead of 10/100/1000 like the ION chipset supports.

Video Card: I bought this machine specifically because it has built-in NVIDIA ION graphics. I wanted to play with CUDA, and I was tired of my Intel 945 software-emulated GPU. It appears that this laptop does not come with NVIDIA ION, it comes with NVIDIA ION LE. I appears to use 128 Megs DDR3 ( other real ION systems use 256 ), and may have some other limitations. I am not amused and will be calling HP about this.

Thoughts: Pretty sweet. Time to set up some games.

Coffee.

Written 2009-12-07

Day 1: Oh man, out of sugar. I already poured this cup and added the cream. Hmm, not too bad.

Day 2: No cream? Well, at least there's sugar.

Day 3: I now take my coffee black.

Srsly Subversion, Srsly?

Written 2009-11-22

configure: Configuring Subversion 1.6.6
configure: creating config.nice
checking for gcc... gcc
checking for C compiler default output file name... a.out
checking whether the C compiler works... yes
checking whether we are cross compiling... no
checking for suffix of executables... 
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking for g++... g++
checking whether we are using the GNU C++ compiler... yes
checking whether g++ accepts -g... yes
checking how to run the C preprocessor... gcc -E
checking for a sed that does not truncate output... /bin/sed
checking build system type... i586-pc-linux-gnu
checking host system type... i586-pc-linux-gnu
checking target system type... i586-pc-linux-gnu
checking for grep that handles long lines and -e... /bin/grep
checking for egrep... /bin/grep -E
checking whether ln -s works... yes
checking for a BSD-compatible install... /usr/bin/install -c
configure: Apache Portable Runtime (APR) library configuration
checking for APR... no
The Apache Portable Runtime (APR) library cannot be found.
Please install APR on this system and supply the appropriate
--with-apr option to 'configure'

or

get it with SVN and put it in a subdirectory of this source:

   svn co \
    http://svn.apache.org/repos/asf/apr/apr/branches/1.2.x \
    apr

Run that right here in the top level of the Subversion tree.
Afterwards, run apr/buildconf in that subdirectory and
then run configure again here.

Whichever of the above you do, you probably need to do
something similar for apr-util, either providing both
--with-apr and --with-apr-util to 'configure', or
getting both from SVN with:

   svn co \
    http://svn.apache.org/repos/asf/apr/apr-util/branches/1.2.x \
    apr-util

XEvil 2.02 on Mac!

Written 2009-11-11

Xevil source patched by Debian project/me to compile on OSX Snow Leopard. Builds both i386 and ppc objects. However, the linker will only make your local architecture, unless you add "-arch ppc -arch i386". Might build on ppc64 and x86_64, but I haven't tested it. Download it here. You might also have better luck compiling this version on other newer unixes as well, since it looks like the codebase stopped development around 2002.

OISC Hardware: DC Supply and Clock Generator

Written 2009-10-24

Although this is nothing special, and likely won't drive the final project, I took some pictures anyways just to keep the documentation complete.

It contains:

78T05 CT 3A Regulator
2x 25V, 3300uF regulation caps( one input, one output)
1 555 Timer as a clock generator producing 9.346 kHz.
1 fairly sketchy capacitor of unknown capacitance and age.
2 47kOhm 20% Precision
4x1 Header for power input( DC+, GND, GND, DC+ )
2x( 3x1 Headers) for output ( Ground, Clock, Power )
12-611B Datek 21-110 CIC ( Small Solder Protoboard)
Two pennies. By the way, if you use older, copper pennies, instead of these zinc ones I happened to have in my pocket, you'll get way better heat dissipation. Whatever you do, don't use steel pennies.

Side Profile Shot:

On second thought, maybe those caps are just a little overkill.

Disappointment with the NComputing L120

Written 2009-09-27

NComputing is a company with a great idea, but terrible implementation. Their product is a series of thin clients. Some run over IP, some run over with a proprietary PCI card. However, in the end you're much better off with a standard Remote Desktop thin client and a Windows 2003 machine.

If you want to use an L120 with Windows, you will need to sacrifice an entire windows machine. The reason is that the software that NComputing provides hack deeply into the internals of Windows. The result is that whenever a Windows patch comes out, you'll need to make a system restore point, as it is very likely nothing will work quite right until NComputing produces another version of their software. You can have security, or functionality, but not both, often for several weeks at a time. There are also some software packages that it is incompatible with. As of writing this article, after the last round of patching, unpatching, and repatching, my L120 will log in, and work for about a minute, before the XP machine dies.

If you want to use an L120 with Linux, good luck. At the time of writing, their Linux server software for the L120 has been in closed beta. I signed up and got a copy, but discovered that the software only works on openSUSE 10.2, which isn't even available from openSUSE anymore. (10.3 and 11.1, which are available, don't work). Also, you'll still need a Windows box to push updates to the devices.

So, on the packaging these products say they're compatible with 'Linux', meaning one specific distribution, one specific kernel version, eventually, somehow, maybe. It sounds like a cheap way to put computers around your house without the work of keeping software up to date, or extra power bills, but it is really a big waste of time. You're better off with a standard X11 or RDP client than the hassle of an NComputer.

*Please note, this was based entirely off of my experience with an L120. Apparently, the X series have a lot better linux support, but do not run over IP, so they have to be placed pretty close to each other, something that prevented me from purchasing one, as I wanted to put these things around my house.

Syntactic Sugar

Written 2009-09-26

In C I am afraid of large functions, because they're often hard to read.

In C I am afraid of small functions, because they usually lead to a long call stack.

In Perl I am afraid of single lines, because of both of the above reasons.

OISC Hardware: Program Counter

Written 2009-09-24

After building a new bench power supply, I present, my first modular program counter!

What's so special about it? You can daisy-chain them! In fact, there are two little four bit program counters running as one eight bit program counter right on that board. This isn't a terribly huge feat for a program counter, but eventually there will be an entire CPU built this way.

Parts / Tools / Technical details:

1 555 Timer as a clock generator producing 9.346 kHz.
2 74LS193 up/down counters used as daisy-chained 4 bit counters.
1 fairly sketchy capacitor of unknown capacitance and age ( the only free one I had at the time ).
2 47kOhm 20% Tolerance resistors (
I had to deal with that unknown capacitor ).
1 Digital Scope from the late 1980s, w/1MHz sampling rate.

The clock overflow pin on the 74ls193 acts as a ~16x clock divisor. The first 74ls193, which given the clock input of 9.346kHz, iterates through all 16 values, then overflows at 584.6 Hz into the second 74ls193. The second 74ls193 ialso iterates through its 16 values, but ~16 times slower, then overflows at 36.56 Hz, which is right on the money for what I'd expect from the input.

For right now, the LOAD and CLEAR lines were manipulated by hand. In the final design they will be exported across modules. LOAD will be driven by the ALU's output, and CLEAR will be part of the reset control.

Attach a DC Utility Port For Extra Peripheral Power

Written 2009-09-14

So, I'm building a computer for my car. I'm trying to make a USB interface for all of my peripherals. Mostly this involves embedding a USB->Serial Adapter in a few things. However, some of my devices require a little more power than USB can supply, or they require just under the limit, and I'm hesitant to attach too many of them to the computer, especially since some of the USB cables may be around 15 feet long.

Enter the utility port. By adding a DC Barrel plug socket to my devices, I can run a two wire power line along with my usb cables. The Power line takes the current of the peripheral, while USB still supplies power to the USB->Serial or USB->Can/SPI/I2C converters.

First you'll need a spare back plate.

Now you'll need some banana posts/terminals. I ordered some from digikey.

Drill some 1/4 inch holes in the back cover plate for the banana plugs. Add as many as you like. I went with four holes, 2 red posts with 5 Volts, and 2 Black Posts for grounding, because I'm only running 5 Volt gear on this machine. Use the nut provided with the banana plug to attach it to the cover plate. Be Careful: make sure to put the 2.18mm hole facing away from the other banana connectors. I used some gorilla glue to make them stronger.

Next you need some Ring Connectors, and a molex socket. I got these ring connectors from the ACE hardware up the street. I forget what size you need, but they were ACE's smallest blue collar connector. You can also just bring a banana socket in for sizing. For a molex socket, I used an old cpu fan connector from a 486 I was throwing away.

Assemble the molex and ring connectors in whichever configuration you like. I have both grounds connected, and connected to both black terminals and the molex socket. I have both red terminals connected together, then connected to the molex socket.

Who Knew, GNU?

Written 2009-08-09

"The right to sell copies is part of the definition of free software." I've known this for a long time, but only today did I think it ironic to sell 'free' software. As a side note, you can buy Tux Racer at Microcenter.

Hacking up the Computer Lab International et3000w and et5000n

Written 2009-08-08

So as part of a bulk deal, I picked up a number of these boxes. Computer Lab(what a name) makes a number of these machines, with customizable hardware as well (you pick ram, rom, and OS).

The first machine is a Computer Lab International et3000W. This little guy comes with:

32 Megs of Ram (SD133, in a slot)
8 Megs Flash Rom, in a 32Pin Dip Socket
some boot rom (in a socket, unknown type)
300 Mhz AMD Geode Processor
VGA, RS232, IEEE1284, Ethernet(10?), USB(1/1.1/2?), 2xPS/2 ports, and Audio I/O.
Windows CE .net 4.2
1x unused IDE port.
Standard IEC C13 Power Socket.
Case can be opened without screwdriver :)

Next is the Computer Lab International et5000n.

128 Megs of Ram( SD133, in a slot, 2 slots total )
16 Meg IDE SSD (Power Quotient International)
Via C3 Processor @ 533 MHZ
VGA, RS232, IEEE1284, Ethernet(10/100), 2xUSB(1/1.1/2?), 2xPS/2 ports, and Audio I/O.
Windows CE.net 4.2
1x unused IDE port.
Standard IEC C13 Power Socket
PCI slot, with PCI->PCMCIA adapter (PCMCIA usable from outside)
Case can be opened by removing 3 screws on the back plate.

The et3000w's motherboard appears to be some sort of custom, system specific board. Indeed, even the power supply is built into it. Due to the custom ROM chip, this looks like a bit of a dead end. Now, after opening the 5000, I realized it was nothing more than a mini-ITX board. There's a little tiny ATX PSU off to the side, a spare molex power plug, 2x IDE ports this time, 2 SDRam slots, and a bios battery. The 3000 might have one too, but I put it away for now.

When these things boot up, they show a Computer Lab logo, then boot Windows CE. If you look at Computer Lab's website, you could order similar machines with Windows XP/Embedded or Linux. Also, you could pick how much RAM you wanted.

Back to the battery. If you take it out, wait a minute or so, and put it in, guess what happens on the next boot - a standard Award Bios comes up complaining about invalid configuration. It seems all they did was upload a new boot image, turn on fast boot, and disable the keyboard messages. However, they seemed to do a poor job. The bios has options for a floppy drive, of which there is no connector, and for some other non-existant ports. So now we can load code to this guy. A standard IDE drive should do the trick. Also, the PCMCIA is really just a PCMCIA->PCI adapter, so potentially you could add almost anything.

The 16 Meg SSD is pretty neat, and very small. Also, it requires no external power, unlike most CF->IDE adapters. It contains a Windows CE .net 4.2 Build with a load of terminal emulation software and some version of Internet Explorer. The 5000 boots up noticeably faster than the 3000.

*Note: The numbering scheme seems to go something like this: et<motherboard identifier><Original OS Install identifier>

Modular RISC Machine: Design Process

Written 2009-07-16

So, I'm going to build this little modular RISC machine.

Core isn't the right word for a division in this processor. I think `Channel`, is a bit more accurate. Although the design should scale for an arbitrary number of channels, and for channels with an arbitrary bit width, I plan to make a dual-channel, 4bit channel demonstration( an 8 bit computer). With any luck, I should be able to extend the design to a 16 bit computer, which is the limit of the largest SRAM I've been able to find for a low price. So the sections I need to design/purchase:

Modular Program Counter, parallel load, with Overflow output.
Modular ALU with carry-in and carry-out
Registers
Control Unit
2->1 Multiplexers
TriState Buffers

Parts List:

3 Registers
1 ALU
1 Program Counter
1 Address Latch

Ghost in the Shell

Written 2009-07-13

I hope the cables in 'Ghost in the Shell' are firewire.

Core Failure

Written 2009-06-28

So, for the past few days, my computer has been acting very strangely. It often locked up or rebooted all by itself. Today I finally learned what had caused it. I happen to own of the earlier dual-core processors, a Pentium D with hyperthreading. Running HTOP, I could see only 2 processors, where there should be four. Usually, CPUs in linux count first the real cores, then the virtual cores. In my computer, Cores 0 and 2 were the only ones running. I feared a corrupted bios had turned off my second core, but alas, Core 1 is dead. I'm quite happy with the fail-state though. Once the core completely died, my computer is running fine...well, half as fast as it used to, but it's still pretty good.

Cheapest webcam on ebay, now with OpenCV support!

Written 2009-06-23

I while back I bought a Sigma Micro USB webcam. These little guys will run you about $5 with overseas shipping included. They're supported by the UVC-Video drivers in linux, but until recently, OpenCV didn't support their output format. But with OpenCV 1.1, all that is fixed. The standard webcam interfaces work great with this camera, and I've started playing with some corner detection on it.

However, there are a few catches:

Don't expect to get more than a few frames per second right now. With no processing, I get 6fps.
Linux can only drive the camera at 640x480 for right now
The Auto White Balacing is terrible.

All-in-all, probably worth about five dollars. At least it allows you to do basic opencv camera work.

A New Idea about Realtime Image Processing

Written 2009-06-12

The more image processing I deal with, the more I'm convinced a better solution is needed than pre-made image toolkits. Evaluating a 640x480 image, even just once, can be processor intensive. Every processing toolkit I've seen has the same basic parts, an image class/interface, some I/O functions, and some image modification functions. OpenCV at least gives you raw pointer access to the image, but the problem here is that each of these transforms are always applied sequentially.

Some time ago, I wrote a simple image-segmentation algorithm. However, because this was a realtime task, it needed to be made as fast as possible(see here). What I completed was a giant mess of c/c++, with loops hand-unrolled to remove edge conditions from them. With optimization, 50fps worst-case (P4, 3.2). However, later I needed to add some functionallity to the segmentation algorithm. Re-opening this code was a nightmare. Even though I had plenty of comments, it would've been about 1/6 the size if I hadn't been crunched for CPU power.

So, raw image access and image-optimized code are still necessary for real-time image processing (I look forward to doing it in python, whenever that becomes doable).

Also, the fewer passes you can make through an image, the better( usually ). The problem with most(perhaps all) premade image toolkits, is that they encourage sequential execution of functions. For example, if I wanted to double contrast, then apply a threshold, and then halve the contrast, the image would be contrasted, then thresholded, then contrasted. We've made a total of three passes through the image to do our three functions. O(N) on the number of functions isn't bad, but it isn't necessary either. It would be faster to say, Pixel(x,y) = (composition of multiple rules). In this way, the pixels, which now have a single, CPU intensive step, only have to be read and written once. I don't know of a single compiler that can optimize this case.

If we can break our image processing down to the pixel level, we can apply multiple complex transformations in one step. If we can do that, we can minimize memory bandwidth usage and increase our cache utility.

So, apparently MT works in ELinks

Written 2009-04-30

Finding a New Fileserver OS: Step Four - Debian Saves the Day

Written 2009-04-11

Debian/kFreeBSD. Wow. Everything I've ever wanted.

Stable/Fast kernel
ZFS support
OpenGL Acceleration
GNU Userland

Right now I've got it in a virtual machine, but it is amazing. Just apt-get whatever you need.

Design of an Modular, Arbitrary word length RISC machine

Written 2009-04-07

I've always been impressed with the simplicity of the OISC ( one instruction set computer). One day, I had an interesting idea-Why not build it? In about six hours I had finished my first design on a whiteboard.

Now, the 7400 logic series contains a lot of 4-bit chips(registers, a counter I could use for a program counter, tri-state buffers, multiplexers, and full adders). A four bit OISC machine with direct addressing has only 16 words. Keep in mind that an instruction is 3 words. So a 4-bit oisc could only hold 5 instructions, with a 4-bit bank for data. So I set out to design an 8-bit machine.

The first step in designing the 8bit machine was implementing effectively 8bit components out of 4bit parts. Turns out, that almost all the necessary constructs are simply 2x4bit constructs. Here's an image of that first machine, partially implemented in oregano.

Did you notice the symmetry? You can implement an 8bit component with 2 4bit components and some control glue.

Sadly, I had forgotten one very important step in the OISC architecture, and so my design was worthless-or so I thought. My roommate Ryan Meuth (of irontaco.com fame) showed me that what I had designed was a cellular automata simulator. It was pretty cool, and I'll upload the schematics someday. However, it still wasn't a turing complete machine, so I set out to design it again.

Finding a New Fileserver OS: Step Three - Giving Up

Written 2009-04-07

Well, I didn't make it. I spent a while with OpenSolaris, which was the closest I've come to a non-linux posix I could love. Original Solaris is still a load of outdated software, and has the least user-friendly cli I've ever used. Nexenta is a half implemented joke. FreeBSD would never boot, and fuse doesn't support things like mmap.

So I'm back to linux Raid5/jfs. It's a little slow, but my data is all safe. In fact, I had a sata cable come loose recently, and it emailed me, and I plugged it in, no questions asked, a few minutes later the drive was done.

Finding a New Fileserver OS: Step Two - Testing

Written 2008-11-22

Having decided on Sun's ZFS, I am left with the following list of operating systems to test:

Solaris
OpenSolaris
Nexenta
FreeBSD
Linux2.6/Fuse

Virtualbox Testing Setup(real setup):

1(4) CPU

1(4) Gig Ram

ide0,0: 16 gig hard drive-6 for OS, 10 for ZFS (80 gig OS drive)

ide0,1: DVDRom

ide1,0: 10 gig for ZFS

ide1,1: 10 gig for ZFS

(4x500Gig Sata)

Solaris

I've worked with Solaris before, but I can't say I enjoyed it. It was a lot of work to install, but has been running non-stop save power outages for a while now. It's actually remarkably efficient, and a rather slim kernel, but I really would like to use this box as a desktop now and then.

OpenSolaris

OpenSolaris has come a long way. I mount the ISO, boot off the CD, it asks me to choose a kernel with GRUB(I choose x86 + gui), and then the machine boots into a full Gnome Desktop. Next a fancy graphical installer installs to the 6 gig partition with standard ZFS flawlessly. I can even play gnibbles in the VM. After a reboot, everything works perfectly. Creating the RaidZ pool was equally easy (for someone who knows the Solaris command-line). It even has a graphical package manager, not the same as aptitude, but I can deal.

Nexenta

Nexenta is supposed to be the Ubuntu of Solaris. However, it seems a lot like they repacked Solaris packages as DEBs. There's a ton of packages named SUNW....blahblahblah...blah...packagename. Creating a RaidZ pool was rather easy though, as it should be, and completed the creation in less than 30 seconds. However, Nexenta doesn't support ZFS compression or encryption, which is a huge bummer. ZFS compression, applied sparingly, often increases write speed at the expense of some minor CPU performance.

FreeBSD

Odd - I couldn't get FreeBSD to install in VirtualBox. Backburnered.

Linux2.6/Fuse

This worries me: http://zfs-on-fuse.blogspot.com/ So, it seems that ZFS/Fuse currently suffers from the Gigantic Raid Write Whole. You have to disable your drive write buffers if you're using anything like MD or LVM with it, which often makes things very slow. Backburnered.

Finding a New Fileserver OS: Step One - Filesystems

Written 2008-11-12

Recently, I started to run out of hard drive space. Previously, I was using this machine as a Linux desktop/fileserver, but as I've become more accustomed to running OSX/Windows2003/Linux2.6 on my laptop, I've lost the need for a local console.

Previous setup:
2x250Gig 7200RpM Sata drives in Raid-0
1x80Gig drive for backup

New Setup:
4x500Gig 7200RpM drives in Raid5/Raid6/ZFS/BTRFS
assuming 20MB/s = 160Mb/s
1x80Gig drive for OS

Comparisons:
FSCK Checking Time
Compression
OverHead
Data Safety
Speed ( I want 400Mb/s from the array)

ZFS
I've used ZFS with Solaris 10 before, and I must say I'm impressed. It offers dynamic compression, multi-drive mounts, and redundancy up to a level of your choosing. We used ZFS on the ACM fileserver. It handles 12 drives (18Gig, 15kRpM scsi drives) with double redundancy (3 failures must occur before data loss). The main bottleneck in this system is the 100Meg Sun HappyMeal Ethernet card, which it can effortlessly keep filled with AFS data. Also, the CPUs in this box are 400Mhz each, so I figure 4xP4s will have no trouble at all with fewer drives. What about a filesystem check? ZFS keeps things stable at all times. A full scrub is relative to the amount of data stored, not the size of the disks. Supported Operating Systems: OSX Snow Leopard, FreeBSD, Solaris, OpenSolaris, and Linux/FUSE.

BetterFS
BetterFS is supposed to replace Ext4 when it is done. It isn't done yet though. When done, it should feature multi-drive mounts (but only Raid0/Raid1 style), CopyOnWrite for everything, ACLs, and online FSCK. However, at time of writing, it isn't done yet. Supported Operating Systems: Linux.

Raid-5 with a standard filesystem
Raid-5 is a method for adding checksumming to a group of block devices. It presents you with one virtual drive consisting of many hardware drives. However, you lose 1 drive out of n for parity, so your efficiency is (n-1) drives, and there have been some problems with Raid5 scaling to very large numbers of drives. Supported Operating Systems: Linux, FreeBSD, NetBSD, Solaris, OpenSolaris, and Windows Server.

Raid-6 with a standard filesystem
Raid-6 is much like Raid-5, but you use the equivalent of two drives for parity, so your efficiency is (n-2). This also takes more cpu time to write, the Raid has to be checked in its entirety, not just the space with files on it, and still has scalability problems. Supported Operating Systems: Solaris/OpenSolaris, or any other with a hardware card.

HammerFS
HammerFS is a relatively recent filesystem for DragonFlyBSD. It's neat in the way that it supports clustering, data checksums, multidrive mounts, and dynamic rollback/versioning. However, I only have one machine, so its not the best of options. Supported Operating Systems: DragonFlyBSD.

Decisions, Decisions
Factors:

I don't want anything that isn't supported by at least two operating systems. So long HammerFS and BetterFS.
I don't want to be locked to a single model hardware card. So long Raid-6
I'd like Linux(liveCD rescue) and OSX support (laptop rescue)
Fast Fast Fast

I'm really just left with ZFS. I mean, Raid-5/EXT3 and Raid-5/JFS have worked for me in the past, but I imagine checking a 2TB array will take forever with that setup. I've seen ZFS perform in the past, so it will be the first FS I test. Now the question is: what OS do I run?

JAUS: a testament to poor design and early adoption.

Written 2008-06-10

Suppose two very different robots meet each other some day. Wouldn't it be wonderful if they could handshake, and then talk to each other in a very nice, simple, predefined way. It's such a wonderful concept. In fact, it is such a wonderful concept that the US Department of Defense wanted it done, so it put together a plan. JAUS is what they invented. I went to a conference last week to hear about it.

JAUS (the Joint Architecture for Unmanned Systems) is an acronym for exactly what it does not do. Joint-usually meaning that the team that worked on it is from a diverse group, but I suspect that the group did not contain any experienced network programmers because of one glaring error.

Networks have been big endian since the beginning of time(teletypes), and from the time x86 workarounds have been in place, programmers have been using things like htons() and ntohs() to make sure their data is in the proper format for networks( big endian ). Not so with JAUS! With JAUS, x86 programmers no long have to worry about network byte safety, because everyone else will!

That's the level of enthusiasm about this particular item the speaker had. I suspect that the JAUS folks first started laying out some code and later discovered that JAUS on Macs couldn't talk to JAUS on PCs. Next they probably figured out what endian-ness is. The speaker basically recommended I keep two copies of all code: One for big-endian, and one for little-endian.

Now, they also call JAUS an architecture. To say that, is to say that something can be built on JAUS. However, the only thing that JAUS is good at doing, is message passing. You know what else is good for message passing? UDP. JAUS is a decent multi-component message passing standard, but because the rest of JAUS is so dysfunctional, you can't really rely on JAUS messages to go where you want them to, do what you intend them to do, or accomplish much of anything useful. You know what else is good at putting data where it belongs? IP addresses and TCP ports.

Back to the two robots meeting in a field. Robot A has data for Robot B. These two robots are within PAN(personal area network) range of each other, so they can send data back and forth. The only question is: can they understand each other? Probably not. What if Robot A and Robot B were designed, built, and programmed by the same people? Well...maybe. The problem is that JAUS has its own identification system built into the JAUS header. The JAUS identification system lets you specify which component you want to message behind the same IP address. So, suppose A wants to ask B where it is located. A would look up the identifier for position that the programmers for A used and query B with it. The problem? B might not use the same identifier as A did. At best, B will ignore unknown messages from A. At worst, a "where are you?" from A could be a "fire weapon" command. JAUS does have some default records bolted on to the architecture now, but don't trust it: if your robot has to communicate with an older JAUS implementation, you've got nothing and JAUS gains you nothing. You'll have to keep records of each set of JAUS protocols for every revision, which is remarkably hard to do with a closed standard.

Searching, Sorting, and the Memory-Time Tradeoff

Written 2008-06-08

Case A: Suppose you needed to search through a black and white ( real B&W, not greyscale ) picture and pull out a list of white objects from the black background?

Case B: Suppose you needed to sort 50 Gigabytes of unsigned shorts?

Most people will tell you that the limit for sorting is n*log(n). But that's not really the whole story. The hard limit to any sorting algorithm is actually linear time. I say this, because in Case B, you could sort them rather easily if you knew where to put each item, irrelevant of the others. Suppose there are ten numbers. I look at the first number, and if I instinctively know where to put it, I can put it where it needs to go, and continue sorting the array in the same way. Many, many problems do not lend themselves to this type of solution, but Case B, and in a way, Case A both do.

Solving Case B: There's 50GB of unsigned shorts. An unsigned short ranges from 0 to 65535. Great for postal codes, not so great for much of anything else. An unsigned short also happens to take up 2 bytes, so there are 25,000,000,000 unsigned shorts to sort. Assuming they are all the same (worst case), we'll need a data type that can hold all of them...how about an unsigned long long(ULL)? A ULL can hold 18446744073709551616 distinct values, and there happens to be a prebuilt structure for them in gcc. Now that we've established some definitions...on to the game plan.

So we read in the first unsigned short. For our purposes, a 1337. By it's very nature, we can make some inferences about where to put it. Assuming there's an even distribution, we can assume it'll end up somewhere on the left of the mean...or better yet, we can use its absolute position.

We're going to need an array of counters for this next trick, something like "unsigned long long counters[65536];" And we'll just increment the 1337 counter by one. For the next 24,999,999,999 unsigned shorts, we'll do exactly the same thing. Now we have an array containing counts from one to (at worst) 25,000,000,000. All we have to do is traverse the array, reassembling the sorted structure.

So how long would it take to sort 50GB of files holding Unsigned Shorts? About as long as it takes to read the unsorted files and write the sorted files. I discovered this algorithm last week on a robotics competition, while trying to solve Case A. (it's actually been done before-right here is a great paper on it-I wasn't the first. I was a little disappointed.) But anyhow, it has some great features:

Time complexity is linear for large collections of objects, or constant for small collections.
It is easily extended to sort complex data structures, although there are some rules.
It's a stable sort for larger objects, if implemented properly. (Use an array of queues instead of the counters)

Sadly, it has the following disadvantages

It requires an onto mapping for all objects to be sorted
It works great on standard memory, but performance degrades for files or NUMA machines, because the underlying data structure must contain all objects in ram.
The characteristic to sort on must be limited by a maximum value, and that maximum value should be small in order to prevent the constant time component from growing too large.
Only works well for large collections of objects. For most collections of objects smaller than the range of the characteristic to sort upon, you're probably better off with introsort or one of it's ancestors.

I've posted example code and runtimes here. If you look at the file 'run' you'll see the number of unsigned shorts sorted, and the second column is the seconds required to sort them. Keep in mind that each unsigned short also needs a call to rand(), which takes up the majority of processing. Also keep in mind that this example doesn't reassemble the data, which is trivial, but also builds into the time complexity MAX(linear term, constant term).

Anyways...how does this help solve Case A? The hard limit of A is linear against width, and linear against height. Assuming a fixed ratio (in this case 4:3), the hard limit is quadratic against either.

The first algorithm I thought of was pretty simple, and horrendously slow. Lets say each pixel will belong to a "blob" and each "blob" can have multiple pixels. In my case, I used an STL vector.

So to start, search through the picture until a white pixel was found. Set the pixel to black, check if it can be stored in a known blob (if a blob is adjacent to it), otherwise create a new blob, and recursively scan the 8 pixels around this found pixel.

This algorithm has two failing points. The first, is that the recursive factor will scan each pixel more than once. This isn't optimal, especially if we're looking to make this go fast. Secondly, since my blobs are made of vectors, not only must we search through each blob, we must search through every single pixel. Thankfully, I added a little hack, some range checking. Each blob knew the ranges it spanned(MaxX, MaxY, MinX, MinY), and so often checking an entire blob for adjacency with a pixel would only take a few operations, but other times it took an entire linear search. I could have sorted each blob internally by some characteristic, say X or Y, but this is just building on a broken machine, like ripplesort is to bubblesort.

The first addition is to make the algorithm non-recursive. Ryan Meuth used this in his approach and solved the 'U' problem (finding two parts of a blob before finding the center) by the addition of a lookup table.

So, in much the same way, search through the image, creating new blobs if needed and appending to existing blobs. But, if you ever find a pixel that is adjacent to two blobs, mark it in the table and combine them later.

This is a little slower, because each pixel has to be compared against every single blob, regardless of a previous match, but it is much faster than my recursive solution above. What may perform even faster, is combining blobs on the spot.

The second fix is to store each blob as a 2-dimensional array of pixels. In fact, if you can spare the memory, allocate an entire image-worth. To check if a pixel is adjacent to a blob, you can take its position in the array, and look around it to see if there are other set pixels. In this way, a lookup on a blob now has a constant time of eight lookups. Inserting a pixel into the blobspace will still be linear over the number of blobs, but as far as I can tell, that can't be helped. We still have one small problem: each blob has its pixels stored in an obnoxiously hard-to-use format, being a picture. The solution is simple: create a hybrid structure. Each blob has two containers: a matrix and a queue. Insert a pixel into the blob, and you insert it into both. To obtain a list of all pixels in a blob, simply process the queue.

Complex Computational Systems: Planning is Key

Written 2008-05-31

I'd have to say that programming a robot from the ground-up is the most complex project I've ever worked on. Granted, we built this robot from aluminum, motors, gears, and wire. The three items we didn't build are a set of stereo vision cameras, a computer, and a small power-controller. And how do we integrate the separate software projects?

Earlier this year, the electrical, mechanical, and computing groups sat and decided upon the best way to integrate these systems. We created a document outlining which other systems a given module would need to communicate with. Groups were then responsible for creating message passing interfaces between their modules.

Fast forward to today. Every module works quite well in its own environment. The vision system can pick out white road-lines and construction signs when the robot is parked. The drive system is so simple it requires only three lines of code to move the robot. The AI can drive through a virtual model every time. But we're at competition now-and nothing works. One person is gluing it all together. He asked me for help, and there are global pointers everywhere. After a few hours, it started to work. It is the most complex, ugly thing I've seen. Every so often, there are segfaults whenever code ignored mutexes (there are actually pointers to mutex-protected memory).

For such a volatile system, it's working surprisingly well. This system goes against every design standard I've seen-it is a barrel full of monkeys all shouting commands at each other, and every so often monkeys shout at the same time, and scream louder, making the other monkeys act funny. But all-in-all, it looks like its working, even if we neglected to test before the competition.

Robotics, Pervasive Computing and Mobile Connectivity

Written 2008-05-29

It wasn't too long ago when the only way to get a data connection was a large brick-like phone-sort of like on "Congo". Very few people could justify a mobile connection for everyday use. Now, you can purchase a data plan for $5 or $10 per month attached to your phone, and have access almost anywhere. In the past, I've listened to streaming radio on a sailboat in the middle of the Lake of the Ozarks. Now, I've ridden from Rolla, MO to Rochester, MI with internet connectivity the entire way. Not only was there internet on my phone, but with a few lines of Linux shell script, my laptop became an access point, dialed through my phone, and connected them with a NAT so that team members in both vehicles could work on code.

Mobile connectivity is driving the pervasive computing market forward. While on the road, I was able to do mapping, look up email addresses, and read online. However, there was one computer with us that was even more connected. Aluminator, our prize competition robot, has a harness for a cell phone and a gps receiver. It also has motor sensors detecting how far/where/how fast it moves.

There are three items in the area of Mobile Computing that I'd like to see in my lifetime:

A global positioning system that works indoors, even in basements, possibly a little underground. The key: These sensors would have to become small enough and cheap enough that almost every device would have one. Your TV could know where your remote is, anywhere in your house.
I want to see a pervasive wireless network. Right now I can get online in what I estimate is at least half of the United States. Eventually, I want to be able to travel the world without fear that I won't be able to check my email, or buy train tickets online. The key: These chipsets/antennas would have to be small enough and mass produced enough to fit inside of every notebook computer or PDA and cheap enough that every producer would include one.
Secure, two+ factor authentication for everyone.

With these technologies, mobile computers will become an extension of the individual. My PDA would record my car's position in a parking lot and point me back to it (actually, Garmin Colorados do this now) But there would be no reason for so many separate devices. Right now I carry a phone, a proxcard, a calculator, and a laptop in my bag. Eventually these devices will fuse into one. One of my systems already has more than enough processing power. Another has a powerful mathematics language. Another is part of my authentication. The last finds me the closest pizza in any town. Someday, somehow, even if I don't live to see it, everyone will have affordable, fast, mobile connected computing.

A New Home

Written 2008-05-26

So, I finally broke down and purchased a new dns name, and put together an MT blog. It did take about 12 hours to install all the perl modules it wanted. I spent about six hours mostly working on robot code, and then pulling down a new perl module and building it whenever the terminal went inactive. After starting to build imagemagick, I realized that Solaris Make doesn't handle multiple cpus, not even when instructed to. So halfway through, I downloaded Stallman's GNU Make, built it, ran a make clean;make -j8 , and it was done in a few minutes. What's the point of Big Old Solaris Iron without multithreaded code? As of this posting, the server hosting this is loaded with:

4x400 Mhz UltraSparcIIs (4mb cache)
12x18GB(min) SCSI SCA drives (11 of which are in RAIDZ2)
1792 MB RAM (ECC, informs you when sticks are dying)
Solaris 10 (Ubuntu actually ran pretty well, and had graphics support for the CG4, but a bad SCSI driver destroyed the RAID5 after a few reboots-yay backups)
Happy Meal Ethernet!

All being said, it makes a pretty stable and responsive webserver. Now if only we could get it off of that laggy shared T3. Now for a little history. Years ago, in the dark ages, there was a teacher named Fickret Ercal. Fickret had something to do with the school vision processing lab and was in charge of buying them a new system. He choose a then-$64,000 Sun server. Keep in mind that at this point in time, Sun was one of the few companies offering such power in such a small space. The only other possibilities were IBM/POWER, DEC/Alpha, and SGI/MIPS. Eventually, the machine was given to ACM, where it sat in the office for untold ages. Flash forward to last spring. I was elected ACM server administrator for our local chapter. We had an Athlon XP running as the shell/webserver. Something happened and it started scanning the campus for SSH servers. Tim noticed it and disconnected it from the network. My roommate at the time, Doug Kelly, recommended Solaris and installed it on the old beast. Why Solaris? We really wanted RAIDZ/ZFS. I was willing to settle for Ubuntu, but it died after a few reboots and lost all of the hard drive labels for its software RAID5. No BSD supported RAIDZ on Big Endian at the time (FreeBSD does now, but it's an ugly hack). And so, only Solaris was left. It took a few tries to get all the configs set up. That and the fact that the expansion boards have to go into certain slots or they won't work. But eventually, a serial console came up, and it was installed. Next the old Athlon XP was set up with Debian and OpenAFS was used to communicate between them-11 hard drives on the Sun make a ZFS Zpool, which is exported under /afs/mstacm.org. Users can now login to acmshell.device.mst.edu, and edit the same files stored on acm.device.mst.edu. You can also go to here to learn about how to set up your personal computer to connect to our AFS space. Overall, I'm pretty happy with the machine. It isn't blindly fast (except for access times), but it is rock solid. A few weeks ago the web pages started acting a little slow. The culprit? A bad hard drive says dmesg. The disk was audibly clinking as the head slammed across the drive. A few zpool commands and it was running fine again, just with degraded redundancy (equivalent to RAID5). When we got the new drive, a few more zpool commands and it was happily updating blocks and metadata from the other ten drives. Keep in mind that there was no reason for downtime in this period of time. When the drive went out, the webpages were slow only until cached. When the drive came out and the zpool was scrubbed, there was some high cpu usage, but when files were rechecked and marked as good, their access time/speed went back to normal. Replacing the disk was much the same. Hopefully we can get a bigger UPS for it and then there will be no more downtime. Right now, it's up between storms. So you want to know why I'm pumped? Well, because the title to this post is one letter away from being a Star Wars title.

Written 2013-02-21

Tags:USB CLA

Written 2013-01-17

Tags:x264 h.264 video latency

Written 2013-01-08

Tags:Video UART Serial h.264

Source Code

Test Setup

Building the code

Getting it running

More Fun

Written 2012-12-16

Tags:Disassembly objdump BeyondCompare

Left-hand-side

Right-hand-side

Results

Written 2012-12-04

Tags:Monte-Carlo-Simulation Cache-Line-Sharing Caches Atomic-Operations

The Algorithm

The Synchronization Options

Binary Semaphore/Mutex

Atomic Operations

Thread-based storage in a shared area

Thread-based storage local to each thread

The Machines

Intel CoreDuo-T2300 running Linux 3.2

Intel Core2Duo-T7600 running Linux 3.2

Intel dual Xeon 5650 running Linux 3.2

Texas Instruments MicroSPARC running OpenBSD 5.1

Intel Atom N455 running OpenBSD 5.2

IBM PowerPC Quad-G5 running OSX 10.5

Conclusions

Atomic operations aren't a magic bullet

Semaphore cost can vary widely over different platforms

Often times, a single-thread can do pretty well

Costs of synchronization should be mitigatable

Source Code

Written 2012-11-30

Tags:Zip Arc Compression Future-Proofing

Written 2012-09-30

Tags:Blogging

Written 2012-09-23

Tags:Latency contention lock reader-writer-lock semaphore

Written 2012-09-02

Tags:ARM FIQ IRQ Interrupt Latency hook

Written 2012-08-23

Tags:CpE315 Kraken Kraken-II MST UMR github

Written 2012-05-03

Tags:audio oscilloscope

Written 2012-01-07

Tags:Atari Atari Flashback Pickin' Sticks Serial Synchronous Serial arduino hack pinout serial

Written 2012-01-03

Tags:Atari Atari Flashback serial spi teardown

Written 2012-01-02

Tags:Arduino NTSC PAL Pickin' Sticks game potentiometers video

Written 2012-01-02

Tags:GNU Languages Programming Python pyGNU python

Written 2012-01-02

Tags:Bsides femtocell security sprint verizon

Written 2011-10-22

Tags:FAT File system HP HPFAT Printer driver filesystem

Written 2011-10-12

Tags:Camera Canon Remote Remote Shutter time-lapse transistor

Power Supply

Power Usage

Frequency Range

Output Coupling

Picking Parts

Written 2011-10-11

Tags:cupcake9 robot

Written 2011-10-09

Tags:Canon Remote Shutter Time-Lapse

Written 2011-10-01

Tags:modem satellite serial teardown

Written 2011-09-29

Tags:CCCKC Canon Remote Remote Shutter SLR dSLR

Written 2011-09-25

Tags:Canon Electronics Remote Shutter dSLR

Written 2011-09-25

Tags:Ademco CCCKC CCTV DVR FAT RapidEye VxWorks teardown