MASSCAN: Mass IP port scanner

This is the fastest Internet port scanner. It can scan the entire Internet in under 6 minutes, transmitting 10 million packets per second.

It produces results similar to nmap, the most famous port scanner. Internally, it operates more like scanrand, unicornscan, and ZMap, using asynchronous transmission. The major difference is that it's faster than these other scanners. In addition, it's more flexible, allowing arbitrary address ranges and port ranges.

Building

On Debian/Ubuntu, it goes something like this:

$ git clone https://github.com/robertdavidgraham/masscan
$ cd masscan
$ sudo apt-get install libpcap-dev
$ make

This puts the program in the masscan/bin subdirectory. You'll have to manually copy it to something like /usr/local/bin if you want to install it elsewhere on the system.

While Linux is the primary target platform, the code runs well on many other systems. Here's some additional build info:

• Windows w/ Visual Studio: use the VS10 project
• Windows w/ MingGW: just type make
• Windows w/ cygwin: won't work
• Mac OS X /w XCode: use the XCode4 project
• Mac OS X /w cmdline: just type make
• FreeBSD: type gmake
• other: I don't know, don't care

PF_RING

To get beyond 2 million packets/second, you need an Intel 10-gbps Ethernet adapter and a special driver known as "PF_RING DNA" from http://www.netop.org. Masscan doesn't need to be rebuilt in order to use PF_RING. To use PF_RING, you need to build the following components:

• libpfring.so (installed in /usr/lib/libpfring.so)
• pf_ring.ko (their kernel driver)
• ixgbe.ko (their version of the Intel 10-gbps Ethernet driver)

You don't need to build their version of libpcap.so.

When Masscan detects that an adapter is named something like dna0 instead of something like eth0, it'll automatically switch to PF_RING mode.

Regression testing

The project contains a built-in self-test:

$ make regress
bin/masscan --regress
selftest: success!

This tests a lot of tricky bits of the code. You should do this after building.

Performance testing

To test performance, run something like the following:

$ bin/masscan 0.0.0.0/4 -p80 --rate 100000000 --router-mac 66-55-44-33-22-11

The bogus --router-mac keeps packets on the local network segments so that they won't go out to the Internet.

You can also test in "offline" mode, which is how fast the program runs without the transmit overhead:

$ bin/masscan 0.0.0.0/4 -p80 --rate 100000000 --offline

This second benchmark shows roughly how fast the program would run if it were using PF_RING, which has near zero overhead.

Usage

Usage is similar to nmap. To scan a network segment for some ports:

# masscan -p80,8000-8100 10.0.0.0/8

This will:

• scan the 10.x.x.x subnet, all 16 million addresses
• scans port 80 and the range 8000 to 8100, or 102 addresses total
• print output to <stdout> that can be redirected to a file

To see the complete list of options, use the --echo feature. This dumps the current configuration and exits. This output can be used as input back into the program:

# masscan -p80,8000-8100 10.0.0.0/8 --echo > xxx.conf
# masscan -c xxx.conf --rate 1000

How to scan the entire Internet

While useful for smaller, internal networks, the program is designed really with the entire Internet in mind. It might look something like this:

# masscan 0.0.0.0/0 -p0-65535

Scanning the entire Internet is bad. For one thing, parts of the Internet react badly to being scanned. For another thing, some sites track scans and add you to a ban list, which will get you firewalled from useful parts of the Internet. Therefore, you want to exclude a lot of ranges. To blacklist or exclude ranges, you want to use the following syntax:

# masscan 0.0.0.0/0 -p0-65535 --excludefile exclude.txt

This just prints the results to the command-line. You probably want them saved to a file instead. Therefore, you want something like:

# masscan 0.0.0.0/0 -p0-65535 -oX scan.xml

This saves the results in an XML file, allowing you to easily dump the results in a database or something.

But, this only goes at the default rate of 100 packets/second, which will take forever to scan the Internet. You need to speed it up as so:

# masscan 0.0.0.0/0 -p0-65535 --max-rate 100000

This increases the rate to 100,000 packets/second, which will scan the entire Internet (minus excludes) in about 10 hours per port (or 655,360 hours if scanning all ports).

The thing to notice about this command-line is that these are all nmap compatible options. In addition, "invisible" options compatible with nmap are also set for you: -sS -Pn -n --randomize-hosts --send-eth. Likewise, the format of the XML file is inspired by nmap. There are, of course, a lot of differences, because the asynchronous nature of the program leads to a fundamentally different approach to the problem.

The above command-line is a bit cumbersome. Instead of putting everything on the command-line, it can be stored in a file instead. The above settings would look like this:

# My Scan
rate = 100000.00
output-format = xml
output-status = all
output-filename = scan.xml
ports = 0-65535
range = 0.0.0.0-255.255.255.255
excludefile = exclude.txt

To use this configuration file, use the -c:

# masscan -c myscan.conf

This also makes things easier when you repeat a scan.

By default, masscan first loads the configuration file /etc/masscan/masscan.conf. Any later configuration parameters override what's in this default configuration file. That's where I put my "excludefile" parameter, so that I don't ever forget it. It just works automatically.

Getting output

The are two primary formats for output. The first is XML, which products fairly large files, but is easy to import into anything. Just use the parameter -oX <filename>. Or, use the parameters --output-format xml and--output-filename <filename>.

The second is the binary format. This produces much smaller files, so that when I scan the Internet my disk doesn't fill up. They need to be parsed, though. In the util subdirectory there is a program scan2text.c that will scan in the binary format and produce text.

Comparison with Nmap

Where reasonable, every effort has been taken to make the program familiar to nmap users, even though it's fundamentally different. Two important differences are:

• no default ports to scan, you must specify -p <ports>
• target hosts are IP addresses or simple ranges, not DNS names, nor the funky subnet ranges nmap can use (like 10.0.0-255.0-255).

You can think of masscan as having the following settings permanently enabled:

• -sS: this does SYN scan only (currently, will change in the future)
• -Pn: doesn't ping hosts first, which is fundamental to the async operation
• -n: no DNS resolution happens
• --randomize-hosts: scan completely randomized
• --send-eth: sends using raw libpcap

If you want a list of additional nmap compatible settings, use the following command:

# masscan --nmap

Transmit rate (IMPORTANT!!)

This program spews out packets very fast. On Windows, or from VMs, it can do 300,000 packets/second. On Linux (no virtualization) it'll do 1.6 million packets-per-second. That's fast enough to melt most networks.

Note that it'll only melt your own network. It randomizes the target IP addresses so that it shouldn't overwhelm any distant network.

By default, the rate is set to 100 packets/second. To increase the rate to a million use something like --rate 1000000.

Design

This section describes the major design issues of the program.

Code Layout

The file main.c contains the main() function, as you'd expect. It also contains the transmit_thread() and receive_thread() functions. These functions have been deliberately flattened and heavily commented so that you can read the design of the program simply by stepping line-by-line through each of these.

Asynchronous

This is an asynchronous design. In other words, it is to nmap what the nginx web-server is to Apache. It has separate transmit and receive threads that are largely independent from each other. It's the same sort of design found in scanrand, unicornscan, and ZMap.

Because it's asynchronous, it runs as fast as the underlying packet transmit allows.

Randomization

A key difference between Masscan and other scanners is the way it randomizes targets.

The fundamental principle is to have a single index variable that starts at zero and is incremented by one for every probe. In C code, this is expressed as:

for (i = 0; i < range; i++) {
 scan(i);
}

We have to translate the index into an IP address. Let's say that you want to scan all "private" IP addresses. That would be the table of ranges like:

192.168.0.0/16
10.0.0.0/8
172.16.0.0/20

In this example, the first 64k indexes are appended to 192.168.x.x to form the target address. Then, the next 16-million are appended to 10.x.x.x. The remaining indexes in the range are applied to 172.16.x.x.

In this example, we only have three ranges. When scanning the entire Internet, we have in practice more than 100 ranges. That's because you have to blacklist or exclude a lot of sub-ranges. This chops up the desired range into hundreds of smaller ranges.

This leads to one of the slowest parts of the code. We transmit 10 million packets per second, and have to convert an index variable to an IP address for each and every probe. We solve this by doing a "binary search" in a small amount of memory. At this packet rate, cache efficiencies start to dominate over algorithm efficiencies. There are a lot of more efficient techniques in theory, but they all require so much memory as to be slower in practice.

We call the function that translates from an index into an IP address the pick() function. In use, it looks like:

for (i = 0; i < range; i++) {
 ip = pick(addresses, i);
 scan(ip);
}

Masscan supports not only IP address ranges, but also port ranges. This means we need to pick from the index variable both an IP address and a port. This is fairly straightforward:

range = ip_count * port_count;
for (i = 0; i < range; i++) {
 ip = pick(addresses, i / port_count);
 port = pick(ports, i % port_count);
 scan(ip, port);
}

This leads to another expensive part of the code. The division/modulus instructions are around 90 clock cycles, or 30 nanoseconds, on x86 CPUs. When transmitting at a rate of 10 million packets/second, we have only 100 nanoseconds per packet. I see no way to optimize this any better. Luckily, though, two such operations can be executed simultaneously, so doing two of these as shown above is no more expensive than doing one.

There are actually some easy optimizations for the above performance problems, but they all rely upon i++, the fact that the index variable increases one by one through the scan. Actually, we need to randomize this variable. We need to randomize the order of IP addresses that we scan or we'll blast the heck out of target networks that aren't built for this level of speed. We need to spread our traffic evenly over the target.

The way we randomize is simply by encrypting the index variable. By definition, encryption is random, and creates a 1-to-1 mapping between the original index variable and the output. This means that while we linearly go through the range, the output IP addresses are completely random. In code, this looks like:

range = ip_count * port_count;
for (i = 0; i < range; i++) {
 x = encrypt(i);
 ip = pick(addresses, x / port_count);
 port = pick(ports, x % port_count);
 scan(ip, port);
}

This also has a major cost. Since the range is an unpredictable size instead of a nice even power of 2, we can't use cheap binary techniques like AND (&) and XOR (^). Instead, we have to use expensive operations like MODULUS (%). In my current benchmarks, it's taking 40 nanoseconds to encrypt the variable.

This architecture allows for lots of cool features. For example, it supports "shards". You can setup 5 machines each doing a fifth of the scan, orrange / shard_count. Shards can be multiple machines, or simply multiple network adapters on the same machine, or even (if you want) multiple IP source addresses on the same network adapter.

Or, you can use a 'seed' or 'key' to the encryption function, so that you get a different order each time you scan, like x = encrypt(seed, i).

We can also pause the scan by exiting out of the program, and simply remembering the current value of i, and restart it later. I do that a lot during development. I see something going wrong with my Internet scan, so I hit to stop the scan, then restart it after I've fixed the bug.

Another feature is retransmits/retries. Packets sometimes get dropped on the Internet, so you can send two packets back-to-back. However, something that drops one packet may drop the immediately following packet. Therefore, you want to send the copy about 1 second apart. This is simple. We already have a 'rate' variable, which is the number of packets-per-second rate we are transmitting at, so the retransmit function is simply to use i + rate as the index. One of these days I'm going to do a study of the Internet, and differentiate "back-to-back", "1 second", "10 second", and "1 minute" retransmits this way in order to see if there is any difference in what gets dropped.

C10 Scalability

The asynchronous technique is known as a solution to the "c10k problem". Masscan is designed for the next level of scalability, the "C10M problem".

The C10M solution is to bypass the kernel. There are three primary kernel bypasses in Masscan:

• custom network driver
• user-mode TCP stack
• user-mode synchronization

Masscan can use the PF_RING DNA driver. This driver DMAs packets directly from user-mode memory to the network driver with zero kernel involvement. That allows software, even with a slow CPU, to transmit packets at the maximum rate the hardware allows. If you put 8 10-gbps network cards in a computer, this means it could transmit at 100-million packets/second.

Masscan has its own built-in TCP stack for grabbing banners from TCP connections. This means it can easily support 10 million concurrent TCP connections, assuming of course that the computer has enough memory.

Masscan has no "mutex". Modern mutexes (aka. futexes) are mostly user-mode, but they have two problems. The first problem is that they cause cache-lines to bounce quickly back-and-forth between CPUs. The second is that when there is contention, they'll do a system call into the kernel, which kills performance. Mutexes on the fast path of a program severely limits scalability. Instead, Masscan uses "rings" to synchronize things, such as when the user-mode TCP stack in the receive thread needs to transmit a packet without interfering with the transmit thread.

Portability

The code runs well on Linux, Windows, and Mac OS X. All the important bits are in standard C (C90). It therefore compiles on Visual Studio with Microsoft's compiler, the Clang/LLVM compiler on Mac OS X, and GCC on Linux.

Windows and Macs aren't tuned for packet transmit, and get only about 300,000 packets-per-second, whereas Linux can do 1,500,000 packets/second. That's probably faster than you want anyway.

Safe code

A bounty is offered for vulnerabilities, see the VULNINFO.md file for more information.

This project uses safe functions like strcpy_s() instead of unsafe functions like strcpy().

This project has automated unit regression tests (make regress).

Compatibility

A lot of effort has gone into making the input/output look like nmap, which everyone who does port scans is (or should be) familiar with.

Authors

This tool created by Robert Graham: email: robert_david_graham@yahoo.com twitter: @ErrataRob

Troll? That’s how people write Verilog1. At my old company, we had a team of formal methods PhDs who wrote a linter that typechecked our code, based on our naming convention. For our chip (which was small for a CPU), building a model (compiling) took about five minutes, running a single short test took ten to fifteen minutes, and long tests took CPU months. The value of a linter that can run in seconds should be obvious, not even considering the fact that it can take hours of tracing through waveforms to find out why a test failed2.

Lets look at some of the most commonly used naming conventions.

Pipeline stage

When you pipeline hardware, you end up with many versions of the same signal, one for each stage of the pipeline the signal traverses. Even without static checks, you’ll want some simple way to differentiate between these, so you might name them foo_s1, foo_s2, and foo_s3, indicating that they originate in the first, second, and third stages, respectively. In any particular stage, a signal is most likely to interact with other signals in the same stage; it’s often a mistake when logic from other stages is accessed. There are reasons to access signals from other stages, like bypass paths and control logic that looks at multiple stages, but logic that stays contained within a stage is common enough that it’s not too tedious to either “cast” or add a comment that disables the check, when looking at signals from other stages.

Clock domain

Accessing a signal in a different clock domain without synchronization is like accessing a data structure from multiple threads without synchronization. Sort of. It’s worse. Much worse. Driving combinational logic from a metastable state (where the signal is sitting between a 0 and 1) can burn a massive amount of power3. Here, I’m not just talking about being inefficient. If you took a high-power chip from the late 90s and removed the heatsink, it would melt itself into the socket, even under normal operation. Modern chips have such a high maximum power possible power consumption that the chips would self destruct if you disabled the thermal regulation, even with the heatsink. Logic that’s floating at an intermediate value not only uses a lot of power, it bypasses a chip’s usual ability to reduce power by slowing down the clock4. Using cross clock domain signals without synchronization is a bad idea, unless you like random errors, high power dissipation, and the occasional literal meltdown.

Module / Region

In high speed designs, it’s an error to use a signal that’s sourced from another module. This will insidiously sneak through simulation; you’ll only notice the problem when you look at the timing report. On the last chip I worked on, it took about two days to generate a timing report5 (and you couldn’t concurrently generate a new one for each build, due to the license costs6). If you accidentally reference a signal from a distant module, not only will you not meet your timing budget for that path, the synthesis tool will allocate resources to try to make that path faster, which will slow down everything else7, making the entire timing report worthless8.

PL Trolling FTW

I’d been feeling naked at my new gig, coding Verilog without any sort of static checking. I put off writing my own checker, because static analysis is one of those scary things you need a PhD to do, right? But, all the things I’ve talked about are simple enough that half an hour after starting, I had a tool that found seven bugs, with only two false positives. I expect we’ll have 4x as much code by the time we’re done, so that’s 28 bugs from half an hour of work, not even considering the fact that two of the bugs were in heavily used macros.

Huh. That wasn’t so bad. I’ve now graduated to PL troll.

There are more complex checks you can do; for example, checking if things aren’t clock gated or power gated when they should be (not too difficult to do dynamically, but non-trivial to check statically). But, I’ll leave those for another day.

P.S. If you want to try this at home, you don’t have to write a complete parser. You probably don’t even want to write a complete parser (the SV spec is 1300 pages long, vs 800 for C++, 500 for C, 300 for Java, and 30 for Erlang). The thing that was stopping me from trying this, for weeks, was that one of the formal methods PhDs at my old company, who’s a much better programmer than me, spent the better part of a quarter writing a parser. Having a complete parser is nice, but not necessary.

First, for any high-speed design, there’s not enough fast (wide) interconnect to go around. Gates are at the bottom, and wires sit above them. Wires get wider and faster in higher layers, but there’s congestion getting to and from the fast wires, and relatively few of them.. There are so few of them that people pre-plan where modules should be placed in order to have enough fast interconnect to meet timing demands. If you steal some fast wires to make some slow path fast, anything relying on having a fast path through that region is hosed. Second, the synthesis tool tries to place sources near sinks, to reduce both congestion and delay. If you place a sink on a net that’s very far from the rest of the sinks, the source will migrate halfway in between, to try to match the demands of all the sinks. This is recursively bad, and will pull all the second order sources away from their optimal location, and so on and so forth.

Last week, Russell Brand was in hot water again after cracking a Nazi joke at the expense of GQ award sponsors, Hugo Boss. Here, he gives his side of the story:

I have had the privilege of scuba diving. I did it once on holiday, and I'm aware that it's one of those subjects that people can get pretty boring and sincere about, and sincerity, for we British, is no state in which to dwell, so I'll be brief. The scuba dive itself was numenistic enough, a drenched heaven; coastal shelves and their staggering, sub-aquatic architecture, like spilt cathedrals, gormless, ghostly fish gliding by like Jackson Pollock's pets. Silent miracles. What got me, though, was when I came up for air, at the end. As my head came above water after even a paltry 15 minutes in Davy Jones's Locker, there was something absurd about the surface. How we, the creatures of the land, live our lives, obliviously trundling, flat feet slapping against the dust.

It must have been a while since I've attended a fancy, glitzy event, because as soon as I got to the GQ awards I felt like something was up. The usual visual grammar was in place – a carpet in the street, people in paddocks awaiting a brush with something glamorous, blokes with earpieces, birds in frocks of colliding colours that if sighted in nature would indicate the presence of poison. I'm not trying to pass myself off as some kind of Francis of Assisi, Yusuf Islam, man of the people, but I just wasn't feeling it. I ambled into the Opera House across yet more outdoor carpets, boards bearing branding, in this case Hugo Boss, past paparazzi, and began to queue up at the line of journalists and presenters, in a slightly nicer paddock who offer up mics and say stuff like:

"Who are you wearing?"

"I'm not wearing anyone. I went with clobber, I'm not Buffalo Bill."

Noel Gallagher was immediately ahead of me in the press line and he's actually a mate. I mean, I love him: sometimes I forget he wrote Supersonic and played to 400,000 people at Knebworth because he's such a laugh. He laid right into me, the usual gear: "What the fook you wearing? Does Rod Stewart know you're going through his jumble?" I try to remain composed and give as good as I get, even though the paddock-side banter is accompanied by looming foam-tipped eavesdroppers, hanging like insidious mistletoe.

In case you don't know, these parties aren't like real parties. It's fabricated fun, imposed from the outside. A vision of what squares imagine cool people might do set on a spaceship. Or in Moloko. As we come out of the lift there's a bloody great long corridor flanked by gorgeous birds in black dresses, paid to be there, motionless, left hand on hip, teeth tacked to lips with scarlet glue. The intention, I suppose, is to contrive some Ian Fleming super-uterus of well fit mannequins to midwife you into the shindig, but me and my mate Matt just felt self-conscious, jigging through Robert Palmer's oestrogen passage like aspirational Morris dancers. Matt stared at their necks and I made small talk as I hot stepped towards the pre-show drinks. Now, I'm not typically immune to the allure of objectified women, but I am presently beleaguered by a nerdish, whirling dervish, and am eschewing all others. Perhaps the clarity of this elation has awakened me. A friend of mine said: "Being in love is like discovering a concealed ballroom in a house you've long inhabited." I also don't drink, so these affairs where most people rinse away their Britishness and twitishness with booze are for me a face-first log flume of backslaps, chitchat, eyewash and gak.

After a load of photos and what-not, we descend the world's longest escalator, which are called that even as they de-escalate, and in we go to the main forum, a high ceilinged hall, full of circular cloth-draped, numbered tables, a stage at the front, the letters GQ, 12-foot high in neon at the back; this aside, though, neon forever the moniker of trash, this is a posh do, in an opera house full of folk in tuxes.

Everywhere you look there's someone off the telly; Stephen Fry, Pharrell, Sir Bobby Charlton, Samuel L Jackson, Rio Ferdinand, Justin Timberlake, foreign secretary William Hague and mayor of London Boris Johnson. My table is a sanctuary of sorts; Noel and his missus Sara, John Bishop and his wife Mel, my mates Matt Morgan, Mick and Gee. Noel and I are both there to get awards and decide to use our speeches to dig each other out. This makes me feel a little grounded in the unreal glare, normal.

Noel's award is for being an "icon" and mine for being an "oracle". My knowledge of the classics is limited, but includes awareness that an oracle is a spiritual medium through whom prophecies from the gods were sought in ancient Greece. Thankfully, I have a sense of humour that prevents me from taking accolades of that nature on face value, or I'd've been in the tricky position of receiving the GQ award for being "best portal to a mystical dimension", which is a lot of pressure. Me, Matt and Noel conclude it's probably best to treat the whole event as a bit of a laugh and, as if to confirm this as the correct attitude, Boris Johnson – a man perpetually in pajamas regardless of what he's wearing – bounds to the stage to accept the award for "best politician". Yes, we agree: this is definitely a joke.

Boris, it seems, is taking it in this spirit, joshing beneath his ever-redeeming barnet that Labour's opposition to military action in Syria is a fey stance that he, as GQ politician of the year, would never be guilty of.

Matt is momentarily focused. "He's making light of gassed Syrian children," he says. We watch, slightly aghast, then return to goading Noel.

Before long, John Bishop is on stage giving me a lovely introduction, so I get up as Noel hurls down a few gauntlets, daring me to "do my worst".

I thanked John, said the "oracle award" sounds like a made-up prize you'd give a fat kid on sports day – I should know, I used to get them – then that it's barmy that Hugo Boss can trade under the same name they flogged uniforms to the Nazis under and the ludicrous necessity for an event such as this one to banish such a lurid piece of information from our collective consciousness.

I could see the room dividing as I spoke. I could hear the laughter of some and louder still silence of others. I realised that for some people this was regarded as an event with import. The magazine, the sponsors and some of those in attendance saw it as a kind of ceremony that warranted respect. In effect, it is a corporate ritual, an alliance between a media organisation, GQ, and a commercial entity, Hugo Boss. What dawned on me as the night went on is that even in apparently frivolous conditions the establishment asserts control, and won't tolerate having that assertion challenged, even flippantly, by that most beautifully adept tool: comedy.

The jokes about Hugo Boss were not intended to herald a campaign to destroy them. They're not Monsanto or Halliburton, the contemporary corporate allies of modern-day fascism; they are, I thought, an irrelevant menswear supplier with a double-dodgy history. The evening, though, provided an interesting opportunity to see how power structures preserve their agenda, even in a chintzy microcosm.

Subsequent to my jokes, the evening took a peculiar turn. Like the illusion of sophistication had been inadvertently disrupted by the exposure. It had the vibe of a wedding dinner where the best man's speech had revealed the groom's infidelity. With Hitler.

Foreign secretary William Hague gave an award to former Telegraph editor Charles Moore, for writing a hagiography of Margaret Thatcher, who used his acceptance speech to build a precarious connection between my comments about the sponsors, my foolish answerphone scandal at the BBC and the Sachs family's flight, 70 years earlier, from Nazi-occupied Europe. It was a confusing tapestry that Moore spun but he seemed to be saying that a) the calls were as bad as the Holocaust and b) the Sachs family may not've sought refuge in Britain had they known what awaited them. Even for a man whose former job was editing the Telegraph this is an extraordinary way to manipulate information.

Noel, who is not one to sit quietly on his feelings, literally booed while Charles Moore was talking, and others joined in. Booing! When do you hear booing in this day and age other than pantomimes and parliament? Hague and Johnson are equally at home in either (Widow Twanky and Buttons, obviously) so were not unduly ruffled, but I thought it was nuts. The room by now had a distinct feel of "us and them" and if there is a line drawn in the sand I don't ever want to find myself on the same side as Hague and Johnson. Up went Noel to garner his gong and he did not disappoint: "Always nice to be invited to the Tory party conference," he began, "Good to see the foreign secretary present when there's shit kicking off in Syria."

Noel once expressed his disgust at seeing a politician at Glastonbury. "What are you doing here? This ain't for you," he'd said. He explained to me: "You used to know where you were with politicians in the 70s and 80s cos they all looked like nutters: Thatcher, Heseltine, Cyril Smith. Now they look normal, they're more dangerous." Then, with dreadful foreboding: "They move among us." I agree with Noel. What are politicians doing at Glastonbury and the GQ awards? I feel guilty going, and I'm a comedian. Why are public officials, paid by us, turning up at events for fashion magazines? Well, the reason I was there was because I have a tour on and I was advised it would be good publicity. What are the politicians selling? How are they managing our perception of them with their attendance of these sequin-encrusted corporate balls?

We witness that there is a relationship between government, media and industry that is evident even at this most spurious and superficial level. These three institutions support one another. We know that however cool a media outlet may purport to be, their primary loyalty is to their corporate backers. We know also that you cannot criticise the corporate backers openly without censorship and subsequent manipulation of this information.

Now I'm aware that this was really no big deal; I'm not saying I'm an estuary Che Guevara. It was a daft joke by a daft comic at a daft event. It makes me wonder, though, how the relationships and power dynamics I witnessed on this relatively inconsequential context are replicated on a more significant scale.

For example, if you can't criticise Hugo Boss at the GQ awards because they own the event, do you think it is significant that energy companies donate to the Tory party? Will that affect government policy? Will the relationships that "politician of the year" Boris Johnson has with City bankers – he took many more meetings with them than public servants in his first term as mayor – influence the way he runs our capital?

Is it any wonder that Amazon, Vodafone and Starbucks avoid paying tax when they enjoy such cosy relationships with members of our government?

Ought we be concerned that our rights to protest are being continually eroded under the guise of enhancing our safety? Is there a relationship between proposed fracking in the UK, new laws that prohibit protest and the relationships between energy companies and our government?

I don't know. I do have some good principles picked up that night that are generally applicable: the glamour and the glitz isn't real, the party isn't real, you have a much better time mucking around trying to make your mates laugh. I suppose that's obvious. We all know it, we already know all the important stuff, like: don't trust politicians, don't trust big business and don't trust the media. Trust your own heart and each another. When you take a breath and look away from the spectacle it's amazing how absurd it seems when you look back.

Still, Chavarria has to hop on buses to meet clients throughout Phoenix because Arizona won't give her a driver's license. The state considers her to be in the country illegally, even though she recently obtained a two-year reprieve from deportation under the Obama administration's deferred action program.

She may not drive, but along with thousands of other young people who entered the country illegally, Chavarria has found a way to make a living without breaking the law.

Although federal law prohibits employers from hiring someone residing in the country illegally, there is no law prohibiting such a person from starting a business or becoming an independent contractor.

As a result, some young immigrants are forming limited liability companies or starting freelance careers — even providing jobs to U.S. citizens — as the prospect of an immigration law revamp plods along in Congress.

Ever since 1986, when employer sanctions took effect as part of the immigration overhaul signed by President Reagan, creating a company or becoming an independent contractor has been a way for people who are in the country illegally to work on a contract basis and get around immigration enforcement.

But organizers who help immigrants said the idea has taken on new life in recent years, often among tech-savvy young people who came into the country illegally or overstayed visas.

Chavarria, whowas 7 when shecrossed into Arizona from Mexico with her mother, said her parents told her from a young age that anything was possible in her newly adopted country.

"We're taught as young kids that this is the land of opportunity," she said. "They told me, 'You could be anything you want to be if you work hard, you're a good person, obey your parents and go to school.'"

But when she graduated from high school in Phoenix, Chavarria discovered that her lack of legal status was a roadblock to becoming a graphic designer. Although she won a scholarship, she said, she could afford to take only two classes at a time at Scottsdale Community College because she wasn't willing to risk working with fraudulent documents to pay for school.

Congress delivered another blow to Chavarria in 2010 when it failed to pass the Dream Act, which would provide a path to legalization for young adults who were brought into the country illegally as children.

The next year, after she became more involved with the Dream Act Coalition, she discovered a way she could sell her designs to others without fear of repercussions.

How is this possible? Though the issue is complex, the answer boils down to how labor law defines employees, said Muzaffar Chishti, an expert on the intersection of labor and immigration law at the Migration Policy Institute.

For example, employees often have set hours and use equipment provided by the employer. Independent contractors make their own hours, get paid per project by submitting invoices and use their own tools. Also, someone who hires an independent contractor isn't obligated by immigration law to verify that person's legal status.

At a workshop hosted by immigrant rights activists, Chavarria learned about these intricacies of labor law — and how to register as a limited liability company. "I didn't know it was possible," Chavarria said. "And it wasn't that hard."

It was as easy as downloading the forms from the Internet, opening up a bank account and turning in paperwork to the state along with a $50 fee. Proof of citizenship is not required. Regulations vary, but similar procedures exist in other states. In California, the fee is a bit higher and there's an annual minimum tax of $800, but the process is similar to Arizona's.

It's unclear how many entrepreneurs there are like Chavarria. Immigration experts say anecdotal evidence suggests interest in such businesses has grown in recent years as more states have adopted tougher illegal-immigration laws. But research is scant.

Indications of a trend could be found, however, in a Public Policy Institute of California report on the effects of Arizona's 2007 mandatory E-Verify law, which forced businesses to use a federal system intended to weed out people working in the country illegally.

The study found that 25,000 workers living in Arizona illegally became self-employed in 2009. That was an 8% jump over the number a year earlier. They probably formed limited liability companies, created their own businesses or even left employers to become independent contractors.

Freddy S. Pech, 25, a Mexican national who lives in East Los Angeles, said he decided to remain an independent graphic designer rather than form a limited liability company.

A wildcard may be used, at the start of the name only, for example "*.wikipedia.org". To search for external links to pages on this site, start with

The search pattern is case-sensitive after the first slash (/).

• See also Meta:Help:Linksearch.

For some reason many people discuss the problem of estimating software development timeframes without properly understanding the issue. There is a famous answer on Quora that exemplifies this. Lots of people like that story, even though it’s inaccurate and misguided. “Software development” is such a huge endeavor that it doesn’t even make sense to talk about estimates without an understanding of the kinds of problems software can solve. To put this in context, let’s forget software for a second and look at a few tangible problems of different magnitudes.

• You are a medical researcher. A new disease makes headlines. It’s a virus. It seems to be spreading through sexual contact. How long is it going to take you to find a cure?
• It’s 1907, and you’ve just built the first airplane. The government wants you to build a spaceship to fly to the moon. How long will it take?
• You are in charge of a construction company that has built hundreds of buildings in your metropolitan area. I want you to build me a twenty-story apartment complex very similar to the one you just finished on the other side of town. When will it be done?
• You own a chair factory that produces 1000 chairs per month. I need 3500 chairs. Can you make them in a month?

The four questions above are radically different in nature. The first two involve significant unknowns, and require scientific or technological breakthroughs. The other two, not so much. Meta-question: does software development look like the first or the second kind? Another meta-question: what kind of software development are we talking about?

Let’s focus on the construction company. People have been constructing buildings for centuries. There’s relatively little variation in the effort and costs of putting up vanilla high-rises (we’re not talking about Burj Khalifa). Of course there is uncertainty: economic conditions could change, suppliers could go out of business. The new mayor could have a personal vendetta against your company because your big brother bullied him in school. All those things have happened before, perhaps in combination. Let’s say you can give me an estimate of 15 to 24 months with 98% confidence based on past data. Sounds good to me.

If buildings were software you could break out a template, customize it a bit, install it on my plot of land in the cloud, the end. There are companies doing this for software, for example Bitnami (disclosure: I’m an investor). The process is so quick that you don’t even need to ask for a time estimate. You just see it happen in real time.

Let’s imagine that it were impossible to clone software at almost zero cost, like it is for physical things. Most software developers would be like monks copying manuscripts. If you have been handwriting pages for long enough, you can confidently tell me that it will take you at least 15 years to produce a high-quality copy of the entire Bible (it can be done 4x faster today, not sure about the quality though). You could get sick, or suffer interruptions. However, the number of absolute hours you’ll need is well known. There is a type of software development that works like this: porting old applications to a new language (say COBOL to Java back in the day).

Of course, the more rewarding problems in software development look nothing like this. I enjoy trying to solve problems that nobody has solved before. The malleability of software makes it easy to explore an open problem. Some problems are deceiving; at first they may look like building a house, and as you discover unknowns they sometimes mutate to resemble a quest for an AIDS vaccine. If a problem is solvable with software, it may take weeks or months to come up with an imperfect solution. It will probably take years to build one that’s scalable and robust. The meta-problem is that some problems cannot be solved with software, or at least not yet, or not by me / my team / my company. I might give you an estimate that would look like:

• less than a month: 30% chance
• less than a year: 40% chance
• never: 30% chance

Another kind of software development somewhere in the middle, and it may be the one that generates the most software jobs. Usually an organization wants a solution to a problem that has already been solved by others (e.g. building a cluster manager for a social network graph). Even though you don’t have access to the design or the code, you have an idea of what the solutions look like. You don’t know what key issues they ran into, how good the teams were, how lucky they got. Still, you know the problem can be solved by companies that look like yours for a reasonable cost. There is still quite a bit of uncertainty:  you can estimate small tasks reasonably well, but you cannot predict which “week-long” task might expand to several months (e.g. it turns out that no open source tool solves X in a way that works for us, we’ll have to write our own).

The gist of why estimates are hard: every new piece of software is a machine that has never been built before. The process of describing how the machine works is the same as building the machine. The more your machine looks like existing ones, the easier it is to estimate its difficulty. Of course it won’t be exactly the same machine; that can happen with a chair but not with software. On the other hand, you may want to boldly build what no one has built before. In that case, you’ll most likely adjust your scope so that you can build something that makes sense for the timeframes you work with. The solution to the original problem might take iterations over generations. Not necessarily generations of humans, perhaps generations of product versions, teams or even companies. You may set out to put a man on the moon, and your contribution would be the first airplane.

Discuss on Hacker News

When it was time for the parents to leave, their children kissed them goodbye — fully independent for the first time and on the brink of academic adventure — and headed, en masse, to their next session, one that catapulted them directly from orientation to the job market: “From College to Career.”

Rock music played as the students entered the school’s chapel, and then Andy Chan, vice president in charge of the Office of Personal and Career Development — the O.P.C.D., it is called — introduced his team with a video spoof of the television show “The Office.” The students played a game in which they could guess, by text, which majors had been chosen by various gainfully employed alumni of the school. (Human-capital analyst at Deloitte? And the answer is . . . German!) And a panel of students shared their own glamorous work experiences: a fellowship in Paris, an internship at a start-up.

Staff members from the O.P.C.D. had handed out forms asking the students what fields they’d like to work in and where they’d like to live. At the end of the session, Chan directed the students to fold them into paper airplanes. Then, as a group, they let them fly, a symbol of “them launching their careers,” as he had put it. Some planes soared, others took nose-dives. A young man flinched when someone else’s plane clocked him on the side of his head. Celebratory in midair, planes quickly littered the floor. The career-office staff, who would input the information on the forms into databases, walked around with boxes, picking them up. A few students helped, but many watched, as if curious, but not that curious, to see what would happen next to their professional dreams.

For years, most liberal-arts schools seemed to put career-services offices “somewhere just below parking” as a matter of administrative priority, in the words of Wake Forest’s president, Nathan Hatch. But increasingly, even elite, decidedly non-career-oriented schools are starting to promote their career services during the freshman year, in response to fears about the economy, an ongoing discussion about college accountability and, in no small part, the concerns of parents, many of whom want to ensure a return on their exorbitant investment.

The University of Chicago has extensive pre-professional programming and a career center that engaged with roughly 80 percent of its freshmen last year. Wesleyan University in Middletown, Conn., has a new career center prominently located on campus; its Web site urges freshmen to stop by and start their four-year plan. Michael S. Roth, the school’s president, says he wants the career program “to work with our students from the first year to think about how what they’re learning can be translated into other spheres.” Like Chan, Roth believes that the process can make for more thoughtful, meaningful careers choices; but he also told me that the demand from parents for better career services has pushed resources in that direction (for those schools that can afford it; many schools have been forced to cut back their career-center budgets). “My parents didn’t expect me to have an easy time when I graduated,” said Roth, who recalled finishing college in the challenging economy of the late ’70s. “I think families at these, dare I say, fantasy schools — they’re used to kids getting what they want, and they expect that to happen at graduation.”

September 12, 2013

Around three months ago, the company I worked for got suddenly closed.

The reasons for this, are quite irrelevant to this article, but in short, the investor backed out and the ongoing development of a product I was mainly involved in, had been suspended.

I didn’t plan for this scenario. The timing was off, I just came back from vacation and had had a semi long-term plans based on a stable situation with my current day job.

But it had happened and, practically, overnight I had to figure out my next moves for the next couple of weeks.

The idea of working remotely has always been compelling for me. Somehow I always knew that at some point in time, I’d like to “escape the nine to five”, travel the world with my laptop, make money doing freelancing gigs, experience life to the fullest, and yet maintain the technical skills without becoming rusty.

And suddenly - here I was, without a lot of attachments, no mortgage, kids, with some savings on my account and no immediate plans for the nearest future.

Having said that, the decision was pretty simple - this was a great opportunity to do some traveling, that I’ve always wanted to do.

Choosing the location was also quite obvious and for a number of reasons I finally bought a ticket to Bangkok.

Southeast Asia FTW!

I’ve already done some traveling in Europe and wanted to see some other parts of the world.

I’ve spent a year in States and would love to come back, but I wanted to depart as soon as possible and American visa policy is both expensive and still quite hard to get for Polish citizens.

Australia is just too damn expensive for longterm travel on budget, especially when, like me, you’re into scuba diving, kite surfing and numerous other outdoor activities that are generally quite pricey, even in “cheap” countries.

So, Southeast Asia was an obvious choice. It’s the mecca of budget travelers. There are countless blogs owned by people who work online and travel this region and, I guess, that it’s just a thing that one has to experience at least once in his or hers lifetime.

Fortunately, I had some minor clients that I worked for after hours when I still had a job, so while much, much smaller, some income while on the road was pretty much secured.

After around one month of preparations (vaccines, gear, minor planning), me and my girlfriend packed our bags (42 liter Northface Duffels - highly recommended, although not the best choice if walking long distances) and was off to Bangkok with a general idea to travel around Thailand, Vietnam, Laos and Cambodia (which, by the way, we later corrected to Thailand, Vietnam, Cambodia and Indonesia, but this is different story).

I’ve been on the road for two months now, working online for the clients back in Poland. The gigs mostly include maintaining and modifying existing websites, but I’ve lately managed to secure one larger Rails project that needs to be done from scratch.

So far, this has been a very rewarding, educating (although not really in the technical sense) experience, but there are some major drawbacks to this kind of lifestyle, that I’d like to share.

They mainly involve two recurring, significant problems that I found I was facing:

1) Limited types of projects, you are able to tackle while on the road;

2) Illusion of freedom of choice concerning your location;

Only small, simple projects

This is a huge disadvantage for me. Initially I was OK with doing some minor gigs that would just keep me afloat, but the reality of this is that now, I have a constant feeling of underachievement and wasting time.

Yes, it’s fun to open your laptop in your bungalow on Koh Phangan, right next to a beautiful beach, chat with your client or hack around on the project, knowing that after the work is done, you’ll jump in a 30 degree (Celsius) warm waters of the Gulf of Thailand, or better yet, go free diving on numerous reefs around, but this comes with a price.

Also, I don’t believe that long-term travel is actually waste of time. There are numerous advantages and opportunities to grow while traveling, but you can’t have the best of both worlds all at once unfortunately.

If you like challenges and have the need of constant self development in your area of expertise, it’s just very hard to find the necessary balance here.

Of course it all depends on the kinds of projects you manage to get, and technical problems you face during implementation, but let’s be honest here - knowing that you’re on a constant move, quite often facing unpredictable internet access quality and desire to make the most out of your traveling experience you just can’t be doing some substantial, challenging work, that involves hours of analysis, discussions and complicated programming.

There are just too many distractors around and too much stuff to see after (and sometimes during) work.

I can already tell, that some people are now thinking that it’s just a matter of self-control, planning and proper preparation, but for me it never was the case.

I think of myself as a very self-motivated guy. My clients are happy with the work that I’m doing and I always manage to meet the agreed deadlines, while maintaining high quality standards I set for myself.

This is not the point. The point is, that having some significant experience on mid-sized projects, involving some non-trivial programming, I observe, that most of the kind of work I’ve been doing at my day job, would be either impossible or extremely challenging to get done while on the road.

This limits my options to taking only the kinds of projects that I know I’ll be able to easily handle by myself, within very predictable timelines and involving limited amount of research.

And to be completely honest, it’s just not enough. I believe that working with a team of people that are more experienced and smarter than yourself is crucial for your development, and as a freelancer, doing minor gigs involving some MVC/CRUD application programming, you just miss out on a world of possibilities to grow and learn.

Don’t get me wrong though - I keep busy, try to learn something new everyday and strive to continue to develop my skills as a programmer. For the past two months, however, I just felt I lack the exposure to real-world problems that you get to face, when on the job in an organization that everyday tackles problems way more sophisticated than you’d be able to tackle on your own. Whether it’s a small startup or a large corporation is irrelevant here.

The location independence illusion

Now, I absolutely think that working remotely is possible and it’s a great way for (a) developers to stay “location independent” and (b) companies to cut costs a bit, making use of available technologies.

The problem here is quite different though. Working on the road gives you a kind of illusion of location independence. While it’s true you can work for one company being physically “anywhere” in the world, in order to get some substantial amount of work done, you actually need a comfortable spot, good internet connection, peaceful environment and ideally some facilities around like a gym/yoga school park or a bar - whatever you’re into.

Even though it might be obvious, during my travels I found out the hard way that creative, meaningful work, requires some routine. Changing your location once a week, working from benches, hammocks, cafes, bars and hostel floors is a cool way to fund your vacation, but it certainly doesn’t help you when tackling hard programming problems.

Thinking about different possibilities and layers of these problems, currently I have a couple of solution ideas listed below:

1) get a remote-friendly job that fits your skill set and ambitions. Move every quarter or so. Stay in one place for a while. Immerse yourself in the culture and rhythm of the chosen destination, rent a nice room or apartment with a legit desk and a chair you can actually sit in for longer than 3hrs (unless you’re one of the standup-work-environments kind of guy, than forget it), get the routine going. Live somewhere you like, and when you feel like change, do some research and go for it.

2) get a real job, in real office, do stuff you love and challenge yourself (actually, always do that, ofc), negotiate long-term vacations and go traveling for instance, for two months every year or so (details irrelevant).

3) keep working on that “passive income” wordpress site and/or travel blog that makes you tons of cash and travel till you puke ;)

SIDENOTES:

• I personally dislike the ‘digital nomad’ term. There is something inherently ‘douchy’ about this title.
• If interested, please follow my travel adventures at airseasummit.com

On January 23, 1961, a B-52 packing a pair of Mark 39 hydrogen bombs suffered a refueling snafu and went into an uncontrolled spin over North Carolina. In the cockpit of the rapidly disintegrating bomber (only one crew member bailed out safely) was a lanyard attached to the bomb-release mechanism. Intense G-forces tugged hard at it and unleashed the nukes, which, at four megatons, were 250 times more powerful than the weapon that leveled Hiroshima. One of them "failed safe" and plummeted to the ground unarmed. The other weapon's failsafe mechanisms—the devices designed to prevent an accidental detonation—were subverted one by one, as Eric Schlosser recounts in his new book, Command and Control:

Unable to deny that two of its bombs had fallen from the sky—one in a swampy meadow, the other in a field near Faro, North Carolina—the Air Force insisted that there had never been any danger of a nuclear detonation. This was a lie.

Here's the truth: Just days after JFK was sworn in as president, one of the most terrifying weapons in our arsenal was a hair's breadth from detonating on American soil. It would have pulverized North Carolina and, depending on wind conditions, blanketed East Coast cities (including New York and Washington, DC) in lethal fallout. The only thing standing between us and an explosion so catastrophic that it would have radically altered the course of history was a simple electronic toggle switch in the cockpit, a part that probably cost a couple of bucks to manufacture and easily could have been undermined by a short circuit—hardly a far-fetched scenario in an electronics-laden airplane that's breaking apart.

The anecdote above is just one of many "holy shit!" revelations readers will discover in the latest book from the best-selling author of Fast Food Nation. Easily the most unsettling work of nonfiction I've ever read, Schlosser's six-year investigation of America's "broken arrows" (nuclear weapons mishaps) is by and large historical—this stuff is top secret, after all—but the book is beyond relevant. It's critical reading in a nation with thousands of nukes still on hair-trigger alert.

In sections, Command and Control reads like a character-driven thriller as Schlosser draws on his deep reporting, extensive interviews, and documents obtained via the Freedom of Information Act to demonstrate how human error, computer glitches, dilution of authority, poor communications, occasional incompetence, and the routine hoarding of crucial information have nearly brought about our worst nightmare on numerous occasions.

While casual readers will learn a great deal about the history and geopolitics of our nuclear arsenal, Schlosser's central narrative is built around a deadly 1980 explosion at a missile silo in Damascus, Arkansas, where our most powerful weapon, a W-53 thermonuclear warhead,  sat atop a Titan II missile. He puts us on site as the catastrophe unfolds, offering an intimate window on the perspectives and personalities of those involved. It's a gripping yarn that shows how the military concept of "command and control"—the process that governs how decisions are made and orders are executed—functions in practice, and how it can unravel in a crisis.

"Command and Control" will leave readers with a deep unease about our ability—let alone, say, Pakistan's—to handle nuclear weapons safely.

Absent the Soviet threat, it's easy to forget that these ungodly devices are still all around us. An entire generation, as Schlosser told me recently, is blissfully unaware of the specter of nuclear devastation. But Command and Control will leave readers of any age with a deep unease about our ability—to say nothing of, say, Pakistan's—to handle these weapons safely. Schlosser wrote the book in the hope of reviving America's long-dormant debate about "the most dangerous machines ever invented." Fortunately, he delivers a page-turner, not a doorstop.

So below you'll find the first chapter. It's just a tease, but it'll give you a taste of what's in store. The book is available September 17. Buy it. Read it. Make noise about it. And don't miss my chat with Schlosser about his epic project, and why he believes "it's remarkable—it's incredible!—that a major city hasn't been destroyed since Nagasaki."

*******

The following excerpt is reprinted by arrangement with The Penguin Press, a member of Penguin Group (USA) LLC, a Penguin Random House Company. Copyright © Eric Schlosser, 2013.

Not Good

On September 18, 1980, at about 6:30 in the evening, Senior Airman David Powell and Airman Jeffrey Plumb walked into the silo at Launch Complex 374-7, a few miles north of Damascus, Arkansas. They were planning to do a routine maintenance procedure on a Titan II missile.

They'd spent countless hours underground at complexes like this one. But no matter how many times they entered the silo, the Titan II always looked impressive. It was the largest intercontinental ballistic missile ever built by the United States: 10 feet in diameter and 103 feet tall, roughly the height of a nine-story building. It had an aluminum skin with a matte finish and U.S. AIR FORCE painted in big letters down the side. The nose cone on top of the Titan II was deep black, and inside it sat a W-53 thermonuclear warhead, the most powerful weapon ever carried by an American missile. The warhead had a yield of nine megatons—about three times the explosive force of all the bombs dropped during the Second World War, including both atomic bombs.

The silo was eerily quiet, and mercury vapor lights on the walls bathed the missile in a bright white glow.

Day or night, winter or spring, the silo always felt the same. It was eerily quiet, and mercury vapor lights on the walls bathed the missile in a bright white glow. When you opened the door on a lower level and stepped into the launch duct, the Titan II loomed above you like an immense black-tipped silver bullet, loaded in a concrete gun barrel, primed, cocked, ready to go, and pointed at the sky.

The missile was designed to launch within a minute and hit a target as far as 6,000 miles away. In order to do that, the Titan II relied upon a pair of liquid propellants—a rocket fuel and an oxidizer—that were "hypergolic." The moment they came into contact with each other, they'd instantly and forcefully ignite. The missile had two stages, and inside both of them, an oxidizer tank rested on top of a fuel tank, with pipes leading down to an engine. Stage 1, which extended about 70 feet upward from the bottom of the missile, contained about 85,000 pounds of fuel and 163,000 pounds of oxidizer.

Stage 2, the upper section where the warhead sat, was smaller and held about one fourth of those amounts. If the missile were launched, fuel and oxidizer would flow through the stage 1 pipes, mix inside the combustion chambers of the engine, catch on fire, emit hot gases, and send almost half a million pounds of thrust through the supersonic convergent-divergent nozzles beneath it. Within a few minutes, the Titan II would be 50 miles off the ground.

The two propellants were extremely efficient—and extremely dangerous. The fuel, Aerozine-50, could spontaneously ignite when it came into contact with everyday things like wool, rags, or rust. As a liquid, Aerozine-50 was clear and colorless. As a vapor, it reacted with the water and the oxygen in the air and became a whitish cloud with a fishy smell. This fuel vapor could be explosive in proportions as low as 2 percent. Inhaling it could cause breathing difficulties, a reduced heart rate, vomiting, convulsions, tremors, and death. The fuel was also highly carcinogenic and easily absorbed through the skin.

The missile's oxidizer was classified as a "Poison A," the most deadly category of man-made chemicals.

The missile's oxidizer, nitrogen tetroxide, was even more hazardous. Under federal law, it was classified as a "Poison A," the most deadly category of man-made chemicals. In its liquid form, the oxidizer was a translucent, yellowy brown. Although not as flammable as the fuel, it could spontaneously ignite if it touched leather, paper, cloth, or wood. And its boiling point was only 70 degrees Fahrenheit. At temperatures any higher, the liquid oxidizer boiled into a reddish brown vapor that smelled like ammonia. Contact with water turned the vapor into a corrosive acid that could react with the moisture in a person's eyes or skin and cause severe burns. When inhaled, the oxidizer could destroy tissue in the upper respiratory system and the lungs. The damage might not be felt immediately. Six to twelve hours after being inhaled, the stuff could suddenly cause headaches, dizziness, difficulty breathing, pneumonia, and pulmonary edema leading to death.

177 Days of GitHub

This is the story of how I contributed to GitHub for 177 consecutive days, nearly half a year. I will talk of why I did it and why you should too, or at least why you should do something similar. This is a story about accepting challenges, making habits, and knowing when to stop. I learned a lot through this, and I hope to teach you all something. Let’s begin.

We can't all fight habits as well as Calvin.

The 30-day GitHub challenge

It all started as a 30-day challenge. A good friend of mine Shuhao messaged me on the 21st of February, challenging me to contribute to GitHub every day for 30 consecutive days and he would do the same. At that time my longest streak on GitHub had been a measly 5 days. It was a lack-luster streak that I wanted to improve but just couldn’t find enough motivation. This opportunity to both challenge myself and challenge him was just what I needed to make it all happen. I accepted the challenge under a few simple conditions:

I started this streak off with a bang, contributing nearly 30 commits to my website and other various projects on the first day. This actually happened to be the most contributions in a single day that I made throughout the entire half-year streak. I was motivated to make it to 30 days and maybe even keep going until one of us declared defeat, so this was the best way to start.

One week

A week into the challenge, difficulty started to set in. I was nearly forgetting to commit or otherwise contribute, waiting until the last possible moment to submit my daily contribution. Shuhao ended up dropping out of the challenge at 8 days. I think he just forgot, which I don’t blame him.

But I kept going…

Two weeks

Two to three weeks in it was easier to remember to commit, it was becoming more of a habit. Soon internal reminders to contribute were paired with the reminder to brush my teeth, something I rarely forget to do. Around the same time I picked up a book called The Power of Habit which educated me on habits and how to understand and change your habits for your benefit. I recommend you read this book if you have any interest in how your brain deals with habits. It’s quite fascinating, one story in the book spoke of a man who couldn’t remember how to navigate his own home and yet could go for daily walks around the block and not get lost, all thanks of the power of habits. But I digress. Before I knew it, the power of habit had brought me to…

30 days!

Thirty days was exciting! I had made it! And all I wanted to do was keep going! After 30 days, it just became a way of life. Eat, sleep, commit. It became normal and I no longer found it difficult to include in my day-to-day rituals. But like any habit, it was hard to be 100% consistent, so there were definitely some close calls on busy days. I commited from my phone. I committed after getting home extra late from hanging with friends. I even commited in a state of Ballmer Peak. This reminds me of a few tricks I learned along the way regarding GitHub contributions:

GitHub contribution tricks

Though these tricks admittedly saved my ass when it came to my GitHub streak, I was keeping within the simple conditions originally outlined above. You can see more tips from GitHub here.

100 Days

One hundred days was a big moment for me as well. At this point I was in San Francisco, interning at Mozilla. You can read all about that experience on my blog. The great thing about working at Mozilla is that everything is open source, so I more or less could just work and keep my streak going. It wasn’t as easy as that, but being at Mozilla really helped.

Mozilla had their fair share of internal competition as well, including Shuhao who joined me in interning there this summer and was back at his streak, this time with much more success, as well a silent competitor I never knew personally but whom I heard about through the Mozilla grapevine.

150 Days

There reached a point where my motivation to keep the streak going took a nose-dive. I contribute this to a lack of ideas, a drive for a fresh challenge, and the realization of the inevitability that my streak would eventually end. All this combined with the fact was that there was just too many ways to “cheat” the system, like this guy did, and thus the challenge started to turn into a chore.

I wanted to escape. I wanted time to refresh and forget about GitHub for just a little bit. I just wanted to be free to think about new ideas and not worry about making that commit just for the sake of it. I decided to end the streak on a nice round number. I decided to continue until 200 days were reached and then I would graciously sit back and watch the streak come crumbling down.

Oops

200 days didn’t come. I forgot to commit on day #178, and with this tragic realization a sudden weight was lifted off me and I felt simultaneously relieved. It was all finally over.

Looking back

My 177-day streak taught me a lot about habits. It taught me how good habits can turn evil when excessively executed. It taught me a lot about the power of ”don’t break the chain”, the motivational technique behind successful GitHub streaks. I still commit to GitHub on a regular basis but now I make less required and more desired contributions. I take regular days away from the computer to refresh my ideas and come back stronger than when I left. I recommend everyone try to break a habit or start a habit using this technique, but it can be really powerful so much as to be draining so watch out!

If you know of anyone with a longer GitHub streak than me, please let me know! I haven’t found anyone yet! ;)

Happy contributing! Follow me on GitHub! :) Discuss on HN.

My Postmodern Adventure

This is the story of one computer professional's explorations in the world of postmodern literary criticism. I'm a working software engineer, not a student nor an academic nor a person with any real background in the humanities. Consequently, I've approached the whole subject with a somewhat different frame of mind than perhaps people in the field are accustomed to. Being a vulgar engineer I'm allowed to break a lot of the rules that people in the humanities usually have to play by, since nobody expects an engineer to be literate. Ha. Anyway, here is my tale.

It started when my colleague Randy Farmer and I presented a paper at the Second International Conference on Cyberspace, held in Santa Cruz, California in April, 1991. Like the first conference, at which we also presented a paper, it was an aggressively interdisciplinary gathering, drawing from fields as diverse as computer science, literary criticism, engineering, history, philosophy, anthropology, psychology, and political science. About the only relevant field that seemed to lack strong representation was economics (an important gap but one which we don't have room to get into here). It was in turn stimulating, aggravating, fascinating and infuriating, a breathtaking intellectual roller coaster ride unlike anything else I've recently encountered in my professional life. My last serious brush with the humanities in an academic context had been in college, ten years earlier. The humanities appear to have experienced a considerable amount of evolution (or perhaps more accurately, genetic drift) since then.

Randy and I were scheduled to speak on the second day of the conference. This was fortunate because it gave us the opportunity to recalibrate our presentation based on the first day's proceedings, during which we discovered that we had grossly mischaracterized the audience by assuming that it would be like the crowd from the first conference. I spent most of that first day furiously scribbling notes. People kept saying the most remarkable things using the most remarkable language, which I found I needed to put down in writing because the words would disappear from my brain within seconds if I didn't. Are you familiar with the experience of having memories of your dreams fade within a few minutes of waking? It was like that, and I think for much the same reason. Dreams have a logic and structure all their own, falling apart into unmemorable pieces that make no sense when subjected to the scrutiny of the conscious mind. So it was with many of the academics who got up to speak. The things they said were largely incomprehensible. There was much talk about deconstruction and signifiers and arguments about whether cyberspace was or was not "narrative". There was much quotation from Baudrillard, Derrida, Lacan, Lyotard, Saussure, and the like, every single word of which was impenetrable. I'd never before had the experience of being quite this baffled by things other people were saying. I've attended lectures on quantum physics, group theory, cardiology, and contract law, all fields about which I know nothing and all of which have their own specialized jargon and notational conventions. None of those lectures were as opaque as anything these academics said. But I captured on my notepad an astonishing collection of phrases and a sense of the overall tone of the event.

We retreated back to Palo Alto that evening for a quick rewrite. The first order of business was to excise various little bits of phraseology that we now realized were likely to be perceived as Politically Incorrect. Mind you, the fundamental thesis of our presentation was Politically Incorrect, but we wanted people to get upset about the actual content rather than the form in which it was presented. Then we set about attempting to add something that would be an adequate response to the postmodern lit crit-speak we had been inundated with that day. Since we had no idea what any of it meant (or even if it actually meant anything at all), I simply cut-and-pasted from my notes. The next day I stood up in front of the room and opened our presentation with the following:

This bit of nonsense was constructed entirely out of things people had actually said the day before, except for the last ten words or so which are a pastiche of Danny Kaye's "flagon with the dragon" bit from The Court Jester, contributed by our co-worker Gayle Pergamit, who took great glee in the entire enterprise. Observing the audience reaction was instructive. At first, various people started nodding their heads in nods of profound understanding, though you could see that their brain cells were beginning to strain a little. Then some of the techies in the back of the room began to giggle. By the time I finished, unable to get through the last line with a straight face, the entire room was on the floor in hysterics, as by then even the most obtuse English professor had caught on to the joke. With the postmodernist lit crit shit thus defused, we went on with our actual presentation.

Contrary to the report given in the "Hype List" column of issue #1 of Wired("Po-Mo Gets Tek-No", page 87), we did not shout down the postmodernists. We made fun of them.

Afterward, however, I was left with a sense that I should try to actually understand what these people were saying, really. I figured that one of three cases must apply. It could be that there was truly some content there of value, once you learned the lingo. If this was the case, then I wanted to know what it was. On the other hand, perhaps there was actually content there but it was bogus (my working hypothesis), in which case I wanted to be able to respond to it credibly. On the third hand, maybe there was no content there after all, in which case I wanted to be able to write these clowns off without feeling guilty that I hadn't given them due consideration.

The subject that I kept hearing about over and over again at the conference was deconstruction. I figured I'd start there. I asked my friend Michael Benedikt for a pointer to some sources. I had gotten to know Michael when he organized the First International Conference on Cyberspace. I knew him to be a person with a foot in the lit crit camp but also a person of clear intellectual integrity who was not a fool. He suggested a book called On Deconstruction by Jonathan Culler. I got the book and read it. It was a stretch, but I found I could work my way through it, although I did end up with the most heavily marked up book in my library by the time I was done. The Culler book lead me to some other things, which I also read. And I started subscribing to alt.postmodern and now actually find it interesting, much of the time. I can't claim to be an expert, but I feel I've reached the level of a competent amateur. I think I can explain it. It turns out that there's nothing to be afraid of.

We engineers are frequently accused of speaking an alien language, of wrapping what we do in jargon and obscurity in order to preserve the technological priesthood. There is, I think, a grain of truth in this accusation. Defenders frequently counter with arguments about how what we do really is technical and really does require precise language in order to talk about it clearly. There is, I think, a substantial bit of truth in this as well, though it is hard to use these grounds to defend the use of the term "grep" to describe digging through a backpack to find a lost item, as a friend of mine sometimes does. However, I think it's human nature for members of any group to use the ideas they have in common as metaphors for everything else in life, so I'm willing to forgive him.

The really telling factor that neither side of the debate seems to cotton to, however, is this: technical people like me work in a commercial environment. Every day I have to explain what I do to people who are different from me -- marketing people, technical writers, my boss, my investors, my customers -- none of whom belong to my profession or share my technical background or knowledge. As a consequence, I'm constantly forced to describe what I know in terms that other people can at least begin to understand. My success in my job depends to a large degree on my success in so communicating. At the very least, in order to remain employed I have to convince somebody else that what I'm doing is worth having them pay for it.

Contrast this situation with that of academia. Professors of Literature or History or Cultural Studies in their professional life find themselves communicating principally with other professors of Literature or History or Cultural Studies. They also, of course, communicate with students, but students don't really count. Graduate students are studying to be professors themselves and so are already part of the in-crowd. Undergraduate students rarely get a chance to close the feedback loop, especially at the so called "better schools" (I once spoke with a Harvard professor who told me that it is quite easy to get a Harvard undergraduate degree without ever once encountering a tenured member of the faculty inside a classroom; I don't know if this is actually true but it's a delightful piece of slander regardless). They publish in peer reviewed journals, which are not only edited by their peers but published for and mainly read by their peers (if they are read at all). Decisions about their career advancement, tenure, promotion, and so on are made by committees of their fellows. They are supervised by deans and other academic officials who themselves used to be professors of Literature or History or Cultural Studies. They rarely have any reason to talk to anybody but themselves -- occasionally a Professor of Literature will collaborate with a Professor of History, but in academic circles this sort of interdisciplinary work is still considered sufficiently daring and risqué as to be newsworthy.

What you have is rather like birds on the Galapagos islands -- an isolated population with unique selective pressures resulting in evolutionary divergence from the mainland population. There's no reason you should be able to understand what these academics are saying because, for several generations, comprehensibility to outsiders has not been one of the selective criteria to which they've been subjected. What's more, it's not particularly important that they even be terribly comprehensible to each other, since the quality of academic work, particularly in the humanities, is judged primarily on the basis of politics and cleverness. In fact, one of the beliefs that seems to be characteristic of the postmodernist mind set is the idea that politics and cleverness are the basis for all judgments about quality or truth, regardless of the subject matter or who is making the judgment. A work need not be right, clear, original, or connected to anything outside the group. Indeed, it looks to me like the vast bulk of literary criticism that is published has other works of literary criticism as its principal subject, with the occasional reference to the odd work of actual literature tossed in for flavoring from time to time.

Thus it is not surprising that it takes a bit of detective work to puzzle out what is going on. But I've been on the case for a while now and I think I've identified most of the guilty suspects. I hope I can spare some of my own peers the inconvenience and wasted time of actually doing the legwork themselves (though if you have an inclination in that direction I recommend it as a mind stretching departure from debugging C code).

The basic enterprise of contemporary literary criticism is actually quite simple. It is based on the observation that with a sufficient amount of clever handwaving and artful verbiage, you can interpret any piece of writing as a statement about anything at all. The broader movement that goes under the label "postmodernism" generalizes this principle from writing to all forms of human activity, though you have to be careful about applying this label, since a standard postmodernist tactic for ducking criticism is to try to stir up metaphysical confusion by questioning the very idea of labels and categories. "Deconstruction" is based on a specialization of the principle, in which a work is interpreted as a statement about itself, using a literary version of the same cheap trick that Kurt Gödel used to try to frighten mathematicians back in the thirties.

Deconstruction, in particular, is a fairly formulaic process that hardly merits the commotion that it has generated. However, like hack writers or television producers, academics will use a formula if it does the job and they are not held to any higher standard (though perhaps Derrida can legitimately claim some credit for originality in inventing the formula in the first place). Just to clear up the mystery, here is the formula, step-by-step:

Step 1 -- Select a work to be deconstructed. This is called a "text" and is generally a piece of text, though it need not be. It is very much within the lit crit mainstream to take something which is not text and call it a text. In fact, this can be a very useful thing to do, since it leaves the critic with broad discretion to define what it means to "read" it and thus a great deal of flexibility in interpretation. It also allows the literary critic to extend his reach beyond mere literature. However, the choice of text is actually one of the less important decisions you will need to make, since points are awarded on the basis of style and wit rather than substance, although more challenging works are valued for their greater potential for exercising cleverness. Thus you want to pick your text with an eye to the opportunities it will give you to be clever and convoluted, rather than whether the text has anything important to say or there is anything important to say about it. Generally speaking, obscure works are better than well known ones, though an acceptable alternative is to choose a text from the popular mass media, such as a Madonna video or the latest Danielle Steele novel. The text can be of any length, from the complete works of Louis L'Amour to a single sentence. For example, let's deconstruct the phrase, "John F. Kennedy was not a homosexual."

Step 2 -- Decide what the text says. This can be whatever you want, although of course in the case of a text which actually consists of text it is easier if you pick something that it really does say. This is called "reading". I will read our example phrase as saying that John F. Kennedy was not a homosexual.

Step 3 -- Identify within the reading a distinction of some sort. This can be either something which is described or referred to by the text directly or it can be inferred from the presumed cultural context of a hypothetical reader. It is a convention of the genre to choose a duality, such as man/woman, good/evil, earth/sky, chocolate/vanilla, etc. In the case of our example, the obvious duality to pick is homosexual/heterosexual, though a really clever person might be able to find something else.

Step 4 -- Convert your chosen distinction into a "hierarchical opposition" by asserting that the text claims or presumes a particular primacy, superiority, privilege or importance to one side or the other of the distinction. Since it's pretty much arbitrary, you don't have to give a justification for this assertion unless you feel like it. Programmers and computer scientists may find the concept of a hierarchy consisting of only two elements to be a bit odd, but this appears to be an established tradition in literary criticism. Continuing our example, we can claim homophobia on the part of the society in which this sentence was uttered and therefor assert that it presumes superiority of heterosexuality over homosexuality.

Step 5 -- Derive another reading of the text, one in which it is interpreted as referring to itself. In particular, find a way to read it as a statement which contradicts or undermines either the original reading or the ordering of the hierarchical opposition (which amounts to the same thing). This is really the tricky part and is the key to the whole exercise. Pulling this off successfully may require a variety of techniques, though you get more style points for some techniques than for others. Fortunately, you have a wide range of intellectual tools at your disposal, which the rules allow you to use in literary criticism even though they would be frowned upon in engineering or the sciences. These include appeals to authority (you can even cite obscure authorities that nobody has heard of), reasoning from etymology, reasoning from puns, and a variety of other word games. You are allowed to use the word "problematic" as a noun. You are also allowed to pretend that the works of Freud present a correct model of human psychology and the works of Marx present a correct model of sociology and economics (it's not clear to me whether practitioners in the field actually believe Freud and Marx or if it's just a convention of the genre).

You get maximum style points for being French. Since most of us aren't French, we don't qualify for this one, but we can still score almost as much by writing in French or citing French sources. However, it is difficult for even the most intense and unprincipled American academician writing in French to match the zen obliqueness of a native French literary critic. Least credit is given for a clear, rational argument which makes its case directly, though of course that is what I will do with our example since, being gainfully employed, I don't have to worry about graduation or tenure. And besides, I'm actually trying to communicate here. Here is a possible argument to go with our example:

Of course, no real deconstruction would be like this. I only used a single paragraph and avoided literary jargon. All of the words will be found in a typical abridged dictionary and were used with their conventional meanings. I also wrote entirely in English and did not cite anyone. Thus in an English literature course I would probably get a D for this, but I already have my degree so I don't care.

Another minor point, by the way, is that we don't say that we deconstruct the text but that the text deconstructs itself. This way it looks less like we are making things up.

That's basically all there is to it, although there is an enormous variety of stylistic complication that is added in practice. This is mainly due to the genetic drift phenomenon I mentioned earlier, resulting in the intellectual equivalent of peacock feathers, although I suspect that the need for enough material to fill up a degree program plays a part as well. The best way to learn, of course, is to try to do it yourself. First you need to read some real lit crit to get a feel for the style and the jargon. One or two volumes is all it takes, since it's all pretty much the same (I advise starting with the Culler book the way I did). Here are some ideas for texts you might try to deconstruct, once you are ready to attempt it yourself, graded by approximate level of difficulty:

Beginner:

Intermediate:

Advanced:

Tour de Force:

So, what are we to make of all this? I earlier stated that my quest was to learn if there was any content to this stuff and if it was or was not bogus. Well, my assessment is that there is indeed some content, much of it interesting. The question of bogosity, however, is a little more difficult. It is clear that the forms used by academicians writing in this area go right off the bogosity scale, pegging my bogometer until it breaks. The quality of the actual analysis of various literary works varies tremendously and must be judged on a case-by-case basis, but I find most of it highly questionable. Buried in the muck, however, are a set of important and interesting ideas: that in reading a work it is illuminating to consider the contrast between what is said and what is not said, between what is explicit and what is assumed, and that popular notions of truth and value depend to a disturbingly high degree on the reader's credulity and willingness to accept the text's own claims as to its validity.

Looking at the field of contemporary literary criticism as a whole also yields some valuable insights. It is a cautionary lesson about the consequences of allowing a branch of academia that has been entrusted with the study of important problems to become isolated and inbred. The Pseudo Politically Correct term that I would use to describe the mind set of postmodernism is "epistemologically challenged": a constitutional inability to adopt a reasonable way to tell the good stuff from the bad stuff. The language and idea space of the field have become so convoluted that they have confused even themselves. But the tangle offers a safe refuge for the academics. It erects a wall between them and the rest of the world. It immunizes them against having to confront their own failings, since any genuine criticism can simply be absorbed into the morass and made indistinguishable from all the other verbiage. Intellectual tools that might help prune the thicket are systematically ignored or discredited. This is why, for example, science, psychology and economics are represented in the literary world by theories that were abandoned by practicing scientists, psychologists and economists fifty or a hundred years ago. The field is absorbed in triviality. Deconstruction is an idea that would make a worthy topic for some bright graduate student's Ph.D. dissertation but has instead spawned an entire subfield. Ideas that would merit a good solid evening or afternoon of argument and debate and perhaps a paper or two instead become the focus of entire careers.

Engineering and the sciences have, to a greater degree, been spared this isolation and genetic drift because of crass commercial necessity. The constraints of the physical world and the actual needs and wants of the actual population have provided a grounding that is difficult to dodge. However, in academia the pressures for isolation are enormous. It is clear to me that the humanities are not going to emerge from the jungle on their own. I think that the task of outreach is left to those of us who retain some connection, however tenuous, to what we laughingly call reality. We have to go into the jungle after them and rescue what we can. Just remember to hang on to your sense of humor and don't let them intimidate you.

Before we get started, I should admit something: I have no idea how a country goes from being poor to being rich.

I know that my own country did this, sure, and that Western Europe and Australia and Japan and Korea and lots of other places did too. But I don’t know anywhere that did it without totally unique circumstances, without an ugly century or two on the way.

In America, getting from poverty to here meant crowded factories, tenement housing, belching smokestacks, diseases caused by human shit in the drinking water. In other places development was born out of devastation, revolution, authoritarianism—nothing we would ask other countries to emulate.

Zambia is poor—that much is clear as soon as you arrive. To get to Kitwe, a city of 500,000 people in the Copper Belt Province, you land at Ndola airport an hour away. “Airport” is putting it grandiosely. It’s a strip of runway next to a low building the size of an exurban Starbucks. You get off the plane, walk 100 feet across the tarmac, and wait under an awning until a tractor pulls up, towing a cart with your luggage. Everyone crowds in, grabbing their bags, and you do, too—it’s all over in about 45 seconds. At no point are you indoors. As you leave, you hear a European in a suit remark, “I wish they did it like this everywhere.”

Zambia is not failed. It is simply very, very poor. Sixty-four percent of the population lives on less than $1 per day, 14 percent have HIV, 40 percent don’t have access to clean drinking water.

Probably Zambia isn’t a country you’ve thought about much. It’s a landlocked patch of 14 million people smack-dab in the middle of Sub-Saharan Africa. Most of its immediate bordermates—Zimbabwe, the Democratic Republic of Congo, Angola—you know from the lowest quintile of various corruption, failed state, and poverty indices.

But relative to its neighbors, Zambia is actually doing pretty well. The religious groups (mostly Christian, some Muslims) largely get along, as do the various ethnicities. There’s no flamboyant dictator, no child soldiers, no celebrity adoptions, just downright boring elections. People do not disappear in the night, nor are they beaten or tortured for how they vote or who they spend time with. Crime waves, food riots, power outages—they’re not unheard of here, but they’re not everyday occurrences either.

So Zambia is not failed. It is simply very, very poor. Sixty-four percent of the population lives on less than $1 per day, 14 percent have HIV, 40 percent don’t have access to clean drinking water. Almost 90 percent of women in rural areas cannot read or write. Name a category—schools, health care, environment—and I’ll give you statistics that will depress the shit out of you.

That’s actually why I’m here. I work for an international development NGO. Part of my job is traveling to developing countries to gather information on the conditions there, to meet people who are working to improve them.

Given what I just admitted, maybe it’s a bit weird that helping countries go from poor to rich is part of what I do for a living. But the more I do this, the less I’m sure of. Like Tolstoy’s unhappy family, every poor country is poor in its own way, and everyone I meet has a narrative, a creation myth, for how it got this way and why it remains so.

I will spend the next 10 days meeting NGO activists, government officials, and business representatives. They will tell me that Zambia is terrible, that Zambia is fine, and that Zambia is getting better, respectively.

I’m not here to determine which of those statements is true. I’m here for the numbers, the information I can’t get back home. Somewhere between the handshakes, the spreadsheets, the PowerPoints, the annual reports, a story will emerge about Zambia, a story of a country watching its mineral wealth disappear, a country making everyone rich but itself.

I can tell we’re getting close to Kitwe because the number of people crossing the highway increases. The highway has no streetlights, the only light is from the cars, and about halfway there we start to see silhouettes of people in twos and threes running across the road. Our driver never slows down, even as the groups increase to six, seven people, crossing our headlights, stopping in the road to let a car whiz by, running again. I could ask him to slow down, but instead I just look.

We stop at a police checkpoint. The cops look in, see me in the backseat, wave us on.

“What was that about?” I ask the driver.

“No reason,” he says. “Just checking us out.”

01. THE MINES
If you look up Kitwe on Google Maps, you’ll see that it’s dotted with what look like lakes, but are actually tailings dams, where the mines dump their runoff. For more than 150 years, the only reason to come to Kitwe—to Zambia, really—was the copper. Everyone, locals and foreigners alike, are here because of the mines—to work in them, to buy something they produce, to produce something they need.

That is why Jonathan Mutambwa (like everyone else in this story, that’s not his real name) is here, too. He works at an NGO in Kitwe that helps communities negotiate with the mining companies. He’s skinny, with a kind face, round glasses, and the same baggy polo shirt both days I see him.

Jonathan takes me to his regular lunch spot, a nightclub in the elbow of Kitwe’s two main streets that operates as a restaurant until dark. It’s buffet-style, and he picks lunch for us both: Fish, maize, a green paste I assume is vegetables. It’s only when we get to the table that I realize he didn’t get us any silverware.

“That’s where you wash your hands,” Jonathan says, pointing to a water cooler with a slotted milk crate under it. I start washing my hands and the owner shrieks and runs over. “The bucket, the bucket!” she says. I was letting the water run onto the floor. Jonathan is laughing at me, along with the rest of the restaurant.

He tells us about his home village, where most people work without contracts as waiters or security officers. “You find out your salary when you are paid. If you are paid.” People go six, eight months without pay. If you complain, you’re gone.

While Jonathan teaches me how to eat—you ball up the maize dough, then pinch the veggies and sauce into it—he talks about Kitwe, how mining created and destroyed it.

Zambia has the 9th richest copper deposits in the world, he says. In fact, that’s how it got its crushed-Coke-can shape. When the British and the Belgians were negotiating the line between Zambia and what is now the Democratic Republic of Congo, they both knew about the fat, lucrative spike of copper running vertically between the two territories.

The British got it first. They marked the borders of their territory by painting blue marks on all of the trees, giving themselves most of the copper. Later, the Belgians drew their own border, this time with beacons driven into the ground, and kept the copper on the Belgian side.

When the British protested, the Belgians pointed out that there was no way to prove where the original borders had been, since all the trees had mysteriously been cut down. The copper stayed in the Democratic Republic of Congo. (Jonathan admits that this story is apocryphal, but he can’t resist retelling it.)

For the rest of the world, the history of Zambia is basically the history of copper in Zambia. In 1890, the British South Africa Company (owned by everybody’s favorite colonialist, Cecil Rhodes) arrived here and called it, with the humility characteristic of the man and the era, Northern Rhodesia.

The copper was so easy to get to that European explorers reportedly saw African tribesmen wearing copper bracelets. The company, backed by the British government, promptly started mining, recruiting the men strong enough to work and taxing everyone else.

My favorite game-show fact about Zambia is that it’s the only country to start the Olympics as one country and finish it as another. In 1964, decades of discontent with British rule, including the flight of all that copper revenue, culminated in a vote for independence. Athletes arrived in Tokyo as Northern Rhodesians and left as Zambians.

They returned to a Zambia whose language was English, religion was Christianity, legal system was common law, and, for better or for worse, economy was married to the world price of copper. Over the next 35 years, Zambia’s economy would suffer simply due to a slump in copper prices. The World Bank and the IMF have been telling Zambia as long as anyone can remember that it needs to diversify, but even now, Zambia’s fortunes (or at least its GDP) rise and fall in sync with copper.

If Zambia is married to mining, then Kitwe is handcuffed to it. Most of the buildings in Kitwe, the roads, the health clinics, the schools, were built by the national mining company. At its peak, the Zambia Consolidated Copper Mines company employed more than 65,000 Zambians and carried out services like water delivery and waste collection for five cities in the Copper Belt Province.

This topic comes up a lot in Kitwe, not just in official meetings, but in chit-chat with waiters, cab drivers, people in waiting rooms. They talk about a time when a mining company provided public services as glory days, the peak from which Kitwe, and Zambia, has fallen.

And fallen it has. The problem with the state-run mining company providing all that infrastructure, all those services, was that it ate into profits. Starting in 1996, the government sold it off in chunks to private bidders. Mining employment has dropped to just 30,000, half of its glory-days peak, and the job of maintaining all that company housing and infrastructure has reverted back to the government.

Today, all of the mines are private. Zambia’s GDP growth has been above six percent for over a decade, riding the wave, as always, of increasing copper prices. Copper is 40 percent of Zambia’s GDP and 95 percent of its exports, but little of that money makes it here. The stats identify Switzerland as Zambia’s primary export market. This is not an indicator that Zambia hosts a thriving chocolate and suspenders sector, but rather that its copper trades are booked in the jurisdiction where they are least likely to be taxed.

Many of the mining companies pay just 0.6 percent royalties to Zambia, far below the already-meager industry standard of three percent. One mine reportedly got the low rate by telling the government “this mine’s basically tapped out, we’ll just be here two years.” That was more than 13 years ago, and the royalty rate hasn’t gone up, even though the price of just about every mineral you can squeeze out of Zambia has.

And then there’s the Chinese. They arrived like a well-packed picnic, everything in shipping crates ready to be unpacked. Their own materials, their own equipment, their own workers, their own fences. If you were designing a foreign investment not to benefit the host community, this is what it would look like.

So far this story—shitty colonizers, problematic independence, Chinese economic invaders—is the one we hear over and over from this part of the world, a sort of Mad Libs history of Sub-Saharan Africa. But once I’m here, I’m dying to know: What does this story mean for the people who live here?

02. THE CULTURE
Our guesthouse is at the top of a hill, next to a belt of power lines we’re told leads all the way to Lusaka. It’s owned by a woman named Helen, who tells us she is “one of the last white Zambians” with the same pride people tell you their family came over on the Mayflower.

One of her employees is Thomas Sonkwe. He is the waiter at our hotel, making us cappuccinos, asking us what time we want dinner and breakfast. He is 43 but looks 21, shaved head on a wiry frame, here from 7 a.m. ‘til 8 p.m. every day but never weary-looking.

“You work in human rights,” Thomas says one night after he’s served dinner, a statement, not a question. “I’m trying to start a human rights organization myself. I want to stop child abuse. The children selling water for the parents in the streets.”

He tells us lots of people here in Kitwe have been laid off (“retrenched” he says, like most people here, the language of a corporate boardroom), and they send their kids into the city to sell clothes, water, fruit, whatever they can arbitrage for a few extra kwachas.

“The Environmental Management Act says developers are allowed to discharge [pollutants] to a certain threshold,” Noah says. “But the same agency can’t determine whether that level is going to harm the environment, and has no equipment to test whether the emissions are within the law.”

“Things look fine here in Kitwe, the city,” he says, and I nod, though this is not true, “but out in the villages, it would shock you the way people treat each other.” He tells us about his home village, where most people work without contracts as waiters or security officers. “You find out your salary when you are paid. If you are paid.” People go six, eight months without pay. If you complain, you’re gone.

Over the next few days, I get Thomas’ life story piece by piece. He was born to a housewife and a chauffeur, one of two boys and five girls. He has sickle cell anemia, and his mother told him “As weak as you are, you’ll never get a hard job, so you’ll need to go to school.”

She told him to be a lawyer: “From the go, my mother said I should be in the court, because there you will be just sitting.”

He is the only person in his family who completed high school. When I ask him what his sisters do for a living, he says they are married.

Ten years ago, one of his sisters died of AIDS. Thomas offered to take in her two children, Aaron and Leonard, whose fathers had died the same way. Thomas wanted to send them to school, but his family resisted, saying there was no point. And his nephews, for their part, “they wanted easy things now,” Thomas says.

By 16, Aaron was married, with a pregnant wife and no job. By 18, his wife was a prostitute, the only viable way to earn a living to support their baby. Three years later, they were both dead of AIDS.

This is why Thomas wants to start an NGO, why he saved up for a year of law school by working as a security guard, why he’s working at this guesthouse, so he can pay for another one. He tells me he knows the syllabus of his next year of law school, so he’s reading the books now, since he’ll have to work while he’s in school and won’t have the time to keep up.

“It’s not a problem with policy, but within families.” This is Namwile Uzondile, the director of a rural health education project. I meet her the morning after Thomas tells me about his nephews. She looks like Pam Grier in Jackie Brown, and talks fast, like she doesn’t have time to wait for you to agree with her on your own.

We are talking about poverty here in Kitwe, what it looks like, what causes it.

“It’s you,” she says.

“… White people?” I ask.

“No, you men,” she says. “The men here are jealous of women’s earnings and education, and block them from getting employment.”

In rural areas, women do most of the farming, but the men are the ones who go to the market, sell the crops, pocket the money. Before they even get home, many of those kwachas have already disappeared into beer, sunglasses, and brand names. The wives have to resort to asking nicely for money for school fees, medicines, next season’s seeds and fertilizers.

Namwile is regularly invited to the mines to lecture employees on spousal abuse, which she describes as an epidemic. Some of the mines have their own on-site battered women’s shelters.

Last year Namwile conducted a survey of prostitutes here in Kitwe, and found that at least half of them had education certificates, but couldn’t find work. Most had been married off early, 15 or 16, and since then had either left their husbands or lost them to AIDS.

“These are decent ladies. If she’s lost her husband and she remains with the family, two or three children, she has nothing to do, nothing to feed the children, the only option is to go out there and look for money,” Namwile says.

Thomas has six kids of his own. He can only afford to send the two oldest to school. The rest, he teaches himself.

“They’re doing better than kids going to grade one,” he says. “The 2nd and 3rd, they can do mathematics because I teach them.”

One of them has sickle cell, and Thomas spends a lot of time at the hospital. A few months ago, while he was there, his landlord came into his house looking for the rent and, not finding Thomas, found his daughter, and beat her up as a punishment for the debt.

I ask him why he had so many kids if he doesn’t have very much money.

“Where I’m from, we are not many,” he says. He wants his kids to get educated and move back to his home village, people it with better skills, a better culture. “I want to make sure,” he says, “my children enjoy what their parents leave for them.”

03. THE LAND
Here’s how you buy a plot of land in Zambia:

First, you go to the tribal chief. Ninety-four percent of the land in Zambia is customary or traditional, no one has a title to it. It’s not just sitting there, people are living on it, farming, grazing animals, it’s just technically under the control of a chief.

The word chief conjures up images of thatched huts and grass skirts, but the chiefs in Zambia are more like governors. They get a government salary, they wear suits, some of them live in the capital.

And they have a lot of power. The chiefs get to approve every purchase of land on their territory, so if you want a chunk of what’s theirs, you have to ask nicely. In Zambia most of the chiefs require a gift just to get a meeting. This might mean taking them lunch at a restaurant in Lusaka, or it could mean buying their daughter a car—it’s up to them.

For most Zambians—say everyone and you’re only off by only a few percentage points—this is the job market, these are your options: Farming a small plot of land or selling consumer products a few at a time.

However you do it, you get the chief to approve the land you want to buy. Then he writes a letter to the District Land Council saying, “I’ve allocated this acre to Steve,” or whatever. Then the government sends a surveyor out to the site to find out if anyone’s currently using the land, whether it’s culturally significant, if there’s any good reason you shouldn’t have it. If the district government approves, it sends a letter up the chain to the Ministry of Land: “We agree, Steve can have this acre now.” The minister issues a title, and it’s yours.

And that, usually, is when the trouble begins.

At each stage of this process, all the power resides in one individual. At the tribal level it’s the chief. At the district level it’s the surveyor. At the national level it’s the minister. Each of these individuals can either block you from getting the land or, if you get on their good side, give you more than you’re entitled to.

The law only allows the chief to distribute 250 hectares at a time, for example, but he’ll give you more if you get on his good side. The surveyor, if he’s so inclined, can expand the plot of land the chief gave you, or ignore evidence that the land is already being farmed.

After the title has been issued, the Minister of Land is the only person you need to keep happy. Once land is converted from traditional ownership to individual, the chief and the district lose all their power over it. The land is yours to use however you want, and whatever promises you made to the chief and the district before they gave you the land are, legally speaking, null and void.

This system is problematic enough at the individual level, but imagine that the purchaser of the land isn’t a person, but a multinational company.

The way it’s explained to me over and over again is this: First, a company comes in and gets the chief to allocate a big plot of land by promising to feed his tax base, build schools and roads, offer jobs to his friends and neighbors. Then the company pays off the surveyor to give the company more than the chief originally allocated, and to file a report saying the land is vacant, out of use, culturally insignificant.

As soon as the ink on the title is dry, the company starts negotiating with the national government to get a tax holiday so it can avoid paying the taxes that were the condition for the chief allotting the land. The chief complains—where are my schools, my hospitals, my jobs?—but by now the land isn’t under his authority anymore, it’s designated as “state land,” and he has as much a claim to it, statutorily speaking, as Donald Trump does to the White House.

I’m simplifying this, obviously, but this is the basic story that every NGO we meet with in Kitwe tells us. Subsistence farming communities of 200, maybe 2,000 people, many illiterate, all poor, going head-to-head at the negotiating table against multibillion-dollar companies.

Many of these negotiations are about compensation. Zambian law says people who get kicked off their land to make way for a mine or a farm have to be fairly compensated, but neglects to specify a number, a negotiation process, or even the definition of fair. Communities have to work all this out between themselves, the local administration and the company across the negotiation table.

The first sticking point in these negotiations is usually related to the value of the land. Government assessors are in charge of determining what the land is worth, but each ministry has its own method for this, its own formula for paying off communities to leave their land and give up their income. The companies know this, of course, so they shop around, looking for the lowest compensation rate, the stingiest assessors.

Sometimes the companies fast-forward to the payouts. In one recent case, a multinational mining company offered community members 10,000 kwacha each (about $2,000) to move off the land it had just purchased.

“For someone earning 1,000 kwacha a year, that’s a lot,” says Mary Mwane, who works at a land NGO. We’re sitting in her second-floor, bars-on-the-door offices across from a stand selling auto parts. “They don’t see that that’s a one-time payment.”

Mary is the one who has told me all this, the way it works here, the monotony of the promises-made, promises-broken cycle of community relations. She’s visibly upset. She spends a lot of her time in the kinds of communities she’s telling me about, she’s seen men in suits appear from out of nowhere (well, out of the U.S., Europe, or South Africa) commit to build schools and hospitals, then disappear, letter to the District in hand, never to be heard from again.

The companies aren’t required to disclose the size of the land they purchase, the length of the lease, or the procedure they followed to get it. The people living on the land are in the dark until one day a company man knocks on their door with a deed to it and, if they’re lucky, a payout if they agree to leave.

“I have money, I know the procedure, I can’t even get land,” Mary says. So why is it so easy for these companies?

04. THE SKILLS
“There are no environmental lawyers in Zambia.” This is Noah, he works for a regional development agency. He’s agreed to meet at our guesthouse, he arrives wearing a crisp pastel shirt and pants. He looks like the temptation from a Tyler Perry movie.

I am crosslegged in a flea market T-shirt with pink letters shouting MUTE across the front. Noah was 90 minutes late getting here, so an hour ago I changed out of my work clothes. I keep apologizing.

Noah is saying that one of the reasons Zambia is so stagnant, that the World Bank and IMF’s orders to modernize and diversify go unfollowed, is that the country simply doesn’t have any technical capacity to spare.

“The Environmental Management Act says developers are allowed to discharge [pollutants] to a certain threshold,” Noah says. “But the same agency can’t determine whether that level is going to harm the environment, and has no equipment to test whether the emissions are within the law.”

In this economy, having a job—a paycheck, a name tag, a pension—is a rare thing, and having one that is secure and well-paid is practically unheard of. There are a million reasons for this—historical, cultural, economic, take your pick—but the explanation I hear the most is political.

The only actors that have the equipment to measure pollution in groundwater, particles in the air, are the companies themselves, and they unanimously report that they’re within the legal limit.

“This country doesn’t have the skills to meaningfully monitor whether promises are being kept,” Noah says.

One reason for this is simply a matter of money. Later, in Lusaka, a representative of the Ministry of Energy tells me that 30 percent of government jobs are funded but are sitting empty because they can’t find someone with the skills to fill them. Or at least they can’t at the salaries being offered.

Just yesterday, the representative says, she had a high-skilled machinery operator turn down a job because she could only offer him 4,000 kwachas a month, about $800. That’s one-half to one-third what he would make doing the same thing for a company.

Even herself, even on a government middle manager’s salary, she can’t afford a car and had to pay to build her house one bag of concrete at a time.

This problem is everywhere you look. As of 2012, the Ministry of Labor only had 13 inspectors for the whole country. There are so few judges in the courts that people have reportedly waited in pre-trial detention for up to 10 years.

I ask her what she would do if her budget doubled overnight.

“I’d hire more people,” she says. “I’ve got three people working in the geological survey. I need 12.”

Another reason Zambia lacks skills is that some parts of the workforce operate as cartels. Take lawyers. Zambia only has 1,000 of them, and they’re concentrated where the money is: Lusaka (government), Copper Belt (mining) and Livingstone (safari tourists). Some provinces don’t have any lawyers at all. The government operates a kind of legal bookmobile, a team of lawyers that travels around the country offering basic services, but it only comes to each province once every two months. If you miss it this time, you’re out of luck until it returns.

Last year, only six lawyers were admitted to the bar out of 164 who took the exam. The year before that, it was 16 out of 145. Keep in mind, these aren’t people coming in off the streets. These are people who have a law degree.

The Zambia Institute of Advanced Legal Education (ZAILE), which administers the test, blames … well, everything. In a speech this year the board chairman called out“poor academic background of law graduates, lack of sponsorship, lack of student accommodation, lack of adequate facilities, poor lecture delivery, [and] unsatisfactory pupilage.”

Maybe his explanation is right. Maybe the teachers are bad, the students lazy, the accommodations shoddy. But ZIALE, or any professional licensing body really, has no incentive to open its doors more than a crack. More lawyers means more competition, and more competition means lower wages.

Back at the guesthouse, I shake Noah’s hand and he drives away in a plume of dust. At dinner, Helen tells us she has the same problem filling positions.

“They’re lazy and they don’t want to work,” she says. She repeats a South African saying I’ve heard in this part of the world before: “They’re born tired and they live to rest.”

She says this in front of Thomas, who is serving dinner. I look for his reaction, but he doesn’t seem to be listening. He’s looking through us, like his mind is somewhere far away.

05. THE POLITICIANS
The most prominent feature of Lusaka, Zambia’s capital, is the dust. It’s everywhere—frosting the paved roads, burying the unpaved ones, coming off your hands in big brown drops no matter how many times you wash them.

Lusaka sprawls, one story tall, across an area roughly the size of Philadelphia. The restaurants, gyms, shops, NGOs—most of them are converted three- or four-bedroom houses, squat stucco behind broken glass-tipped walls. Window shopping in Lusaka means driving slowly, reading movie poster-size signs on the walls telling you what’s behind them.

By 7 a.m. the city is already awake, stands selling vegetables, vendors walking into traffic with bags of fruit and rolls. Ninety percent of the population is, in NGO-speak “informally employed.” Or, as Thomas put it: “Children selling things for their parents in the streets.” For most Zambians—say everyone and you’re only off by only a few percentage points—this is the job market, these are your options: Farming a small plot of land or selling consumer products a few at a time.

There are the cops that pull you over to ask for 50 kwacha ($10); the schools with slots reserved for paying parents; the hospitals that swear the earliest appointment, the only available medicine, is six months away until you reach into your pocket.

In this economy, having a job—a paycheck, a name tag, a pension—is a rare thing, and having one that is secure and well-paid is practically unheard of. There are a million reasons for this—historical, cultural, economic, take your pick—but the explanation I hear the most is political.

The first problem is how the money comes in. More than 60 percent of Zambia’s government revenue comes from the copper mines. That sounds like a lot, but thanks to the low royalties it’s just a few drips off the waterfall of Zambia’s mineral wealth.

The rest of the revenue is supposed to come from taxes, but looking around, how would that even work? So many people are barely getting by, eating vegetables they grow in their backyards, doing piecework where they can find it. Taxing all this informal activity would be costly in both resources and voter goodwill. In 2012, Zambia collected just $2.3 million in income taxes from its citizens.

You’d think this would be a government priority: Formalize all those fruit stalls and day laborers, hand out tax cards, threaten audits. But the problem with more taxes is that they create more taxpayers. If you make people give away a percentage of their income every month, they’re going to want to know where it goes, what they’re getting in return.

That brings us to the second problem: Zambian politicians. You know how when a football player gets big, he buys his mom a house and hires all his buddies to be his managers and security guards? From what everyone here tells me, the Zambian equivalent of the NFL is national politics.

It goes as high up as you want to follow it. Michael Sata, the president of Zambia, appointed his uncle the finance minister, his nephew the deputy finance minister, his niece the local government minister, and cousins as ambassador to Japan and chief justice.

In a rather bald statement of the status quo, Daniel Munkombwe, a member of parliament who has been in office since independence in 1964, told a journalist earlier this year: “There is nobody who goes into Parliament naked, we go to Parliament because of allowances. … I know people will say Munkombwe has gone into government because he wants to eat, but who does not want to eat?”

The ruling party has recently been accused of paying opposition MPs to switch sides. There’s no actual smoking-gun evidence, but Zambia’s cabinet has ballooned to 20 ministers and 47 deputy ministers, the largest in Africa. With salaries three to four times higher than opposition MPs and each ministerial post bundled with perks like a company car, free fuel, house servants, and mobile phone talk-time, you get the feeling politicians aren’t jumping from opposition into government on moral sentiment alone.

It’s easy to paint all of the problems in Zambia with this brush, to talk about kleptocrats wringing their privilege for as much income, as many perks, as they can squeeze. But even if Zambia was run by a coalition of charitable technocrats and Mormon philanthropists, that wouldn’t solve the most fundamental problem of all: There simply isn’t that much money to go around.

In 2011, Zambia spent a total of $4.3 billion running itself. Stretch that to cover every man, woman, and child, and it amounts to just $325 per person per year. That amount—less than a dollar per person per day—has to cover education, health care, infrastructure, law enforcement, foreign debt … everything.

I think of something Jonathan said the day after I arrived in Zambia, leading me through the market. “Politics here is like a kind of blindness,” he said. “They’re only in it for themselves and their families.”

Before the elections in 2011, he met with opposition MPs who spoke with the same language, the same passion as he did. Right after the election it all stopped.

“I talked to guys that, when they were in the opposition, they would say ‘No more Chinese companies in Zambia!’” he says. “As soon as they won the election, they held a banquet for Chinese investors.”

Like Munkombwe said, whether you’re a company or a politician, who doesn’t want to eat?

06. THE CORRUPTION
On my last working day in Zambia I meet a sustainability representative for a mining company. We meet in an office building in central Lusaka, six stories of glass and steel looking like a radio beacon next to the one-story stucco surrounding it.

Jane has been at the mining company for two years. She worked at several Zambian NGOs before this, and gives me the rundown of who I should meet with and what I should ask them. When she talks sometimes she sounds like the other NGO representatives I’m meeting with, all crusade and outrage (“these communities aren’t in control of their own destinies!”). Other times, she has the flat affect and careful wording of a corporate PR exec on cable news (“stakeholders will be consulted in due course”).

She tells me about her company’s human rights policy, how they train company staff on women’s rights, how they meet with people living near their mining sites. It sounds like a different country than the one the NGOs have been describing, one where people whose sole possession is a backyard farm sit down for civilized negotiations with the multinational company that wants to buy that farm out from under them, one where miners wear hardhats and proofread their wives’ business plans.

I meet a Zambian expat in Zimbabwe. She says every time she returns, there are more cars, more roads, more restaurants, bars, gyms, decent cappuccinos. “It’s all malls,” she says. “Zambians love to go to the goddamn mall.”

I ask her about the problems the NGOs told me about, the paltry compensation, the regulator-shopping, the entire inspection bureaus that could fit in the back seat of a Chevy Malibu .

“The government assesses the value of the land,” she says, in cable-news mode. “They have taken into account the full value of the land.”

“But what do you do in a country with so little government capacity?” I ask, thinking of all those empty civil service positions, the missing lawyers, the outdated equipment.

Now she goes all NGO. “Little government capacity,” she says, is the nicest way to put it. “There are simply no systems for routine government services,” she says. Getting a license, a permit, certificates, approvals to start work, visas for expats to fly down here—nothing is in one place, nothing is fast or easy.

And that’s just the bureaucracy. Then there are the cops that pull you over to ask for 50 kwacha ($10); the schools with slots reserved for paying parents; the hospitals that swear the earliest appointment, the only available medicine, is six months away until you reach into your pocket.

“People are desperate for jobs,” she says. “Everyone is trying to get a job in the formal sector. So they try to bribe our HR staff to hire them.”

But wait, it gets worse. “Sometimes we have to pay for the inspectors to come to our mines,” Jane says.

The conversation goes like this: Jane tells the local certification body that she needs an inspector to sign off for a permit. The local certification body tells her that they would be happy to come out to the site, but they don’t have fuel for their cars, or enough petty cash to pay per diems. Jane offers to pay their costs, but only their costs, and the payments aren’t related to clearing the inspection.

“We ask them for a breakdown, government per diem is this, salary is this, fuel is this,” she says. “It must be well-documented, obviously.”

“Are all the mining companies doing this?” I ask, trying not to show my aghast-ness.

“Sometimes [the inspectors] say, ‘Your friends come here, and they offer us this and this. You guys are so stingy.’” This is a point of pride in the company. We only pay for their costs.

The company has even paid the police to follow up on complaints or to investigate thefts. “They say, ‘We don’t have this in our budget’ or ‘We’ll need you to pay for it,’” Jane says. So the company fixes the police cars, covers their travel expenses, treats them to lunch.

The government, Jane tells me, has been experimenting with new procedures to cut down on petty bribery. Now, when she files permit applications or visa forms, she leaves them at a desk without seeing a clerk. The fewer people you see, goes the logic, the fewer opportunities you’ll have to bribe them.

“If people keep shunting you from office to office, they are asking you for a bribe,” she says, whether there is anyone at the desk or not. “We tell them, ‘The company I work for, we’re not going to pay up.’ But at the end of the day, they know you’re on a short timeline, and they aren’t.”

I’M NOTGOING TO end with some sweeping declaration about why Zambia is poor and the one thing it needs to be richer. I’ve been here for a week; I’m not going to tell Zambia how to run itself, what it needs to fix and in what order. The explanations I heard, they aren’t the whole puzzle, they aren’t even the biggest pieces.

The only thing I’m able to conclude after my trip here is that it’s incredibly difficult for a poor country to go about getting un-poor. Just when you think you’ve got the right narrative, another one comes bursting out of the footnotes. It’s the informality. No, it’s the taxes. No, it’s the mining companies. No, it’s the regulators.

And that’s what makes fixing it so difficult. Does Zambia need better schools? Debt relief? Microfinance? Nicer mining companies? Better laws? Stronger enforcement? Yes. All of them. And all at the same time.

You can’t fix the land issues without tackling the corruption. You can’t fix the corruption without tackling the politics. You can’t fix the politics without addressing the culture. Thomas’ family told him his nephews didn’t need to be in school. From their perspective, that’s not totally irrational. In a country with so few formal jobs and so much competition for getting them, I can see how spending hundreds of hours, thousands of kwachas, on education would seem superfluous. Thomas’ daughter wants to become a lawyer. You could almost forgive Thomas if he told her that the bar exam failure rate is more than 90 percent, so what’s the use?

So honestly, I have nothing to say, no prescriptions, no lessons. I don’t even have a clear narrative for how my own country went from Deadwood to Real Housewives, much less a guidebook for how a poor country would make that transition now. Zambia is trying to make it in a world of cell phones, broadband Internet, Range Rovers, international flights. I don’t know if that makes development easier, or harder.

A week after I leave Lusaka, I meet a Zambian expat in Zimbabwe. She left Lusaka four years ago, and she says every time she returns, there are more cars, more roads, more restaurants, bars, gyms, decent cappuccinos.

I tell her that in Lusaka I saw construction cranes on the horizon in every direction.

“It’s all malls,” she says. “Zambians love to go to the goddamn mall.”

That’s not the only reason for optimism. Inflation is down to seven percent from 20 percent last decade. International investors pledged $750 million last year to build infrastructure. The new draft of the constitution limits presidential powers and confronts the MP-hopping problem. Fundamentally, Zambia is a stable country sitting on top of an El Dorado of fertile land and lucrative minerals. In the long run, things will probably get better.

On my last day in Zambia I go on a daytrip, a safari lodge an hour outside Lusaka. South Africans, Europeans, rich Zambians, all lounging around a pool. The surroundings look like the Africa you see in movies: tall grass, spindly trees, termite mounds, and antelopes.

The guide points out animals, talks about what they eat, how they mate, how long they live. He says all of them are available to hunt, as long as I’m willing to pay. Bagging a warthog will cost $500. An antelope, $1,000. The lions, when they’re too old, are set free for the hunters to have at them. That’ll cost me $10,000.

As I drive back to the lodge at the end of the day, I pass a Jeep going the other direction. Hanging halfway out of the cab is a huge female impala, on its back, blood everywhere, eyes still open. Maybe it’s just the bouncing of the Jeep, but it looks like it’s still kicking.

“Are they going to eat it?” I ask.

“No, it’s a trophy,” he says. “They’re just going to take it away.”

By Peiter Zatko
On September 14, 2013

I was in Germany for Chaos Congress 2009, a hacker conference, and after attending a series of talks I was headed back to my hotel when I spotted Julian Assange. This predated my working as a project manager at DARPA as a hacker-in-residence, if you will. It was also before Wikileaks released the video “Collateral Murder” and hundreds of thousands of diplomatic cables, the Swedish rape allegations, and Julian ending up a virtual prisoner, holed up in Ecuador’s diplomatic mission in London.

Given that I hadn’t seen Julian in more than a decade, I greeted him, and we agreed to grab some dinner and catch up. After reminiscing about the old gang – he and I had been part of the same hacker “milieu,” as Julian would say – I asked why he had skipped out on the hacking scene to form Wikileaks.

Last I heard Julian had left to study mathematics and physics at the University of Melbourne. He was working on a “duress” based crypto-file system, also known as the “rubber hose” file system. The theory was that if your disk was encrypted and someone threatened to beat you with a rubber hose unless you decrypted it, you could fool him by decrypting a secondary innocuous file system without revealing the true content of your still cloaked files. The baddies would be convinced they had what they wanted and let you go while you would keep your secrets. (There’s a problem with this, of course. If the baddies think you have a rubber hose file system, even if you decrypt your drive they may think you’re holding out on them and continue to beat the crap out of you. But I digress.)

Julian told me his graduate work had been funded by a US government grant, specifically NSA and DARPA money, which was supposed to be used for fundamental security research. It was a time when the Bush Administration and Department of Defense were seen to be classifying a great deal of fundamental research and pulling back on university funds. These universities were getting the message that they could no longer work on the research they had been conducting, and what they had already done was classified. In a Joseph Heller-like twist, they weren’t even allowed to know what it was they had already discovered.

According to Julian, the US government cast such a wide net that even general scientific research, whose output had always been published openly, was swept up in America’s secrecy nets. As you can imagine this did not sit well with Julian, because his work had also been funded by one of these fundamental research funding lines and yanked.

So here you have a non-US citizen at a foreign university doing graduate work studies, and the United States government came barreling in and not only snuffed out the funding and killed his studies, it also barred him from knowing what it was he had been funded to research.

It was at that moment, Julian told me, that he decided he would devote himself to exposing organizations that attempted to keep secrets and withhold information in an effort keep the masses ignorant and disadvantaged.

So you see, depending on who you ask, the US government actually helped create of Wikileaks. And the rest, as they say, is history.

[Image courtesy nehavish]

Solar time is a reckoning of the passage of time based on the Sun's position in the sky. The fundamental unit of solar time is the day. Two types of solar time are apparent solar time (sundial time) and mean solar time (clock time).

Fix a tall pole vertically in the ground; at some instant on any sunny day the shadow will point exactly north or south (or disappear, if the Sun is directly overhead). That instant is local apparent noon: 12:00 local apparent time. About 24 hours later the shadow will again point north/south, the Sun seeming to have covered a 360-degree arc around the Earth's axis. When the Sun has covered exactly 15 degrees (1/24 of a circle, both angles being measured in a plane perpendicular to the Earth's axis), local apparent time is 13:00 exactly; after 15 more degrees it will be 14:00 exactly.

The problem is that in September the Sun takes less time (as measured by an accurate clock) to make an apparent revolution than it does in December; 24 "hours" of solar time can be 21 seconds less or 29 seconds more than 24 hours of clock time. As explained in the equation of time article, this is due to the ellipticity of the Earth's orbit and the fact that the Earth's axis is not perpendicular to the plane of its orbit.

So a clock that runs at a constant rate — the same number of pendulum swings in each hour — cannot follow the actual Sun; instead it follows an imaginary "mean Sun" that moves along the celestial equator at a constant rate that matches the real Sun's average rate over the year.[1] This is "mean solar time", which is still not perfectly constant from one century to the next but is close enough for most purposes. Currently a mean solar day is about 86,400.002 SI seconds.[2]

The two kinds of solar time (apparent solar time and mean solar time) are among the three kinds of time reckoning that were employed by astronomers until the 1950s. (The third kind of traditional time reckoning is sidereal time, which is based on the apparent motions of stars other than the Sun.)[3] By the 1950s it had become clear that the Earth's rotation rate was not constant, so astronomers developed ephemeris time, a time scale based on the positions of solar system bodies in their orbits.

Apparent solar time or true solar time is based on the apparent motion of the actual Sun. It is based on the apparent solar day, the interval between two successive returns of the Sun to the local meridian.[4][5] Solar time can be crudely measured by a sundial.

The length of a solar day varies through the year, and the accumulated effect produces seasonal deviations of up to 16 minutes from the mean. The effect has two main causes. First, Earth's orbit is an ellipse, not a circle, so the Earth moves faster when it is nearest the Sun (perihelion) and slower when it is farthest from the Sun (aphelion) (see Kepler's laws of planetary motion). Second, due to Earth's axial tilt (known as the obliquity of the ecliptic), the Sun's annual motion is along a great circle (the ecliptic) that is tilted to Earth's celestial equator. When the Sun crosses the equator at both equinoxes, the Sun's daily shift (relative to the background stars) is at an angle to the equator, so the projection of this shift onto the equator is less than its average for the year; when the Sun is farthest from the equator at both solstices, the Sun's shift in position from one day to the next is parallel to the equator, so the projection onto the equator of this shift is larger than the average for the year (see tropical year). Also, in June and December when the sun is farthest from the celestial equator a given shift along the ecliptic corresponds to a larger shift on the equator. So apparent solar days are shorter in March and September than in June or December.

These lengths will change slightly in a few years and significantly in thousands of years.

Mean solar time is the hour angle of the mean Sun plus 12 hours. Currently (2009) this is realized with the UT1 time scale, constructed mathematically from very long baseline interferometry observations of the diurnal motions of radio sources located in other galaxies, and other observations.[7][8] The duration of daylight varies during the year but the length of a mean solar day is nearly constant, unlike that of an apparent solar day.[9] An apparent solar day can be 20 seconds shorter or 30 seconds longer than a mean solar day.[6][10] Long or short days occur in succession, so the difference builds up until mean time is ahead of apparent time by about 14 minutes near February 6 and behind apparent time by about 16 minutes near November 3. The equation of time is this difference, which is cyclical and does not accumulate from year to year.

Mean time follows the "mean sun", best described by Meeus:

The length of the mean solar day is slowly increasing due to the tidal acceleration of the Moon by the Earth and the corresponding slowing of Earth's rotation by the Moon.

Many methods have been used to simulate mean solar time. The earliest were clepsydras or water clocks, used for almost four millennia from as early as the middle of the 2nd millennium BC until the early 2nd millennium. Before the middle of the 1st millennium BC, the water clocks were only adjusted to agree with the apparent solar day, thus were no better than the shadow cast by a gnomon (a vertical pole), except that they could be used at night.

But it has long been known that the Sun moves eastward relative to the fixed stars along the ecliptic. Since the middle of the first millennium BC the diurnal rotation of the fixed stars has been used to determine mean solar time, against which clocks were compared to determine their error rate. Babylonian astronomers knew of the equation of time and were correcting for it as well as the different rotation rate of the stars, sidereal time, to obtain a mean solar time much more accurate than their water clocks. This ideal mean solar time has been used ever since then to describe the motions of the planets, Moon, and Sun.

Mechanical clocks did not achieve the accuracy of Earth's "star clock" until the beginning of the 20th century. Today's atomic clocks have a much more constant rate than the Earth, but its star clock is still used to determine mean solar time. Since sometime in the late 20th century, Earth's rotation has been defined relative to an ensemble of extra-galactic radio sources and then converted to mean solar time by an adopted ratio. The difference between this calculated mean solar time and Coordinated Universal Time (UTC) determines whether a leap second is needed. (The UTC time scale now runs on SI seconds, and the SI second, when adopted, was already a little shorter than the current value of the second of mean solar time.[12])

For many years, i’ve been convinced that programming needs to move forward and abandon the Algol family of languages that, still today, dampens the field. And that that forward direction has been signalled for decades by (mostly) functional, possibly dynamic languages with an immersive environment. But it wasn’t until recently that i was able to finally put my money where my mouth has been all these years.

A couple of years ago i finally became a co-founder and started working on a company i could call my own (i had been close in the past, but not really there), and was finally in a position to really influence our development decisions at every level. Even at the most sensitive of them all: language choice.

During my previous professional career i had been repeatedly dissapointed by the lack of conviction, when not plain mediocrity, of the technical decision makers in what seemed all kinds of company, no matter how cool, big or small: one would always end up eaten alive by the Java/C++ ogre, with a couple testimonial scripts (or perhaps some unused scheme bindings) to pay lip service to the hipsters around.

Deep down, the people in charge either didn’t think languages make any difference or, worse, bought into the silly “availability of programmers” argument. I’m still surprised anyone would believe such a thing. If i guy came telling me that he wanted to program only in, say, Java because that’s what he knows best and that he doesn’t really feel prepared or interested in learning and using, say, Clojure (or any other language, really), i wouldn’t hire him in a million years, no matter what language my project were using, and no matter how many thousands of candidates like this one i had at my disposal.

Give me programmers willing to learn and eager to use Lisp or Haskell (or Erlang, ML, OCaml, Smalltalk, Factor, Scala...) any day, even if we happen to be using C++, for goodness sake! Those are the ones one needs, scanty as they might be.

Given the sad precedents, when i embarked in my new adventure, i was careful not to try to impose on anyone my heretical ideals: they had to be accepted on their own grounds, not based on my authority. But i was finally lucky enough to meet a team of people with enough intellectual curiosity and good sense (which is, again, actually a pre-condition to be in a startup). People, let me tell you, with no experience whatsoever in languages outside the mainstream, but people that, nonetheless, were good programmers. And when you give a good programmer a good language, good things happen.

Come to think of it, it’s true that, with good programmers, the language doesn’t matter: they’ll choose the best one, or follow the advice of more experienced colleagues, and quickly take advantage of any extra power the novel environment has to offer.

Our experience so far could hardly be a better counterexample against algol naysayers.

Our backend is 99.4% coded in Clojure, and 66% of the team had never programmed seriously in any lisp, let alone Haskell or Prolog (heck, not even i (the remaining 33%) had actually tried anything non-mainstream for real in a big project!) Maybe some Ruby, and lots and lots of Java and C and C++. But they accepted the challenge after reading around and learning the basics, and 3 months later you couldn’t take Clojure from their prying hands. More importantly, they had fun discovering they could also be Dr Jekyll.

Of course, lots of effort is a must, and someone with a bit of experience to guide the newbies is probably necessary. In particular, extensive code reviews were of the essence in our case, and i had never read and criticised this many lines of code in a similar amount of time. But know what? I prefer to “waste” time in code reviews to employ at least as much writing factories of factories of factories and similar boilerplate (configured of course using XML) and chasing bugs in the resulting soup. Not to mention that, again, you need code reviews no matter the language you use: the trick of having a code base so ugly that nobody would review it doesn’t fly.

So yes, it’s more difficult to find hackers, but you need to find them anyway. And yes, it may require more serious software engineering, but, again, you need it anyway. So why shouldn’t we use the best tools at our disposal? I can finally tell you: been there, done it, and it does work.

Windows 8.1 is now released to manufacturing and those with MSDN or Technet Subs can download it now.

I have the RTM version now set up on home and work machines and have been running the Preview versions on both my Surface RT and Surface Pro. Windows 8.1 has some glaring errors.

SkyDrive integration now built in, removes features compared to the old Desktop client

You no longer have to install the SkyDrive app separately as it is now built into the OS. Windows 8.1 makes a concerted effort to force you to use SkyDrive.

The above is a screenshot of a screen displayed during the upgrade process. Can you see the error?

This is nonsense – if you delete a file from your local machine, it will be deleted from the cloud. This is NOT a backup. Live mirroring is NOT a backup. If a sync goes wrong – your file disappears completely. There is no backup.

On the left is Windows 8’s SkyDrive integration after installing the SkyDrive Desktop app. On the right is Windows 8.1’s built-in SkyDrive integration. This is now a system-level folder and doesn’t even have the syncing icons available. This folder is now virtualized and you aren’t guaranteed that the actual file will be present. Opening the file will sometimes download it from SkyDrive.

The “Windows 7 File Recovery” backup system has been removed in 8.1

Windows 8.0 actually contained two backup systems, the new “File History” and the awesome old backup system from Windows 7, threateningly renamed “Windows 7 File Recovery” as a warning that this will be removed. Lets compare the systems:

File History is Microsoft’s attempt to copy OS X’s Time Machine, except Time Machine actually backs up all your files and lets your restore the entire OS partition, just like Windows 7 did! At least you had the choice in 8.0 to use the old system. In 8.1, Windows 7 File Recovery has been removed completely, you can’t even restore your old backups!

Edit: Turns out the system image part of Windows 7 File Recovery survived, but is only accessible through Powershell.

It gets worse. File History is even more useless in 8.1.

“File History” no longer backs up your SkyDrive folder in 8.1

Microsoft really don’t want you to have a local backup of your SkyDrive files. Take a look at this:

Above: Windows 8 File History

Above: Windows 8.1 File History

No problem you think? Just add your SkyDrive folder to a Library and it should back up? Nope – all the SkyDrive files are ignored completely, even if you manually add them to a Library.

The response from Microsoft on this is beyond tragic (from here):

Your files “are protected by the cloud in case user lose/damage their device”. What about protection from user error or viruses or badly written programs? If your files get corrupted the corrupted files will sync to the cloud and then sync to all your other devices.

Conclusions

It appears that Microsoft are desperate to push SkyDrive, even at the expense of the computing safety of their customers – customers you’d hope were being educated about safe computing. Now I am on 8.1 I am personally stuck with no built-in backup system. My experience with File History has been awful – it appears to even ignore an additional Library I’ve created to include non-library files. I literally cannot get it to back up files on my computer, it is useless. I am going to have to go with a third-party backup system like CrashPlan now.

Windows 8.1 was Microsoft’s chance to undo the wrongs of Windows 8. Users are now faced with the prospect of upgrading and being faced with no backup solution, or even worse their existing backups just stopping working with no warning.

Sort it out Microsoft.

Edit: Some excellent discussion on this over at Hacker News: https://news.ycombinator.com/item?id=6388431