Reflections on Engineering
Very early in my career, I was lucky enough to stumble on Web development before it really became a “thing” and landed an apprenticeship (and later job) in a new, small experimental arm of an established consulting firm which primarily worked on embedded systems. I was employee #2 of this new effort and we worked in an attic loft. Nearly all of my experience before this was from books and pure experimentation/willpower to accomplish something. I worked there for about a year, mostly by myself churning out code in somewhat typical fashion: Someone said “make it look like this picture” or “this stuff needs to come from a database” and that was pretty much it. I worked too hard, I was usually scared to death I'd never be able to figure out that one new technology that each project introduced me to and quite often it was just struggle to finish and make something work.
I was learning a lot about APIs and the languages I was working with, but very little about how to do my job. If this sounds confusing to you, you're not alone: I've been writing software professionally for about 20 years and I've worked on a lot of teams with a wide array of people and initially nearly all of them hold the position that that is my whole job. So, let me explain the counterpoint and how I came to hold this idea about what my job really is.
My First Lesson...
About a year into my first job, an opportunity came up for a project that the boss didn't want to pass up: A former client with a startup who had gotten into some trouble with a deadline had heard we were doing this sort of thing now. Success would likely mean more work — but it was considerably over our heads. It dealt with some ideas for which there were only two books we could find and it seemed at the time like pretty high-end nerd stuff. We'd be working with the author of those books directly. The deadlines looked clearly impossible even if it were something we'd understood well. Still, the boss signed the contract on faith and almost immediately we lost the only other employee we had — it was just me.
So the boss negotiated with an old partner who had left the company to come back and help out. His name was George and, actually, he was the one that recommended me for the position in the first place. He was friendly and quiet — a lanky guy with beard and glasses, a seasoned engineer who was great with C but he knew nothing about the Web. He came back on one condition: He was running the team — no questions or pressure or status inquiries from the boss. It seemed strange to me coming from this quiet and very amiable guy.
On the first day, we had a meeting. I brought a thick printout of 4 .asp pages — each tens of thousands of lines of vbscript that represented 4 pages in their application that they had nearly completed. They said “use these as a guide and do these other 6 pages in the next 2 months — we'll do the rest”. They had a virtual army of people banging away on keyboards because in 2 months they had to ship something like 40 “pages” (their application) and in the past 3 months they had completed just the 4.
“Let’s go over these and I can explain what I've already figured out…” I said.
“No, tell me how a Web page works” he replied. I gave some quick explanation of HTML documents and elements and tried to move on.
“No, but tell me how it works.”
After much back and forth for a couple of hours I said “ok, are you ready to get started?” He cocked his head to the side, “What do you mean? We are started.”
Literally all day long we drew diagrams on the board and discussed urls, query strings, HTTP methods, their relation to pages and state and HTML elements, frames, JavaScript, .asp and on and on.
That night I didn’t sleep. I pondered over those printouts, I made notes. I edited code and tested my understanding of it (usually to ill effect). I did math involving the number of people typing, the lines of code, the work cut out for us and how much it cost us to have lost an entire day talking.
The next day I felt a little better: All I needed was some progress then surely I’d settle down. But that day too, we talked — we talked about OLAP and SQL and COM and how you'd use them with .asp and IIS and HTTP. How the server was communicating and what we painting on the screen. Where was the state and so on. Every so often the boss would peek in and look worried and George would quickly dismiss him.
On day 4 I had something of a meltdown: “This is killing my career,” I thought. I confronted him “George, I don't know what’s going on here. Maybe you can afford to do this, but I have to pay my rent — I work here day in and day out. That guy pays my salary and this contract will pay us both and give us more work. We've nothing to show and we're closing in on a significant fraction of the time we have which was already too short. By the end of next week, we have to have one of these done. The boss wants to see progress. The client wants to see progress. My job is to write this code, not to help you feel like you're learning or have interesting conversations. I don't know what happened between the two of you, but you need to get past it — I can't let some power trip or revenge take me down like this.”
George was quiet.
He took a deep breath through his nose and looked thoughtful about my comments. He reached out to the table and slowly opened a box of chocolate covered cherries that was sitting there, unwrapped it and popped it into his mouth and shook his head. He didn't look angry. He didn't look frustrated or guilty. Instead, he looked more the way a grandfather might look just before he passed down some sage wisdom — just trying to figure out how to explain. I’ll try to summarize some of the wisest words that ultimately came...
Any monkey can write code, it doesn't always turn out well. Why do you think I am here? I’m not the only engineer in town — you have others right here. Your boss is a skilled engineer and just next door you could get two guys at half the price. I'll tell you why: I'm here because this project is in trouble. Coding as fast as you can won’t work, and your boss knows that I won’t do that.
Imagine that we did. Even if we didn't encounter problem after problem with code we didn't fully understand — working 15 hours a day odds are still about 99% that we'd fail. Who would that serve? We'd be battered and broken — not us. Would it make your boss happy that you came up short? Not likely. Would it make his client happy that he failed to ship? Would you get follow-on work? Not a chance.
That’s our job Brian — successfully shipping something that actually meets their needs. It involves programming — hopefully good programming — but the programming and the lines of code isn't what the client wants, it’s the end result. They have to ship. This is make or break for them. You think I'm somehow not taking it seriously but I am: I want them to succeed, that’s all I want. That’s what your boss understands, and why he had the faith to bring me on. So I'm asking you to trust me — we're going to get this done and give them what they really want — the thing they really need to be done. Trust me, the rest will take care of itself.
Over the course of the next 2 weeks it was slow — we completed one page. It was a tiny fraction of the amount of code it had taken them. It seemed like a minor win, but I was still full of dread: We were behind. It took a week and a half to complete the next one, a lot of that was readjusting some earlier assumptions that turned out to be wrong, but the code actually got smaller. The last 4 fell like dominoes in just a few days and we were done — early. Each one was really easy to craft and hardly took any code at all.
When we synced up with the main team they confessed that they were way behind. They were impressed — despite having all of those people they'd completed only 10 and they were more than a little worried about their quality. Could we do a few more? In the end, in fact, the lion’s share of the total work was done by our team (mostly George too because as it turned out, I had planned a vacation thinking we were done) in an amazingly short time.
Everyone was happy. I learned an incredible amount of engineering which has continued to help me throughout the years. George was a great teacher. But the real value I took away was much bigger and more abstract than this.
The end goal is to help the client be successful.
George wasn't religiously following engineering practices because some study had shown that over time it would lower maintenance cost or something abstract. Of course, he employed good principles which helped with that, but — he wanted to ship. That’s what everyone wanted. He was simply using whatever powers he had at his disposal to solve the right problems rather than the wrong ones.
I, my boss, and even the client mistakenly understood the problem to be “how do we measure and knock down units of work to do variants of what someone clumsily hacked together before they really understood the problem” . George was able to see the greater issue. Originally, their product shipped on time with two completely different approaches inside — that’s not great engineering, in fact, it’s terrible. George never tried to tell them they should do otherwise — he simply showed them a better way, he delivered and optimized everyone’s happiness. If they had any sense, they went back and learned the right lessons.
On my own...
Over the years I've frequently been in situations where I've had to apply these lessons. It’s not always easy and sometimes the hardest part is the fact that under a lot of pressure, no one can see the forest for the trees. No one wants to hear you — you can frequently be misunderstood as shooting for unaffordable perfection or just “difficult” even if that is pretty clearly not the case. But let me share just one of many stories about how this matters in a very real way — and why it’s worth caring and having the integrity to focus on the non-programming bits.
Several years ago I was contracted by a large company for 6 months to complete a simple task: They had a number of systems which were written in Java or .NET. After years of discussion and planning they had timetables and budgets and a directive to convert all of their existing .NET applications to Java. I was hired to do one. They practiced a waterfall methodology and had already spent tons in codifying the requirements and analysis of the existing system and so on. More or less a straight port. They wanted me to potentially suggest a few minor UI tweaks since I was pretty good with JavaScript and CSS, but otherwise straightforward. I spent my first two weeks reading requirements, looking at code, trying to get set up and run through the existing system and so on. At the end of all of this, I had a nagging problem: I still had no idea what this system was actually for and I couldn't even get it to do simple things without crashing — even the one in production.
At beginning of the third week I was really torn. I just didn't feel like I understood the problem. I could very easily “port” the NET code to Java more or less class for class, but how could I be sure I was doing it right — how could I test it? How could I suggest UI enhancements if I didn't understand what I was enhancing? I honestly couldn't afford to lose this job by pissing someone off, I'd just bought a house and my car died — things were going badly and I was stretched far beyond my limits. Part of me wanted to come to work, do the conversion and take the paycheck and hope. The other part of me won that day though as I remembered George’s words and I walked into my boss’ office and asked him for a moment of his time. I explained all this and asked if perhaps he could point me to some actual users of the system who could just let me observe how they used it for an hour or so so that I could actually understand the job I was trying to accomplish — not the programming, the actual thing the program was supposed to do: The intent. After a while, to his credit, he agreed and he sent a few of us, not just me.
Afterward, we met again to compare notes and, without fail the experience was universal: They don’t use it.
Wait...What?
Why?
It’s pretty simple actually: It doesn't work for what it is designed for — and it wasn't designed to do what they actually need. So instead, they do all these other crazy things with other systems… They each do it their own way — but they're all just solving the same problems on their own. What’s more, it’s not that involved — just clumsy.
So, in the end — the budget to convert a useless system from one language to another was applied toward the valuable task of just writing the (much, much simpler) system that they actually needed in the ‘right’ language in the first place. I'd helped them not just to be successful but to understand what success meant in the first place: The real problem wasn't just “we have an application written in a language we don't want to maintain anymore” it was “we have a useless application and we never actually solved the problems that needed solving in the first place”.
In short...
Many of us, and those we work with would frequently like to imagine that our job is just programming. It’s easy to forget that, and while yes, you have that rare skill — your real job is in making the project successful. It’s that same quality of solving problems that occasionally gives you an insight that others can’t have. What someone is asking you for is frequently abstract, it often lacks an understanding of the medium and that means various choices incur significantly more cost than others. It’s quite possible (at least in my experience) that with just a little cooperation and effort you can adjust expectations, correct problems with deadlines and budgets and just generally increase the overall happiness of everyone. It doesn't always work — sometimes people will misunderstand your position or even be frustrated by it. But, after ~20 years of doing this, I can honestly say that in the end my results have been overwhelmingly positive. In the end, people tend to come around and frequently those who most misunderstood initially have later come to be some of the best partners.
I think my friend Mark Nottingham summed it up well recently on Twitter. I’m not sure he meant precisely this, maybe I just see this as a nice summary because it’s on my mind, but I like it regardless…
Eventually, you figure out that the code itself doesn't matter for shit; its effect on the world around you is what matters.— Mark Nottingham (@mnot) June 20, 2015