John D. Cook has a post on one of my favorite management topics, when to delegate.

My thoughts on delegation are probably fairly radical. The first principle is that if a project strikes me as interesting to work on, I must delegate it immediately. Managers almost never have the bandwidth to tackle interesting technical problems, it’s the nature of the beast. When I delegate projects, I can remain engaged to a degree that allows me to learn and to hopefully add some value, without stalling progress because I’m too busy dealing with the other million distractions that are part of manager life. I tend to stick with working on the tasks that are too boring or repetitive to bother delegating to other people – they’re generally about the right size to get out of the way when I’m between other things.

The second principle is, delegate everything, especially if you’re in a new role. When you take on a new role, there is no way to estimate how much work is going to be coming your way, or to list what all the demands on your time will be. All of the stuff you were doing is either going to keep you from succeeding in your new role, or just isn’t going to get done, unless you can delegate it to someone else. Of course, in reality you’ll never be able to delegate everything, but proceeding as though you can will hopefully keep you mindful of the fact that you should always be looking for opportunities to delegate.

The point I’m really getting at is that most people are far too reluctant to delegate, and far too selfish when it comes to delegating. Usually when you’re in a position to delegate, you are also at least partially responsible for the professional well-being of the people you’re delegating things to. The things you delegate should hurt a little to give away.

Gloria Steinem was asked by Yahoo whether she thought Miley Cyrus’ recent risqué performance and video were setting the feminist movement back, and she replied, said, “I don’t think so.” She said:

I wish we didn’t have to be nude to be noticed … But given the game as it exists, women make decisions. For instance, the Miss America contest is in all of its states … the single greatest source of scholarship money for women in the United States. If a contest based only on appearance was the single greatest source of scholarship money for men, we would be saying, “This is why China wins.” You know? It’s ridiculous. But that’s the way the culture is. I think that we need to change the culture, not blame the people that are playing the only game that exists.

Another, complementary game that exists for young men is the sports game, addressed in this week’s issue of The Atlantic, in the cover story How Sports are ruining High School.

(Via HuffPo)

Last year I started working in the world of Big Data, and at the time, I didn’t know that “data science” and “data engineering” were separate things. At some point, I looked at what my team is working on and realized that the distinction between the two is important, and that the team is firmly entrenched in the data engineering camp.

Data scientists get all the glory and attention, but without data engineering, there’s no way for data scientists to practice real science. I’ll talk more about this in another post. In this first post, I want to talk about the four basic layers of the data engineering stack. These apply whether you’re working to enable people to collect analytic data for a Web-based business, or enable people to collect and analyze rainfall data. The layers are:

1. Instrumentation
2. Data crunching
3. Data warehousing
4. End-user tools

Let’s look at an example from Web analytics, because that’s what I understand the best. A tool like Google Analytics provides all four layers. End users only have access to two of them. When you add the Google Analytics JavaScript to your Web site, you’re setting up the instrumentation. Google crunches the data they collect, and they warehouse it for you. You can then view reports using the Web interface. Google Analytics is a great general purpose tool, but is limited by the fact that you have little control over layers one and four and no control at all over layers two and three.

At Etsy, we have our own custom instrumentation, our own Hadoop jobs to crunch the logs the instruments write to, our own data warehouse, and, for the most part, end-user tools for exploring that data that we wrote ourselves.

All of our projects involve at least one layer of the stack. For example, we worked with our mobile team to add instrumentation to our native iOS and Android apps, and then we made changes to our Hadoop jobs to make sure that the new incoming data was handled correctly. The new mobile data also has implications for our end-user tools.

Data quality is a huge part of data engineering. It’s possible to lose data at every layer of the stack. If your instrumentation is built using JavaScript, you lose data from browsers that don’t have it enabled. Your instruments usually log through calls to some kind endpoint. If that endpoint is down or the connection is unreliable, you lose data. If people close the browser window before the instruments load, you lose data. If your data crunching layer can’t properly process some of the data from the instruments, it’s lost. Data can be lost between the data crunching layer and the data warehouse layer, and of course bugs in your end-user tools can give the appearance of data loss as well. Understanding the potential sources of loss and figuring out how to measure that loss is a huge part of our work.

In terms of skills and daily work, data engineering is not much different than other areas of software development. There are cases where having a background in math or quantitative analysis can be hugely helpful, but many of the problems are straightforward programming or operations problems. The big problems tend to be scaling each of the layers of the stack (especially the first three) to accommodate the volume of data being collected, and doing the hardcore debugging and analysis required to manage data loss effectively.

That’s a quick description of what life in data engineering is like. I am planning on writing a lot more about this topic. If you have questions, please comment.