Talk to someone a few years in to working in one company in one specific industry, any industry. Ask them to think for a moment about whether coverage of their focus area is accurate, particularly in the easily accessible places. The answer you’ll probably get is a resounding NO.

And yet that’s what we all rely on to keep up on the professional world outside the narrow area of our everyday focus and that’s what you probably rely on in figuring out early career choices. What’s more is that even within a sector, say information technology, the heuristics people develop can be an inaccurate representation of reality.

So why is that? You could blame the internets, the sensationalist bloggers, the stretched journalists, or any number of others. But its likely deeper than that. In areas as disparate as politics and investing and over eras preceding the internet, we see a phenomena called various names: groupthink, herd mentality, echo chamber, and so on. This phenomena seems to result from the combination of two effects.

First, as described by Daniel Kahneman in Thinking, Fast and Slow, we have two modes of thinking which he calls System 1 (fast, emotional, intuitive) and System 2. (slow, logical, deliberate). Interestingly, even when we seek to use System 2 to analyze a problem, we create a representation of the problem in our mind quickly using System 1. And we tend to give undue attention to certain features.

Second, this System 1 lens makes us susceptible to the bandwagon effect, a formal term for the observation of “the probability of any individual adopting [a belief] increasing with the proportion who have already done so.” This applies more obviously to fashions and fads, but the first effect above makes it relevant even in this analysis context.

And this is one form of cognitive bias. Many others likely play a role. Confirmation bias, for example, gets us actively seeking information that validates our view than information that invalidates it.

Now imagine this process playing out in the minds of each person, those who read, discuss, and write about professional trends and make implicit choices about what to focus on and what not to.

As one recent example, Dan Primack’s piece Why we’re paying attention to the wrong deal summarizes the relative coverage of the acquisition of Tumblr by Yahoo and of a revenue-generating mid-sized drug maker:

Price being paid for Warner Chilcott: $8.8 billion
Price being paid for Tumblr: $1.1 billion

# of Warner Chilcott employees: 2,700
# of Tumblr employees: 175

Warner Chilcott 2012 revenue: $2.5 billion
Tumblr 2012 revenue: $13 million

# of online Actavis/Warner Chilcott stories today: 192
# of online Yahoo/Tumblr stories today: 1,059.

So back to our original question. What industry and region tops the list for producing the most companies that achieve $100m in revenue? In Companies That Matter, Paul Kedrosky finds that of the hundreds of thousands of companies founded in the US each year, about 125 to 250 companies in each cohort achieve $100 million in revenues in a reasonable timeframe. While information technology is important, the largest contributors are industrials and consumer discretionary. The most productive region is the U.S. southeast.

In the next post, I’ll take a closer look at the many “hidden” industries.

Please sound off. How do you get reliable information about industries and trends to inform your career choices? What data do you wish you was available? Fire away in the comments or in email.

Sign-up for updates by email.

The data in the above comes from Energy Information Administration. Many others have explained the drivers of this dramatic change. In order of rough importance, they include:

• Historically low natural gas prices, paired with substantial natural gas generation capacity built when competition was introduced to the electricity market in the late 1990s
• An aging coal generation fleet, facing modern health and environmental standards enacted by the Environmental Protection Agency
• State mandates in 29 states for a growing share of electricity from wind, solar, hydro, geothermal, biomass and similar sources

In his inaugural address, the President said, “The path towards sustainable energy sources will be long and sometimes difficult. But America cannot resist this transition; we must lead it.” With it, the President was signaling a significant push on energy innovation in the second term and we’re likely to hear about it in the State of the Union tonight. I explore what a broader vision for the Department of Energy (DOE) could look like, sharing observations from working in DC, with DOE from outside DC, and conversations with many friends who have worked at DOE at some point. It looks like Rob Day came to some similar conclusions in November that I’ll riff off; I recommend reading his whole post.

Focused Mission

Focus, Focus, Focus. It’s something you hear a lot about DOE. Just two recent examples are Secretary Chu’s departing letter calling DOE the Department of Nuclear Security and the Department of Science and Ray Rothrock highlighting this at the Energy Innovation 2013 forum on Panel 2.

From its ad hoc origins in the Carter administration much like the Department of Homeland Security in the Bush administration, the DOE has lacked a central focus. In the President’s second term, the DOE and its Secretary needs the ability to focus on energy innovation, and not on nuclear weapons security, weapons environmental cleanup, particle physics, or running government-owned utility businesses all of which take up time and attention.

A focused mission on energy innovation could include the following components:

• Continuing to invest in disruptive technology innovation, as recommended by everyone from the National Academy of Sciences (NAS) to the American Energy Innovation Council (AEIC) to the National Petroleum Council (NPC)
• Enabling market innovation by creating larger and better functioning markets for emerging technology, free of regulatory friction
• Beginning a dialogue about long-term institutional innovation around energy infrastructure
• Being an ambassador for energy innovation and driver of an inclusive national dialogue, in many ways the most obvious and yet most challenging role

While the first is clear, the others require some explanation. “Market innovation” has recently gotten more attention in a new wave of successful energy startups, from enterprise energy management software-as-a-service to rooftop solar-as-a-service to LED lighting-as-a-service to most recently mobility-as-a-service. What all of these have in common is fundamental innovation on the customer side, e.g. value proposition, channel, relationship, rather on the technology side, e.g. with the concomitant long-lead-time research and development involved. While market innovation driven companies have the potential to scale rapidly, their growth is often hindered by various regulatory frictions and market fragmentation. Enabling larger and better functioning markets is really about shepherding the removal of these. The following is an illustrative list: rules hindering 3rd party ownership of distributed generation; rules hindering innovative “sharing economy” businesses that increase resource efficiency indirectly; and lack of openness and standardization across existing energy efficiency resource standards (EERS) and renewable portfolio standards (RPS) in the states.

The next on my list is long-term institutional innovation around energy infrastructure, where a modest goal would be to begin a dialogue to address some big hairy long-term problems. This one has 3 parts: power market structure, the role of public and private sector in infrastructure, and the regulatory complexity of building anything. The United States itself in a limbo with a partially restructured power market, where market participants fear the whims of 50 state regulators and legislators when putting capital at risk and the public sector no longer finances and constructs 60- and 100-year assets. At some point, we will have to move in one direction or the other to enable greater capital formation and innovation and DOE could lead a renewed dialogue around this. Actually, the challenge goes beyond the power market to the role of private and public sectors in infrastructure more broadly, with the United States lagging in innovative models like PPP which have been used elsewhere. Finally, the regulatory complexity of building infrastructure have grown overtime, with up to ten agencies just at the Federal level involved in review. (Ironically, we may have to wrangle the regulatory-environmental complex to enable responding to climate change in a post-denial world).

Finally, being an ambassador is a critical role. The country can be inspired around energy innovation in a way that it hasn’t been yet. As one example of what might be possible, ARPA-E’s budget request for 2013 is $350m. Private philanthropic giving in the US to causes other than religion, education, health, and other human services, for example the environment and arts, is over $40b. A public-private partnership could be powerful. Moreover, it has the potential to be inclusive. And there are several urgent problems to be addressed on which there can be more agreement, from security against weather events and cyber-threats to more stable and affordable prices and to local air quality and health. In many cases, reduced greenhouse gases are side-benefits that we get for free. Energy innovation and climate change are not synonymous.

Structure

One key step would be to move major responsibilities not directly connected to energy innovation to more appropriate agencies and integrate major responsibilities from other agencies directly connect to energy innovation into the DOE. The following is an illustrative quick hits list, ignoring complexities and nuances.

Hand-off:

• National Nuclear Security Administration (NNSA) reports to the Secretary of Defense.
• DOE Environmental Management (EM) program which cleans up old weapons site reports either to the Secretary of Defense as well or to the EPA Administrator.
• The Power Marketing Authorities are basically Federally-owned utilities and could be managed either cooperatively at the state and local level or by the private sector.
• Office of Science programs and labs report to the head of an expanded National Science Foundation.

Integrate:

• Energy credit programs at Treasury such as the Investment Tax Credit and similar credit programs at HUD and elsewhere report to Secretary of Energy.
• Fuel economy standards for vehicles report to the Secretary of Energy, in addition to DOE’s existing program on performance standards for appliances.

Remain as is:

• ARPA-E
• Energy programs
• Credit programs
• Product standards
• FERC, with an improved dotted line relationship

Team & Culture

The focused above requires a tremendous amount of stakeholder leadership towards specific outcomes and a depth of experience about market realities. The new DOE would need a team that’s a better fit. Its a commonly held view in the business that DOE’s team and culture have stagnated. In part, this is outside DOE’s control as a result of an antiquated personnel policy across the Federal government. The Administration could consider trying a new personnel system more in line with current private sector practice. But the Bush Administration’s experience with trying something new at the Department of Homeland Security should create pause.

Honestly, I’m not sure if there’s a solution here. There have been various commissions that have tried to tackle this over the years (e.g. the 2009 Volcker Report) but the stalemate continues. While this one is third on the list, without a plan for addressing it, it will be challenging to take on the previous two in a meaningful way.

Making it Happen

None of these are easy and each would require collaborating with Congress, most significantly to provide the President the reorganizing authority he needs. In fact, I think this is unlikely to happen as much for institutional reasons as for partisan reasons. By institutional reasons, I mean, for example, legislators who sit on certain committees and want to retain jurisdiction over certain programs and an Administration who could accomplish more near-term with one or two specific initiatives rather than taking on the thankless job of spending political capital to set up a department for long-term success.

So that’s my list. How do you think DOE could be more effective in a second Obama term?

Everyday we throw around words like sector, industry, space, and so to describe some category of companies or products or market segments that we have in mind. Most of the time, precise definitions don’t matter because people get what we’re trying to say. But in certain contexts, these categories become important—for example, when they help talent define and signal themselves as “enterprise” or “consumer” or when they help investors define and signal focus areas in their portfolio.

For lack of a better term, I’ll call these special categories “market categories.”

Defining Market Category

So what’s a market category anyways and why does it matter? Here are a couple of ways I think about market categories, one more intuitive within the tech startup world and another formal in the public markets.

A market category in the tech startup world is a grouping of companies that people use as shorthand for a shared set of qualities either on the customer side or the product side. On the customer side, these are segments & value propositions and a shared business talent pool. On the product side, these are the underlying technologies & processes and a shared engineering talent pool.

Market categories defined by the customer side include global ones like enterprise and consumer or vertical ones like ed-tech, health-tech, and fin-tech. Market categories defined by the product side include things like analytics, collaboration, marketing automation, and so on.

Companies within a market category share more with each other than with those outside the category. As a result, within a market category there are common patterns that arise around business models, risk profiles, and exposure to macro trends.

In the public markets, there are more formalized categories around similar principles. The most commonly used in the financial investment and advisory space are the Industry Classification Benchmark, the Global Industry Classification Standard, the Thomson Reuters Business Classification, and the Morningstar Global Equity Classification Structure.

Each of these is a market-based classification—rather than a production activity based one like NAICS used by the Federal government. To quote a Cornell study comparing GICS with NAICS and SIC: “Our results show that GICS classifications are significantly better at explaining stock return co-movements, as well as cross-sectional variations in valuation-multiples, forecasted and realized growth rates, R&D expenditures, and various key financial ratios.”

Cleantech

Cleantech as a whole shares neither customer side qualities nor product side qualities. In general, customers do not usually seek out “clean” as the primarily value proposition nor is “clean” usually a defining quality about the underlying technologies and processes. For example, enterprise energy management companies like Vigilent or FirstFuel or C3 Energy probably have more in common with enterprise software companies like Salesforce or Workday than they do with say FloDesign (wind turbines) or Project Frog (prefab construction). Tesla has more in common with luxury automotive and manufacturing, than with any of the above. The exception here might be a narrower eco-tech category that is primarily about products for eco-conscious consumers or for sustainability / EHS compliance in the enterprise.

It’s possible these may be obvious examples to experienced entrepreneurs and investors. But it doesn’t seem to be reflected in how the industry is covered in the press—traditional or startup. And based on many recent conversations, it doesn’t come across to new talent and entrepreneurs exploring the space. As more great folks driven by mission seek to make a difference, it’s important for them to be well-informed to make strategic career and startup and investment choices. Being clear about these things helps make sure one understands the implications of working there and the expertise to look for when gauging the likelihood of success.

Actually, looking at the same companies and problems with fresh eyes for actual market categories may suggest broader opportunities. There are likely other opportunities in software around “buildings-tech” or in industrials around lighting or in consumer goods & services around home appliances and transportation services.

So if cleantech is not a market category, what might it be? Cleantech is still a valuable grouping in the same way as Socially Responsible and many others. It groups companies whose products tread lightly on the planet and which cut across market categories—ranging from consumer goods & services to industrials to energy and basic materials.

In fact, Advanced Energy Economy, which I helped build over the last couple of years is also founded on a different and valuable grouping around “advanced energy.” In founding AEE, we had a couple of significant observations. The value chains around energy services from electricity to transportation were regulated at multiple levels, often by similar policy winds and underlying public sentiments. They had a long history of lack of openness to and investment in innovation to improve cost, reliability / security, and environmental performance. And these shared interests were the basis for a business relationship and industry consortium. These are reflected in AEE’s mission to “influence public policy, foster advanced energy innovation and business growth, and provide a unified voice for a strong U.S. advanced energy industry.”

Bottom line

Cleantech may be an important grouping of companies but it is not a market category as commonly used either in the startup world or in the public markets. Talent, entrepreneurs, and investors should all be clear about this when exploring cleantech and when defining themselves as cleantech.

Do you agree, disagree, or think it doesn’t matter? Fire away in the comments!

The Internet provides the perfect medium to aggregate the long tail of fragmented and illiquid markets. This dynamic has made the opportunity for creating online marketplaces so compelling.

The post includes an update to an earlier popular graphic by his colleague Andrew Parker that shows emergence and growth of (and need for) a range of focused marketplaces from what used to be aggregated in Craigslist (or managed by various intermediaries):

Summarizing a similar thesis, Chi-Hua Chien of Kleiner Perkins said in a TechCrunch NYC Disrupt interview:

We’re now entering an era around the democratization of commerce

The interview also puts this in the context of the last two decades of tech innovation:

The mid- to late-90′s saw the democratization of information — companies such as Google made data available to everyone, no matter where or who they were. After that came the democratization of distribution, with services such as Twitter and Facebook allowing anyone to broadcast their content and potentially attract an audience. The democratization of computing has occurred as well, with billions of people in the world now having access to computers because of the availability of low-cost mobile devices…

The past, he said, has been about “mass aggregation,” with companies such as Safeway and Wal-Mart rising to the top of the commerce space by simply being the best at aggregating a suite of products into one space. These big companies also built up their own brand names to make shoppers feel secure in buying things from them. Today, though, we are starting to “see an unwinding of aggregation of commerce as technology starts to disrupt” the industry, Chien said.

At TechCrunch Disrupt, Jim Goetz of Sequoia said:

“It’s shocking we don’t see more engineers and entrepreneurs interested in enterprise” the Sequoia Capital partner said earlier this week at the TechCrunch Disrupt conference in San Francisco.

Twice as many enterprise startups have become billion-dollar companies compared to consumer startups, he says.

“At Sequoia, upwards of a hundred entrepreneurs a week present and if we’re lucky, maybe a dozen of them are focusing on the enterprise,” he told Disrupt attendees. ”In the last 10 years, there have been 56 IPOs in the enterprise space that have gotten north of a billion [dollars in market capitalization] and just 23 in consumer.” …

Goetz said the enterprise IPO market will stay hot for at least a couple more years.

Enterprise remains an “enormous opportunity” because companies are spending about $500 billion a year with legacy enterprise companies and those budgets are ripe for the plucking.

“We believe most will be disrupted in the next decade,” he says…

“The business model, thanks to Marc Benioff, is now a weapon for all of you,” he said, referring to cloud software and subscription billing.

And an earlier TechCrunch story on Peter Fenton of Benchmark pointed out just how fundamental a shift is happening around the consumerization of enterprise:

Marc Andreessen and Ben Horowitz have made it clear that software has begun to eat the world, a belief that is reflected in their own investment thesis, which focuses almost exclusively on promising software startups.

But it’s not the old software model that is consuming the planet. When we talk about this not-to-be-underestimated change in software companies, we’re talking about the consumerization of enterprise…

The truth is that we’re seeing a fundamental shift from sales-driven companies to product-driven companies. The companies that are leading the way there (Fenton cites Dropbox and, of course, portfolio companies Zendesk and New Relic) let this consumer and product focus permeate (read: define) the culture of their companies.

“It’s a fundamentally different value system,” he says. “These companies don’t try to sell to non-users.”…

The cloud has lowered the barriers to scale, making enterprise solutions instantly horizontal and global, providing an increasing number of access points to much larger markets. Fenton says that he’s been keen on finding smart, product-focused entrepreneurs and either turning them on to enterprise opportunities or backing those already beginning to focus there. There’s a much greater opportunity for those companies to reach scale than it is for their sales-driven counterparts.

On a similar theme, Doug Pepper of Interwest highlighted  earlier this year the  attractiveness of solid SaaS business models:

So, why are SaaS companies so attractive to public market investors and trading up in the aftermarket? I asked that question of a friend at a hedge fund that invests in IPOs. Here is his response (I bolded what I believe are his key points):

• SaaS companies are driven less by media hype and more by the investor appetite for attractive recurring revenue business models offered by the SaaS platform.
• These companies are growing rapidly as a result of customers shifting functions away from in-house solutions to more flexible and enhanced platforms that help increase revenues, improve productivity and reduce costs.
• Investors are not concerned by the lack of GAAP earnings, because there is a comfort driven by the 90-95% customer retention rates and an understanding that investing capital back in the business makes sense during the early phase of adoption.
• These factors lead to more defensible businesses that make for attractive takeout candidates.

In a nutshell, he is reiterating what Bill Gurley noted in his excellent post about what constitutes high quality revenue. In this case, SaaS companies solve critical problems for enterprises and generate sustainable and predictable recurring revenue with rapid growth.

• What skills and body of knowledge does a specific individual need to learn for him or her to be a contributing member of society?
• And is it possible to learn the skills–like “how to learn” or critical thinking or quantitative reasoning or managing projects–in the context of a body of knowledge that’s relevant to doing something in the world, rather than knowledge that–in most cases–is an intellectual or academic or test-driven pursuit?

As one albeit narrow example of that, mathematician Arthur Benjamin suggests, in the short TED video below, building quantitative learning up towards statistics rather than calculus. Would parts of algebra fall in the same category as calculus?

What are your thoughts on the framing questions or pointers to others’  good thinking?

In recent years, many top universities have tried to guide their students into careers other than finance…

Wall Street — like a few other professions, including law, management consulting and Teach for America — is taking advantage of the weakness of liberal arts education.

For many kids, college represents an end goal. Once you get into a good college, you’ve made it, and everyone stops worrying about you. You’re encouraged to take classes in subjects like English literature and history and political science, all of which are fine and interesting, but none of which leave you with marketable skills…

What Wall Street figured out is that colleges are producing a large number of very smart, completely confused graduates…

Wall Street is promising to give graduates the skills their university education didn’t. It’s providing a practical graduate school that pays students handsomely to attend…

So it seems universities have been looking at the problem backward. The issue isn’t that so many of their well-educated students want to go to Wall Street rather than make another sort of contribution. It’s that so many of their students end up feeling so poorly prepared that they go to Wall Street because they’re not sure what other contribution they can make…

The fact that Teach for America … is one of the few recruiting systems competitive with Wall Street suggests that graduates are open to paths that aren’t remotely as remunerative as finance and aren’t based in New York or San Francisco. They’re just not seeing all that many of them.

Making a similar point, Bryan Caplan, economics professor at George Mason University, writes:

Think about all the time students spend studying history, art, music, foreign languages, poetry, and mathematical proofs.  What you learn in most classes is, in all honesty, useless in the vast majority of occupations.  This is hardly surprising when you remember how little professors like me know about the Real World.  How can I possibly improve my students’ ability to do a vast array of jobs that I don’t know how to do myself?  It would be nothing short of magic.  I’d have to be Merlin, Gandalf, or Dumbledore to complete the ritual:

Step 1: I open my mouth and talk about academic topics like externalities of population, or the effect of education on policy preferences.

Step 2: The students learn the material.

Step 3: Magic.

Step 4: My students become slightly better bankers, salesmen, managers, etc.

Yes, I can train graduate students to become professors.  No magic there; I’m teaching them the one job I know.  But what about my thousands of students who won’t become economics professors?  I can’t teach what I don’t know, and I don’t know how to do the jobs they’re going to have.  Few professors do.

Many educators sooth their consciences by insisting that “I teach my students how to think, not what to think.”  But this platitude goes against a hundred years of educational psychology…

At this point, you may be thinking: If professors don’t teach a lot of job skills, don’t teach their students how to think, and don’t instill constructive work habits, why do employers so heavily reward educational success?…

According to the signaling model, employers reward educational success because of what it shows (“signals”) about the student.  Good students tend to be smart, hard-working, and conformist – three crucial traits for almost any job.  When a student excels in school, then, employers correctly infer that he’s likely to be a good worker.  What precisely did he study?  What did he learn how to do?  Mere details.  As long as you were a good student, employers surmise that you’ll quickly learn what you need to know on the job.

1. Give teams goals, and the autonomy to hit those goals.  Don’t legislate the process – only the end results
2. Keep teams small. Don’t let the room get too large so people can make decisions and move forward.
3. Identify a DRI (designated responsible individual).  That person owns the decision, not the entire room.  This is critical for cross functional decisions.
4. Require transparency. Hold people accountable to these decisions by reviewing the success of the decision, and then keep iterating.

 Read the entire post.

Updated:  The same topic was covered in Fred Wilson’s MBA Mondays in a guest post by Scott Kurnit founder of About.com on company culture:

Input, not consensus: This may be the biggest for me since it’s the major thing I can point to for why AOL crushed Prodigy in the pre-internet online world. I still have nightmares of 18 people sitting around a table trying to make a pricing decision. It took Prodigy over a year to adjust pricing to be more in line with – and trump AOL and it took Steve Case’s AOL one measly day to respond. One year… one day. I still get chills. Rather than have the indecision of 18 people, pick one to be the decider as the very first action. Trust me, that person feels the weight and authority when they own the decision. They’ll get a ton of input… rather than having endless discussions. Group decision-making makes people fearful of engaging with the concern that it will never end. When one person’s in charge… they want to hear it all. And fast. And get it right. And crisp. And done!

It’s exciting to see the growing number of entrepreneurs and investors taking on education in disruptive ways–both K-12 and higher ed. FWIW, there are some “serious” folks that think the time is right:

Already in August 2011, Marc Andreessen said in the seminal “Why Software is Eating the World“:

Health care and education, in my view, are next up for fundamental software-based transformation. My venture capital firm is backing aggressive start-ups in both of these gigantic and critical industries. We believe both of these industries, which historically have been highly resistant to entrepreneurial change, are primed for tipping by great new software-centric entrepreneurs.

In February 2012, Vinod Khosla included in his “Unhyped” New Areas in Internet and Mobile, Education 2.0:

“Education models that dramatically reduce the cost and increase the availability of quality learning.” The puzzling question is why education has not already changed. My guess is we have not experimented enough with non-linear, rapidly evolving, out-of-the-box approaches but have instead tried to force-fit ‘multi-media textbooks’ and other traditional (often broken) ideas into the “computerized” model. We have also had too much punditry from experts in education instead of just trying hundreds of new ways of doing things. This is starting to change; it makes me optimistic that what has not worked so far can now work

In March 2012, Paul Graham in Frighteningly Ambitious Startup Ideas included “Replacing Universities.” (He mentions high school but thought that would be harder to start with.):

One of the more surprising things I’ve noticed while working on Y Combinator is how frightening the most ambitious startup ideas are. In this essay I’m going to demonstrate this phenomenon by describing some… The biggest startup ideas are terrifying…You’d expect big startup ideas to be attractive, but actually they tend to repel you. And that has a bunch of consequences. It means these ideas are invisible to most people who try to think of startup ideas, because their subconscious filters them out. Even the most ambitious people are probably best off approaching them obliquely…

3. Replace Universities

People are all over this idea lately, and I think they’re onto something. I’m reluctant to suggest that an institution that’s been around for a millennium is finished just because of some mistakes they made in the last few decades, but certainly in the last few decades US universities seem to have been headed down the wrong path. One could do a lot better for a lot less money.

I don’t think universities will disappear. They won’t be replaced wholesale. They’ll just lose the de facto monopoly on certain types of learning that they once had. There will be many different ways to learn different things, and some may look quite different from universities.