I think Jacob VanWagoner is spot-on.
1. Business model: The primary issue here is the investment it takes to make something that beats the competition. For most semiconductor companies, IP is market leadership, recognition and most importantly revenue. This is what separates one company from the competition.
This is not particularly true for software companies. The business model is completely different.
In short, I would say,
Software business: Make something people pay money to use
Hardware business: Make something that enables a completely new market. This is not mainly for consumers directly.
2. Software hides a lot too: Of course, there are exceptions. You might ask why does Microsoft hide its source code when all it wants to do is sell software? Because OS is a completely different beast of software that is closely coupled with hardware. Linux might be popular among engineers but Microsoft (and Apple iOS) will always lead because they can make money off of their closed IP. Same could be said about Google and their primary products. When has Google talked about their search strategies? PageRank algorithm barely scratches the surface of their search engine.
3. The software you read about online is barely engineering: Making apps, using them, storing stuff in a database is all very very easy. Web and app development is commoditized. Someone has done all the dirty work so that everyone can use it and share basic knowledge about it. And rightly, folks are sharing knowledge openly.
An analogy I like to use is cars. We don’t need to know anything about IC engines to drive one.
Also, not all software is similar. Web development is completely different from making a new database software. There are many reasons to not release database code, the primary reason being a lot of proprietary techniques implemented inside it that would give away competitive advantage. Open source software is *not always* state-of-the-art. I am not against open source development. But more often than not, open source works only for consumer facing applications that make money by selling a “service” to others. A lot of software is quite hard to make and the proprietary IP is what sets one apart from the other.
I personally think that it is not true that semiconductor companies are super-secretive. They release a lot of details. The details are enough for their competitors to know what is coming but not enough for them to just replicate it. But yes, semiconductor engineers are not inclined towards sharing the architecture changes they are doing on hacker news.