Tag Archives: geek

Software Morghulis

In George R R Martin’s books “A Song of Fire and Ice” (which you may know by the name “A Game of Thrones”), the people of Braavos,
have a saying – “Valar Morghulis” – which means “All men must die.” As you follow the story, you quickly realize that this statement is not made in a morbid, or defeatist sense, but reflects on what we must do while alive so that the death, while inevitable, isn’t meaningless. Thus, the traditional response is “Valar Dohaeris” – all men must serve – to give meaning to their life.

So it is with software. All software must die. And this should be viewed as a natural part of the life cycle of software development, not as a blight, or something to be embarrassed about.

Software is about solving problems – whether that problem is calculating launch trajectories, optimizing your financial investments, or entertaining your kids. And problems evolve over time. In the short term, this leads to the evolution of the software solving them. Eventually, however, it may lead to the death of the software. It’s important what you choose to do next.

You win, or you die

One of the often-cited advantages of open source is that anybody can pick up a project and carry it forward, even if the original developers have given up on it. While this is, of course, true, the reality is more complicated.

As we say at the Apache Software Foundation, “Community > Code”. Which is to say, software is more than just lines of source code in a text file. It’s a community of users, and a community of developers. It’s documentation, tutorial videos, and local meetups. It’s conferences, business deals and interpersonal relationships. And it’s real people solving real-world problems, while trying to beat deadlines and get home to their families.

So, yes, you can pick up the source code, and you can make your changes and solve your own problems – scratch your itch, as the saying goes. But a software project, as a whole, cannot necessarily be kept on life support just because someone publishes the code publicly. One must also plan for the support of the ecosystem that grows up around any successful software project.

Eric Raymond just recently released the source code for the 1970s
computer game Colossal Cave Adventure on Github. This is cool, for us greybeard geeks, and also for computer historians. It remains to be seen whether the software actually becomes an active open source project, or if it has merely moved to its final resting place.

The problem that the software solved – people want to be entertained – still exists, but that problem has greatly evolved over the years, as new and different games have emerged, and our expectations of computer games have radically changed. The software itself is still an enjoyable game, and has a huge nostalgia factor for those of us who played it on greenscreens all those years ago. But it doesn’t measure up to the alternatives that are now available.

Software Morghulis. Not because it’s awful, but because its time has

Winter is coming

The words of the house of Stark in “A Song of Fire and Ice”, are “Winter is coming.” As with “Valar Morghulis,” this is about planning ahead for the inevitable, and not being caught surprised and unprepared.

How we plan for our own death, with insurance, wills, and data backups, isn’t morbid or defeatist. Rather, it is looking out for those that will survive us. We try to ensure continuity of those things which are possible, and closure for those things which are not.

Similarly, Planning ahead for the inevitable death of a project isn’t defeatist. Rather, it shows concern for the community. When a software project winds down, there will often be a number of people who will continue to use it. This may be because they have built a business around it. It may be because it perfectly solves their particular problem. And it may be that they simply can’t afford the time, or cost, of migrating to something else.

How we plan for the death of the project prioritizes the needs of this community, rather than focusing merely on the fact that we, the developers, are no longer interested in working on it, and have moved on to something else.

At Apache, we have established the Attic as a place for software projects to come to rest once the developer community has dwindled. While the project itself may reach a point where they can no longer adequately shepherd the project, the Foundation as a whole still has a responsibility to the users, companies, and customers, who rely on the software itself.

The Apache Attic provides a place for the code, downloadable releases, documentation, and archived mailing lists, for projects that are no longer actively developed.

In some cases, these projects are picked up and rejuvenated by a new community of developers and users. However, this is uncommon, since there’s usually a very good reason that a project has ceased operation. In many cases, it’s because a newer, better solution has been developed for the problem that the project solved. And in many cases, it’s because, with the evolution of technology, the problem is no longer important to a large enough audience.

However, if you do rely on a particular piece of software, you can rely on it always being available there.

The Attic does not provide ongoing bug fixes or make additional releases. Nor does it make any attempt to restart communities. It is
merely there, like your grandmother’s attic, to provide long-term storage. And, occasionally, you’ll find something useful and reusable as you’re looking through what’s in there.

Software Dohaeris

The Apache Software Foundation exists to provide software for the public good. That’s our stated mission. And so we must always be looking out for that public good. One critical aspect of that is ensuring that software projects are able to provide adequate oversight, and continuing support.

One measure of this is that there are always (at least) three members of the Project Management Committee (PMC) who can review commits, approve releases, and ensure timely security fixes. And when that’s no longer the case, we must take action, so that the community depending on the code has clear and correct expectations of what they’re downloading.

In the end, software is a tool to accomplish a task. All software must serve. When it no longer serves, it must die.

Event report: ApacheCon North America, 2017, Miami

Event Report, ApacheCon North America 2017

May 15-19, 2017

(This is an abridged version of the report I sent to my manager.)

Last week I attended ApacheCon North America in Miami. I am the conference chair of ApacheCon, and have been for on and off for  about 15 years. Red Hat has been a sponsor of ApacheCon almost every single time since we started doing it 17 years ago. In addition to being deeply involved in specific projects, such as Tomcat, ActiveMQ, and Mesos, we are tangentially involved in many of the other projects at the Apache Software Foundation.

Presentations from ApacheCon may be found at https://www.youtube.com/playlist?list=PLbzoR-pLrL6pLDCyPxByWQwYTL-JrF5Rp (Yes, that’s the Linux Foundation’s YouTube channel – this ApacheCon was produced by the LF events team.)

I’d like to draw specific attention to Alan Gates, at Hortonworks, who has developed a course to train people at the company in how to work with upstream projects. He did this because, as the company expanded from a small group of founders who deeply understood open source, to thousands of employees who kinda sorta got it, but not always.

Also of great interest was the keynote by Sandra Matz about what your social media profile tells the world about you. It’s worth watching all the way to the end, as she doesn’t just talk about the reasons to be terrified of the interwebs, but also about how this kind of analysis can actually be used for good rather than evil.

Event report: Red Hat Summit, OpenStack Summit

Event report: Red Hat Summit, OpenStack Summit

May 1-5, 2017 and May 8-11, 2017

During the first two weeks of May, I attended Red Hat Summit, followed by OpenStack Summit. Since both events were in Boston (although not at the same venue), many aspects of them have run together.


On the first day of Red Hat Summit, I received the mini-cluster, which had been built in Brno for the April Brno open house. There were one or two steps missing from the setup instructions, so with a great deal of help from Hugh Brock, it too most of the first day to get the cluster running. We’ll be publishing more details about the mini-cluster on the RDO blog in the next week or two. However, most of the problems were 1) it was physically connected incorrectly (ie, my fault) and 2) there were some routing table changes that were apparently not saved after initial setup.

Once the cluster was up, we connected to the ManageIQ cluster on the other side of our booth, and they were able to manage our OpenStack deployment. Thus, we were able to demonstrate the two projects working together.

In future events, we’d like to bring more projects into this arrangement – say, use Ceph for storage, or have ManageIQ managing OpenStack and oVirt, for example.

After we got the cluster working, in subsequent days, we just had to power it on, follow the startup instructions, and be patient. Again, more details of this will be in the RDO blog post in the coming  weeks.

Upcoming CentOS Dojos

I had conversations with two groups about planning upcoming CentOS Dojos.

The first of these will be at Oak Ridge National Labs (ORNL), and is
now tentatively scheduled for the first Tuesday in September.  (If you saw my internal event report, I mentioned July/August. This has since changed.) They’re interested in doing a gathering that would be about both CentOS and OpenStack, and draw together some of the local developer community. This will be held in conjunction with the local LOPSA group.

The second Dojo that we’re planning will be at CERN, where we have a great relationship with the cloud computing group, who run what we believe to be the largest RDO installation in the world. We have a tentative date of October 20th, immediately before Open Source Summit in Prague to make it easier to combine two trips for those traveling internationally. This event, too, would cover CentOS topics as well as OpenStack/RDO topics.

If you’re interested in participating in either one of these events, you need to be on the centos-promo mailing list. Send mail to centos-promo-subscribe@centos.org to subscribe, or visit https://lists.centos.org/mailman/listinfo/centos-promo for the
clicky-clicky version.

General Impressions

The Community Central area at Red Hat Summit was awesome. Sharing center stage with the product booths was a big win for our upstream first message, and we had a ton of great conversations with people who grasped the “X is the upstream for Red Hat X” concept, seemingly, for the first time. The “The Roots Are In The Community” posters resonated with a lot of people, so huge thanks to Tigert for pulling those together at the last minute.

The collaboration between RDO and ManageIQ was very rewarding, and helped promote the CloudForms message even more, because people could see it in action, and see how the communities work together for the greater good of humanity. I look forward to expanding this collaboration to all of the projects in the Community Central area by next year.

The space for Red Hat Summit was huge, making the crowd seem a lot smaller than it actually was. The opposite was true for  OpenStack Summit, where it was always crowded and seemed very busy, even though the crowd was smaller than last year.


Where next?

In three weeks I’ll be heading to the High Performance Computing event in Frankfurt. My mission there is to talk with people that are using CentOS and RDO in HPC, and collect user stories.

No, Apache did not send you spam

Today, the ASF received yet another complaint from a distraught individual who had, in their opinion, received spam from the Apache Software Foundation. This time, via our Facebook page. As always, this is because someone sent email, and in that email is a link to a website – in this case, www125.forcetwo.men , which is displaying a default (ie, incorrectly configured) Apache web server, running on CentOS.

This distraught individual threatened legal action against the ASF, and against CentOS, under FBI, Swedish, and International law, for sending them spam.

No, Apache didn’t send you spam. Not only that, but Apache software wasn’t used to send you spam. Unfortunately, the spammer happened to be running a misconfigured copy of software we produced. That’s the extent of the connection. Also, they aren’t even compentent enough to correctly configure their web server.

It would be like holding  a shovel company liable because someone dug a hole in your yard.

Or, better yet, holding a shovel company liable because someone crashed into your car, and also happened to have a shovel in their trunk at the time.

We get these complaints daily, to various email addresses at the Foundation, and via various websites and twitter accounts. While I understand that people are irritated at receiving spam, there’s absolutely nothing we can do about it.

And, what’s more, it’s pretty central to the philosophy of open source that we don’t put restrictions on what people use our software for – even if they *had* used our software to send that email. Which they didn’t.

So stop it.


Open source stats – but what do the numbers *mean*?

I recently sent a report to project management containing some numbers that purport to describe the status of the RDO project.

I got a long and thoughtful response from one of the managers – we’ll call him Mark – and it seems worthwhile sharing some of his insights. To summarize, what he said was, don’t bother collecting stats if they don’t tell a story.

1. Focus on the goals

Listing a bunch of numbers without context – even with pretty graphs – doesn’t tell us anything unless you relate them to goals that we’re trying to achieve.

Several weeks ago I presented a “stakeholder review” to this same audience. Any statistics that I present in the future should be directly related to a goal in that review, or they are just meaningless numbers, and possibly a distraction, and, worse still, might cause people to work towards growing the wrong metric. (Google for “be careful what you measure” and read any of those articles for more commentary on this point.)

2. Focus on the people

One of the stats that I provided was about how certain words and phrases feature in the questions on ask.openstack.org. Mark looked beyond the numbers and saw three people who are very active on that website, two of whom are not obviously engaged in the RDO community itself. Why not? How can we help them? How can they help us? What’s their story? Why are we ignoring them?

3. Focus on the blips

In February, our Twitter mentions, retweets, visits, and so on, went through the roof. Why? And why didn’t we do that same thing again in March?

As it turns out, in February there were two conferences that contributed to this. But, specifically, we captured a lot of video at those events, and the Twitter traffic was all around those videos. So clearly we should be doing more of that kind of content, right?

4. Ignore the stuff that doesn’t seem to mean anything

We track “downloads” of RDO, which roughly speaking means every time someone runs the quickstart and it grabs the RPM. Except RDO is on a mirror network, so that number is false – or, at best, it reflects what the trends might be across the rest of the mirror network. So we have no idea what this metric means. So why are we bothering to track it? Just stop.

5. Ask not-the-usual-suspects

This last one wasn’t one of Mark’s observations, but is what I’m taking from this interaction. We tend to ask the same people the same questions year after year, and then are surprised that we get the same answers.

By taking this data to a new audience, I got new answers. Seems obvious, right? But it’s the kind of obvious thing we overlook all the time. Mark provided insight that I’ve been overlooking because I’m staring so hard at the same things every day.

By the way, I’ve presented Mark’s insight very bluntly here, because it’s important to be clear and honest about the places where we’re not doing our job as well as we can be. Mark’s actual response was much kinder and less judgmental, because Mark is always kind and supportive.


Reposting from an email I sent a while back:

As several people have asked about my todo list within the last 2 weeks, I thought I’d share the goodness with everyone.

I’ve been using todo.txt for about a year now. http://todotxt.com/

Don’t let the website fool you. todo.txt isn’t (primarily) a gui app, or a phone app. The todo list is in a plain text file. There’s a dozen different tools that you can use to manage it, but I just use the command line:

t ls – what’s in my list?
t add ITEM – Adds ITEM to my todo list
t pri ## A – Makes item ## priority A
t do ## – Marks item ## as done, moves it to DONE list for later reference
t ls blarg – Lists todo items that match ‘blarg’
t lsp A – Show me all the things that are priority A
done – An alias to ‘cat ~/Dropbox/todo/done.txt’ which shows me what I’ve done most recently

If you happen to store your todo list in your Dropbox directory, you can then also use the free Android app to manage your todo list from your phone. (I’ve heard it also work with google drive, or owncloud, or a variety of other things.)

As someone who has used every possible todo list out there, including a dozen issue trackers, and writing a few different todo list webapps, sticking with a single tool for a whole year is unprecedented. Being able to work from the command line made all the difference for me, since that’s where I always am anyways.

OpenStack PTG, trip report

last week, I attended the OpenStack PTG (Project Teams Gathering) in Atlanta.

Even more in depth: PTG info at https://www.openstack.org/blog/2016/05/faq-evolving-the-openstack-design-summit/


1) This is a hugely productive event, with project teams getting an enormous amount of work done without the distractions that are usually present at a conference.

2) I remain very concerned about how this event will effect the
character of OpenStack Summit – removing the bulk of the engineers from that event, and making it more product/marketing/sales focused. Time will tell

At the gathering, I did 23 interviews with Red Hat engineers about what they did in the Ocata release. You can see some of those interview on the RDO YouTube Channel. I’m not done editing them all yet, but they will appear over the coming weeks as part of various blog posts, as well as all of them appearing in that YouTube playlist.

I am constantly blown away by the passion, expertise, and
professionalism of the folks I get to work with. Wow.

Anyways, more about the PTG.

I was (and, really, still am) very skeptical about this new format.
Splitting OpenStack Summit into four events rather than two has already had significant impact on travel budgets, not just at Red Hat, but also at other companies involved in OpenStack. A lot of companies, for example, didn’t send anyone to FOSDEM, and we had a hard time staffing the OpenStack table there. Usually people work one shift at the table, but this year several of us worked 4 and 5 shifts to cover all the slots.

I am concerned that splitting the engineers off into their own event
will significantly change the character of OpenStack Summit from being a community-centric, tech-centric event, to more of a sales and marketing event, light on technical depth.

But this event, for what was intended, has already been amazing.
Everyone is commenting on how much is getting done, how much less distracted the team meetings are, how much better the teams are gelling than they have at any previous event. This is a working event, and people are here to get work done. They are meeting all day, every day, working on plans and blueprints, and fighting out agreements on things that would take weeks in email, and everyone seems VERY pleased with outcomes.

So, perhaps the trade off will be worth it. Time will tell. Regardless, Erin Disney and her team put on an amazing event that fulfilled, and exceeded, its goals.

On Wednesday  night, everyone that has ever contributed a patch to RDO was invited for drinks and hors d’oeuvres at the SideBar, and while there the RDO Ocata release announcement was sent out.

We had about 50 people in attendance, who ate and drank up all of my budget in about 2 hours.

Here’s some pictures.


Features, not lies

A colleague is attending the nginx conference in Austin this week, and shared with me several anecdotes in which a speaker preached misinformation – or if I want to be generous, grievously outdated information – about Apache httpd, to support the notion that nginx is better.

This led to the following:


Each time I have encountered nginx people at conferences, and attended their talks, they have compared nginx to grossly misconfigured, 10 year old installations of Apache httpd 2.2 to support their claim that nginx is leaner, faster, and easier to administer.

Here’s the thing. nginx is a solid project. I have zero beef with the software itself. I have used it myself, when the need arose. What I object to is the habit of the fans of nginx to lie (or exaggerate, or just spout uninformed opinions) to make themselves look better. If you must compare, compare our latest to your latest, and have experts correctly configure each. That way, each will show where it shines, and where it doesn’t.

It is possible to configure ANY software badly. This is why it’s almost always a bad idea for an expert on SoftwareA, who knows little or nothing about SoftwareB, to compare them head to head – they’ll invariably be comparing a well-configured A to a less than optimally configured B. And in the case of nginx vs Apache httpd, these guys almost always use 2.2 or 1.3 as an example of … well, all of the things that 2.4 fixed. 5 years ago.

Any intro to marketing class will tell you that you need to talk about your own strengths more than you talk about the other guy’s weaknesses. This is a message that nginx and presidential candidates seem to have missed. And, in the case of software, it’s even more important, because whereas Donald Trump will always be a monster, every time you point out a legitimate shortcoming in Apache httpd, we fix it.

Cacti and the Asus RT‑N66U

I discovered a few days ago, quite by accident, that the Cacti project is still quite alive and well. I don’t know why I thought it wasn’t. I thought it would be kind of cool to set it up to graph my network traffic here at home.

I have an Asus RT-N66U wireless router, which I’ve been very pleased with since I acquired it.

Step one was getting Cacti running, which has always been something of a challenge. The installation instructions, while extensive, miss a lot of prerequisites that you encounter along the way. (Install packages are not, apparently, available for CentOS7.) Notably, you have to install Spine (the stats collection daemon) from source, and it required the -devel version of several of the items that the docs mention. So, things like php-devel, mysql-devel, snmp-utils, which are not mentioned in the installation instructions. No big deal, but it did make the process a little longer, finding and installing these prerequisites.

Step two was getting file permissions set up correctly in my Apache httpd document directory. This turned out to be a combination of missing directories (log/ and rra/ in Cacti’s home directory) and the fact that my vanilla installation of php had logging turned off, so everything just silently failed. Those directories, once created, and ownership changed to the newly created cactiuser user, Cacti itself started running. Awesome.

Step three is enabling SNMP on the router, which isn’t hard, but is a little time consuming. Instructions for doing this may be found on the My gap in the void blog, and I will not copy them here.

Finally, there’s the step of getting Cacti to talk to the N66U. This turned out to be absurdly easy. Under ‘devices’, I clicked Add. I gave it the name and IP address of the router, and selected “Generic SNMP-enabled Host” from the Host Template dropdown, and pressed ‘Create’.

On the server, in the cactiuser’s crontab, add:

*/5 * * * * php /var/www/html/poller.php > /dev/null 2>&1

Then, click ‘New Graphs’, select the “select all” checkbox, and press ‘Create’.

Finally, under ‘Graph Management’, select the ‘select all’ checkbox, select ‘Place on a tree (default tree)’, and press go.

And, you’re done. Wait a few hours for data to accumulate:


Server Not Found

Up until about 9 years ago, I had a server rack in my home office. At one point, there were as many as 12 servers running on it, running websites, DNS services, email , NNTP, and a variety of other services.

I blogged about this a while back.


As time went on, I started to recognize the benefits of running services in the cloud. This meant everything from moving email to GMail, to running web and DNS on a hosted server at Rackspace (then known as SliceHost).

Eventually, I had no computers in the house at all, except for my work-issued laptop.

A little while ago, I started to miss my servers, for many reasons. I still run some mail services on my VPS at Rackspace, to do things like mail aliasing for a variety of domains, and of course I still run all my own web servers there. But there’s something about having a server that you have to physically maintain that keeps your skills going in ways that you just don’t have to when it’s out there somewhere.

A year or so ago, I found a refurb Dell machine for next to nothing, and put CentOS 7.1 on it. That released in April, so it must have been about then. For a while I didn’t do much with it, other than test OpenStack installs. But after a while I brought up a Minecraft server on it, running the Bukkit distribution of Minecraft, to play with family members. Then I opened it up to a few friends.

treasure_huntThis Christmas, one friend did a treasure hunt for her kids in the Minecraft world, which was incredibly cool. I’ve seen them on the server a number of times since then.

When I posted on Facebook about this a few days ago, someone from ArcLight responded, saying I should host my server there, which brought on this trip down memory lane. No, Seth, while I appreciate the offer, and I’m certain you’d do a better job than I, running it myself was the entire point. Thanks, though, and I wish you well with your business.