Things I’ve learned at ISC HPC

1 Comment

I came into the ISC event pretty ignorant. Here’s some of the things I’ve learned.

Supercomputers run Linux. All of them. This isn’t even a topic of discussion. Yes, I’m sure there are some that don’t, but everyone here just assumes that you are running Linux. And probably two or three Apache products.

Supercomputing isn’t about software. This is a hardware conference.

Supercomputing is primarily about how fast you can get rid of heat. And these people are serious about cooling. I’ve seen some amazingly cool cooling rigs. Perhaps the coolest of them was this one: https://youtu.be/hs9WG0ZA79Q  That unit is called the AIC24, and is manufactured by Asperitas, and is a full submersion rack. You lower your blades into oil, which is in turn cooled by a water cooling pump. This is much quieter than fans, and much more efficient. The oil was cool enough to touch. Enormous supercomputing centers are locating on the edge of lakes specifically so that they can pump cool water from the lake into cooling units like this.

I also saw this cool demo: https://youtu.be/aaEQN8DH0kM  You can actually see the oil boiling on the processor. The vapor is then condensed on a cooling unit in the back and trickles back down into the tank.

I have also been blown away by the Student Cluster Competition. These kids have access to hardware that would have blown my mind when I was in school. There’s 11 teams competing on a variety of metrics, and they have these astonishing supercomputers at their disposal. I was also amazed to discover that LINPACK is still one of the standard benchmarks. I used that when I was in college!

The student hardware is all sponsored by the vendors that are here at this event – presumably so that they can benefit from the publicity when they win the contest. Check out some of these rigs:

I was pleasantly pleased to discover that of the 11 teams competing, 8 are running CentOS. One other was running Fedora – they wanted to run CentOS, but needed a newer kernel for something (I wasn’t very clear on what that was. I’ll try to go find out more information today.) The other two were running Ubuntu. CentOS  also appears to be the preferred platform for the various research institutes I’ve talked to. However, these are the groups that chose to come over to the Red Hat booth and talk to me, so I do acknowledge that this is a rather self-selected sample. The sign on the SuSE booth claims that SuSE is the Linux “most used by the top 100 supercomputers.” More research is warranted here. But it appears clear, at least from this small sample, and from conversations with the students, that CentOS is just What You Run when you’re doing supercomputing.

And finally, I’ve learned (not that it’s a big surprise) that one year of high school German, 30 years ago, is not a great deal of help. And that people are amazingly patient and kind with my ignorance – something that I’ve discovered almost everywhere in the world

 

Software Morghulis

In George R R Martin’s books “A Song of Fire and Ice” (which you may know by the name “A Game of Thrones”), the people of Braavos,
have a saying – “Valar Morghulis” – which means “All men must die.” As you follow the story, you quickly realize that this statement is not made in a morbid, or defeatist sense, but reflects on what we must do while alive so that the death, while inevitable, isn’t meaningless. Thus, the traditional response is “Valar Dohaeris” – all men must serve – to give meaning to their life.

So it is with software. All software must die. And this should be viewed as a natural part of the life cycle of software development, not as a blight, or something to be embarrassed about.

Software is about solving problems – whether that problem is calculating launch trajectories, optimizing your financial investments, or entertaining your kids. And problems evolve over time. In the short term, this leads to the evolution of the software solving them. Eventually, however, it may lead to the death of the software. It’s important what you choose to do next.

You win, or you die

One of the often-cited advantages of open source is that anybody can pick up a project and carry it forward, even if the original developers have given up on it. While this is, of course, true, the reality is more complicated.

As we say at the Apache Software Foundation, “Community > Code”. Which is to say, software is more than just lines of source code in a text file. It’s a community of users, and a community of developers. It’s documentation, tutorial videos, and local meetups. It’s conferences, business deals and interpersonal relationships. And it’s real people solving real-world problems, while trying to beat deadlines and get home to their families.

So, yes, you can pick up the source code, and you can make your changes and solve your own problems – scratch your itch, as the saying goes. But a software project, as a whole, cannot necessarily be kept on life support just because someone publishes the code publicly. One must also plan for the support of the ecosystem that grows up around any successful software project.

Eric Raymond just recently released the source code for the 1970s
computer game Colossal Cave Adventure on Github. This is cool, for us greybeard geeks, and also for computer historians. It remains to be seen whether the software actually becomes an active open source project, or if it has merely moved to its final resting place.

The problem that the software solved – people want to be entertained – still exists, but that problem has greatly evolved over the years, as new and different games have emerged, and our expectations of computer games have radically changed. The software itself is still an enjoyable game, and has a huge nostalgia factor for those of us who played it on greenscreens all those years ago. But it doesn’t measure up to the alternatives that are now available.

Software Morghulis. Not because it’s awful, but because its time has
passed.

Winter is coming

The words of the house of Stark in “A Song of Fire and Ice”, are “Winter is coming.” As with “Valar Morghulis,” this is about planning ahead for the inevitable, and not being caught surprised and unprepared.

How we plan for our own death, with insurance, wills, and data backups, isn’t morbid or defeatist. Rather, it is looking out for those that will survive us. We try to ensure continuity of those things which are possible, and closure for those things which are not.

Similarly, Planning ahead for the inevitable death of a project isn’t defeatist. Rather, it shows concern for the community. When a software project winds down, there will often be a number of people who will continue to use it. This may be because they have built a business around it. It may be because it perfectly solves their particular problem. And it may be that they simply can’t afford the time, or cost, of migrating to something else.

How we plan for the death of the project prioritizes the needs of this community, rather than focusing merely on the fact that we, the developers, are no longer interested in working on it, and have moved on to something else.

At Apache, we have established the Attic as a place for software projects to come to rest once the developer community has dwindled. While the project itself may reach a point where they can no longer adequately shepherd the project, the Foundation as a whole still has a responsibility to the users, companies, and customers, who rely on the software itself.

The Apache Attic provides a place for the code, downloadable releases, documentation, and archived mailing lists, for projects that are no longer actively developed.

In some cases, these projects are picked up and rejuvenated by a new community of developers and users. However, this is uncommon, since there’s usually a very good reason that a project has ceased operation. In many cases, it’s because a newer, better solution has been developed for the problem that the project solved. And in many cases, it’s because, with the evolution of technology, the problem is no longer important to a large enough audience.

However, if you do rely on a particular piece of software, you can rely on it always being available there.

The Attic does not provide ongoing bug fixes or make additional releases. Nor does it make any attempt to restart communities. It is
merely there, like your grandmother’s attic, to provide long-term storage. And, occasionally, you’ll find something useful and reusable as you’re looking through what’s in there.

Software Dohaeris

The Apache Software Foundation exists to provide software for the public good. That’s our stated mission. And so we must always be looking out for that public good. One critical aspect of that is ensuring that software projects are able to provide adequate oversight, and continuing support.

One measure of this is that there are always (at least) three members of the Project Management Committee (PMC) who can review commits, approve releases, and ensure timely security fixes. And when that’s no longer the case, we must take action, so that the community depending on the code has clear and correct expectations of what they’re downloading.

In the end, software is a tool to accomplish a task. All software must serve. When it no longer serves, it must die.

Three best features of open source events

As part of Stormy’s ongoing blog challenge, here’s my take on “Three best features of open source events.”

1. The hackathon

While there is considerable evidence that the term “hackathon” should be avoided (No, I can’t find the article right now. I’ll keep looking), the collaborative space at an event is, in my opinion, the most important part of an open source event.

Open source events are educational, of course. You can attend a talk and learn things. But most of the information that you need to learn is available, free, online. So to me the most important part of an event is the opportunity to meet and collaborate with the other people on the project.

Defining a specific space for this is critical to get people to sit down and play along. Signs identifying project teams or topics is even more welcoming. Having a white board where people can identify specifically what they are working on gives a way for introverts to be overtly welcoming of other people with similar interests.

Publicizing the collaborative space well in advance of  the event gives the opportunity for people to discuss what they might work on, and gives some people the added incentive to show up at all.

2. The after-party

While it’s indeed a cliche (because it’s true!) that open source events have too much alcohol, having an after-event, with or without food and/or drinks, is a critical part of the event. It gives a specific time and place for your community to get to know one another in a less formal atmosphere, and talk about something other than code. These kinds of community bonds will simply never happen on the mailing list, which is by design focused on the project, the code, the design, and so on, rather than on the personalities.

Open source communities fail because of personality issues at least as often as they do because of code issues. Providing a specific time and space to address these issues saves communities. As we say at Apache, Community > Code.

3. The keynotes

Picking good keynotes is really hard, because keynotes should be inspiring. As such, they don’t always have to be directly related to the topic of the event, but should be, in some way, of interest to the audience.

A keynote should be delivered by someone who is engaging and eloquent. And it should have some kind of call to action, or end on a note that inspires the audience to go do something.

I’ve been attending technical conferences for 20 years, and I remember only a handful of keynotes. I remember Douglas Adams because he was funny. I remember Hugh Howie because I got to sit on stage with him and grill him about the process of being an author and engaging your fans. I remember an OSCon keynote about maps, and one by a guy from WETA about the making of the Lord of the Rings movies. I remember Gema Parreño talking about using data to save the earth from collision with space objects. And most recently, Sandra Matz talking about what your social media profile says about you.

But there have also been a lot of clunkers, and a lot of product pitches, and a lot of Hey Look At Me talks. I can do without those.

Oh, and if you have any suggestions for great keynotes, please let me know. 🙂

Event report: ApacheCon North America, 2017, Miami

Event Report, ApacheCon North America 2017

May 15-19, 2017

(This is an abridged version of the report I sent to my manager.)

Last week I attended ApacheCon North America in Miami. I am the conference chair of ApacheCon, and have been for on and off for  about 15 years. Red Hat has been a sponsor of ApacheCon almost every single time since we started doing it 17 years ago. In addition to being deeply involved in specific projects, such as Tomcat, ActiveMQ, and Mesos, we are tangentially involved in many of the other projects at the Apache Software Foundation.

Presentations from ApacheCon may be found at https://www.youtube.com/playlist?list=PLbzoR-pLrL6pLDCyPxByWQwYTL-JrF5Rp (Yes, that’s the Linux Foundation’s YouTube channel – this ApacheCon was produced by the LF events team.)

I’d like to draw specific attention to Alan Gates, at Hortonworks, who has developed a course to train people at the company in how to work with upstream projects. He did this because, as the company expanded from a small group of founders who deeply understood open source, to thousands of employees who kinda sorta got it, but not always.


Also of great interest was the keynote by Sandra Matz about what your social media profile tells the world about you. It’s worth watching all the way to the end, as she doesn’t just talk about the reasons to be terrified of the interwebs, but also about how this kind of analysis can actually be used for good rather than evil.

Event report: Red Hat Summit, OpenStack Summit

Event report: Red Hat Summit, OpenStack Summit

May 1-5, 2017 and May 8-11, 2017

During the first two weeks of May, I attended Red Hat Summit, followed by OpenStack Summit. Since both events were in Boston (although not at the same venue), many aspects of them have run together.

Mini-cluster

On the first day of Red Hat Summit, I received the mini-cluster, which had been built in Brno for the April Brno open house. There were one or two steps missing from the setup instructions, so with a great deal of help from Hugh Brock, it too most of the first day to get the cluster running. We’ll be publishing more details about the mini-cluster on the RDO blog in the next week or two. However, most of the problems were 1) it was physically connected incorrectly (ie, my fault) and 2) there were some routing table changes that were apparently not saved after initial setup.

Once the cluster was up, we connected to the ManageIQ cluster on the other side of our booth, and they were able to manage our OpenStack deployment. Thus, we were able to demonstrate the two projects working together.

In future events, we’d like to bring more projects into this arrangement – say, use Ceph for storage, or have ManageIQ managing OpenStack and oVirt, for example.

After we got the cluster working, in subsequent days, we just had to power it on, follow the startup instructions, and be patient. Again, more details of this will be in the RDO blog post in the coming  weeks.

Upcoming CentOS Dojos

I had conversations with two groups about planning upcoming CentOS Dojos.

The first of these will be at Oak Ridge National Labs (ORNL), and is
now tentatively scheduled for the first Tuesday in September.  (If you saw my internal event report, I mentioned July/August. This has since changed.) They’re interested in doing a gathering that would be about both CentOS and OpenStack, and draw together some of the local developer community. This will be held in conjunction with the local LOPSA group.

The second Dojo that we’re planning will be at CERN, where we have a great relationship with the cloud computing group, who run what we believe to be the largest RDO installation in the world. We have a tentative date of October 20th, immediately before Open Source Summit in Prague to make it easier to combine two trips for those traveling internationally. This event, too, would cover CentOS topics as well as OpenStack/RDO topics.

If you’re interested in participating in either one of these events, you need to be on the centos-promo mailing list. Send mail to centos-promo-subscribe@centos.org to subscribe, or visit https://lists.centos.org/mailman/listinfo/centos-promo for the
clicky-clicky version.

General Impressions

The Community Central area at Red Hat Summit was awesome. Sharing center stage with the product booths was a big win for our upstream first message, and we had a ton of great conversations with people who grasped the “X is the upstream for Red Hat X” concept, seemingly, for the first time. The “The Roots Are In The Community” posters resonated with a lot of people, so huge thanks to Tigert for pulling those together at the last minute.

The collaboration between RDO and ManageIQ was very rewarding, and helped promote the CloudForms message even more, because people could see it in action, and see how the communities work together for the greater good of humanity. I look forward to expanding this collaboration to all of the projects in the Community Central area by next year.

The space for Red Hat Summit was huge, making the crowd seem a lot smaller than it actually was. The opposite was true for  OpenStack Summit, where it was always crowded and seemed very busy, even though the crowd was smaller than last year.

 

Where next?

In three weeks I’ll be heading to the High Performance Computing event in Frankfurt. My mission there is to talk with people that are using CentOS and RDO in HPC, and collect user stories.

Admitting when you’re wrong

Today I received email from a service I use – Expensify. In this message, the CEO of the company acknowledged that the name that they had chosen for one of their services was a bad choice, and they were consequently changing it:

2) “Wingman” renamed to “Copilot”
Remember how we had the genius idea of naming our amazing delegated access feature (where one user can sign in to another’s account to help them out) “Wingman”? As a child of the 80’s I just assumed that name conjured up images of Top Gun fighter jets and double high-fives in everyone. But it turns out that to the children of the 90’s and beyond, it means cruising bars and picking up chicks — who knew? Actually, almost everyone it seems. So, bowing to the wisdom of the crowd, “Wingman” is now the less-offensively named “Copilot”. My bad!

Meanwhile, the President of the United States of America made a typo on Twitter (no big deal, we’ve all done it) and then, rather than just saying “oops”, sent his press secretary out on stage to claim that it was intentional – a coded message, no less.

There’s a video at http://thehill.com/homenews/administration/335809-spicer-offers-cryptic-explanation-for-trump-covfefe-tweet if you missed it, or don’t believe me.

One of the benchmarks of becoming an adult is an ability to admit an error. One of the marks of being a child is that one defends one’s mistakes, even when they are inconsequential, like a typo.

One day, I hope we have an adult in the White House again.

Why I love ApacheCon 

This is the lightning talk I gave this evening at ApacheCon: 

ApacheCon is a high point of my year, every year, going back to March of 2000.

In late 1999, Ken Coar told me I should submit a talk for ApacheCon. Astonishingly, my talks were all accepted, and I found myself in Orlando speaking in front of a few hundred people who thought I knew what I was talking about. I have since made a career out of that particular game.

This is the 28th ApacheCon since the creation of the Apache Software Foundation. 29 if you count the event in 1998 before there was a Foundation. I don’t count it, because I missed it. I also missed the ApacheCon in Sinsheim, Germany, in 2012, for which I will never forgive my boss at the time. But I *think* I have been to more ApacheCons than anyone else. 27 of them.

I love being on stage. With hundreds of people looking at me, hanging on my every word, believing I know what I’m talking about.

But there’s other reasons I love ApacheCon. It’s the place I go to see some of my oldest friends – many of whom I first met at ApacheCon, including some new ones this week.

I love ApacheCon because it shows me that I’m not alone. As C. S. Lewis said, we read to know that we’re not alone. Except that he didn’t say it. It’s actually just a quote from a movie about him.

I love ApacheCon because I love Apache. And Hawaiian shirts. Shane’s lightning talks are another high point of my year, because they are both entertaining and informational. Except I hear he’s not giving one this year.

I love ApacheCon because of the passion that I see in the people that attend. People that love Apache, and also love solving actual real world problems. The sessions here at ApacheCon range from the esoteric and philosophical to the deeply practical, but at the heart of each one is a desire to solve problems in the real world. To scratch your own itch, as the saying goes.

I love ApacheCon because of our sponsors. Talking to sponsors about why they are here at ApacheCon has the effect of re-centering us. Sure, open source is about having fun and tinkering, but it’s also about solving problems for real people that rely on us. People that depend on Apache because we have a reputation for vendor-neutral, high-quality software which is sustainable because of those esoteric philosophies that we cling to even in the face of practical realities.

I love ApacheCon because of the time I’ve put into it. I’ve worked on ApacheCon for 18 years now. I often refer to ApacheCon as my life’s work. I spend hundreds of hours on it, and so do many other people, including our amazing producers, our numerous volunteers, our tireless Infra contractors, our beloved Melissa, and our supportive board of directors. ApacheCon is my sweat and tears, literally and figuratively. It’s older than two of my kids, and the oldest kid grew up knowing that Dad loves ApacheCon. The wall in my office is covered with ApacheCon attendee badges – 27 of them. And ApacheCon has become a part of my identity.

So as we look forward to the next ApacheCon (details coming very, very soon, I hope) we need to figure out what *you* want ApacheCon to be, and make it that, rather than doing it just because it’s what we do, and what we’ve always done. ApacheCon is about building community, more than it’s about anything else, and that’s really why I love ApacheCon. I love seeing communities come together around a common goal, and believing that I was a catalyst in making that happen.

So, thank you so much for coming to ApacheCon, my friends. I hope you’ll come again, and I hope that you’ll come to love it as much as I do. But that might not be possible.

No, Apache did not send you spam

Today, the ASF received yet another complaint from a distraught individual who had, in their opinion, received spam from the Apache Software Foundation. This time, via our Facebook page. As always, this is because someone sent email, and in that email is a link to a website – in this case, www125.forcetwo.men , which is displaying a default (ie, incorrectly configured) Apache web server, running on CentOS.

This distraught individual threatened legal action against the ASF, and against CentOS, under FBI, Swedish, and International law, for sending them spam.

No, Apache didn’t send you spam. Not only that, but Apache software wasn’t used to send you spam. Unfortunately, the spammer happened to be running a misconfigured copy of software we produced. That’s the extent of the connection. Also, they aren’t even compentent enough to correctly configure their web server.

It would be like holding  a shovel company liable because someone dug a hole in your yard.

Or, better yet, holding a shovel company liable because someone crashed into your car, and also happened to have a shovel in their trunk at the time.

We get these complaints daily, to various email addresses at the Foundation, and via various websites and twitter accounts. While I understand that people are irritated at receiving spam, there’s absolutely nothing we can do about it.

And, what’s more, it’s pretty central to the philosophy of open source that we don’t put restrictions on what people use our software for – even if they *had* used our software to send that email. Which they didn’t.

So stop it.

 

What does a community manager do

Last week I took some long-postponed vacation time and missed Stormy’s blogging prompt. But this particular question needs answering.

What does a community manager do?

Whenever people ask me what I do, I find myself struggling for words, because being a community manager is a daunting mishmash of interrelated tasks that range from marketing to HR to event management to journalism to … everything else.

During the Obama years, the term “community organizer” was bandied about, and that comes close to what a community manager does. And then there’s the confusion around the term “manager”, because I’m not a manager, and I don’t manage the community.

So what do I do?

Well, it’s a bit of this and a bit of that.

Cheerleading

A major task of community management is to be the lead cheerleader for the community. I’m the community manager for RDO, and so it’s my job to promote RDO to the various audiences that might care.

In the case of RDO, that includes the larger OpenStack community, cloud operators in the general IT work force, and also internally to people at Red Hat who need to be informed about what’s happening in the “upstream” community, and how it affects our “downstream” product, Red Hat OpenStack Platform.

To this end, I manage our Social Media presence, post updates to our mailing lists and blog,  make announcements on IRC, and so on. I also show up at conferences and answer questions from attendees, put together presentations to tell people about what we do, and speak at events.

Event planning

At Red Hat, we have an actual event planning team, and they are awesome at what they do – booking facilities, shipping stuff around, planning and setting up and tearing down the physical booths at events, coordinating and communicating with all of the people that will attend.

I don’t do that.

But I plan and coordinate events that my part of the community will show up to. I plan small social events around the main events. I make sure that those events work with the schedules of the people that I’m responsible for, and encourage them to come and bring their friends.

And I help out with small regional meetups, providing swag, as well as helping with (i.e., funding) travel and lodging for speakers that might not have budget to do that on their own.

Make personal connections

A major job of the community manager is to become an expert on the community members. Who knows what. Who is willing to speak on a particular topic. Who has the connections to make things happen.

Then, when a question is asked, or when someone has a need, it’s my job to connect them with the right person that can help them.

So … kind of like first-tier support, triaging problems, solving the ones that I’m able to, and referring to second-tier support those problems that are beyond my expertise.

There’s also some mentoring that goes in here – helping people new to the project know where to fit in, and where to find the information they need.

Look for stories

I look for stories of people using our project, and tell those stories to the world. In this part of the role, I’m the journalist, looking for ways to tell the story about how our project makes life easier. Much of this is amplifying stories that other people are telling. And then some of it is going out and gathering the stories, and then finding ways to tell those stories.

Everything else

While I don’t write code for the RDO project, I do a little bit of everything else – whatever people need done, I try to fill in, or find the right people to do those tasks.

So, that’s why it’s difficult to know how to answer when people ask me what I do at work.

The Margin Is Too Narrow