Standalone Sysadmin

I do have a confession to make. It’s not a confession that should require the services of the clergy, but a lot of people have been surprised when I’ve told them about it.

I have an iPad.

There are lots of good arguments against it becoming the ubiquitous computing platform, however as a standalone interface, I’ve found it to be a pretty good tool that’s useful in more ways than I imagined before I bought it.

Honestly, one of the ways that it has surprised me is how useful it is in meetings. If I needed to research something, I’d have to do it on a laptop. Then if I wanted to show someone, I’d turn the laptop around, and it would be on the screen. Now, with the iPad, I hand it to them. It seems like such a small insignificant difference, but there’s a visceral change when you hold a piece of information in your hands literally. It has both literal and figurative weight.

Of course, if it were just a web browser, it would be a novelty. The key is that you can install apps on it. Yes, there are lots of problems with the app store (and their app approval methods), and a lot of the apps in the store are rubbish, but there are some gems in there as well.

Deep down at the very bottom of my sysadmin heart, I’m pragmatic. If something works, I use it. If not, I don’t. The iPad works for what I use it for, so I use it. Also, I absolutely love Harbor Master.

So aside from web browsing, what can the iPad do for us in our job roles? As it turns out, quite a lot, provided we abide by some ground rules that are enforced by the limitations of the device.

1. No tactile keyboard

If you want to type a significant amount, you need a bluetooth keyboard, which severely limits the mobility of the device.

2. The resolution is fixed.

If you attempt to access resources with higher resolutions, the display will either scale (and become nigh unreadable) or you will scroll

3. The only VPN access on the iPad is for a Cisco ASA-series SSL VPN.

If you want to access corporate resources, you need an Adaptive Security Appliance from Cisco, or you need to be on your internal network via wifi, or you need to have your corporate resources available over the internet.
OK, as many people have pointed out, I was basing my words on old information. Apparently it’s not just SSL, it’s also IPSec, and it’s not just Cisco. That being said, I’ve heard of many, many problems from many vendors who aren’t Cisco. I’ve personally got Juniper Netscreens, and have never heard of anyone getting a VPN to work using this. Your Mileage May Vary.

That last one is the killer for me. It essentially means that I can only use the tools on my iPad while I’m at work. Lots of people have the Cisco hardware and licenses to make the VPN work, though, and there are a decent number of people doing business on the public cloud, in which case the tools may be handy. If so, great! Just be warned about what you’re up against.

So now that you know whether or not you can use the apps, what apps should you get? There are tons. Over 250,000. Lots of them are crap, though. Even ignoring the stupid fart apps, what makes a sysadmin-related app good? In my mind, there are a few things to look for.

1. It doesn’t see itself as the center of the world

There are a lot of apps I’ve seen that do very useful things, except that they make big assumptions. Computer Inventory is a great example. Having an app on the iPad that could enter information into your asset database would be great! Unfortunately, Computer Inventory writes to an internal database. As far as I can tell, you can’t even export it to a CSV file. Lovely.

Much better would be Decision Manager, which ties into OCS / GLPI, which is software you run on an inventory server. This app does have the disadvantage of being written in French, but even so, it’s still more useful than a piece of software which can’t export the information entered into it.

2. It doesn’t do things wholly unfit for a mobile device

Would you use a pocket camera to do security monitoring of your front door? Of course not, you want to carry it with you, and it’s not set up to do the kind of continual feed that is useful for that anyway.

So why, then, are there so many crappy network monitors that do monitoring from the iPad? If you’re doing network monitoring from your iPad, trust me, you’re doing it wrong.

Instead, use something like TouchMon for Nagios or iPRTG for PRTG. These are apps which connect to the dedicated servers already running on your network and display the information locally.

The strength of the mobile device is that it’s mobile, not that it’s a computing powerhouse.

3. If you’re going to pay for an app, make sure it does something worth paying for

Would you pay a buck for an app that does nothing except lets you query whois? Me neither. But someone wrote it and charges that for it. That’s pretty miserable. There are a plethora of other apps that do the same thing, some of them with added functionality that’s also very basic. Network Ping, for instance, will let you ping, traceroute, telnet, and even ping a subnet. Of course, it’s $4, and that seems expensive to me for something that I can do at a shell prompt.

I don’t want to use this blog to advocate anything illegal, and fortunately, I don’t have to. I recommend jailbreaking your iPad. Yes, it’s unsupported, but it’s also easy to do, and it’s easy to reverse. Doing so gives extra functionality to your device, and allows you to install software that you’re probably going to be very familiar with, at least if you’re a UNIX/Linux admin. All of the functionality of the $4 app above (and way, way more) is available via the command line utilities that do the same thing. They just need installed using Cydia or one of the other software installers available to a jailbroken iOS device.

There are some apps that I’ve installed and use, and there are others that people have recommended to me on twitter. I can’t personally speak for all of them, but they sound useful and good for the most part. Several of them do similar things, so I’ll try to group them by utility.

Remote Access (terminal)
iSSH is the gold standard for remote terminal apps. It includes “VT100, VT102, VT220, ANSI, xterm, and xterm-color terminal emulator over SSH and telnet, integrated with a tunneled X server and VNC client”. Say it with me…”that’s hot”.

The existence of good terminal emulators hasn’t stopped people from solving specialized problems, sometimes very cleanly. MyRouters Pro, for instance, simplifies logging in to your Cisco routers and devices. It supports multiple concurrent connections and also macros. Interesting sounding software!

Remote Access (GUI)
Desktop Connect is a pretty smooth looking single pane of glass for RDP and VNC. At $15, it’s expensive, but not as expensive as solutions like iTap RDP and iTap VNC, each of which are $12.

Coming in at $15 as well is the Wyse PocketCloud Remote Desktop. It seems to be the market leader, in terms of remote desktop solutions, and from the screenshots, you can see why. The unique mouse pointer menu system looks handy, and while I haven’t shelled out money for one of these solutions yet, if I did, it would be for this one.

As far as I can tell, LogMeIn Ignition is a remote desktop solution, but only for Macs. I was under the mistaken impression that LogMeIn Ignition was Mac-only, but it has been pointed out to me that this is not the case. In fact, according to commenter Dan, the pro version includes some pretty sweet Windows-specific things, too. Check out the user guide for more details.

Remote Control
Sometimes, all you want is to control the screen in front of you. Having a keyboard and mouse on your couch is bulky and uncomfortable. Having one for every machine attached to a NOC display may not be possible. The software I use for my media-PC-sans-keyboard-and-mouse problem is Mobile Mouse. Essentially, it lets the iPad function like a giant touchscreen, and it works great. At $3, it’s slightly cheaper than the other option I found out about, creatively called touchpad, which costs $5.

Service & Solution Administration
Citrix caught on to the remote desktops pretty quickly and wrote Citrix Receiver to fill the need for an iPad app to access your network-based machines simply.

If you’ve got a Rackspace server (or several), you may already know about Rackspace Cloud Pro, an app that allows you to manage your servers and storage.

I don’t run Mac servers anymore. They’re too…well…weird, for my tastes (at least, the old headless Xserve compute nodes we had were). If I did still run macs, You can bet that I’d have this app. It’s Server Admin Remote, which manages the services as though you were sitting at the administrative console in Server Manager.

Troubleshooting
UDP Tools is a cool idea. It’s sort of like a graphical netcat which only does UDP. There’s also one for TCP, but that’s called “telnet”

syslogger is a tool which allows you to send a syslog message to the syslog server of your choice. It only speaks UDP, but assuming you’ve got syslog listening for remote connections, you can spit out messages at it from your iPad using this.

I’m not usually a fan of crappy visual traceroute programs. They’re typically kludgy, buggy, and most of the apps cost money for something that is, in all reality, pointless. Vtrace, however, is at least free. So if you like maps with dots and lines, try this one.

Monitoring & Security
The aforementioned touchmon is an iPhone app for checking the statuses reported by a Nagios server. It doesn’t require any modifications to the Nagios server itself. It seems to just scrape the html and interpret the statuses itself, so if you’ve customized your CGIs, it probably won’t work, but it seems pretty handy otherwise.

iStat should be included just for the interface. It’s a way to see the stats of a remote machine at a glance, and it’s pretty. Unfortunately it also requires installation of a monitoring program, so it’s probably not worth it.

If you run network-based security cameras, there’s at least a decent chance that they’re by Axis. If they are, then you, too, can feel like you’re on CSI with the Viewer for Axis Cameras. I don’t have any of these, so I don’t have this program, but I imagine that it allows you to view as many cameras as you’ve got. If you hear anything different (or you try it out) let me know!

Aanval, which connects to an Aanval server, which performs snort & syslog IDS.

iCacti Server Monitor is about all I can ask for in a an app, aside from the $4 price tag. It connects to an existing monitoring server, displays graphs clearly and cleanly, and has real value in displaying trends to people in meetings.

Reference & Information Management
The Omnigraffle app is probably going to be the most expensive on this list at $50, but if you make diagrams (and you have a Mac with Omnigraffle), it’s absolutely worth it. You can share diagrams and stencils from the full Mac version, and being able to organize your thoughts on the ipad is great. The interface is complex, because there are so many options, and it takes a while to get the hang of it, but the reward is worth the time spent.

Evernote is a great piece of software for organizing those bits of information that you’d otherwise jot down on napkins or spare bits of paper, or whatever you’ve got with you. Since your computer can’t follow you around, the iPhone and iPad are natural extensions of this software. The best part is that your notes are synced everywhere, so if you make a note on your iPad, when you get back to your computer, it’s there too. Very handy.

dhcp-options provides on-the-fly reference of all of the dhcp options available, and what they mean.

In the end, if the iPad doesn’t fit your work flow, there’s no reason to use it, but if you are always looking for better ways to access information, then maybe you should give it a shot. Although it’s not the end-all be-all of computing and it certainly won’t ever be my primary means of administration, I’ve found it to be a useful tool, and I have to admit, sometimes I feel like I’m on Star Trek when I’m using a piece of hardware that small and that powerful.

I want to thank everyone who pitched in app suggestions on twitter. I’m sure that I left off some great ones. If I didn’t include your favorite app, paste it in the comments below. Thanks!

For the past few days, our NYC office has had incredibly irritating problems with the internet connection. We’ve got service through a local Metro-E provider, but they’re a CLEC, which means they don’t own the lines, they just lease them from the ILEC, who is in this case, Verizon.

The root of the issue is that the wiring at the building we’re in is crap. It’s a small 5 story building that used to be apartments and has been converted to offices, and the wiring is just not up for the job. We went through several pairs of copper pairs looking for one that was good enough to carry the metro-E signal, and it was all we could do. Before metro-E, we had DSL, where we capped out at just over 1Mb/s…and this is in Manhattan.

Unfortunately, the circuit is currently in the middle of dying, so it’s working sometimes and failing others. I first opened this ticket on Monday, and have exchanged emails with our provider a dozen times or so. They’ll see the issue, but symptoms are vague as to whether it’s their equipment, our equipment, or the line running between our equipment, or (what I’m fairly sure the problem is), the lines entering the building from Verizon.

It wasn’t until last night when they finally saw enough errors on the bridge to have Verizon to commit to a service call tomorrow evening to add a loop. Every other time, everything on the line was hunky-dory. This is why intermittent problems take so long to solve…because all the stake holders have to be monitoring at exactly the right time for anything to get done.

Meanwhile, I’ve been having to apologize to my users, and give them instructions on how to forward their desk phones to their cells.

Even though the problem isn’t actually with my provider, I would love to get a secondary network connection, because the lines here are just too unreliable. No cable companies will give us service, no fiber companies will touch the building…it’s pretty much just Verizon and their CLECs at this point.

I think we’ve only got 2 more years on the lease?

There’s an amusing thread on the LOPSA Discuss list going on right now. It’s called “What Animal is a System Administrator“.

I was leaning toward the beaver until I saw the post by Paul Graydon, who recommends the Pooka, aka the Púca:

The púca has the power of human speech, and has been known to give
advice and lead people away from harm. Though the púca enjoys
confusing and often terrifying humans, it is considered to be
benevolent.

It’s like I’m looking in a mirror.

The other day, I caught a message that KSplice was available for Fedora. I thought I’d be a wiseguy and I replied “Yeah, great. Call me in 20 years when it’s available for for RHEL”. Well, as several people pointed out, it turns out the joke is on me.

As you can see, it’s actually available for many Linux-based OSes at various prices. I suppose my confusion stemmed from the fact that I misunderstood what ksplice was.

My impression from a long time ago, when it first came out on Ubuntu, was that it was essentially a kernel patch that dynamically loaded patches and provided the ability to rebootstrap a kernel that was already loaded. As it turns out, it’s a commercial product that offers the ability to not have to reboot your machine to update the kernel. Let me be frank: I’m all about that.

The part that I kind of object to is in the press release, of all things. It’s the opening line of the company profile:

Ksplice is an enterprise software company making reboots a thing of the past.

Please, lets be honest. Reboots are inevitable. Using this product as a stop-gap for untimely reboots may be handy (at the low low price of $50 per year per server), but it can’t (and shouldn’t!) replace regular reboots.

The reasons for scheduled rebooting of machines are numerous. The primary one is that regular reboots assure that the machine is configured to boot correctly. If you’ve got a machine that’s got over 100 days of uptime, how do you know it will start correctly? You last booted it last quarter…what has happened to that machine since then? Changes in installed services, mountpoints, etc…it’s hard to tell if it’s going to be in a known-good state when it comes back up after a power failure.

Another reason to reboot occasionally is to clean up the running state of the machine. What’s that you say? Your machine is running fine? Well, sure, it may be, but how much cruft is left hanging that isn’t obvious? Have you ever used kill -9? Do you know for sure that there aren’t any memory leaks in your running services? Any processes hang while reading I/O and is now stuck in uninterruptible sleep?

Yes, there are lots of things that happen to servers over the course of doing their jobs. A reboot fixes many of them. The only argument against it is uptime.

I’ve written about uptime before, and I still feel the same way. Modern system administration has advanced beyond a single server providing a service. Uptime needs to be measured from the outside in, and according to the availability of the service, not the individual servers comprising that pool.

Feel free to disagree. Let me know if you’ve got an uptime of a year plus and you’re proud of it, or if you would be ashamed to be in that position.

Edit
This entry is causing quite a stir on Reddit. Cxunix from twitter also weighed in on his blog, servermanaged.it (link is in Italian, English translation here).

This is apparently the “time to schedule your conference trips” part of the year, because there is news on the SysAdmin conference front.

First, and most pressing, the LISA10 conference schedule has been released! I’ve got to say, I’m digging the theme of the website, too. More important, though, is the content. Interestingly, all sessions and tutorials are available in half-day increments this year. This means that you can attend the first half of one session then migrate to another session after lunch. I’ve got mixed feelings about this, but I’m interested in how it will pan out. More flexibility is nice, though, and sometimes the first half of a session is really review (though there are a lot of arguments against that, too).

As always, there are discounts available for certain groups, and you do get a lower admission price if you’re a member of LOPSA, USENIX, or SAGE.

Check out the registration page for the fees. There’s an early-bird special going on until October 18th, so make sure you register soon. The return on investment for this conference is amazing.

I’m going to be there as a conference blogger, along with Matthew Sacks, Ben Cotton, and Marius Ducea. We’ll be publishing entries on the USENIX blog (which I’ll be linking to from here as well, of course).

Come to LISA and have a great time. And if you do decide to come, find me and say hello. I always love meeting readers.

Shifting gears a little bit, I’m sure you remember the PICC conference that LOPSA-NJ hosted. Well, we had a blast, and last year’s conference chair, William Bilancio, did an amazing job. It’s a bit much to do that twice in a row, though, so he was looking for someone to take the responsibility for this year’s conference, and after running it through my head a while, I decided that I’d take the job if he thought I’d do alright. Here’s his email announcing it:

It is with a great sigh of relief that Matt Simmons has decided to be
the Program Chair for PICC ‘11.

Last year Matt was the head of the marketing team and did a great job
at getting the word out about the conference and was a key person in
making last years conference a success.

Tom and I feel that he will do a great job as the Program Chair and
will make PICC ‘11 a great conference.

In other news I will be getting in contact with the hotel and get the
date locked in, in the next few weeks and then we can start really
working on the conference.

Please start thinking about sponsor ideas as well as any new people
you think will be able to help make PICC ‘11 another great conference.

Again thank you Matt for taking PICC ‘11 Program Chair job and good luck.

William

I want to thank William and everyone who was involved with last year’s conference. Everyone I’ve talked to had a great time and has been looking forward to this coming year. I’m going to work hard to try to improve on William’s example, and really grow the community of system administrators in New Jersey and the rest of the northeast. I’m going to need help, though, so if you helped out last year, I’ll be calling on you now. If you weren’t involved last year, now is a great time. Drop me an email or comment on this story to let me know that you’re interested in volunteering. We can definitely use the help.

In addition, I was talking to Lee Damon, who let me know about a SysAdmin conference called “Cascadia IT Conference” (aka “CasITConf”), and it’s happening in the Pacific Northwest. It’s being put on by SASAG, the Seattle-Area System Administrators’ Guild.

So there you go. Three sysadmin conferences in one post. It’s going to be a busy year for everyone, so get involved and lend a hand to someone in your area!

My datacenter migration (or renovation, as I’m referring to it) includes a fair amount of added virtualization. We’ll be maxing out the memory and processor power of three machines at each site, and those will act as a VMware HA cluster (we’re buying the vSphere Essentials Plus license kit for each site).

Of course, I’ve got to have some VMs to run. I could reinstall all of my machines using cobbler (which would invoke the gods of trial and error, not to mention incur Murphy’s Wrath), or I could convert the machines that already exist from physical to virtual (p2v). That second option sounds much less error prone.

That being said, converting a physical machine to a VM isn’t exactly a fast process. Hoping to get it done the weekend of the move would be foolish, so I need to get it done beforehand. That’s why I’m driving to Philadelphia today.

Last week, I threw a couple of terabyte SATA drives into a spare PowerEdge 1950 server, upped the RAM a bit, and installed a freshly minted copy of vCenter Hypervisor 4.1 (formerly known as ESXi). I’m trucking this machine down to our secondary data site today so that I can begin the p2v conversion process. I’ve got enough disk space that I won’t run out (I’m only putting the root partitions in the VMs, since all the data is stored on the SAN), and I don’t need to actually run the machines, so RAM won’t be a problem. This will just be a holding tank until I get the VM hosts setup during the conversion weekend.

The actual conversion will be done using VMware Converter, a free tool by VMware that I’ve been really impressed with. It does want an ESXi…err..vCenter Hypervisor server to connect to, but that’s free too.

Once this is down there, I’ve got some decisions to make. Namely, I need to decide how long to wait until I do the conversion. Not a lot of data changes on the root partition. It’s going to be limited to logs, really (since I haven’t gotten a centralized syslog server running yet). The exception to this rule is the domain controller at that site. That needs to be the absolutely last machine I convert, and once I do it, I’ve got to turn off the source, because if the image becomes too far out of sync, well…that’s sort of like crossing the streams.

So, has anyone else pre-converted VMs like this in preparation for a move? Any advice or caveats to watch for?

Edit
Fixed the mistaken Ghostbusters quote. Did I seriously say “crossing the beams”? I am disappoint.

Alright, several people have asked me why I haven’t weighed in on the current “devops” movement. Mostly because no two people can absolutely agree on what DevOps is. I’m outside of that particular community, although I read a lot of the blogs of the key members, so maybe I’m in a good position to comment on my perspective.

First, lets define DevOps. If you strip away all of the touchy-feely stuff that gets associated with the name, devops is, at its core, DevOps is an increased interaction and interdependency between developers and operations staff, whether that operations staff is specifically system administrators or whatever.

This means that the people who develop code no longer have willful ignorance of operational environments, and the people who operate the environments can’t do so in a vacuum of knowledge about the software itself. This increased communication and reliance IS DevOps. That’s it. Nothing more. It’s a methodology. It’s not a panacea and it’s not for everyone. How can you tell if it’s for you?

Let’s answer some questions…

1. Does your organization have programmers?

Developers are necessary for the DevOps relationship…otherwise you’ve just got Ops

2. Do you provide Software as a Service?

DevOps grew up in the web world, around places like Flickr, who provide applications over the web. Other people may just think of them like websites, but in actuality, they’re applications with incredibly large code bases. Since a solid application depends on well-developed code running in a known stable environment, it’s natural that this kind of biosphere would produce methods like DevOps

3. Do you release software updates frequently?

If you’re in an environment where something is broken and gets fixed immediately, then you can say yes here, but it’s not just bug fixes. Features get rolled out, pulled in, and switched around. Agility of this nature isn’t possible without everyone working from the same playbook. It’s also not possible with an environment that can’t change rapidly to match the code.

For the 90% of companies out there without that particular environment, then you probably aren’t using DevOps, and that’s fine, because there’s almost nothing it can do for you. Especially if you don’t have programmers. Because hey, no dev, right?

You’ll notice that nowhere in the preceding text did I mention the tools that DevOps uses. That’s because the tools are completely separate. Using “puppet” doesn’t mean you subscribe to the DevOps methods (or even the mentality), and although DevOps may not be necessary for your environment, you might find puppet extremely useful. Let me say that again, Using the same tools as DevOps shops use does not tie you to the DevOps methodology.

As alluded to in the last answer up above, the shops that run DevOps need environments that can change quickly and absolutely. They needed tools that could do it, because you can’t manually change hundreds of application servers. Because of their need to change that many machines, and have it happen nearly instantaneously, tools to automate this kind of change were developed and implemented.

Other technologies that get lumped into DevOps, cloud computing and virtualization, are also natural off-shoots of the type of environment where you have hundreds of application servers. Of course that kind of environment is going to be heavily into virtualization (if they’ve got an existing large infrastructure) or cloud computing (if they don’t).

Again, DevOps doesn’t “own” these technologies. They just use them (and advance them by writing tools to improve them, in many cases).

So there, that’s my take. For the people who can use it, DevOps is developing into an exciting methodology to ensure increased availability and stability of IT resources.

It’s not for everyone, but you owe it to yourself to take a look at the tools that too many people have been misbranded “DevOps”. There’s a lot of functionality there, and it can decrease the amount of time you spend slogging through administrative tasks.

Edit
It looks like I’m not the only one who’s been thinking about this, too. Benjamin Smith wrote his take as well, and it seems like we agree quite a bit.

I have been to the Ohio Linux Festival once, four years ago. I had a really great time, met interesting people, and made plans to come back the next year. Then I got married the next year on the same weekend that OLF was being held. As much fun as I had at OLF06, I couldn’t really choose it over my own wedding (though frankly, I’m surprised some of our guests picked us over the show). The next year, they had the nerve to schedule it on my anniversary. Jeez!

This year, though, it’s scheduled earlier in September (from the 10th to the 12th), which means I can go! Except that I can’t. I’ve had other stuff scheduled for that weekend for almost a year. Ugh!

On the other hand, you CAN go (and I’m jealous). The schedule looks great, and I’ll let you in on a little tip. It’s directly across the street from Barley’s, home to some of the best burgers and beer in the city. That alone might be worth the trip!

As for the Linux Fest itself, it’s free admission, but if you’re coming (and you are, right?), you should really consider some of the OLFU classes, which are available for a fee. OLFU is the Ohio LinuxFest University, and it’s a day of training put on by LOPSA, the League of Professional System Administrators. I’m a (too-often absentee) member of the committee that is responsible for the classes, and I want to tell you that I’m very excited to see some of the things we’ve got lined up.

The thing that I’m most thrilled about is a class called Datacenters: Planning, Expanding, and Migrating. Finally, a physical infrastructure class! Holy Cow! If I could make it, I would sign up for this class in a heartbeat. How many times have you needed to make changes to the infrastructure, and were told, “Sorry, we can’t have any downtime”. I’m doing a big migration soon myself, and I would love to be part of this class. This alone may be worth the trip.

A course that sounds intriguing is Black Magic: Linux Troubleshooting and System Administration. I’ve talked to the instructor, John Billings, on IRC, and he really knows his stuff. I’m hoping that there are some notes or slides from this class (or maybe you could write a guest blog / review of the class, and I could post it here).

There are a ton of other courses as well. Check out the course list and decide what you want to take. As always with these things, the hard part is narrowing it down.

So go to Linux Fest and have a good time for me. Make sure to bring back all kinds of stories and let me know how it goes. Oh, and LOPSA is looking for volunteers to help them man the booth there, so if you want to volunteer some time, comment on this entry (or drop me an email) and let me know. We appreciate any help we can get!