LCA2015 Sysadmin Miniconf Presentations
Abstracts for the presentations for the Linux.Conf.Au 2015 Syadmin Miniconf are listed below. The speaking programme also includes links to the slides provided by the presenters.
Presentation Titles
(Links sorted by presentation title; Abstracts below are sorted by first name of presenter.)
- Alerting Husbandry - Julien Goodwin
- Configuration Management - A love story - Javier Turegano
- Corralling logs with ELK - Mark Walkom
- Developments in PCP (Performance Co-Pilot) - Nathan Scott
- Documentation made complicated - Eric Burgueno
- etcd: distributed locking and service discovery - Brandon Philips
- FAI -- the universal deployment tool - Thomas Lange
- Healthy Operations - Phil Ingram
- Linux at the University - Randy Appleton
- Managing microservices effectively - Daniel Hall
- Mass automatic roll out of Linux with Windows as a VM guest - Steven Sykes
- Security options for container implementations - Jay Coles
- Untangling the strings: Scaling Puppet with inotify - Steven McDonald
Full Abstracts
(Sorted by first name of presenter.)
- etcd: distributed locking and service discovery - Brandon Philips
etcd provides easy to use distributed locking and service discovery. It has an accessible HTTP+JSON API that exposes a powerful set of primitives inspired by projects like Google's Chubby and Apache Zookeeper. This talk will cover the underlying consensus algorithm, the architecture of the code, introduce the API and survey the libraries and tools that have been built by the etcd community. We will also dive into how etcd is used to power the clustered scheduling on CoreOS.
About Brandon Philips:
Brandon Philips is helping to build modern Linux server infrastructure at CoreOS. Prior to CoreOS, he worked at Rackspace hacking on cloud monitoring and was a Linux kernel developer at SUSE. In addition to his work at CoreOS, Brandon sits on Docker's governance board and is one of the top contributors to Docker. As a graduate of Oregon State's Open Source Lab he is passionate about open source technologies.
- Managing microservices effectively - Daniel Hall
We have all heard how Microservices are changing the way people are building applications. They allow developers to build strict componentization between services and make things much easier to maintain in the long run.
If you are anything like me you are probably worried about the impact of running lots of very small service that have different requirements and need to scale independently. Even with cloud services such as EC2 this can either be easy and costly or complex and cheap.
Luckily there is a way. Using Docker to package up your applications, Marathon to deploy them and Mesos to manage your resources you can easily manage lots of different services even if they have radically different requirements.
This talk is aimed at systems administrators who are grappling with the complexities of deploying web services or want to get a high level overview on how Docker, Marathon and Mesos can work together to make your life easier.
About Daniel Hall:
After spending the last four years at REA doing Systems Administration, web page performance and deploying cloud based services, I have moved on to managing cloud services for IoT (Internet of Things) company LIFX. When I'm not playing with pretty lights I spend my time working on RatticDB.
I also wrote the first book on Ansible, and currently in the process of updating it to a second edition.
- Documentation made complicated - Eric Burgueno
Documentation is probably one of the most tedious tasks we have to do as sysadmins. We all acknowledge and say out loud that keeping good documentation is as important as maintaining the system, yet deep down we all fall short when it comes to it.
Are you confident enough that you can rebuild a server from scratch should all other alternatives fail? Will it really be exactly as it would be after restoring from the latest snapshot or tape backup?
We have at our disposal multiple tools and alternatives when it comes to write documentation up, but they all have their shortcomings: Pure text files are simple and easy to use but extraordinarily limiting. Word/OpenOffice/LibreOffice/etc allow for rich-text formatting and images but really, who writes server documentation in .doc/.odt files? Puppet and Chef are good for configuration management but not really meant to be human-readable.
But regardless of the tool you use, there are two major problems that all documentation faces: outdatedness (the documentation doesn't reflect the current configuration of the server), and inconsistency (the documentation differs between servers that are meant to be identical).
In this presentation I will show you a couple of tips and tricks I have been using to document the environment that I manage. Using MediaWiki and few intermediate level features, I have tried to tackle those issues above.
If time allows, I will also show you how to export the entire wiki to a single PDF, that you can keep handy in case disaster strikes.
About Eric Burgueno:
Eric Burgueño is a geek that went to law school and works as a UNIX Systems Engineer instead. He's been part of big companies like EDS, HP and IBM; but now is a team of one in the scientific research world. He likes doing presentations at work to explain what lies behind the curtain to non-IT people.
- Configuration Management - A love story - Javier Turegano
Who hasn't been in love with some sort of Configuration Management/Automation/Devops practice in the last few years? Difficult not to, as probably some of the hottest topics/practices in IT are related to them somehow.
In this talk I will be telling four different "love" stories about configuration management:
- The first love
- The "other"
- Engaged!
- 7 sisters for 7 brothers
Wait a minute, no buzzwords in your abstract? No, you'll have to come to the talk to find out but I can hint that there will be some well known open source tools involved.
This talk is suitable for all audiences, no adult content is included.
About Javier Turegano:
Javier Turegano is an IT engineer whose passions are open source, web operations and IT leadership.
Currently based in Melbourne he leads the Infrastructure team in REA Group, which hosts some nº 1 realestate advertising sites internationally (realestate.com.au, casa.it, athome.de, etc...). Previously he played different roles including systems architecture lead, project manager and sysadmin in one of the first Open Source consultancy firms in Spain which worked with the objective of empowering Public administration organisations by using Open Source technology.
His love for knowledge sharing and his sense of community commenced in his early years at University where with other enthusiasts he founded a local Linux user group (Linux Albacete) and continues today with different levels of collaboration within the community.
- Security options for container implementations - Jay Coles
Containers have a reputation for being a lower-security option than full hardware virtualisation. This is primarily due to Linux exposing its full userspace API to the container and the processes contained within, including the parts that are not container-aware.
The Linux kernel already has a wide variety of security mechanisms for isolating and restricting processes on a traditional system. As the purpose and scope of a container can be much more narrowly defined than that of a traditional system, these mechanisms can be used to develop tailored security solutions to increase the security of a container, even in the case of a security leak.
This talk is intended for people working with or developing container solutions. Attack models and countermeasures will be presented to show how you can slow down and stop a rogue container from taking control of your machine.
About Jay Coles:
When not working for Anchor Systems, Jay is hacking away and doing stupid things in both Linux and Python. If it's a bleeding edge feature you can be assured that Jay has played with it. In his spare time he enjoys writing parodies of other self referential bios.
- Alerting Husbandry - Julien Goodwin
Adding new alerts to monitoring systems is often easy, but often old alerts don't get any attention.
This talk covers several aspects of "alerting husbandry", including updating thresholds, maintaining playbooks, and more.
About Julien Goodwin:
Julien is a Network Engineer at Google Sydney, working day-to-day maintaining one of the worlds largest IP networks. In the past he has worked as a (primarily Linux) Sys Admin on educational networks, and in small businesses. He has a long history working with the Australian FOSS community and was a member of the LCA2008 team, and has spoken at several previous LCA's and Sysadmin Miniconfs, most recently in 2014.
- Corralling logs with ELK - Mark Walkom
Still grepping logs across tens or hundreds of nodes using multiple, single-use scripts? Tired of having to deal with many "standard" log formats? Want simple, powerful, Open Source powered log analytics with just a few packages? Of course you do!
This talk provides an introduction to using the ELK stack - Elasticsearch, Logstash and Kibana - for getting the most from your log data, whether it's system logs, mail logs or even that custom log format you hate dealing with.
We present a brief introduction about what makes Elasticsearch so easy and fun to work with, how Logstash works its magic to parse and enhance data, and finally how to visualise the results with Kibana. Along the way we'll share some tips on getting the most from the ELK stack as well as some handy third party tools that make life easier.
About Mark Walkom:
Mark is the first Australian employee for Elasticsearch, where he works as a Support Engineer making our customers lives better. He started the Sydney Elasticsearch user group and has previously presented the awesomeness that is the ELK stack to the NSW chapter of SAGE-AU.
Outside of work he has a passion for BBQ, beer, bacon and Star Wars LEGO.
- Developments in PCP (Performance Co-Pilot) - Nathan Scott
The Performance Co-Pilot (pcp.io) is a proven performance analysis tookit for production environments. The project is mature - over 20 years since its original inception - and is especially attractive for those tackling harder systems-level performance problems.
This talk will focus on recent development activity in PCP. Since being adopted at Red Hat, the project has flourished with many new developers, new users, interesting deployments, and many new feature requests!
In particular, its support for analysing containers without a need to install anything into each monitored container will be discussed. We'll explore each of the kernel features that back the userspace container concept, the performance metrics that the kernel makes available for each container, and how this information is extracted and represented in PCP.
About Nathan Scott:
Nathan Scott is one of primary developers in the Performance Co-Pilot project, and currently works at Red Hat in the performance tools group.
Nathan has been involved in PCP development and use in production environments for over 15 years. He has several years of Linux kernel and userspace tool development experience in the storage space.
- Healthy Operations - Phil Ingram
I work remotely at my home office in Operations for a rapidly growing company. Being remote in a polar opposite timezone and geographic locale from our head office has confronted me with several extra challenges above and beyond my day to day job. Adding in dealing with a booming companies growing pains there has been more of an impact on my life and physical condition than any other job I’ve ever had.
Operations is one of the more stressful departments to work for in IT as you are constantly reacting to issues, fighting fires and ultimately being the bottom line. Day to day stresses take their toll and it’s very hard at times to juggle all of the responsibilities while still maintaining a normal and healthy work/life balance. I’ve had some big highs, but also some big lows. This is my story of how I came to be in Operations, some of my victories and defeats and what I’ve learned to manage my day to day to keep this the best job I’ve ever had.
If you have ever worked remotely or had to deal with unrelenting or large stresses then come along, listen and empathise or commiserate. I hope to provide some insight into the stressful side of IT, how some of the badness occurs and what can be done to manage yourself so that you can enjoy the job you love to do.
About Phil Ingram:
Phil started playing with GNU/Linux in 2003 and has hacked away at it ever since. Nearly a decade on he now works in devops administering thousands of servers for a global company from his home office. He's passionate about freedom software, good food, beer and wine. His background is in systems and services, bash/perl/ruby scripting and is slowly trying to learn C. Why C? You'll have to ask him yourself.
- Linux at the University - Randy Appleton
Everyone knows that Universities use Linux. But how, and for what? This talk describes how three different universities (a world class technical school, a regional university, and a smaller school) each use Linux. It describes what seems to work well, and what could be improved on, and where Linux doesn't fit at all. Although there are numbers and charts, this is a fast paced, fun talk with plenty of war stories.
About Randy Appleton:
Randy Appleton is a university professor with a wide variety of Linux experience dating back to the 0.9 kernel days. These days he teaches system administration, operating systems theory, and practical programming. When not on the job, he flies a small airplane for fun.
- Untangling the strings: Scaling Puppet with inotify - Steven McDonald
Every time you make a change to your Puppet manifests, you need to signal the Puppetmaster to initiate a reparse. To accomplish this, you have two choices.
One, you can restart the Puppetmaster to have it reparse everything from scratch, thus picking up your changes immediately. This works well for small deployments with occasional configuration changes, but once your node count starts numbering in the hundreds, every restart costs you precious agent runtime while the Puppetmaster reads all of its manifests.
Two, you can use Puppet's built-in, coarsely-grained cache mechanisms to have a full reparse triggered after a certain amount of time. This allows you to avoid restarting the Puppetmaster, but imposes the trade-off of having to wait some time for your changes to take effect.
That's a trade-off we weren't willing to make. This talk will describe how we solved the problem in our own way, leveraging the Linux kernel's inotify subsystem to provide a running Puppetmaster with notification of changed manifests. I'll also explain the other options we considered, why we settled on the solution we did, and the compromises we had to make in order to accomplish it.
About Steven McDonald:
Steven is probably the same person he was at last year's LCA. He works as a systems administrator for Anchor, and in his spare time enjoys submitting bug reports and writing self-referential biographies.
- Mass automatic roll out of Linux with Windows as a VM guest - Steven Sykes
The Computer Science and Software Engineering department at the University of Canterbury makes extensive use of Linux in its labs and on staff and post graduate computers. However there are times when Windows is required for some courses.
Historically computers had been setup with dual boot Fedora Linux and Windows. While this provided access to both operating systems, the nature of dual boot caused problems. The dormant OS on machines would get behind with software installs and updates that needed to be applied, only catching up when that operating system is manually started.
Machines could get quite behind with updates as students have been observed to sit down at a computer, see that it was running an OS that they didn't want to use and move onto a computer that was. This was despite them knowing they could reboot into the other OS. While automating a change in GRUB to boot Windows at off peak hours to ensure updates occur is easy, reversing that from Windows so Fedora got its updates is problematic.
For the 2014 academic year a new method was used where virtual machines were employed instead of dual booting. The host OS became Linux Mint and Windows runs in the VM. This arrangement has the added benefit of exposing more students to Linux because it was always the default OS.
This talk describes the details of an automatic method of deploying a customised Linux Mint install on the department's computers along with Active Directory authentication with account information drawn from a Linux LDAP server, post install changes, and other tricky problems that got solved along the way.
Additionally, the method of automatically deploying Windows 7 in a virtual machine using custom Linux scripts is described. Detail is shown on how the virtual machine is deployed to computers using VirtualBox as the hypervisor and how the Windows machine account is inserted into the desired Organisational Unit in Active Directory. Simultaneous and safe use of USB sticks and optical media from both operating systems at once is also demonstrated.
The result is a system that encourages use of Linux as the main OS but allows simultaneous use of Windows without any of the problems previously encountered with dual boot.
About Steven Sykes:
Steven has been at the University of Canterbury since 1999 and is a Linux and Windows system administrator in the Department of Computer Science. He prefers a world of open source and enjoys tinkering with low powered Linux systems. Every computing device he owns runs Linux somewhere, including his car.
- FAI -- the universal deployment tool - Thomas Lange
FAI, the Fully Automatic Installation is a network installation system for the installation and configuration of the operation system and all your applications on all your hosts. The whole installation only takes a few minutes without any interaction necessary.
The FAI project startet in 1999 as a bare metal provisioning tool for Debian GNU/Linux only. Today it's also used for deploying different Linux distributions like Ubuntu, CentOS, Scientific Linux or Suse on real hardware or virtual hosts. For FAI there's no difference in installing a real machine, a virtual machine, setting up a chroot environment or creating a Live CD.
Configuration files are shared among groups of similar computers using the class concept, so you need not create a configuration for every new host. Besides network installations, FAI also supports installation from CD or USB stick and can be extended easily.
The talk will show why FAI is the universal deployment tool.
About Thomas Lange:
Thomas Lange, the main author of FAI, has now been working as a system administrator at the University of Cologne for over 20 years. He has also been a Debian Developer since 2000.
In 1999 his laziness prompted him to begin work on the Fully Automatic Installation project.
He has previously given several talks and tutorials about FAI at many conferences, including Linux Kongress, LCA 2006, FrOSCon, DebConf, SANE, FOSDEM, UKUUG, Cebit, LinuxTag.