LCA2014 Sysadmin Miniconf Presentations
Below is a list of presentations accepted for the LCA 2014 Sysadmin Miniconf. The speaking schedule is still in development and subject to change (until the day of the Miniconf!), but all these presentations are confirmed (apart from any last minute issues causing them to be withdrawn.)
The CFP for 2014 is now closed.
Presentation Summary
(Links sorted by presentation title; Abstracts below are sorted by first name of presenter.)
- Ansible and Vagrant
- Better Living Through Statistics
- Coming up to speed on Chef within AWS
- Custom equipment monitoring with OpenWRT and Carambola
- Dualstack Firewalling with husk
- How to keep track of puppet with Foreman
- Howto Reliably Replicate Block Devices, even over Long Distances
- Open Sourcing your entire Puppet configuration
- Optimizing Linux memory usage
- Providing a continuously updated ITIL CMDB
- "Quality of Service", a common misconception
- RatticDB
- Running virtualized Galera instances
- Scalable SOE deployments
- Scaling Graphite
- The Six Stages of systemd
- Thinking outside the box
Full Abstracts
(Sorted by first name of presenter.)
- Providing a continuously updated ITIL CMDB - Alan Robertson
The Assimilation Project integrates zero-network-footprint discovery with extreme-scale monitoring in a single package.
The result of having continually up-to-date discovery is that you have created an ITIL CMDB that's always correct. In our case that means creating a graph model of the service, system, switch and dependencies in your infrastructure. Proper discovery finds what your systems are doing, and also finds the systems you've forgotten about. This is important because 30% of all break-ins come through forgotten machines. By combining discovery with monitoring, it's also easy to see what things you've forgotten about. A great many other benefits com from being able to discover what your systems are doing, and have that rich source of information available in a central database. This talk will concentrate on the benefits of having an up-to-date discovery database.
Alan is a long-time open source developer. He founded the Linux-HA project and led it for 10 years. More recently, he founded the Assimilation project which provides groundbreaking integrated IT discovery and monitoring. Alan is a frequently requested speaker, having spoken in over 30 conferences all over the world on Discovery, Monitoring, High-Availability and Linux.
- Custom equipment monitoring with OpenWRT and Carambola - Andrew McDonnell
Sometimes as a sysadmin you may need to monitor something that doesn't quite "fit". Perhaps you need to keep an eye on the backup generator, read status from an RS-485 connected control system, or talk to something exotic using i2c. But you want the familiarity and control of your Linux operating system. And management has imposed a tight budget, putting that dedicated $1+K monitoring system out of reach.
There are plenty of hacks on the web, mods to home routers and such, but you need engineering, documentation and repeatability. The Raspberry Pi, really aimed at the education/hobbyist market, has limited onboard networking.
In this presentation I will show how to meet all the above requirements using inexpensive, pocket sized, commercially available and supported embedded computers, running the OpenWRT Linux distribution for low up-front cost and only a few hours of time invested.
Topics covered include a brief overview of commercially available devices, configuring and automating installation of OpenWRT, connecting to i2c and other devices, and integrating monitoring into an existing system by means of SNMP, email or other mechanisms.
Andrew McDonnell has been a software engineer for well over a decade, having spent many years before that hacking code since receiving his first Commodore 64 at age 12. He has significant experience programming in C++, Java and Python and a multitude of scripting languages. Outside of family and work he sometimes has time to play with his collection of 8-bit and PC/XT-vintage computers; computing and electronics has always been his passion. He intermittently maintains a blog at blog.oldcomputerjunk.net sometimes posting how he solved a problem in the hope it may be useful to someone else.
- Ansible and Vagrant - Daniel Hall
Last year I talked about using Ansible to do configuration management. This year I'll be explaining how to utilise these two tools to enable you to test your infrastructure on your own laptop.
Ansible written in Python and reuses tools you probably already have set up such as SSH it aims to simplify config management, deployment tasks and ad hoc changes. Ansible is currently undergoing heavy development and getting exciting new features every release.
Vagrant is a tool for building development environments on your laptop that look similar to your own production environment. It was designed to reduce the amount of 'works on my laptop' issues that operations and developers often deal with.
This talk will involve a live demonstration of installing and setting up a python web application that uses a database backend.
I have been working as a systems administrator for the past 5 years, three of them at my current workplace realestate.com.au. I spoke last year about "Ansible Configuration Management" and am currently working on a book on the same subject.
- Scaling Graphite - Devdas Bhagat
This is a user experience report on how Booking.com scaled it's Graphite installation from scratch to 20 million metrics/minute.
Graphite is a useful tool for generating charts from time-series data. Most sysadmins who are familiar with tools like MRTG or Cacti will find Graphite to be a huge step up in user experience due to the additional features in the API. For people not familiar with Graphite, http://graphite.readthedocs.org/en/latest/ should give a brief overview.
This talk will cover the scaling aspects of Graphite, and various problems encountered in doing so in starting from scratch and becoming one of the largest installations in the world.
The target audience for this is sysadmins at all skill levels. The audience is expected to get an overview of how systems can be scaled, real world limits and why horizontal scalability matters.
Devdas is a sysadmin working for Booking.com in Amsterdam, with over a decade of operations experience. He has worked as a sysadmin, a developer, a DBA and dabbled in network engineering. He has spoken at the sysadmin and database miniconferences at LCA before, as well as at various other Linux and networking conferences.
- Open Sourcing your entire Puppet configuration - Elizabeth Krumbach Joseph
In a very useful trend for all of us, many systems administrators now open source and share the Puppet modules they use in production.
The OpenStack Infrastructure team takes this one step further and not only open sources our generic modules, but also publishes our full configurations that specifically pertain to the OpenStack project infrastructure so people get real world examples of the modules being used in production. We leverage Hiera to keep private data private and publish the rest on a public git repository that is mirrored to GitHub.
This is a deeper dive into the Puppet-specific portions of the infrastructure than I'll be covering in my main LCA talk, which will be more of a generic overview of doing systems administration in the open.
Elizabeth Krumbach Joseph is an Automation and Tools Engineer at HP working on the OpenStack Infrastructure team. She also has contributed extensively to the Ubuntu project and currently sits on the Ubuntu Community Council.
She has given the keynote at Fosscon, and spoken at various open source conferences including OSCON, SCaLE and UbuCon.
- How to keep track of puppet with Foreman - Glen Ogilvie
This talk will cover using Foreman with puppet to help you understand and monitor what is going on, along with providing a nice interface to configure nodes. Foreman as an external node classifier solves many of the difficulties of managing variables and parametrized classes with puppet.
I aim to cover the following topics:
- What the Foreman does
- Using Foreman with puppet
- How to install (two slides only)
- Example using Foreman to manage a system with puppet ? Local demo
- Foreman Parametrized classes - when to use them and how
- Foreman vs Hiera
I've been a passionate supporter and contributor of open source since Linux came on floppy disks and I've been working with Linux professionally since 2001. My focus is web infrastructure and spans sysadmin, coding and design work. I've attended every LCA since 2005 and am an advocate for this forum in both professional and personal networks. Ever year I look forward to the sharing and learning that is synonymous with this event.? I work for a Linux infrastructure company (www.oss.co.nz) where I am the Practice Lead for our Automation and Management Practice. This practice deals significantly with technologies such as RHEL, Red Hat Satellite, AWS, Puppet, Splunk. It also spans many other areas of Linux, such as security, virtualization and monitoring. In additional to this I run the local LUG in Auckland and am involved with a number of open source communities, including Mageia and TangleBall (a NZ maker space).? I'm an active facilitator and speaker in Auckland and have speaking experience at LUGs in Auckland, Tokyo and Hong Kong.
Target audience: This talk is aimed at people that are beginners with puppet, or have not used Foreman before.
- Better Living Through Statistics - Jamie Wilkinson
"#monitoringsucks." I can appreciate the sentiment. I used to use Nagios too! However, I can't agree that monitoring sucks. Monitoring is awesome!
We observe our systems to understand their behaviour. We do this in various ways like reading logs or taking measurements and, more recently, storing them in a timeseries database such as collectd or graphite.
However, the standard practice for alerting is still to check the measurement at the time that it is taken and it is this "check script" model of monitoring that is long due for an overhaul.
In this talk, I'll start over from first principles: what do we want monitoring to do for us? I'll deconstruct the "check script" and rebase its essentials on the humble timeseries. I'll demonstrate simple aggregation and apply some maths and stats to show how monitoring can scale to cluster size without increasing maintenance costs.
With worked examples based on real-world situations, you'll learn techniques that you can use to improve the signal to noise of your monitoring systems.
Jamie is a site reliability engineer, working on cloud storage at Google. His interests include monitoring and automation.
- Coming up to speed on Chef within AWS - Josh Mesilane
Josh shares his experiences and challenges encountered in coming up to speed on Chef managed environments, having previously only managed puppet environment in the past.
Using comparative examples of challenges encountered Josh will show an objective view of the two tools, and then also share how chef can be integrated with other tools in order to extend it's functionality and scope in order to ensure the agility, scalability and consistency of platforms.
This is not a Chef vs Puppet talk, but an objective comparison of the two tools, and Josh's thoughts on the two tools as a whole.
Josh is a Systems Administrator of 10 years, recently specialising in the Networking and Linux space. Previously presented at local Infracoders meetup groups in Melbourne, presenting on AWS integration as well as presenting a lightning talk at the 2013 Arduino Miniconf.
Josh has previously worked for companies such as Serraview, CSG, PlayUp, Luma Pictures and now working for Base2Services, Josh has a passion for making his life easier as a sysadmin. Tools such as Chef, Puppet, Jenkins, Git drive Josh to automate and ensure reliable, repeatable results to ensure availability and reliability of production systems in both Cloud and traditional Bare Metal Applications. Josh is all about using open-source software, and whilst not a developer is keen to contribute back to the community by sharing my experiences and challenges with others working with these applications.
In Josh's spare time he has "too many hobbies" including Hardware Hacking (Arduino/AVR/Electronics), Cars, Horses, Miniature Trains, Engineering, Fabrication, Brewing Beer and many more interests.
- "Quality of Service", a common misconception - Julien Goodwin
There're many things people believe about networking, some of them are even true.
This talk is about many of the things that aren't.
Julien is a Network Engineer at Google Sydney, working day-to-day maintaining one of the worlds largest IP networks. In the past he has worked as a (primarily Linux) Sys Admin on educational networks, and in small businesses. He has a long history working with the Australian FOSS community and was a member of the LCA2008 team, and has spoken at several previous LCA sysadmin miniconfs.
- RatticDB - Elizabeta Sørensen
- Scalable SOE deployments - Matthew Cengia
PrisonPC is designed as a secure PILS (Prisoner Information and Learning System) which, among many other things, is based around PXE-bootable SOEs running on diskless desktops. This talk is centred around how we build and upgrade those SOEs, and how we assign them to "realms" allowing us to alter large quantities of desktops' SOEs at once.
Depending on time constraints, I may also delve into some of the security features we include, such as whitelisting optical media for users, and blacklisting USB storage.
I started playing with various Linux distros in 2006.
In the same year I began an Advanced Diploma of Computer Science at RMIT TAFE, which got me a job at Cyber IT Solutions in mid-2008, where I've worked ever since.
After finishing my Diploma at the end of 2007 I began a Bachelor's Degree in Computer Science (Applications Programming) with RMIT. I both studied and worked at Cyber IT part-time until I completed the degree in mid-2010.
At Cyber IT I specialise in systems administration and spend much of my time presently testing and packaging our PrisonPC product (http://prisonpc.com).
When I'm not sitting in front of a computer monitor, it's usually because I'm running a Scout troop north-west of Melbourne; I've been a leader there since mid-2008, after a 2-year break before which I had been a youth member there since 1996.
- Dualstack Firewalling with husk - Phillip Smith
One of the hurdles faced when implementing IPv6 in a dual-stack network environment is firewalling. IPv6 requires duplication of certain network services (dhcpd I'm looking at you), and firewalling is often (or should be!) one of those. The segregation of iptables vs ip6tables in Linux requires a duplication of effort to maintain firewalls, both host and perimeter based. But have I got an offer for you!? My open-source project 'husk' is a clean, simple wrapper around both iptables and ip6tables, allowing management of both IP versions in a common configuration, with some other useful features on the side to simplify your firewall management through re-use and a flexible, human-readable DSL. 'husk' is not only for dual-stack networks and can be used in regular IPv4-only (or IPv6-only) networks.
Phil Smith is a System Administrator from Melbourne, currently masquerading as an IT Department for an Australian manufacturing company but with past roles in Managed Security. Hawker of IPv6, Linux and Security, when he's not on the computer you'll find him at his local Fire Brigade where he has been a volunteer fire-fighter since 2001, or out and about in his 4WD either down and dirty in the mud, or high and dry exploring Victoria's High Country.
- Running virtualized Galera instances - Raghavendra Prabhu
In this talk, Percona XtraDB Cluster (PXC) based on galera will be considered. PXC has been used for synchronous replication in MySQL-verse to create a cluster of homogenous mysql nodes while maintaing data and application level compatibility with MySQL. Both virtualization and clustering share a similar set of goals - high availability (HA) et al.; yet, can cater to a diverse requirements. This commonality can certainly be exploited. Virtualization bundled with clustering is not new, this has already been done in storage ranging from GlusterFS at filesystem level to Sheepdog at hypervisor (QEMU) level. Now, this can be further extended to applications running inside the guest. While this integration may not be as well-knit as the others, a virtualization-aware cluster can take advantage of it. Virtualization is no longer a pariah it once was (in datacenters), now that frameworks like OpenStack are becoming common. Warehouse style computing is dominating data centers, hence clusters or cluster-based applications (Hadoop, Manta et.al.) are too. This talk will glean over the areas where this will work and be beneficial.
It would be unreasonable to assume that a node can have only one instance (being one node of a multi-node cluster) of PXC running, since we are starting to have bulkier hardware which a single head application may not be able to fully saturate. While it may still be possible to run multiple instances of the application itself, from a deployment and HA perspective it is better to have them in their own VMs. At the same time, since we are running a cluster, it wouldn't be wise to have all on same server and instead deploy over LANs or even WANs. Hence, the talk will go over deployment scenarios/flows starting from spawning a new node to live migrations and virtFS.
Raghavendra Prabhu is the Product Lead of Percona XtraDB Cluster (PXC) at Percona. He joined Percona nearly 2 years ago. Before joining, he worked at Yahoo! SDC at Bangalore for 3 years as Systems Engineer, primarily dealing with databases (MySQL).
Raghavendra's main interests include databases and operating systems, specifically linux kernel. He also likes to contribute and has contributed code upstream to several FOSS projects -- for more details on that visit http://wnohang.net/code.
In his spare time, he likes hacking code, reading books on his kindle, playing games over PC and rest of the time spending time over IRC. Visit wnohang.net/about and/or about.me/raghavendra.prabhu for more.
- The Six Stages of systemd - Rodger Donaldson
Everyone knows about the five stages of grieving, but what about the six stages of systemd? Like many sysadmins, I've looked askance at the now-impending apocalypse of pervasive systemd adoption, and I'd like to walk through my own six stages of systemd: Denial, Anger, Bargaining, Depression, Acceptance, and Enthusiasm.
I'll be talking about the fears around this huge change in our infrastructure, what I've learned about systemd that's taken me to the enthusiasm stage, some of the lessons I've learned about both myself and the Linux community, and some suggestions about how some of the hysteria could have been avoided.
Rodger has been wrangling Linux in commercial environments since before it got trendy. He has run Linux on everything from cobbled together skunkworks PCs to get things running to mainframe-class hardware.
His background is in corporate environments, working in the media and finance industries. As well as the Holy Path of systems administration he has worked on the dark side of development; please don't ask him about TCL; it can cause fits. A large part of his career has been spent making broken things work faster, or at all.
- Optimizing Linux memory usage - Sander van Vugt
Too many system administrators still think that swap is bad. Too many Linux system administrators still also think that cache and buffer memory can be cleared without any problem whatsoever. There are even still Linux administrators around who think that virtual memory is the same as swap memory.
This talk is going to change your perspective on Linux memory management. You are going to learn how the Linux kernel uses different kinds of memory and distinguishes between active and inactive memory to decide what exactly to do with available RAM in your server. Based on this knowledge you'll be able to profile your server's memory usage and optimise it accordingly.
Sander van Vugt is an independent Linux consultant and instructor. He also is an author of numerous books on Linux system administration topics, including "Red Hat 6 Real Life Administration". He has also presented on previous LCA conferences about High Availability clustering and SELinux.
- Thinking outside the box - Steven McDonald
It's really great when everything goes according to plan and just works itself out. All your careful, meticulous plans come to fruition, all your long hours spent testing and evaluating different methods finally pay off and your long-awaited solution goes into production.
Unfortunately, we do systems administration in the real world. Unexpected things happen, people make mistakes, hardware fails, software has bugs, and you're going to get woken up at three in the morning by a monitoring alert that needs fixing right now just when you don't have time to plan.
At times like these, it's necessary to have a broader range of techniques at your disposal than what you've tried and tested in advance. You need to be able to come up with innovative solutions that hold things together at the drop of a hat. You need to be able to think outside the box.
In this talk, I will introduce some techniques I've learned as a sysadmin for coming up with unorthodox yet functional solutions in a hurry, as well as some examples where this has saved a seemingly hopeless situation.
Steven is a systems administrator and geek in Sydney, Australia. He currently works for Anchor as part of a large team of sysadmins, and in his spare time makes the occasional small contribution to an open-source project.
He has had experience speaking at local gathering in Sydney, such as SLUG and DevOps meetings, mainly about whimsical topics. He has also spoken about more serious subjects internally within Anchor, and will be giving a semi-related talk in the main conference at LCA 2014.
- Howto Reliably Replicate Block Devices, even over Long Distances - Thomas Schöbel-Theuer
Synchronous replication of block devices works only well over crossover cables (rack-to-rack). If your network is unreliable, or if distances are higher than ~50km, you will need asynchronous replication in order to avoid serious operational problems, such as final damage of your mirror in case of rolling disasters.
This talk shows you how to setup and operate MARS Light, which is similar to DRBD at the surface, and explains the internal differences to DRBD. I will also report some operational experiences from our 1&1 datacenters.
https://github.com/schoebel/mars
Thomas Schöbel-Theuer was an assistance professor in Operating Systems at the University of Stuttgart, and later a substitute professor in Operating Systems and Databases at the University of Applied Sciences in Bingen. He is the original implementor of the dentry cache in the Linux kernel.
He is currently working on asynchronous block storage replication (MARS project) at 1&1 Internet AG, Karlsruhe, Germany.
RatticDB is an open source password management system built using django. Going through security circus and some of the main attractions regarding password security.
Elizabeta Sørensen is a full-time university student, systems administrator with several years experience in vfx and managed services.