LCA2021 Sysadmin Miniconf Presentations
The abstracts for the presentations accepted for the Linux.Conf.Au 2021 Syadmin Miniconf are listed below.
The LCA2021 Sysadmin Miniconf Programme links to all the slides and videos that the speakers made available.
(Links sorted by presentation title; Abstracts below are sorted by first name of the first presenter.)
- API != REST - procmail to the rescue - Bruno Cornec and Frédéric Passeron
- A re-introduction to s3fs - Andrew Gaul
- Building my own border router + a wireguard love story - Arjen Lentz
- Building Raspberry PI Supercomputers - Federico Lucifredi
- Getting started with Docker and Swarm - Matthew Cengia
- Getting started with LinuxBoot Firmware on AArch64 Server - Naohiro Tamura
- MySQL for System Administrators - der.hans
- Rootless containers with Podman - Steven Ellis
- The worst outage I never caused - Julien Goodwin
- “They’re just taking home any laptop they could find!” - Gyle dela Cruz
- Tips and Tricks for Managing and Administering Ceph Clusters - Michael Hackett
S3 file systems are a popular interface to object storage despite their leaky abstractions and performance pitfalls. In this talk we will explore s3fs, one of the most popular FUSE file systems, and when it is an appropriate solution. We will compare it with NFS and also discuss how s3fs has evolved over the last ten years.
About Andrew Gaul:
Andrew Gaul has contributed to a variety of block, file, and object storage systems at various startups. He is a committer to s3fs, created S3Proxy, co-founded Bounce Storage, and was vice president of Apache jclouds. Andrew has previously presented at ApacheCon, Ohio LinuxFest, OpenStack Summit, and various meetups. He lives in Tokyo and works at Google.
I'm not a sysadmin, but sometimes I have to do stuff - and I have to do security, for a living but also to keep the home network safe and performant.
Consumer routers suck: in performance, connection reliability, security, and ability to have a sane configuration.
OpenWRT and DD-WRT are nice, but sometimes one has this weird urge to just do it again, from scratch and using open bits whereever possible.
Maybe just because.
Anyway, I decided to order two PC Engines APU 4d4 boards from the awesome Pascal in Switzerland. These are cheap-ish little fan-less single board computers with a 1 GHz AMD GX-412TC 4 core CPU (64-bit, of course), 4GB RAM, 4x Gbit Ethernet ports, and plenty of other connectors and options. I put in a PCIe SSD card.
I first talked to the board's BIOS via a serial cable (was already up-to-date, good), and installed Debian 10 on it (with some minor hackery) from a USB stick.
And behold, my NBN HFC connection is now stable, and faster.
Maybe you would like to do something similar, or maybe you just want to pick up a few of the things that I've done - you can do any of this with a regular Linux box as well:
- A few of the ports are configured as a switch, using bridged network interfaces.
- One port talks to the NBN HFC modem with PPPoE connection and a VLAN, insisting it's actually the original ISP-provided device.
- IPv4/IPv6 native dual stack, each with subnets (I just have those, from when I ran a company from home)
- Outbound rate-limiting on the HFC connection to keep the stupid NBN happy.
- Appropriate kernel tuning via sysctl - seemingly quite necessary for dealing with funny traffic!
- A hand-crafted effective firewall, providing safety but also (when desired) insights in what fun tries to scan and gain access, how and from where.
- Configuring Unbound DNS to get rid of most ads on the LAN.
- Surviving a reasonably-sized DDoS or other attack without flinching too much.
- Using dynamic geo-blocking, again with options to gain insight.
- WireGuard VPN end-point, both for my own devices and for geo-tunneling (using policy routing) (* see below for a WireGuard love story)
- Optional Suricata intrusion detection/prevention.
- Being invisible for scans, if so desired.
Extra options are adding mobile data backup with an on-board SIM card (may do), and wifi (might do using my 2nd board).
In this talk I will show what it all looks like on the outside and inside, and go over the configuration - understandable for small league nerds.
So you can do this yourself, or just learn more about how the various bits work. (Some entertaining stuff-ups and anecdotes are also included.)
- A WireGuard love story
What if there was a VPN that only requires a few thousand lines of code, and lives inside the kernel?
Horays! Thanks to Jason Donenfeld, WireGuard is now available in recent Linux kernels, and otherwise easy to add.
But how to set it all up? The documentation is kinda all there, but mostly if you already know your stuff.
Tutorials abound, but some things have changed since. Aargh.
So now that the dust has settled, let's look at this from the non-whizz perspective. How to peer, or set up a server with clients, or a network tunnel with policy routing. I'm not the expert, but I've made it work and I can explain what I did.
About Arjen Lentz:
Arjen has done lots of things, from C coding and MySQL documentation,training/consulting, to running a company, and instigating BlueHackers...
He is currently fully entertained as Chief Information Security (CISO) at Catalyst IT Australia.
These days, most people speaking APIs think REST API. It's ubiquitous, uses standard protocols and formats, based on giants shoulders...
For the sysadmin REST API like Redfish is a nice way to have a cross-manufacturer out of band management interface e.g.
However in some case, you have constraints that do not allow you to use a REST API. When your system is not reachable directly using the HTTPS protocol e.g. In that case, you can still use an API of course, but based on other standards, such as the venerable STMP one !
Our use case, a training on demand workshop system, involves a Web front-end to manage user registration to run jupyter notebooks hosted on a back-end hosting the jupyterhub instance as well as all the companion systems needed to perform the various workshops proposed (on Redfish, Git, Rust as visible on https://hackshack.hpedev.io/workshops)
So to make it work flawlessly, we used an SMTP based API, the front-end generating the SMTP content and the back-end using procmail, some scripts and ansible playbooks to manage the setup of the user environment. Once logged on the platform, the user has acces to its own workshop content, with all the links to the other systems available to perform the actions. Why SMTP ? Well our needs were small enough to avoid developing a full REST one (even if if have also one for the front-end), we benefit from the asynchronous aspect of e-mail for free in the management of requests, and it's fun to use old methods to show young engineers that there is more than one way to do it ;-)
Interested ? Well come to hear how we did that and we'll show you how it works, from the automatic deployment of the platform up to the run of a workshop.
About Bruno Cornec:
Bruno Cornec is Engineer of the French Ecole Centrale de Lyon, France (1987)
He has been managing various Unix systems since 1987 and Linux since 1993 (0.99pl14)
Bruno first worked 8 years around Software Engineering and Configuration Management Systems (Build systems, Quality tools) in Unix environments. Since 1995, he is Open Source and Linux (OSL) Technology Strategist and Evangelist, initially for an HPE reseller and since 2000 for Hewlett Packard Enterprise.
Bruno is a contributor to various OSL projects: MondoRescue (2001), Mageia (2003), LinuxCOE (2006), Pause (2007), Tellico (2008), FOSSology (2008), collectl (2009), Ironic (2015), python-redfish (2015). He is also project leader for MondoRescue (GPL disaster recovery solution, 2005), dploy.org (GPL deployment server, 2006), project-builder.org (GPL build service, 2007), UUWL (LGPL/MIT Unix to Unix Wrapper Library, 2011), PUSK (GPL ProLiant USB Setup Key, 2012), python-redfish (Apache, 2015).
He has also been a board member of the AFUL and OpenStack-fr associations. He's president of the FLOSSITA association.
As part of his work he has made numerous presentations for Solution Linux/OWF/POSS in France, and WW at LinuxCon/Open Source Summit, RMLL/Libre Software Meeting, Linux.Conf.au, OSCON, Linux Symposium, Fosdem around various topics (High Availability, Deployment solutions, System management, Disaster Recovery, Package building, Cloud...)
Outside computers, Bruno also likes early and baroque music, singing and playing the recorder. He's married and father of 3 kids.
System administrators often need to help maintain MySQL instances. Sometimes they have databases with little or no DBA support. Learn some basic skills for maintaining MySQL databases. The same skills will help balance roles between sysadmin and DBA.
MySQL is ubiquitous and will be for many years. A little knowledge will go a long way to having a good MySQL setup. Sysadmins can leverage their command line skills when interacting with MySQL.
The presentation will cover *NIX and MySQL changes for good practices. Those skills will help sysadmins use and administer MySQL without needing to become a DBA.
- leveraging shell skills for MySQL
- shell environment for MySQL
- custom MySQL prompts
- MySQL backups and restorals
- choosing a database engine
- MySQL logs
der.hans is a technologist, Free Software advocate, parent and spouse.
Hans is chairman of the Phoenix Linux User Group (PLUG), Promotions and Outreach chair for SeaGL, BoF organizer for the Southern California Linux Expo (SCaLE) and founder of the Free Software Stammtisch. He presents regularly at large community-led conferences (SCaLE, SeaGL, LFNW, Tübix, OLF, TXLF) and many local groups.
Currently a Customer Data Engineer at Object Rocket. Public statements are not representative of $dayjob.
Mastodon - https://floss.social/@FLOX_advocate
Federico discusses what is required to integrate clusters of ARM SBCs, with a focus on Raspberry PI units due to their popularity, the software integration necessary to make them practical, what plumbing is necessary to easily configure nodes, and how to issue commands for cluster management. From the initial spotlight on cluster operations we transition to practical use, and briefly look at how parallel computing is utilized to solve numerical problems and how to code and run numerical workloads using the MPI interface.
This is a live tutorial with a running cluster (or two!), and is meant to be an introduction for those new to Linux clustering.
About Federico Lucifredi:
Federico Lucifredi is the Product Management Director for Ceph Storage at Red Hat and the co-author of O'Reilly's "Peccary Book" on AWS System Administration. Previously, he was the Ubuntu Server product manager at Canonical, where he oversaw a broad portfolio and the rise of Ubuntu Server to the rank of most popular OS on Amazon AWS. A software engineer-turned-manager at the Novell corporation, he was part of the SUSE Linux team, overseeing the update lifecycle and delivery stack of a $150 million maintenance business. A CIO and a network software architect at advanced technology and embedded Linux startups, Federico was also a lecturer for over 200 students in Boston University's graduate and undergraduate programs, and simultaneously a consultant for MIT implementing fluid-dynamics simulations in Java.
He is a frequent speaker at user group and conference events, notably the Linux Foundation's Open Source Summit, the O'Reilly Open Source Convention, The OpenStack Summit, All Things Open, Kubecon, DEF CON, LinuxWorld, and the leading SCALE and LCA Community conferences. Federico is a recognized expert in computing performance issues and consults with Standard and Poor's clients in Free and Open Source Software technical and strategic issues. He participated in the FSF’s GPL v3 drafting process in the large corporation panel, and maintained the man suite, the primary documentation-delivery tool under Linux. Federico is a graduate of Boston College and Harvard University, and holds an ACE from MIT's Sloan School. His writing has been published on Linux Journal and Linux Magazine, he pens the recurring "Performance Tuning Dojo" column for Admin Magazine and writes for O'Reilly Media on topics ranging from Cloud Computing to Open Hardware.
Specialties: All around technical, with an interest in complex, low level problems. I care deeply about my team and enjoy building the orchestra as much as directing it.
It was the worst of times; it was the best of times... The pandemic and the unprecedented sudden lockdowns provided the real test of the business continuity plans (or the lack of it) for many organisations. For someone who worked as a security analyst in a MSSP (Managed Security Services Provider), the early days of the lockdowns was a series of ups and downs. Listen to the stories from the SOC trenches during the time of COVID and pick up some valuable lessons that will be useful even after the lockdowns have been lifted.
About Gyle dela Cruz:
Gyle has a multi-cultural and multi-disciplinary background. She is passionate about contributing to the cyber security industry and wants to empower everyone in understanding how their actions can create a safer cyber world. Her day job at Cyber Research NZ includes using both blue team and red team-related technologies and techniques to help protect and defend her clients. She is based in Melbourne, Australia where the best coffee is available from the different cafes. She was part of the first cohort of the Project Friedman – a joint initiative of Australian Women in Security Network (AWSN) and Women Speak Cyber to encourage more women to speak in cyber security conferences. In her spare time, she mentors other people who are new to the field, presents in different cyber security events and volunteers with different organisations.
In 2017 I came one keypress from causing Google's main backbone to largely fall off the Internet. This is the story of how we used that incident as a learning opportunity, how a lack of buy-in hindered further improvements, and how an existing toolkit of python libraries allowed testing and validation tools to be quickly built, preventing any chance of a recurrence.
About Julien Goodwin:
Julien is a Senior Site Reliability Engineer at Google Sydney, from 2011 to 2018 he worked on Google's production networks, focusing on Internet routing & interconnection. When not at work he does things like designing custom embedded Linux machines & modernising frequency distribution systems.
He was also the 2019 & 2020 Secretary of Linux Australia, the parent organisation for linux.conf.au, and was part of the LCA 2008 team.
Docker has been around for almost 8 years now, but I get the impression that a lot of systems administrators and software developers may only have a basic understanding of its operation, and how its functionality can be leveraged to make software easy to develop, test, and deploy. This talk aims to give an overview of what Docker can be used for, and some concrete examples on how to use Docker and Swarm to package up and deploy apps. We'll cover Dockerfiles, docker-compose.yml files and how they fit together, and, time permitting, how to use Docker BuildX and Docker App to build multi-platform container images, and package collections of containerised services for easy deployment and versioning. Slowly converting apps to work with Docker is a great way to make them automatically more scalable, consistent, and portable. Participants will be able to work with Docker on their local machines, or use Play With Docker via their web browser to follow along.
About Matthew Cengia:
Matt (he/him/his) is a queer, autistic cisgender man from the lands of Wurundjeri people of the Kulin nation (so-called Melbourne, Australia. He has a long background in Linux systems admin and software development, as well as strong interests in communication, empathy, consent, openness and transparency, privacy and security, diversity and inclusion. Matt identifies as a generalist, polymath, or multi-potentialite, and his breadth of interests often give him a unique perspective on how to relate to, and mediate between, people of different specialities.
A quick overview on the Ceph project and improvements made to managing and administering Ceph clusters in our latest release. We will review the Ceph Dashboard as well as improvements made to automation and management of the Ceph cluster. We will touch on the new Cephadm managment component in Octopus and improvements from past management tools.
About Michael Hackett:
Michael Hackett is a Principal Product Experience Engineer at Red Hat with excellent technical, analytical and communication skills demonstrated by certifications and over 13 years of experience.
He has co-published two books on designing, implementing and troubleshooting Ceph Storage systems.
LinuxBoot is one of implementations of Open Systems Firmware (OSF) in Open Compute Project.
The primary goal of OSF is to increase security in Mega Datacenter by making use of Linux as firmware instead of vendor proprietary's.
LinuxBoot consists of Linux Kernel and Initramfs and it can be based on either UEFI PEI, coreboot, U-boot, or Slim Bootloader at this moment.
Linux Kernel requires patch in some cases, and Initramfs can be chosen from either u-root or heads.
If we try LinuxBoot on AArch64 Server defined in ARM SBBR spec, it won't be simple to make Final OS boot from LinuxBoot flashrom since LinuxBoot document is far behind source code and AArch64 implementation is far behind x86_64 as of writing.
This talk discusses LinuxBoot overview, demonstrates to boot CentOS AArch64 from LinuxBoot flashrom in QEMU on x86_64, and finally reveals the biggest and AArch64 peculiar issue we need to solve, that is kernel decompression issue x86_64 doesn't have.
Attendees will get familiar with how to:
- create AArch64 OVMF 32MB Firmware File System
- configure LinuxBoot Kernel and Initramfs
- inject LinuxBoot into AArch64 64MB flashrom
- boot Final OS from local disk
- debug LinuxBoot AArch64 Kernel using QEMU and GDB on x86_64
About Naohiro Tamura:
Naohiro Tamura is a Professional Engineer in the Linux Software Division at Fujitsu Limited.
With a background from BMC and Bare Metal low level C code to application high level functional and logic programming code, he's been consistently working on System and Resource Management Software Development from proprietary to open source project.
- 2019-present AArch64 Server Project and OpenHPC, LinuxBoot
- 2017-2019 FaaS Shell Project and CNCF Serverless WG WorkFlow
- 2014-2017 OpenStack Ironic Bare Metal Provisioning
As more services are becoming containerised the security risks continue to increase. By adopting rootless containers we immediately remove a huge attack surface, in addition to providing the capability for any user to run containers on a host system without requiring admin rights.
This talk will outline the core concepts and benefits of rootless containers, and how Podman provides a simple to use framework that integrates nicely with Systemd. Along the way we'll also touch on some of the other security and performance management capabilities that SELinux and CGroups bring to containerised deployments.
To outline the benefits, and some of the bumpy bits along the way, Steve will use his initial deployments of Home Assistant and Mosquitto as containerised services for home automation, alongside workloads currently unsuitable for rootless containers.
About Steven Ellis:
Steve's is an Open Source Technology Evangelist in the APAC Office of Technology team at Red Hat. Over the last 25+ years he started work as a developer before transitioning to an infrastructure and operations architect across a broad range of Unix and Linux technologies. For most of that period he’s used Open Source technologies to solve business problems. His current role means he gets to help customers across APAC understand some of the latest Open Source tools and technologies.
In his spare time he still hacks on the MythTV project and debugs Open Source on random bits of hardware that really should know better.