Title: Oh for a shorter tea break, or A quest for a faster link time...

Intro;

Who am I?

• Name: Kim Hawtin
• Org: Ratbag Games / Midway Games
• Role: UnixSysAdmin / IT Manager
• Time: 3 years

Who was Ratbag?

• made games on; Peecee, PS2, XBox
• most recently "Dukes of Hazzard" & sprint car games

In the begining;

the first problem, ramdisk

• need large ramdisk to store build files in

build server

• 1U, P4 3.4GHz HT, 4GB Ram, 2 x 10K RPM SATA
• debian sarge, Ratbag Dev environment

Ramdisks over 16mb trash running apps on write!?!?!

• even if you recompile the kernel with larger volumes sizes
• tests with <16mb ramdisk works fine
• hitup mail lists for help

the wonders of tmpfs

• 2gb volume the largest we could create, on the P4 with 4gb
• copy 1.6gb of libraries to ram, generate 600mb elf

why are we doing this?

• to reduce the link times on building the game executable
• working with developers and CTO
• need to pack more work in as milestones approaching fast

background in ps2 game development environment

• dev environment setup about 6 years ago, very little development on it since
• gcc - patched by sony for the ps2 - ee-gcc, ee-gdb, iop-libs originall 2.95, 18 months ago 3.2
• licenced libraries for PS2; havok for pysics, p-mod for sound
• build static executable with dynamic heap, no relocatable libraries
• ps2 tool - the development kit of choice, half pc, half ps2,
• ps2 test kit
• consumer ps2 with Linux kernel load method, ie buffer overrun

whats the most common scenario?

• change one line of code in one source file, compile, relink libraris & elf
• change one & relink time; just over 7 minutes on our P4

Started to look at where the bottlenecks were

• cons does lots and lots of file I/O and some disk I/O
• cons behaves differently to make
• makes dependency checks, only files that changed are rebuilt
• cons checks everything
• not using current version of cons

Change/risk averse

• do no want to change all the config files, if it can be avoided
• developers familiar with Construct files of cons

Scale up hardware to see if 64bit machines and ramdisks are any faster

• eval machine dual core amd64 264, 8gb ram, Tyan MoBo, same SATA disks
• 4gb tmpfs file system
• 32 bit sony gcc works fine on 64bit debian for amd64
• lots of kernel recompilation to get multiple CPUs/cores to work =)

Performance of P4 very similar to amd64

• link times is 7 minutes
• faster on second build, as filesystem cache is much large, link time just under 7 minutes

Tested copying all source into tempfs, but not faster than the 'cached' build

CTO discovers dist-cc, scons and others ...

• dist-cc doesn't help with linking, keep looking
• scons uses same 'Construct' files with small modifications
• deploy scons, first link just over three minutes for full debug build
• much joy, dependency checking more like make
• follow docs from scons, relink time falls to 30 seconds
• after more tweaking the AMD64 builds in about 18 seconds

Ratbag Studio is shutdown, new build system not deployed!

Lessons learned;

• PS2 tools a bit thin on the ground
• not many games studios that actually build using GCC from Sony for the PS2 anymore
• Sonys ee-gcc / build environment is plenty buggy
• gcc uses memory saving features in C++ that commercial compilers don't

• P4 just doesn't have enough RAM!
• with 4GB of physical RAM you have no swap!

• building kernels for multi core/cpu opterons is "interesting", config info on the 'net is still mirky

• lots of RAM, ie 8GB is the fastest CPU acceleration we could find
• RAM in 2GB dimms is EXPENSIVE, almost prohibitively so
• RAM in 4GB dimms is insanely expensive

• other build environments like dist-cc didn't fit our needs, but might for different kinds of projects
• recently found ccache, doesn't fit the relinking problem, would be better for rebuilding codebase from scratch
• theres a whole raft of other tools out there, but mostly around compiling faster, not linking faster