Title: Oh for a shorter tea break, or A quest for a faster link time...
Who am I?
- Name: Kim Hawtin
- Org: Ratbag Games / Midway Games
- Role: UnixSysAdmin / IT Manager
- Time: 3 years
Who was Ratbag?
- made games on; Peecee, PS2, XBox
- most recently "Dukes of Hazzard" & sprint car games
In the begining;
the first problem, ramdisk
- need large ramdisk to store build files in
- 1U, P4 3.4GHz HT, 4GB Ram, 2 x 10K RPM SATA
- debian sarge, Ratbag Dev environment
Ramdisks over 16mb trash running apps on write!?!?!
- even if you recompile the kernel with larger volumes sizes
- tests with <16mb ramdisk works fine
- hitup mail lists for help
the wonders of tmpfs
- 2gb volume the largest we could create, on the P4 with 4gb
- copy 1.6gb of libraries to ram, generate 600mb elf
why are we doing this?
- to reduce the link times on building the game executable
- working with developers and CTO
- need to pack more work in as milestones approaching fast
background in ps2 game development environment
- dev environment setup about 6 years ago, very little development on it since
- gcc - patched by sony for the ps2 - ee-gcc, ee-gdb, iop-libs originall 2.95, 18 months ago 3.2
- licenced libraries for PS2; havok for pysics, p-mod for sound
- build static executable with dynamic heap, no relocatable libraries
- ps2 tool - the development kit of choice, half pc, half ps2,
- ps2 test kit
- consumer ps2 with Linux kernel load method, ie buffer overrun
whats the most common scenario?
- change one line of code in one source file, compile, relink libraris & elf
- change one & relink time; just over 7 minutes on our P4
Started to look at where the bottlenecks were
- cons does lots and lots of file I/O and some disk I/O
- cons behaves differently to make
- makes dependency checks, only files that changed are rebuilt
- cons checks everything
- not using current version of cons
- do no want to change all the config files, if it can be avoided
- developers familiar with Construct files of cons
Scale up hardware to see if 64bit machines and ramdisks are any faster
- eval machine dual core amd64 264, 8gb ram, Tyan MoBo, same SATA disks
- 4gb tmpfs file system
- 32 bit sony gcc works fine on 64bit debian for amd64
- lots of kernel recompilation to get multiple CPUs/cores to work =)
Performance of P4 very similar to amd64
- link times is 7 minutes
- faster on second build, as filesystem cache is much large, link time just under 7 minutes
Tested copying all source into tempfs, but not faster than the 'cached' build
CTO discovers dist-cc, scons and others ...
- dist-cc doesn't help with linking, keep looking
- scons uses same 'Construct' files with small modifications
- deploy scons, first link just over three minutes for full debug build
- much joy, dependency checking more like make
- follow docs from scons, relink time falls to 30 seconds
- after more tweaking the AMD64 builds in about 18 seconds
Ratbag Studio is shutdown, new build system not deployed!
- PS2 tools a bit thin on the ground
- not many games studios that actually build using GCC from Sony for the PS2 anymore
- Sonys ee-gcc / build environment is plenty buggy
- gcc uses memory saving features in C++ that commercial compilers don't
- P4 just doesn't have enough RAM!
- with 4GB of physical RAM you have no swap!
- building kernels for multi core/cpu opterons is "interesting", config info on the 'net is still mirky
- lots of RAM, ie 8GB is the fastest CPU acceleration we could find
- RAM in 2GB dimms is EXPENSIVE, almost prohibitively so
- RAM in 4GB dimms is insanely expensive
- other build environments like dist-cc didn't fit our needs, but might for different kinds of projects
- recently found ccache, doesn't fit the relinking problem, would be better for rebuilding codebase from scratch
- theres a whole raft of other tools out there, but mostly around compiling faster, not linking faster