Industry Broadcast, the podcast for the games industry, has hit 50 articles at this point - about 16 hours of industry wisdom. That means if you have a half-hour commute you'll soak it up in about 3 business weeks. I'm in there, as I've mentioned before, and lately they've got guys like Noel Llopis and Soren Johnson, both bloggers I like to read. Pretty cool.
http://www.industrybroadcast.com/
| | |
| |
Another year, another great conference.
You can read all the talk summaries here - http://www.igda.org/leadership/- video and DVD's will be made available in a few months.
A highlight for me was Mark Cerny's keynote - he really is a kickass speaker, and he went to the trouble to make his slides animated and multimediaish. The important take-home is that "culture trumps process": a culture that values the believer and permits failure is good. "Kill a project with love", he said, or something close to it. (A good culture would adopt good process, it occurs to me, a la Bioware. Even though their games almost always rock they keep trying to improve.)
My own talk (which he sat in on, which stressed me out a bit, because I referred to his DICE talk and had this one teensy little disagreement with it) seemed to go well. Ben Hoyt wrote it up quite well here.
"So what'd you hate about my talk this time?" Mark Cerny asked me. Since my whole talk was about bringing back "grey box prototypes" and his talk had a bit about how "grey box prototypes don't cut it anymore" you'd think I'd take issue with that, but really we're talking about different games. I'm talking about radical new experiments like Wii Sports or Guitar Hero or GTA or The Sims, where what you finally ship isn't really all about the graphics. (Sure, the GTA franchise has awesome graphics *now* - but it took them a while to get there.) He's talking about games like Uncharted and Gears where gameplay innovations, while present, take a back seat to the new challenge of capturing and rendering human motion.
And I got to meet Curt Schilling which was cool even though I'm not much of a baseball fan. (My dad was suitably impressed, anyway.) What I liked about Schilling, the fact that he's willing to say, "I'm not the great game designer I thought I was," is incredible to me. Almost everyone thinks they can design. It's so nebulous, how would you even know if you can't? How would someone prove it to you? (I like to think I can design, but I have no real proof, and as many failures as successes.)
The surprise talk for me was a panel on CMM, PMI, and Scrum; three different methodologies. CMM has a rep for being the sort of waterfally bureaucratic process that gets made fun of by Dilbert and Joel Spolsky, so it was interesting to see Tobi Launier had used one of their programs succesfully to get a DS spider-man game out the door in a mere four months. And Rich Vogel wins my award for best culture-changing tactic: You want people to come into work on time? This was something we wrestled with endlessly at Treyarch, whose culture started as a mosey-into-work-whenever-and-work-late-into-the-night shop. Here's how you do it: serve breakfast, but not quite enough for everybody. The strategy (more flies with honey) is well-known but it never occurred to us to apply it.
My biggest take-home: we need to do two or three orders of magnitude more kleenex / blindtesting. Not QA. The first Cerny method games (Crash Bandicoot, etc) had 200 people play them start to finish, with metrics. Then, Curtis Creamer, in his talk, said Halo 3 had even more. When they were done, four and a half months before ship, 7 groups of 20-40 people each played through the game in its entirety - and that's not counting all the playtests they did on the way to alpha. By way of comparison, with Spider-Man 2, we invited about six people in to play the game all the way through one weekend.
And with Schizoid we had one person play all the way through, on his own time - we weren't even watching. These big playtests are something Torpex isn't equipped to do (we don't have that many friends!) so we may need to consider outsourcing...
I asked myself, is that how you get 90% on gamerankings? I talked to another friend who recently worked on a 90+ game and said they didn't do much, if any, blindtesting, and relied more on everyone playing in the office and the gut of the lead designer. So it is possible to get the 90+ without it, but I expect most games did the big playtest.
| | |
| |
It's almost here. I'll be announcing the project management tracks, and, at the last moment, I volunteered to give a talk about prototyping.
It just so happens we're doing a prototyping deal with Microsoft right now - they're paying us to find the fun in one of their ideas. It's a pretty cool gig, and I'll talk about it some (I can't actually announce anything, so my talk will be circumspect), and I'll go over a bunch of other thoughts I've had lately about prototyping.
| | |
| |
I don't have as much time to read blogs as I used to. (Or update my own, as you may have noticed.)
Fortunately, now there's http://industrybroadcast.com/ - podcasts by various industry bloggers, including myself and Danc of Lost Garden. Now you and I can listen to myself (and others) on our commute, etcetera. And don't worry, it's not actually me doing the reading with my nasal tones and unconscious tongue-clicks - Ryan Wiancko has a voice that wouldn't sound out of place on NPR. He's got a lot of my old *Manager In A Strange Land* articles up, (articles I don't always totally agree with anymore), and more are on their way. (And if you want to filter the other guys out, you closed-minded %$#@, you can go straight to my stuff at http://industrybroadcast.com/
Check it out.
| | |
| |
I'm blogging about Rock Band 2 instead of actually playing it because my 360 made a nice circular scuff mark around the rim and now it likes to crap out right as I start a new challenge.
Rock Band 2's online component is pretty cool. You can invite people to play in your campaign - and why not? You'll get more fans, leaderboard score, etc. if you do that. The problem? When you join someone else's band, all you get is money, which is fairly worthless unless you like to buy bling at the rock shop. You don't get progress, leaderboard scores, or fans...
But this doesn't seem to be too much of a problem - I've never had an issue getting a couple of people to join my band. I guess some people really are motivated by the fake cash. Or maybe it's a truism of online games: everybody wants to join, nobody wants to host. Kudos to Harmonix for giving people real strong motivation to actually host.
Of course, to get the most points, your band needs a bassist. And nobody wants to play bass. So if you want to lead a full band, you're going to have to play bass yourself.
And this is like life! A former girlfriend used to be something of a groupie for indy rock band i found god - and in 1997, after their bassist quit, Stress, the founder, gave up guitar and took up playing the bass. In LA, it's a lot easier to find talented guys who want to play guitar than play bass. Just like rock band.
Which reminds me of something Peter Drucker said, but I'm too lazy to look up the quote, so I'll paraphrase: leaders fill the void. They do the thing that isn't getting done. In other words: CEO's plunge toilets.
Another example, in the early days of Spider-Man 1, I did a lot of tools programming, a not-terribly-sexy job, because other guys were excited to do the rendering and gameplay and whatnot. I did what wasn't being done.
But there are a couple caveats to the "Leaders Play Bass" rule - (my opinions here are subtle and nuanched.)
First of all, it can come dangerously close to "Putting Yourself In As Your Own Utility Infielder" - if you neglect your management duties to do full time tools programming, that's a Bad Thing. So if the void, the thing-that-needs-doing, is a critical path thing, you're really going to have to fix that staffing problem. After the Spidey team ramped up, management was a full-time responsibility so I passed the tools programming off.
Second of all, there's some reason you're a leader. Some reason you're running a project, starting a company - you want to make a difference, make something happen, and you bring a certain x factor to bear on that. If you're giving up your x factor to play bass - why are you being a leader at all? Stress didn't stop writing the songs for ifg, and I always have influence over the design on the games I work on, because that's why I make games.
So...'Playing bass' is a survival measure. You only do it when you have to. But when you have to, you do it.
| | |
| |
I used to love the idea of MS Project.
While working on Spider-Man 2, we'd often hear the same story:
"I was going to work on this, but I couldn't because such-and-such wasn't ready, so I tried to work on this other thing instead, but such-and-such wasn't ready, so I ended up working on item C."
I kept hearing those stories and thinking - we really should be using MS Project. It tracks dependencies! It will help prevent these sorts of problems! It's the piece of the puzzle we're missing! Once we start tracking dependencies, we'll be unstoppable!
So I looked into using Project, and some other scheduling software, and experimented with some dummy schedules and leveling tools and decided that not only did they not work very well, but we weren't ready to switch to a whole new system, not with Spidey 2 already more than halfway done.
Still, when Spidey 3 came along, I spent over a month near the beginning making a detailed schedule with MS Project. We did a bunch of things right. We asked the people doing the work how long the work would take (or their leads if those people weren't actually available yet). We extracted our staffing plan from the schedule. And we deputized some producers to run around and check to see if people were working on what the schedule said they'd be working on and to adjust it if necessary.
And after a month or two (it's been a while, I don't exactly remember) we scrapped it, because the schedule seemed completely unrelated to what people were actually doing in the offices. They still weren't working on "what they were supposed to be working on", despite the wonderful dependency tracking that was supposed to take care of that. Game development is still far too fluid and unpredictable, even when dependencies were being noted, we had problems with staffing and prototyping and changing plans that made dependencies pale in significance.
Now, some people swear by MS Project. I think they break down into a few categories:
* People who use it but not to track dependencies. For example, they might make a list of tasks, select them all and make one giant dependency chain, and then have a pretty (and completely linear) gannt chart they can show off, but under the hood this is no better than using excel and doing a Sum. They might as well use Excel.
* People who have modded the crap out of MS Project. Mike McShaffry's tools are prety cool but they actually don't seem to use Project's ability to automatically track and level dependencies - as he points out at the bottom of the page, his tools are for tracking high-level dependencies, and even then you have to eyeball it: oh look, Joe is starting a level before the World Editor is done.
* People who "do it right" but it still doesn't help them - Erik Bethke's book showed how they used Project and its leveling features for Black 9...a game that never shipped. We can't blame Project for that but it does help to show that we have bigger things to worry about than tracking dependencies.
* People who "do it right" and it does help them. I did an informal poll of the room before my Theory of Drag talk last year and it looked like maybe 1 in 200 people used Project's leveling features and believed it helped. Still they're spending an awful lot of resources on it - resources that could possibly be spent more directly on the game.
Now, let's go back to this original "problem" we had that I thought MS Project would solve. "I couldn't work on Item A or Item B, I had to work on Item C."
Is this really a problem? It's only a problem if Item C is some low-priority item that we might want to cut from the game - chances are, all of these things need to be done and although it might be nice to work on Item A first it's just fine to hit Item C instead.
And if Item C really is optional, or someone really is blocked, there are lots of things you can do that will suck down less resources:
http://www.gamasutra.com/features/20041013/fristrom_01.shtml
http://www.gamasutra.com/view/feature/2170/manager_in_a_strange_land_.php
When I wrote those articles I was still trying to make Project work. I have since given up - now I believe Excel beats Project, hands down. (Microsoft still wins, of course...)
FWIW, dependencies weren't an issue on Schizoid at all - the biggest scheduling issue on Schizoid was that I was an overbooked resource, and this was a blindingly obvious problem that we didn't need any fancy scheduling software to see.
| | |
| |
Hey all, Jason Della Rocca just let me know that the talks from last year's leadership forum are available. There are some really cool talks in there, by Greg Zeschuk, Ray Muzyuka, and Trent Oster of Bioware...by Don Daglow of Stormfront (rest in peace, Stormfront)...Mike Capps of Epic...Tim Gerritsen of Human Head...Clinton Keith...and a pretty funny one where Mike Capps interviews Tim Schafer. And my, let's call it..."controversial"...Theory Of Drag talk is in there too.
Here's the link: http://www.igda.org/leadership/?page_id=64
| | |
| |
Hey, I'm back. I'll have some news about Schizoid soon, and I have the energy to blog again.
A thing about drag. I forgot one of the most important ways to mitigate drag, which is: Don't Start From Scratch. Almost nobody does, anymore - one of the first questions you ask yourself when beginning a game project is "What engine am I going to use?" The main reason to do this is because it saves you a lot of work. But another reason it's helpful is because you schedule more accurately - to keep using the snowball metaphor, the snowball you're rolling uphill is already nice and big: progress starts slow and doesn't get that much slower. In fact, a question I was asked at my IGDA talk went along these lines: "We just do content drops into an existing engine - does this really apply to us?" I fumbled the question at the talk, but now I'd agree and say, yes, although there will always be at least a little increase in drag as a project evolves, it may be so little as to be immaterial.
And I said I was going to talk about measuring velocity. But it's nontrivial, and I don't really have a good way to do it. I've been looking at this old progress graph:
And, if you're at any given point on that graph, what do you use for velocity? The ideal is to find a velocity that will accurately predict when you'll be at at "zero bugs." Here are some options:
- Use when you started the project as a starting point, and your current point as the end point: this always gives you an overly optimistic estimate.
- Use your instantaneous velocity. This gives you a wildly fluctuating estimate. One day, you're going to finish tomorrow. The next day (after a slip), you're never going to finish.
- Use some amount of history. But how much? Something that would have been fairly good for us, though still too optimistic, would be to use the last half of the project.
I tried a bunch of stuff - moving averages, fitting curves, nothing hit. But my math skills in this area are weak. If someone out there can take a graph like the one above and find a good approximating/predicting curve, please let me know how. I did ask my brother-in-law, a financial analyst, if he could use stock market momentum measuring tricks to do it, and he wasn't able to help.
In the end, my best guide was actually my eyeballs. When we were in December, and hoping we'd be finished in January, I was able to say - "Just look at the graph. We're looking at the end of March."
That turned out to be on the money - the first time we hit the zero bug line. Of course, we still weren't finished yet; a little more testing and we were right back in the red again.
One must imagine Sisyphus happy.
| | |
| |
So, yeah, hey, credit where credit's due, Clinton Keith's post was what first got me thinking about this - you can even see my comment there in his post, when Schizoid was young but we were already seeing diminishing returns.
So, what do we do about drag and diminished returns?
1) Mitigate
One reaction is - "Let's get rid of it!" Particularly if we use the tractor-pull metaphor; let's get out of the tractor, clean everything up, and get ready for another pull. That idea doesn't fit so well with the snowball-up-a-hill metaphor, though. You've been working hard to collect that snow, you're not going to get rid of it just so you can push faster.
But still, yes, good idea - let's periodically refactor and pay down our technical debt.
That'll help mitigate the drag somewhat, but it's not going to solve it.
(Another way to mitigate is to bring back waterfall - bigger designs up-front, more rigorous planning up-front. You'll have a less werewolfy curve, because you're A) going slow at first while you do all that planning and B) probably reduce the discovered work. The disadvantages in waste on speculative generality and building the product-you-thought-you-wanted-but-didn't-really-want probably aren't worth it, though, so I'm not recommending this.)
2) Bring Back The Contingency Buffer
Nothing new here. Goldratt in *Critical Chain* spends hundreds of easy-to-read pages explaining why this is a good thing, and he's echoed much more tersely by Tom DeMarco in *Slack.*
Their message: "Don't adjust your estimates by a fudge factor. Instead, use that fudge factor to create a contingency buffer or pad at the end of the project." Because when you set pessimistic goals, people take their time. I'm not a fan of Parkinsonian scheduling (setting unrealistically tight dates) but I'm also not a fan of unrealistically loose dates either. My favorite thing about the contingency buffer is it helps keep scope in check. "Sure, you think you can get it done before our ship date, but can you get it done before we hit alpha?"
Speaking of alpha, game development typically already has a kind of contingency buffer built in, which we call "alpha" or "beta" or "content complete" depending on where you work. The point where you stop "adding features" and just fix bugs or maybe polish. (Of course, the area between bug and feature is a gray one. (The dial is analog, was how I heard Mike McShaffry put it once.))
I have problems with alpha. I have problems with it being defined differently everywhere you go. I have problems with the definition frequently being murky even within a team: ask different guys on your team what alpha means and you might get different answers.
Another problem I have with alpha is it has become too small in this day and age. Back when I did six month projects, we tried to stop adding features a month before we shipped. Now that we're doing three year projects, if we're being proportionate, we'd save six months - but I don't know anyone who does.
But my biggest problem with alpha is it encourages sloppiness - an "add features now, we'll fix the bugs later, in alpha" attitude. Steve Maguire, in *Debugging The Development Process* goes to great lengths to say why that's Bad.
But one thing he doesn't say is, when alpha is treated that way, it is no longer a contingency buffer. For it to be a contingency buffer, your goal has to be to be done by then. Not done-except-for-bugs done. Zero-bugs-done. Done-done-put-it-in-the-box-done.
Schwaber doesn't really do a contingency buffer, either. On the release burndown chart, each estimate is multiplied by a fudge factor. Just what Goldratt says not to do. So what happens? At the beginning of the project, when velocity is high, those fudge factors are canceled out by your high velocity. It looks like you're going to eat through that burndown chart in no time, and Schwaber's method predicts an optimistic release date. Then as velocity decreases the date continually slips back.
So here's my modification to Schwaber. Track your backlog just like Schwaber suggests. (And include all bugs that you intend to fix in your backlog - at Torpex, we use the rule of thumb that 3 bugs equals one story point. YMMV.) But rather than multiplying all the estimates by a fudge factor, add a contingency buffer at the end.
At the beginning of the project, when you have no idea what your progress curve is going to look like, you could use Schwaber's "complexity assessment" to guess how big the contingency buffer should be. I could see it going anywhere from 20% to a well-understood-requirements simple-technology project all the way to 300% for the chaos of poorly-understood-requirements and crazy tech. (Schizoid needed at least a 150% contingency buffer, we know now.)
A problem with the contingency buffer at the end is - if you make your scheduling process visible to your publisher, which most say is a Good Thing - and your publisher sees your large contingency buffer, they're going to ask, "What the hell? Looks like you're billing us for a game much larger than the one you've scheduled!"
So, for the schedule you show your publisher, keep the fudge factors in the original estimates rather than in a large pad at the end. I said this at my talk and Ben Hoyt called me out - "Isn't that dishonest?" he said. At the time I said, "Yes, I suppose," because I tend to cave during Q&A. But in hindsight I don't think it is, really, dishonest - the fudged estimates are your "because this is a project of significant complexity" estimates, and I'd be perfectly willing to share that with a publisher. I'd also be perfectly willing to say that, internally, we have these aggressive goals we're striving to hit, so for internal purposes we don't actually multiply in the fudge factors, but rather leave them as part of a large contingency buffer at the end, but those aggressive goals are not ones we want to be contractually bound to. Trent Oster, in his talk, said similar things - he also said to double your estimates and apparently at Bioware they try to work well ahead of the milestones they've promised their publisher as well.
Now, suppose you're partway into the project, and you're experiencing drag, and you realize even you underestimated your contingency buffer - you're going to sail right past that final deadline with an unfinished game. At that point, a couple things need to happen:
- You have to have that hard conversation with your boss / publisher. The "Please don't sue us for breach, but..." conversation.
- Add a bigger contingency buffer to your next estimate. It's tempting to assume that everything that can go wrong already has and it's going to be smooth sailing for the rest of the way, but then you'll just have to have that hard conversation again with the next slip.
We're trying to approximate this curve:
With these:
On Schizoid, at first, our trajectory had us finishing in April, but we told Microsoft June. Call that our 50% contingency buffer. After our first slip, it looked like we could still get it in under the wire. Then we started optimizing and we slipped again. Our contingency buffer is used up. But it was just a blip, right? We'll start making progress again any day now! No. The blip dragged out and became a plateau.
If we had been honest with ourselves at that point, say, by plotting out where our new trajectory was going to hit and adding a larger buffer to that, we would have ended up with a much more realistic estimate about when we'd be done. An incredibly disheartening estimate, but realistic.
And still wrong - here we are, almost February of next year, and still not done yet! But it would have been much less wrong than what we were going with back then.
Sadly, "Less Wrong" is all I can offer.
Next time (notice I didn't say next week - we just did more focus testing and bug triage and we can't bring ourselves to mark as much stuff "Will Not Fix" as maybe we should and now blogging is starting to feel like a luxury): Just how do you calculate "velocity" anyway?
| | |
| |
Before I go on to Part 3 and what we can do about drag let me deal with some of the discovered work unearthed by the comments on Part 2.
* Graph Quality
Mark Nau calls me out and says he doesn't see the curves. If we still worked in the same office I probably would ask him to help me do an actual rigorous statistical analysis of the data. The curves are there but they look shallow and I admit I was pretty disappointed after generating the graphs that they weren't more obvious.
As for not seeing much of a bump when we brought the new programmer on, I can explain that. He's only part-time - I'd estimate at him increasing our manpower by 25% or so. Factor in Fristrom's Law and it's less. He needed a ramp-up period. And his first task was huge but greatly underestimated. It could have shown on the graph as a simultaneous increase in scope and work, but instead it shows on the graph as an increase in neither.
But that goes to prove Mark's point - the graph isn't accurate. I'm not able to truly tease apart work and scope, and probably won't even bother on future projects. (Though it was nice to see that we actually have been working all these months, something that isn't obvious from the first graph.)
Still, it's not two straight lines. Keep in mind that first graph:
This graph is simply the delta between the two lines in the other graph - though they don't plot out to the same point in time; I made the graphs at different times. If the lines in the other graph were straight, this graph's lines would be straight as well, but this is not a straight line.
* Other curves?
Jake Simpson and Simon Cooke say they see different shaped curves on their projects - they both see sine waves or something approximating sine waves. It is possible on this project we're just looking at a portion of a big sine wave with a really large period. Which would be a good thing, because that means at some point we'll take off like a rocket and ship! I wouldn't be surprised, actually, if we do get a burst in the end, as we put our foot down about feature creep and the remaining bugs are all highly detailed spot fixes. If only there was some way to predict when that change would come.
Cooke sees the opposite from us, in a way: he sees little visual progress in the beginning, during ramp-up, later followed by a lot of visual progress. (Much like our programmer with his big, underestimated task.) He may be more waterfallish than we are - we tend to get something up-and-running *now* and refactor later. For us, the "up and running now" shows up as a lot of progress, and then the refactoring shows up as drag. (Maybe this agile stuff isn't all it's cracked up to be.)
* Feature Creep vs. Discovered Work?
How much scope increase was stuff we wanted vs. stuff we needed?
FWIW. Red & most of green are needed. Blue, yellow, and pink are wanted. Call it 60-40.
Though I hesitate to call that 40% literal 'feature creep' - most of our unnecessary fixes were not "add X" but "X would be better if you do Y". The feature was already there and we were tweaking it.
More interesting would be to see how this changes over time, but I didn't know until late in the project that you could set up Bugzilla to generate snapshot data that you can later graph. It would be fraught with inaccuracy anyway - at the beginning of the project, p3 meant must-fix and later came to mean nice-to-fix.
I'm guessing Busse's point: if you tighten the reins on feature creep, you can mitigate the effects of drag.
I'll try to get back on track next week, unless there are more interesting comments.
| | |
| |
Recent Comments