You don't do a question for us on your C8 today, I just wanted to remind you that we're running out of time.
I think it's on now.
Okay, we'd like to go ahead and get started now if we could.
One important reminder if everyone could turn off their cell phones, pagers, other electronic devices that make any sort of noise.
That'd be great.
I wanted to go ahead and introduce my fellow presenter here first.
At the end we have Dr. Bob Gutman with Zipper Interactive.
He's a software engineer.
And Dr. Bob is actually the lead multiplayer programmer, basically on the client side and working out all the logic for SOCOM.
Glenn Vandada is a director of online technology at Sony Computer Entertainment America.
Glenn is in charge of the SCE-RT group, which is basically an API which Sony distributes to third-party developers.
And it was something that was actually used in creating SOCOM as well.
So it's an API and also a back-end server set.
And of course, I'm Seth Luisi, the senior producer for SOCOM US Navy SEALs.
And actually I had the pleasure of working on a lot of the direction for the game design in SOCOM.
So I was going to come up and talk about the different design goals that we went through when we were creating SOCOM.
So I was basically the person that gave these guys hell for the last year.
So I'd like to first get started talking about why we're here, why you should listen to us.
And I think it also goes to the point of why should you make online console games?
I'd like to go through a few of the facts about SOCOM and how successful it's been.
To date, since we were released in North America, we sold through over a million units of SOCOM.
So it's been extremely successful in North America.
And we still have yet to release in Europe, Japan and Korea.
Europe we're launching at the end of May and Japan and Korea a little later this year.
Currently we have over 300,000 active SOCOM users.
These are players that have logged in and played actively in the last 30 days.
We automatically time out accounts if they haven't been used in the last 30 days.
So this gives us a pretty good representative of the number of players that are currently playing SOCOM,
which is a pretty good size.
Every day we have 50 to 60,000 individual people logging in and playing SOCOM.
So there's a lot of games that don't even sell 50 or 60,000 units,
and we get that many people playing SOCOM online.
On the weekends, players log over 170,000 hours of gameplay on SOCOM.
And they play for an average of about three hours.
So the average person will get in, play SOCOM for three hours.
Obviously some people play more, some people play less.
And I think this is a pretty interesting statistic because basically it says that in one day
we have more game hours than probably we had in our entire test cycle,
which obviously does a lot to expose different things about the game.
So more points.
In the last seven days, we've logged over a million user gameplay hours in SOCOM.
One of our competitors mentioned that their online service had to date 15.7 million user hours.
In the last seven days in SOCOM alone, we had over a million,
so we're pretty excited by how popular it's become.
During the peak hours, which currently right now,
since we're mainly released in North America, is 3 to 7 p.m. Pacific Standard Time,
we get 11 to 12,000 simultaneous people playing SOCOM.
And of course this is a really important number in online gaming
because it determines your server load and bandwidth
and is a very good indicator of how much it costs to actually host an online game.
And one of the things we wanted to point out is that those numbers are actually larger
than our online PC competitors, competitors that are in the same genre as SOCOM.
We have more people playing SOCOM simultaneously every day
than Battlefield 1942, Unreal Tournament 2003,
Medal of Honor, America's Army, or Quake III.
And again, we haven't even released in Europe or Japan or Korea yet.
This is only North America.
But again, the main reason why we're here is not to talk about how successful SOCOM is,
but because we want to encourage people to sort of realize that
online gaming for the console is here and it's already a pretty big market.
And we feel pretty strongly that everyone should be at least investigating
and looking at adding online components to their console games.
In the last six months since the PlayStation 2 Network Adapter and SOCOM has been released,
we've already managed to surpass a lot of PC games as far as number of users and hours played.
So I think that really bodes well for online console games.
I'd like to talk about some of the online goals, design goals for SOCOM when we were creating the game.
SOCOM didn't really start out as an online game.
It mainly started out as a team-based, tactic, single-player game.
We always had some plans for an online component, even when we started the initial concept for the game.
Then about midway through our development cycle, we started focusing and sort of made a commitment to an online game.
One of the first decisions you have to make is, can you actually create a full single-player game and a full online game?
And we felt that being first-generation online PlayStation 2 game, that was extremely important
because you sort of want to pull people in with a single-player game, offer them an experience,
and then entice them with a full online game, a full online experience that they can get when they go online.
And I think it's been worked out really well for us.
We actually managed to get at least a third of the people that bought SOCOM to play online, probably more than that.
Probably up to about 50%.
Of course, creating two full separate games easily equals double the resources and double the work of a standard single-player game.
And we quickly had to try and adapt to that and make sure that we had the resources necessary
to still create a competitive full single-player game along with a competitive full online game.
And that was one of the big challenges and obviously something that needs to be considered.
Another one of our goals was to make sure we provided players with a full online game experience.
This was definitely one of the things we wanted to achieve because up until SOCOM,
and even still a lot of games, they're not nearly as full of an online gaming experience as PC online games.
We wanted to change that with SOCOM.
We wanted to make a game that was equally as large, just as full of an experience as what was currently available in PC games.
Because we wanted to compete in online games and move past online console games and focus on creating a successful online game.
Some of the design goals that came from that is we felt we had to support 16 players.
That's sort of the minimum spec for an online PC game.
And something that we felt was definitely necessary for SOCOM to be competitive in the online gaming arena.
Of course, when you're hosting all the servers on your own, which is usually a good idea for managing players' expectations
and making sure that they have a consistent experience, it tends to cost a lot of money
because you have to invest in servers and bandwidth.
Obviously, there's other business models in peer-to-peer gaming.
It seems to be pretty popular on the console because it diverts a lot of cost.
Of course, the most popular in PC games is having the users host their own servers.
I think there's a good potential for that in online console games as well once you get past the experience for consumers.
I had a feeling that would happen.
Voice Chat.
We started creating SOCOM and the voice in the headset actually came from the single-player game.
We wanted to create a team-based game.
We didn't want to create an every-man-for-himself game where you run out and shoot people and they die
and you can get shot a hundred times.
We really wanted to focus on creating a game which you had to really use your teammates
and they were really a part of the game and very helpful to you
and you can use a lot of more advanced tactics with your teammates.
We came up with using voice recognition and working on that for the single-player game.
Of course, the first thing everyone was asking when they heard we were supporting a headset for a single-player game
is, why are you going to have Voice Chat online?
It was actually something which we didn't initially plan on
but it was such a demanded feature that we felt we had to add it and deliver that
to really make the best use out of the headset and the technology available.
It actually gave us an opportunity to really provide a good integrated voice solution
which hadn't really been done in a game before.
I've been attempted but most of them, you know, a lot of people didn't use the voice
or it wasn't very well integrated.
Another thing that was really important to us was built-in community sport, clans and ladders.
These have turned out to be extremely popular and, you know, people are really getting into creating clans.
You go on the SOCOM and you'll see a tag next almost, you know, everyone's name.
We're all associated with the clan in a clan and, of course, we had a lot of feedback
where people are really concerned about their ladder position and trying to work their way up the ladder
and that's been another very successful aspect of supporting a SOCOM community.
Additional design goals.
Make it easy for console players.
It sounds simple but in a lot of ways people don't take it enough, you know, thought into the online games
and especially the UI.
The UI in a lot of online console games that I've seen is overly complex, overly complicated.
We wanted to make sure with something that was quick and easy to navigate,
you just look at the screen and you knew where you could go and where you were going to.
You didn't get lost in any of the UI screens like happens in a lot of other online games.
You always had a very clear indication of where you were.
We also tried to use as many familiar interfaces as we could.
When you log on to SOCOM it looks suspiciously like, you know, a web page login at Amazon.com or for Yahoo Mail.
You have your username, your password, a little checkbox to save your password and username.
So I think focusing on those sort of design elements, making sure that, you know, it's easy for somebody that's familiar,
maybe familiar with surfing or other aspects but not familiar with online gaming,
can easily navigate and get into a game quickly and easily.
We also, it's important for us to have a persistent appearance and I say appearance
because a lot of things in SOCOM aren't persistent but we make them look like they are.
So we have, you know, different briefing rooms for US East, US West.
But if there's nobody in there, that room doesn't exist.
So when somebody selects it, it's created and then it gives you the feedback on how many players are there.
So, you know, really working with sort of convincing players that it's a persistent experience.
So they know, they can tell their friends.
They can say, oh, meet me in US East one at five o'clock and they'll know where to go and it's always there
and it's really easy to get into that and find your friends and play.
So that was something that we really wanted to do and really provide and I think it's worked out really well for us.
The other thing that we did was sort of psychological grouping of players.
As I mentioned, we had US East, US Central, US West but we don't actually have servers in US East, US Central, US West.
It's just a method of convincing players to go and play in that area because even with a client server model,
you know, some people are farther away than others.
You'll still have some latency issues.
So we wanted to try to convince people to go in group in certain areas where they would be close to the people that they're playing with
but they could also go into another area and play with people that were somewhere else.
And if you go into any of these rooms, go into US East, you know, you have everyone from the East Coast in those rooms.
You go into Central and it's people from, you know, Central America there.
So it really worked out well as far as just thinking about the psychological aspects of what people think about when they go into these rooms.
There's something else we wanted to mention, you know, try to focus.
We tried to focus the online gameplay.
We were trying not to do everything at once.
You know, we were looking at what a lot of other games did but the main focus of the single player game was team play
and we felt that that had to be the main focus of the online game as well.
So a lot of people were saying, oh, you have to have, you know, a deathmatch mode.
You have to have a mode.
Every man for themselves that capture the flag.
Well, you know, we really didn't have time to go in and create and design those to the depth that we wanted to.
So we really wanted to focus on the team play aspects and then sort of build on top of that.
So we do have different game modes but they're still built on the basic teammate game play
and we didn't have to create all new rule sets for how players are spawned and how they work on teams
and how they didn't work on teams and I think that's something that really helped us sort of polish our team play.
And you have to remember that, you know, there's a lot of PC game developers that spend four years creating an online game.
Well, you don't really have that luxury with a console game because you have to make sure that you're actually going to ship something before they're on to the next platform.
So definitely focusing your online game experience is very important because you're not going to have forever to create the online portion of the game like you do in some other areas.
The other big thing that we were really pushing is enabling the community to police itself.
And again, this is something that has really worked well.
We were definitely thinking about, you know, how can somebody easily get somebody out of a game?
So we have a very easy system where you can vote somebody out.
It's on a menu. You just hit the circle button and it pops up and it's one of the first options.
It says vote, vote off and then you see the player's name and you vote them off.
And if, you know, more than 50% of the people on your team vote that person off, he's gone.
And he can't join back in the game for, I think it ended up being 30 minutes or so.
So, you know, if somebody is team killing, if somebody is cheating, you know, this feature gets used all the time.
It's the players in the game, even though the person's on your team, they'll kick them out and they'll police the sort of online community for you.
If they're behaving poorly or doing something else, you know, the consumer sort of takes the actions for you to improve the overall online game experience.
And of course, password games sort of create, you know, so you can just play with your friends online or just the people you know.
You can create a password game, try to make that very easy to do as well.
I wanted to talk about some of our production hurdles, some of the things that sort of hit us when we were creating SOCOM.
One of the biggest things was that everything was new. The hardware was new, the software was new, the low-level networking libraries, the TCP IP stack was new.
So, there was a lot of dependencies on other groups to sort of create that.
Obviously, creating second-generation PlayStation 2 games and third-generation, a lot of those problems have been solved and worked out,
but it was really a challenge for us to keep up with the ever-evolving and technology that was still being created.
The other thing that was huge for us was testing.
16 players in an online game, you need at least 16 players to test it, and then we had the single-player game as well.
So, on average, we had about 24 players minimum testing SOCOM, eight QA guys for single-player and 16 for online,
because there was no other way for us to make sure that it was getting all the features and everything was getting testing,
and obviously that's a pretty large QA group.
One of the other things that can be extremely beneficial, even in console games, especially in console games, I would say,
is doing a really good beta test, public beta test.
Going out there and creating the tools that you need to communicate with your consumer, your public beta test,
participants to get feedback and to sort of work through all the different processes and issues before you release the product,
because obviously it's much more difficult in a console world to make any changes once the product's released,
and you have to find these things out before you release the game.
Another big hurdle for us, which is still kind of a hurdle for most game players or most game developers, is NAT and firewall devices.
It wasn't as big of an issue for SOCOM because the only element that we have that which is peer-to-peer is the voice chat,
but it impacted some of our other games pretty dramatically as well.
Of course, for those that don't know, these NAT firewall devices are like the Linksys or D-Link cable modem broadband routers,
which basically allow you to share one connection.
Unfortunately, all of these are not the same, and they don't follow the same standards,
and they especially don't, most of the manufacturers to begin with, now they're changing the stance, they didn't care about peer-to-peer.
So they have really bad peer-to-peer support, and you have packets being sent to the wrong player.
We had a bug in SOCOM that came up earlier on where one person in one game heard somebody else in another game talking using the voice chat,
and we couldn't figure out how could this be possible.
He's in a completely separate game, and he's hearing voice from another game.
It turned out that the NAT device was sending, the two players were behind the same NAT,
and it was sending the packets to the wrong person.
So we had to make some code adjustments to take that into account as well.
Another thing to mention is post-game release server deployment.
Basically, normally you go gold on a disk, and it hits the shelves, and you take a little vacation,
maybe you work on a European version, Japanese version.
Well, it tends not to be that simple with an online game, especially a successful one.
Even though you try to anticipate everything, once the game comes out and you have thousands of players logging in,
you're going to find something, and luckily we were able to make substantial improvements on the server side.
So we were able to stabilize and make a lot of improvements.
We were even able to fix some issues that we found in the client by making some interesting server side fixes,
things that were impossible for us to really detect earlier on,
because we didn't have nearly the numbers of players that we had when we went final.
So that tends to be an ongoing process.
Another thing I wanted to mention is localization.
In SOCOM, we added in a text chat feature, and it was sort of something that was debated a little bit,
whether we would have it at all or not, and I'm sort of glad that it went in,
but it brought on all kinds of issues when we started working on the Japanese and Korean versions,
because of course if you have text chat input in Japan or Korean, you have to support Kanji input,
and there's no way of inputting Kanji on the PlayStation 2 like there is on the PC,
or you can use Microsoft's input method editor.
So we basically had to create our own Kanji input method editor with help from some other guys,
but we pretty much had to design it and create it from scratch,
and then also creating our own Hangul input method editor,
and this also made it so that we had to support enormous font.
There's more than 5,000 Kanji that we had to keep in video memory,
because we never know what somebody's going to type.
In the single player game, it's easy.
You know what messages are going to be displayed.
You can really have a much smaller font that's based on the messages that are going to be displayed,
but when you support text chat, you have no idea what they're going to put in,
and our texture size for these fonts is enormous,
and we actually definitely had to use dynamic texturing on the PlayStation 2
where every frame were pulling textures for these fonts from the main memory
just to be able to support all the characters.
So it's definitely something to keep into consideration.
Sort of my own personal advice for creating an online game, obviously, is subjective,
but I think it's really important to look at what PC games have done.
PC games, online games have been around for a while.
They've evolved, players have given a lot of feedback,
features have been implemented.
There's really no reason to sort of go back to the drawing board
and reinvent an online game for the console.
I think there's a lot of innovation that can definitely still be done in online console gaming,
but you should always not discount something just because it's being done in a PC game
like a lot of console developers tend to do.
And then the other thing is don't look at the PC games for guidance.
There's definitely a lot of bad things that they do.
The average consumer that is using them is definitely more advanced than the console consumer,
and you have to take that into account.
It still has to be easy.
It still has to be a console game.
You don't want it to feel like a PC game, so it's sort of a give or take there.
You want to take the best elements that are really going to create fun online gameplay
and watch out for elements that are going to add unnecessary complexity.
One of the other things is enable player-to-player communication.
There's been a lot of console games that I've seen that have absolutely no player-to-player communication.
Well, if you have no player-to-player communication, you really have no community for this game.
You have no way of players being able to send messages to one another,
even easy things like, hey, who wants to play a game?
Join my game now.
You can't even get things like that across, and certainly clans and a lot of other options
are impossible without having some format of sending messages to other players.
And then another one, you know, definitely add these community features.
It's been one of the most successful part of SOCOM,
and one of the most requested features has been the ladders and clans and other elements
that really, you know, the players are really getting into it,
so you should definitely take that into consideration.
And then another thing I wanted to bring up is, you know, bring console production values to online gaming.
Online gaming on the PC still tends to be kind of sterile.
Even PC games in general, if you compare them to console games,
console games tend to have a much higher production value in the way that they're presented in the story, in the music.
So when keep those in mind as well when you're creating an online game,
because it's an online game, you shouldn't, you know, eliminate all the production value
and create, you know, sort of standard bare-bones game.
You should still try to maintain these elements,
because that's really what's going to drive consumers and get them excited about it,
and they sort of expect it now in console games.
Sort of my last bit of advice, try to anticipate the worst,
especially when you're in the design phase.
Mainly this is when you're in the design phase of an online game,
because the reality is it will be much worse than what you anticipate.
When we're designing chat features for SOCOM, you know, we had to think about,
okay, how many times can somebody send a chat message?
You definitely have to limit it, because, you know, other games that we've seen didn't limit it,
and, you know, you'll get a player on there that's just spamming a chat channel,
so no one else can send chat messages.
When we designed the voice chat as well, we really worked out a lot of different
rules to prevent player from hogging the chat channel.
They can only use it for 10 seconds.
When their 10 seconds is up, they can't use it again for another five seconds.
So lots of different things to take into consideration when you're creating
an online game, and just try to anticipate, you know, as much as you can,
how players will exploit everything you put in there,
and how they'll use it to their evil advantages.
So that's it for my portion of the presentation.
I'm going to hand it over to Glen Bandata, who is going to talk about the SCE-RT API
and the different technologies that were used in creating SOCOM.
Let's see, testing, testing. Great.
I'm Glen Bandata. I'm the director of the online technology group
at Sony Computer Entertainment America.
I'm responsible for what is called now the SCE-RT SDK, S-C-E-R-T.
And about August of 2000, a company called R-Time was acquired by Sony,
Computer Entertainment America, and this is what the name of the company is now,
or at least the group.
Since this is sort of bringing console gaming online,
my biggest advice that I could give is definitely go with some sort of networking SDK
for consoles.
I would say there must be eight of them out there now.
Don't do the normal PC not invented here unless you've got a lot of resources
to put into this. You really need to make it a simple process.
I specifically put the SCE-RT SDK and infrastructure components up there
to at least show people that what I'm going to talk about here
is very specific to SCE-RT, and it does include things
that I would recommend you put into your product development as well.
This is a summary of what I'll be talking about.
Right now in S-E-E-A, S-E-E and S-E-E-K,
we have approximately 28 first party titles.
I say it's growing by two to three per month.
That's sort of like we're getting introductions to people.
Both first party and third party that we're finding out are doing console games.
A good PS2 title sells millions,
so you really should be expecting with a given percent
that you're going to have tens of thousands of people simultaneously online.
When I say easily 200,000 simultaneous worldwide,
it's just based on some projections that we get on how rapidly the market's growing,
and it's my own opinion of what it is,
what I think we need to design too.
If you look at the number of network adapters sold today,
we've got, it was projected, we put this presentation together earlier,
it was projected that we would have approximately 500,000
if you looked at some of the Sony PR statements.
One thing that's kind of cool about this compared to the PC
is that you have some idea at least of what your user base is
because you know how many network adapters are out there for your console.
It's very similar if you get like an Xbox Live Kit.
In the PC, being part of our time,
we saw in 1995 there were very, very few, if any, online games
that were really considered online games.
By 87,88, it was sort of becoming an entity in itself,
and then by 2000 it was like a check mark that every publisher required you to have.
I really see in talking to first party titles, third party titles,
that that's what's going to happen in the console space as well.
We have two really components to our SDK.
We have the community building component, and we have the game component.
The community building component is what we call the medius SDK.
This is really like the player experience, player matching experience
for online.
The infrastructure of that, which is the back end servers,
include a universe manager which takes people and puts them on the appropriate lobby servers.
It does some authentication for security purposes.
The proxy server is there to do game manipulation.
If you actually create a game, you'll go onto a proxy server.
We have database, and now a new thing we call the information server.
The information server allows you to go from universe to universe.
Any person in a single universe can see or have a buddy that's also in that universe
with the addition of our medius universe information server.
Now you can connect to different universes from a given client.
From the client coding side, we have a medius client and a game communication library.
The game communication library is specifically if you're going to go peer-to-peer game.
Some of the features that we have, which I very highly recommend,
I would add that when we started off development of this medius SDK,
we probably had less than half of these kind of features up there
because we didn't want to go too high risk.
However, working really closely with Seth and Zipper,
we found that in their game design, they were very specific.
This is what we want.
As a real team effort, we came together and said,
okay, well let's make the network SDK exactly what you're going to need
plus more in the actual community building.
Every one of these components from the community building side is extremely important.
We have found that because so much effort was put into the community,
we have much greater numbers for this title online at any one time.
You can see the list of them there,
but we had a period of time where we decided to disable ladders
and the very day that we enabled ladders again,
we got a thousand extra simultaneous players
as if it just kind of fell through the community that,
hey, now you're seeing your rankings.
And so we found from just enabling that that it was extremely important.
I very highly recommend if you get a networking SDK
that it supports multiple platforms from the console and the server side.
In ours, we do PlayStation 2 Linux and Windows from the client side.
It's mainly for doing things like client simulations.
Once you have your community building environment,
you actually want to join a game or create a game.
We launch into a completely separate networking SDK
which is called the DME SDK.
The DME SDK is really tuned for very high packet rates,
a small, small packet size,
whereas a community building doesn't have to be as time critical.
What this allows us to do is separate the two components,
including the groups that are doing the development on.
Ours has a native object interface.
It also has the ability to do broadcast scheduling,
which gives you, instead of the application itself deciding when to update a network object,
the broadcast schedule will go in and do either error-limited or rate-limited decisions
on when should the network layer update the network.
It makes it pretty simple for the developer, from our perspective.
We have client side, server side filtering, a global time base,
very, very highly recommend some sort of time-based synchronization across the games,
specifically from the client perspective.
You're going to have weapons which are like projectiles or some sort of time-based event.
If you don't have a global time base,
I don't know if you would possibly be able to do those.
We have reliably and latency critical communications.
That really means that we support TCP, UDP, and a reliable UDP layer.
Make sure you're looking for each of those.
Definitely, security is a must.
We have both authentication at the PlayStation level, at the application level,
and then at the packet encryption level, if they want that, if the title needs that.
We're not too worried right now about peer-to-peer packet level security
because you're only affecting a small number of users.
We actually can handle a peer-to-peer connection, an integrated server,
meaning the PlayStation 2 is the server itself,
or we have what we call a user-distributed server
where someone can host a PC in their house and have clients connecting to that.
One of the things that's not really up here that was mentioned in CES is
that's extremely important is the NAT situation.
If you're using a middleware solution or some sort of networking SDK,
make sure they support NATs and firewalls.
This is really a basic figure of our architecture of the DME.
The DME is something that we put together at our time in 1995,
and so here in the PlayStation 2 generation, it's about the fifth implementation of this.
The medius components are actually built on top of this as well.
It's pretty aggressive.
You really have a standard resource, which we call the DME server,
and dynamically, depending on what application you're using
and what number of users you want,
a world gets created that the client specifies,
I want 16 players in this world,
I want 256 people in this lobby.
If that lobby gets full,
then you can have the client application launch another lobby that's an identical copy,
which is what SOCOM does.
The clients are PS2s.
We have a concept of a Session Master.
Go ahead and hit these buttons.
The Session Master can either be running on an individual PS2,
or it can be a separate computer.
This Session Master is really meant to do game-specific logic.
The servers that you see in this architecture are really generic servers
and don't have any concept of what the game logic is.
They're sending persistent object state or messages from each client to another client.
The Session Master is really an important entity, though,
because it does allow for, if somebody that's in the game as a Session Master leaves,
you can migrate that Session Master to another client.
That way, that same user control or object control in the game is still controlled,
and the game can continue.
I'm going to really quickly go over some of the other things.
When you're doing these online games,
you have to estimate what your simultaneous load is going to be,
and you really do need to have client simulators for that.
Ours are Linux-based.
We get about 1,000 clients on our production machines,
about 500 on a development machine,
which means we have machines that we can either use as clients, simulators,
or actually in production as server machines.
We use the development servers in our labs to give us about 500.
Right now, you can very accurately describe the lobby components,
which are what our medias components are,
but in terms of game logic, you really can't simulate the game logic at this point.
You'd basically be writing a completely separate game for a new platform,
so we're not suggesting that, but you can mimic the game traffic.
So we have our game simulators generating the same traffic that an actual client would have,
and we use Etherpeak to find out what those type of data rates are.
When we say loosely based on user behavior,
it's because now that we've launched, we do a prediction, we do our tests,
we launch the actual product, and then we go back through and say,
well, what are the actual users doing?
And we have found that even in that model, you have to keep revisiting what are your users doing.
They're doing much different things four or five months down the line.
This represents the approximate percentage of what we test and what our users are doing.
So if we're simulating 20,000 clients, we have this kind of distribution,
and that's what the game data has shown us as well.
If you're hosting the games yourself, or if you've got your own data center,
you're going to be extremely interested in this.
If you outsource the hosting, you're going to be extremely interested in the fact
that your hosting service is providing this kind of information to you.
This is really the server monitoring tools that will show you how healthy is the system.
One of the reasons I believe that we like to host the servers ourselves
is then we have the ability to do the user monitoring pretty easily.
So we can then see for this given application what are going to be some of the application counts,
how many hours are played, the same sort of data that Seth talked about.
It's this user data that's extremely valuable.
We can put new features into the game based on what our users are doing.
They look extremely sinusoidal.
Here's an example of one of the programs we have that goes in
and basically counts all the users that are in the universe.
The first graph, the blue shaded graph, is the number of users that are in the lobby.
So we can find out at any given point that the time scale is actually in minutes.
So at any point in any 24-hour period, we can figure out how many people are in there right now
by looking at these charts in like an Excel spreadsheet.
It also shows in the orange how many people are actually in the game at any given point.
These are extremely valuable.
If you look at the way SOCOM was working, we had two weeks into production.
We were at about just over 2,500 right there.
Ten weeks into production, we were at, looks like, not quite 4,000, but pretty close.
Now, we launched in September.
Now we're over three times that, which is showing the numbers from Seth.
This data shows some really interesting other things as well, not any of these charts.
But we had a period of time that we called, not only the latter showed us going up,
we had a period of time where we called it the Grand Theft Auto Effect,
where we had about 1,000 people for about a week that it went down.
And then it kind of came back up to match these sort of growth rates that we expected.
We've never really proved it.
It just happened to coincide with Grand Theft Auto coming out of Vice City.
That's it for me. Thank you very much.
Test for level? Okay, good.
Glenn works for an organization in Sony.
He pronounces S-C-E-R-T.
There are a few of us, however, who are a little disappointed in that pronunciation.
We actually thought we had a theme going here.
I mean, if we're going to be the zipper, he's going to be the skirt.
Anyway, I won't take that any further, Glenn.
I guess I've introduced what company I work for.
And I'm going to have to move right along through these charts because we have to be out of here quickly
at four o'clock, more or less, and maybe we'll get a question or two in even.
Let me just kind of fundamentally cut to the chase here.
Talk is centered around the in-game aspects of what Glenn was talking about.
That is not the community-building aspects.
Just what happens once you get in the game and want to play the thing.
Let me proceed to this chart, which is basically Glenn was talking about a summary of lots of servers
and worlds and APIs and so on.
And from my perspective, it all boils down basically to this.
I'm working with it via Glenn's API, working with a client server system.
That is, we send our messages, anything we want anybody else to hear or see to a server,
and then the server decides who else to send it to, all the individual players that it really needs to go to.
We have two modes of sending out messages with this API.
One is basically, actually, I'll start with the second one, the messages.
And that's what actually I had been more familiar with before seeing this API from Glenn as well.
But it's just sort of, here's a socket on the network, go crazy with it, you know, sort of thing.
So it's just basically send to whomever, whatever you want.
The objects, actually, the other mode of sending out data lend a lot more structure to what gets sent out.
In particular, with each of these objects, we have a schedule associated with it so that we can determine exactly when it gets sent out.
And that actually works to our advantage to drastically reduce the bandwidth that we would normally expect to use to play a game like this.
The DME takes care of the networking interface.
I think this is probably a good time to admit to the fact that I'm not a networking expert.
I couldn't dissect the TCP packet to save my life.
You get very familiar with it, but you don't have to be, and that's kind of the point of this,
you don't have to be a networking expert to use this.
I have more experience and background in the area of physics and dynamics and, you know, how to smooth and make the characters look more natural and so on once they get out there.
So I don't have to worry about all the nits of exactly what bits need to get set to get a message out there.
His interface and his library takes care of that sort of thing.
As also as Glenn mentioned, one player is designated as a session master.
And this session master has both some duties on the DME side, such as keeping track of the global time base and so on.
And into the game, we give him certain duties as well.
In particular, he's the scorekeeper. He's the master timekeeper.
He transitions from state to state.
He says when the game, you know, when we're done loading and now we transition into the game itself and so on.
The other note I wanted to make here is that the voice chat that we used in SOCOM was not based on any of the SCE-RT code.
But they've developed such an interface and I think in the future we're probably going to tend to look very closely at that as opposed to doing something custom.
And once again, since time is running so short, I'm going to frankly cut to the chase here.
I was going to talk about some examples of those messages and network lot, objects and so on.
But I think basically I'm going to kind of come down to the end here.
And from my perspective, this is sort of a hard chart to write to begin with, but how to stay out of trouble when you're developing a network game.
In particular, there are a couple of things to take away from this.
It's much easier if you're aware from the very beginning that you're designing a network game and not actually trying to transition a single player game into a networking game.
It's much harder to go that route.
It's easier if you know from the very beginning design it that way.
Although as we talked about before, I think that the world doesn't always like that.
So often you start with a single player game and make it a multiplayer game.
The other, some of the other bullets here basically, harking to the fact that what you want to do is not spread the networking.
Aspects of the game throughout the game, keep the interfaces very clean, keep the networking code localized.
Just probably some very obvious bullets for somebody who's been through this before, but that's the sort of situation that we have to live with and try to define these interfaces and keep them just every day.
The final bullet on this is that test as many ways as possible.
You can't test too much.
You can't develop too many test fixtures and you just can't invent enough ways to test this and determine what's going wrong when something actually does go wrong.
So plan on spending some time building up your test library inventory.
What went wrong with this particular game?
And the first thing I'm going to do is pick on Glenn.
And actually I'm not picking on Glenn.
I'm just sort of telling kind of the way it was.
Because the way it was turned out to be that he was porting his DME code from fundamentally from the PC to the PS2 just as we were trying to use it for the very first time.
And so that we were fundamentally both developing code.
He was developing the library and the API as we were coming along trying to use this thing at the same time.
Of course when we're doing that documentation always takes a hit in there someplace.
So we were on the phone a lot and I'll talk about that in a second here.
In server scalability that's just something we've been aware of from the beginning and I think Glenn mentioned that.
You just never aware what happens when you get 10,000 people out there when you're first developing this sort of thing.
A couple of other things.
Be careful how much code and functionality you actually get into scripts as opposed to the code itself.
Particularly if whatever scripting mechanism you come up with or are using is just not originally network compatible.
Once again test tools.
I won't dwell on that any longer.
Voice chat integration was an issue for us because once again it turns out a third yet another party developed that code.
And we had an integration problem on this IO process around the PS2 and that's a little bit of a tougher place to integrate code from several vendors.
And it just turned out to be an issue.
Obviously we got over it but it was more time than we thought.
What went right here.
And now I get to give Glenn some credit I think.
Because every time we found a bug or even beyond a bug an issue that we said okay hey we got to have this feature in here to make this thing work.
That group was very responsive so.
That kind of a tight coupling between code developers and network library support people just turned out to be to save the day frankly.
And of course their code stabilized probably slightly before ours so that we finally got everybody talking on the right track.
These interfaces I talked about before with the other aspects of the code are between rather the other aspects of the code.
I think we were able to maintain and hold those to workable levels.
And we had good interface between zipper and some of the other subcontractors.
It actually worked more on the media or community building part which is BEI 11's enterprises out of Idaho.
Okay I think that's I think we're more or less out of time.
I would like to take an unscripted minute here and thank Seth and his colleagues and Glenn and his group and as well as Sony QA and the IT groups from Sony for making this game the success that Seth described earlier.
Okay I don't know if we have time for a question or two or not.
Okay well I do want to remind people to fill out the yellow sheets.
Those are actually quizzes and I've been told the correct answer is five.
So I think we have time for maybe a couple quick questions if anyone has any.
So go ahead right here in front.
Well the question was about monthly bandwidth and cost associated with it in our levels and I think Glenn would probably be the best person to talk about that.
Since he knows more about the server status than I do.
Well I don't actually know what the bandwidth costs are because our IT department goes ahead and monitors that.
But our strategy using a generic engine these servers that I mentioned that we use for client simulations and we can use for servers we're continually improving the scalability.
So we're getting the number of servers that we're using down we've got when one application does really well we can take away from applications that aren't doing so well.
It's more of a manual grid process if you've heard of the IBM grid architecture but they are generic services.
So what we're doing for popular titles I mean at this point in time we're using the fact that there are online titles like a SOCOM that are selling through more numbers because they have online in them.
We're not actually charging here.
Now in the other territories around the world which I don't know if we can say that we're going to Japan and that they are wanting to charge for these titles.
So they have a different model financial model than we have in the States right now.
Yes I think from a production standpoint we sort of saw online as a you know a very big feature for the game and something that definitely is helping driving the sales of the game.
If you buy a PlayStation 2 network adapter it's kind of obvious that you're going to buy SOCOM along with it.
I think that's been you know had a very positive impact on our sales for the game in general as well.
Okay so I guess the question was about glitches or you know I guess in a lot of ways we refer to them more as exploits in the game that have been discovered and also sort of consumer awareness or community building.
As far as I think I did mention it in the slide which is the fact that you know one day we have more player hours than we had in our entire test development cycle.
I think if you look at online PC games you won't find a single game that doesn't have a patch following you know two weeks or so afterwards.
So I mean we certainly did the best that we could to address as many issues on the server side as we can.
And but obviously there's when you have so many people playing the game and they're hammering it so hard they're going to find exploits are going to find things that you certainly weren't aware were in the code in the game pretty rapidly.
Just because you have so many such a large number of people pounding on it.
As far as the community aspect I mean that's something we're still working on you know with the PlayStation 2 online just being released.
We're definitely starting to focus a lot more on that as we're moving out past it.
So I think that's something that you'll see a pretty big improvement on in both upcoming titles and for existing titles as well.
That's something we're definitely focusing on.
But of course something we want to do right as well.
In SOCOM the question is can you patch the client.
It is possible you know EverQuest online adventures does you know sort of patching updates to the memory card.
That wasn't really a viable option for SOCOM.
So there really is no way of patching the client.
Well I mean that's again that that sort of goes back to the point of public beta tests and making sure you get as much testing on it before you release product as possible because there is no easy way of taking care of that when you're working with a fixed based medium.
That can't really be updated easily.
So I mean that is definitely one of the challenges in creating the game to try to get as much testing and work out these issues as soon as possible before.
I'll quickly add that when when Seth was mentioning that it's the players themselves that start policing that has been really effective.
So a glitch a one kind of glitch was somebody got behind a wall that you weren't supposed to be able to get behind almost every game I've played and we've got a lot of people in here from QA played with somebody doing that unless they were trying to all do that.
The games the people would bump them out of the game even on their own team and you have to be on your own team to bump them out and that's the first thing they would do.
What would you give an example of it doesn't work for.
Yeah I mean it's it's a challenge no doubt.
But I think that's it we need to wrap up so the next group can come in.
So thank you very much for coming and please fill out the surveys as well.
Thank you.