Netcode optimizations (progress and good news)

[Saul]

Hi all,

I am a new programmer on the CSP team and I've implemented a few features and a few bugs that made their way into 1.07. However if you've been looking at the update page (http://update.cspromod.com) recently, you'll see that I'm already busy on 1.08. So far I've fixed bugs like:

Spectator jerkiness
Broken crouching in spectator
Viewpunch in spectator
etc.


But today I have exciting news. After playing a few games in 1.07 on populated public servers, it became apparent that the network is using far too much bandwidth. I've spent a good few hours today trying to find the cause of this and debugging several fixes. These are:

Un-networking some variables
Optimising the way we currently transmit network variables
and more!


To test my work I added 20 bots to my team and had them run/spin around while I watched the netgraph (if you want to reproduce this, make sure net_chokeloop is enabled). If you take a look at the image below:

http://i.imgur.com/xFWt3.png

You will notice that the choke in CSP 1.07 is ~50 with this activity in the game. This is a huge problem. The value at the end of the second line (50.8/s) shows us that we're only receiving 50 world updates per second, despite cl_updaterate being set to 100. The rate variable limits the amount of data the server will send to/receive from us per second. Because so much data is being sent to us per world update, we're choking on data. However, after my changes to the netcode, the choke is now 0 in 1.08.

The other value in the image that is highlighted red (568 and 370 for 1.07 and 1.08 respectively) is the
size of the last world update. Using this data I've estimated that player bandwidth has been reduced by approximately 35%. This means that in 1.08 there will be no more choke (at rates >=75,000) and a vast improvement in smoothness.

I was going to simply post this as a status update but I thought more information would be a good read for the people in our community who are technically minded or just interested in the netcode.

Note: these tests were completed with rate 75,000 and cmd/update rates at the default of 100.

[[EA]]

Good write up Saul. Keep these updates coming!

[Otsche]

n! work

p.s. one of the first times, that the community gets an real informative devlop status, plz keep this spirit

[skykov]

amazing work! but shouldnt the lerp be 10.0? the correct value for cl_interp_ratio is 1 not 1.1

[majkool]

Great news!
Maybe you will upload 1.08 in few days then we could already test it.

[howligH]

Sounds awesome!

Hope you dont let us wait too long for at least a little update to the game.

[wHEElEr]

Wow Saul you conquered that netcode :D

Great job!

[Saul]

amazing work! but shouldnt the lerp be 10.0? the correct value for cl_interp_ratio is 1 not 1.1
Yes, I have no idea why it's 1.1 (that seems to be the default)


Great news!
Maybe you will upload 1.08 in few days then we could already test it.

Sounds awesome!

Hope you dont let us wait too long for at least a little update to the game.
Unfortunately when we release isn't my call. If it was up to me we'd have an auto-update system and this would've already been rolled out :p

[majkool]

Saul

So do it! :D
Auto-update system will be awesome for CSPromod future.

[hare89]

Great job saul ,i am just curious is the gameplay smoother and is the interp issue fixed ? :D

[THEiNTERNETS]

Is it not possible to release these via steamworks and create smaller update cycles? Or is VALVe uninterested in having you guys on there?

[dsLoL]

This is what CSP NEEDS regular updates. Great work guys!

[Otsche]

Saul

So do it! :D
Auto-update system will be awesome for CSPromod future.

you can ask gabe friendly, but i think he dont likes csp very much :D

[Saul]

Great job saul ,i am just curious is the gameplay smoother and is the interp issue fixed ? :D
I would definitely say so (spectating is smooth as silk now too). And it depends which interp issue you are referring to :P


Is it not possible to release these via steamworks and create smaller update cycles? Or is VALVe uninterested in having you guys on there?
It is possible to create smaller update cycles but once again it's not my call, and we wouldn't need Steamworks for auto-update, we would just roll our own.

[Otsche]

and we wouldn't need Steamworks for auto-update, we would just roll our own.

and why do you dont do this?

that would be an incredible important step to get a growing community

[majkool]

Please do it, it's not that huge for You but for players is it.

We need 0 choke to pleasant play.

[hare89]

I would definitely say so (spectating is smooth as silk now too). And it depends which interp issue you are referring to :P


It is possible to create smaller update cycles but once again it's not my call, and we wouldn't need Steamworks for auto-update, we would just roll our own.
Well i am just guessing that its interp , maybe it has something with hitboxes or tagging, but sometimes its really hard to get one bullet kill or tap kill with m4a1/ak47 like you can get with cs 1.6 , dunno maybe it was caused by this choke we shall see :D

[sIMson]

This shitload of choke is quite a serious problem and should be fixed ASAP, along with some other major bugfixes and released like 1.06 was within a ~month.

[majkool]

hare89

I think it was caused by this netcode bug, in 1.06 version we can clearly 1shot kill.

[mendoza]

I was going to simply post this as a status update but I thought more information would be a good read for the people in our community who are technically minded or just interested in the netcode.

Thanks to take the time to explain us what you did, continue! :)

[wazer]

Thats the fight Saul, now push alex or who is in charge for an auto update tool. :D

[Massey]

Thats the fight Saul, now push alex or who is in charge for an auto update tool. :D

I agree, auto update would be awesome. :)

[juks]

great job Saul, thank you
netcode, the most important thing

Gogo steamworks csp !

[reima]

Huge props to you Saul for taking your time to write this dev log, it was a really good read. Keep up the good work!:)

[xtobix]

nice read. now release it.

[n3krO]

Well i am just guessing that its interp , maybe it has something with hitboxes or tagging, but sometimes its really hard to get one bullet kill or tap kill with m4a1/ak47 like you can get with cs 1.6 , dunno maybe it was caused by this choke we shall see :D

interp has nothing to do with hitboxes...

Saul, i have some questions...
Server has deflauted 75000 sv_maxrate? it would be 2000gb per month if the server is 24/7 full (aproximatly), also, i'm using the deflauted 25000 rate, may i change it to 75000?
Is cmd and backuprate worth to put in an higher number or it would lag the server for so many worldupdates sent? i mean, with 12 players and cmd/updaterate 100 the server would send 1200 world updates per second, with 300 it would be 3600, would that lag server or why don't you let us have higher cmd/updaterate?.
How does interp work in CSP? In cs1.6, interp 0.1 causes the game to interpolate the position of the world 100ms after the last world update sent. In CSP i believe interp 1 = ex_interp 0.1 from cs1.6?
Is it posible to desactivate interp? In cs1.6 if you type interp in console you desactivate it, it's usefull when you are playing with 2/3 ping in lans ;)

Now respecting to spectating, you said now we see 100% what the player we are spectating see, does this apply to the flashbang bug? Also, if i'm dx81 and the player im spectating is with dx95, do i see flashbang effects of dx81 or of dx95? :P

I love to see that updates posted after release a new BETA is huge but, are you thinking about keep us refreshed so frequently or will CSP forum return to his nostalgy after 1 month?

[Saul]

interp has nothing to do with hitboxes...
It does. Because interpolation is rendering in the past, and when you shoot, the server tries to guess what was on your screen at that time. It's called lag compensation (http://developer.valvesoftware.com/wiki/Source_Multiplayer_Networking#Lag_compensation).


Server has deflauted 75000 sv_maxrate? it would be 2000gb per month if the server is 24/7 full (aproximatly), also, i'm using the deflauted 25000 rate, may i change it to 75000?
25,000 as the rate is far too low, and will cause a large amount of choke.

Is cmd and backuprate worth to put in an higher number or it would lag the server for so many worldupdates sent? i mean, with 12 players and cmd/updaterate 100 the server would send 1200 world updates per second, with 300 it would be 3600, would that lag server or why don't you let us have higher cmd/updaterate?.
cl_cmdrate and cl_updaterate should not be higher than the server tick rate (100), so increasing updaterate would do nothing, but I'm not sure about cmdrate (probably nothing too).


How does interp work in CSP? In cs1.6, interp 0.1 causes the game to interpolate the position of the world 100ms after the last world update sent. In CSP i believe interp 1 = ex_interp 0.1 from cs1.6?
Interpolation is documented on the Valve developer wiki in great detail (http://developer.valvesoftware.com/wiki/Source_Multiplayer_Networking#Entity_interpolation ).


Is it posible to desactivate interp? In cs1.6 if you type interp in console you desactivate it, it's usefull when you are playing with 2/3 ping in lans ;)
Nope :P


Now respecting to spectating, you said now we see 100% what the player we are spectating see, does this apply to the flashbang bug? Also, if i'm dx81 and the player im spectating is with dx95, do i see flashbang effects of dx81 or of dx95? :P
No again :P


I love to see that updates posted after release a new BETA is huge but, are you thinking about keep us refreshed so frequently or will CSP forum return to his nostalgy after 1 month?
It won't go dead, either.


Huge props to you Saul for taking your time to write this dev log, it was a really good read. Keep up the good work!:)
Thanks, I'm fixing the crashes at the moment and as soon as the smoke crashes are fixed I'll post another dev log.

[Twelve-60]

/snip

Awesome :D That spectator choppyness was really a big problem in my eyes :) Was it just enabling interpolation for specs that fixed it?

[pR8]

GJ man

As far as i see CSP team shouldn't ask valve for help cuz only thing they could do is fuckd up game even more. Cs was free hl mod now u have to pay for it, what they would do is try to take over csp so next one would be the one you would have to pay for.

CSP team did a lot of work and read a lot of flame from community, so I have a hope there will be no valve in this game

[yinscrubber]

Great work Saul!

Keep these updates coming, especially with the devlogs as well, keeps people interested :D

[free]

Awesome :D That spectator choppyness was really a big problem in my eyes :) Was it just enabling interpolation for specs that fixed it?
No. It needed few fixes and more complicated than interp off -> on :).


I'm really shocked nobody asked 'What smoke crashes?'.

[n3krO]

No. It needed few fixes and more complicated than interp off -> on :).


I'm really shocked nobody asked 'What smoke crashes?'.

I supposed that it was a crash result of one beta test O.o

Anyway, what is that crash? :P

Saul: Are you trying to fix the name of the players thought smoke? If i'm right if we try to look though a smoke we see though +/- 12sprites. You could try to enable the show name when looking at player thought 3 or less sprites. What do you think? :P

[Lordearon]

Saul, have you been working on csp 3 days straight?

Quick, release 1.08 before some game company steals you away from csp?

btw, did you have previous mods you worked on? Happy to have you working on csp, you're doing a great job (together with amogan, free, redstar and the other crew of course!)

ah, found your website:
http://saulrennison.me/

good luck with your objectives for cspromod you have listed.. looking forward to that
you might want to have a look at this:
http://cspromod.com/community/threads/3488-CSP-AI-Competition
http://cspromod.com/community/threads/3149-intelligent-bot-after-1.05?p=47358&viewfull=1#post47358

as csp is still under heavy dev and I have had no time so far anyway, I haven't continue work on that.

[drosalion]

does this help solve the issue of kills always being delayed and you can never instantly tell whether you get the kill or not? (everyone i know who plays CSP has this problem)

[yuri]

does this help solve the issue of kills always being delayed and you can never instantly tell whether you get the kill or not? (everyone i know who plays CSP has this problem)There are many delays. Animation + blood particle delay, lag and the interp itself.

[Mystifax]

Excellent!

[duydangle]

Update update \m/

// Sorry it is so exciting.

[Moffasin]

Great!

Btw:
Why don't you default to 0 lerp?
Everyone is using it in DoD:S.

[ilam]

Saul, thanks for all the effort and fixes! Also, i've been wondering.

It has been known widely that the default rate settings that ship with the CSP installer is way too low (35,000). Why is it that they just didn't set the value higher? I've noticed that maxing out the rate - somewhere around 131000 - makes choke disappear but i'm not sure about how smooth it keeps things since i haven't done quality testing.

Did your testing on this issue end up concluding that with the changes you've made we will be able to finally handle all 100 updates sent by the server (and not half) therefore making it a smoother and more responsive experience? Will 75,000 now be the default Rate value? I don't see the point of making it a lower value since according to you, this works wonders.

[yuri]

Why don't you default to 0 lerp?
Everyone is using it in DoD:S.Most probably the 0 lerp is only a bugged value. 0 ms of interpolation time would mean no interpolation at all.

[Saul]

Awesome :D That spectator choppyness was really a big problem in my eyes :) Was it just enabling interpolation for specs that fixed it?
I changed spectator eye angles to use our m_angEyeAngles interpolated network variable instead of pl.v_angle.


Anyway, what is that crash? :P
If a smokegrenade is active at map change, all/most clients crash when they spawn.


Saul: Are you trying to fix the name of the players thought smoke? If i'm right if we try to look though a smoke we see though +/- 12sprites. You could try to enable the show name when looking at player thought 3 or less sprites. What do you think? :P
Particle effects don't have sprites, but this will be fixed.


Saul, have you been working on csp 3 days straight?
Think so, yeah, I haven't really kept track :P I'm gonna fix the crashes today.


btw, did you have previous mods you worked on? Happy to have you working on csp, you're doing a great job (together with amogan, free, redstar and the other crew of course!)
Not really mods, but I've worked on The Axel Project (more actual engine programming than game), and lots of work on EventScripts (server side plugin).


good luck with your objectives for cspromod you have listed.. looking forward to that
you might want to have a look at this:
http://cspromod.com/community/threads/3488-CSP-AI-Competition
http://cspromod.com/community/threads/3149-intelligent-bot-after-1.05?p=47358&viewfull=1#post47358
Not a priority at the moment but that will be completed by me.


does this help solve the issue of kills always being delayed and you can never instantly tell whether you get the kill or not? (everyone i know who plays CSP has this problem)
Afraid not. I'm not sure what this is but incubus' looking into it I think :)


It has been known widely that the default rate settings that ship with the CSP installer is way too low (35,000). Why is it that they just didn't set the value higher? I've noticed that maxing out the rate - somewhere around 131000 - makes choke disappear but i'm not sure about how smooth it keeps things since i haven't done quality testing.
I'm pretty sure default rate is 100,000 and the default maxrate for dedicated servers is 131072. However, maxrate on listen servers is infact set to 35,000 by default, but this is easy to change in the cfg/listenserver.cfg file.


Did your testing on this issue end up concluding that with the changes you've made we will be able to finally handle all 100 updates sent by the server (and not half) therefore making it a smoother and more responsive experience? Will 75,000 now be the default Rate value? I don't see the point of making it a lower value since according to you, this works wonders.
75,000 will not be a default value for the rate as it won't matter if rate is set to 100,000 or 1,000,000. It's just a bandwidth limiter. If you set your download limit to 1MB/s on uTorrent, but only download at 300kB/s, you will not notice anything different.

[nki1]

Hm anybody knows if the animation delay was in 1.04? If no, then it might be an issue with the anims itsself I guess.

[THEiNTERNETS]

I'm just going to suggest again that releasing updates on a smaller timescale, even if they are much smaller fixes, would improve the morale of the community greatly (not to mention your team, I suspect) and lead to increasing coverage by the rest of the esports community. The whole "promod is dead" bullshit claims that have been haunting you guys are manufactured by the time between these releases coupled with an immature/skeptical community. You can't do anything about the latter, but you could definitely change the former.

Might I also suggest that expanding your beta testing roster would diversify the hardware you're testing on which could make it easier to pinpoint esoteric or hardware-specific bugs.

Finally, I don't see any particular reason to delay the net code fix especially if you can begin applying updates in a way that doesn't require a full rewrite of the mod's files for each change. If you don't have the ability to do this yet and this is the cause for such a drawn-out update cycle, I'd think it should be priority uno.

[n3krO]

I'm just going to suggest again that releasing updates on a smaller timescale, even if they are much smaller fixes, would improve the morale of the community greatly (not to mention your team, I suspect) and lead to increasing coverage by the rest of the esports community. The whole "promod is dead" bullshit claims that have been haunting you guys are manufactured by the time between these releases coupled with an immature/skeptical community. You can't do anything about the latter, but you could definitely change the former.

Might I also suggest that expanding your beta testing roster would diversify the hardware you're testing on which could make it easier to pinpoint esoteric or hardware-specific bugs.

Finally, I don't see any particular reason to delay the net code fix especially if you can begin applying updates in a way that doesn't require a full rewrite of the mod's files for each change. If you don't have the ability to do this yet and this is the cause for such a drawn-out update cycle, I'd think it should be priority uno.

Well, why not weekly updates? and start doing it like this:
CSP 1.07.1
CSP 1.07.2
CSP 1.07.3
-
CSP 1.07.5
-
-
-
CSP 1.07.9
CSP 1.08 and so on? :P

If in 3 days you could spot to many things just take a look how would you work if we get updates weekly?
You could reserve a part on the wiki for weekly updates and delete all weekly updates after 10 weeks (when the cycle finishes and a new one start) and put the link of full 1.08 in the normal download page of the wiki :P
(just an idea).

[tiger]

The fact that Saul did a lot of hard work these last few days doesn't mean it's realistically possible to have weekly updates. Or at least, release worthy ones.

[n3krO]

The fact that Saul did a lot of hard work these last few days doesn't mean it's realistically possible to have weekly updates. Or at least, release worthy ones.

but in 10 weeks you can get a worthy full release no?

1 small update is not worth, 10 small updates are worth.
If community sees the game being updated weekly, it will never run out of players like with last versions.
If you archive do get into steamworks, it would not be hard to do weekly updates (or even update when a fix is full tested and tweeked)

[sIMson]

Hm anybody knows if the animation delay was in 1.04? If no, then it might be an issue with the anims itsself I guess.

1.04 had no death animations, it was pure ragdoll. Also, you might have a point here, I don't remember any lags in 1.04

[n3krO]

1.04 had no death animations, it was pure ragdoll. Also, you might have a point here, I don't remember any lags in 1.04

in fact, 1.04 was very stable. Let's hope 1.08 is like 1.04...

[danterossi1]

nice work, keep it up

thanks for the informative, good job! :)

[oldie]

Wow, great job! Keep it up.

[free]

Just remeber that 1.04 was on a older engine.

[Moffasin]

Most probably the 0 lerp is only a bugged value. 0 ms of interpolation time would mean no interpolation at all.

And what is wrong with having no interpolation? I dont mind it looking glitchy if it shows me more accurately where people are at.

[yuri]

And what is wrong with having no interpolation? I dont mind it looking glitchy if it shows me more accurately where people are at.Do u know what the interpolation does, right?

Because having a fully discrete player positioning system is like the worst idea ever.

[n3krO]

Just remeber that 1.04 was on a older engine.

that's why i hope csp1.08 is as stable as 1.04 was in source2006...


Do u know what the interpolation does, right?

Because having a fully discrete player positioning system is like the worst idea ever.

In cs you can only interpolate to a minimum of 10ms, this means that if you have less then 5ms lantecy, it would be better to use no interpolation (lan matches?). You could desactivate it in GldSrc engine but not in source engine :S

[Moffasin]

All I can say it works fine in DoD:s without any obvious drawbacks so I dont see what the big deal is lol.

[yuri]

In cs you can only interpolate to a minimum of 10ms, this means that if you have less then 5ms lantecy, it would be better to use no interpolation (lan matches?).You are confusing the line/route delay with compute delay. Thus interpolation and prediction mechanisms.

U know, computers are somehow discrete machines in nature (CLK). The entire game world is simulated discretely tick by tick. In case of CSP those ticks occour 10ms after each other => 100 ticks per second (ofc only when a CPU can handle it 100% constantly, which is almost impossible).

Now, imagine 0 line latency. You would have updated the player origin computed with a period of ~10ms. What would happen between those updates!?

--- Update ---


All I can say it works fine in DoD:s without any obvious drawbacks so I dont see what the big deal is lol.Is interpolation turlly OFF or is it just a display bug.


amount of linear interpolation (lerp) = cl_interp_ratio / cl_updaterate

What will happen if you set the interp ratio to 0? The output will surely be zero lerp. Howerver, is the linear interpolation OFF? I don't think so...

[free]

If we turn interpolation off then player model would jump from spot to spot within 10ms delays plus your ping. No walking animation would be there.

If you get an update pos=10 and then after 10ms you get pos=20 then your game will render player at pos=10 and pos=20 respectivley. Interpolation adds a delay and moves playermodel back those 10ms and starts a walk animation from pos=10 to pos=20 when you recieve the pos=20 update. This is how it works in 1.6 and every other game.

[Moffasin]

What will happen if you set the interp ratio to 0? The output will surely be zero lerp. Howerver, is the linear interpolation OFF? I don't think so...

No but I bet it would be closer to zero than 10 would be.

And anyway if it is as you say that it isnt actually 0, what is the harm in putting it to 0 then?

[yuri]

No but I bet it would be closer to zero than 10 would be.

And anyway if it is as you say that it isnt actually 0, what is the harm in putting it to 0 then?Have you actually bothered to read one of my or free's posts?! Or do u have some reading problems?


You are confusing the line/route delay with compute delay. Thus interpolation and prediction mechanisms.

U know, computers are somehow discrete machines in nature (CLK). The entire game world is simulated discretely tick by tick. In case of CSP those ticks occour 10ms after each other => 100 ticks per second (ofc only when a CPU can handle it 100% constantly, which is almost impossible).

Now, imagine 0 line latency. You would have updated the player origin computed with a period of ~10ms. What would happen between those updates!?

[Moffasin]

Looked in the code and it seems like 0.0099999998 is the lowest you can get.

[free]

Can you guys read and reason properly? Interpolation has to be here as in every shooter game if we want movement animations like walking/running etc.

[Saul]

Can you guys read and reason properly? Interpolation has to be here as in every shooter game if we want movement animations like walking/running etc.
Even if you have 1000ms interpolation, it doesn't matter. When the server is processing your movements, it moves the world back in time to what was on your screen when you moved/shot/whatever. This is called server-side lag compensation.

As long as everyone has the same interp ratio, nobody has any advantage (apart from those with lower pings I guess).