A Technical Solution to Cheat Detection from an Engineer
August 9, 2019 5:47PM
edited August 2019
From a technical perspective, computational cost of running battle simulations on the servers is a limiting factor in achieving true automated cheat detection in the game. Simulating a battle is magnitudes greater in terms of computing power required than any other server operation, which of course directly relates to non-negligible costs. Politics and economics aside, EA needs a solution that is not prohibitively expensive.
So, here it is: peer-to-peer battle data validation:
Obviously, the minor technical details need to be hashed out, but from a pure complexity standpoint you achieve reliable (decentralized) validation without significant server side computational overhead. You don't need to go full crypto blockchain on the game to regain the trust of the users, but this would surely go a long way in identifying who is cheating.
So, here it is: peer-to-peer battle data validation:
- After a battle is complete, put the payload data (random seeds, moves, result) into a queue.
- Clients who do not opt-out of participation will automatically retrieve the data, run simulations on the user device, send the results back to the server to either raise alarm or mark the request finished.
Obviously, the minor technical details need to be hashed out, but from a pure complexity standpoint you achieve reliable (decentralized) validation without significant server side computational overhead. You don't need to go full crypto blockchain on the game to regain the trust of the users, but this would surely go a long way in identifying who is cheating.
3
- 61.6K General Discussion
- -35 Developer Q&A Archive
- 2.6K New Players
- 30.7K Arena and Character Strategy
- 3.2K Individual Events
- 5.5K Guild Events
- 4.7K PVP and Rivalry
- 51K Guild Recruitment
- 751 Art and Media
- 4.2K Off Topic
Howdy, Stranger!
Replies
It should not take very long or require a lot of data, to transfer the random seed, the data of the involved characters, the moves played and then run the combat and return the result to the server for verification. Flag for inspection if different, and ban accordingly.
This is definitely doable.
You would at least need a wifi-only setting depending on the overhead. i think overall a lot of people would be more than willing to let this do its thing even if you did leave the option to disable. it does point out the major risk though: having adequate numbers to support the whole thing. worse-case scenario you can scale down or clear part of the queue if it starts getting backed up. but, that's something you can mitigate.
All jokes aside, thank you for trying to come up with solutions to problems. I truly enjoy this game, and would hate to see it die due to rampant cheating.
If we cannot trust data sent from cheating device, there will always be a loop hole.
Plus I'm not a fan of any processing done on battery powered device which I didn't ask for. It could be quite controversial feature. I wouldn't be surprised if this was against some policies on Apple Store or Google Play.
If I understand correctly, most cheats involve the client sending the wrong combat result to the server. It does not involve manipulating char/ship stats permanently, because I would suspect the server would pick up such a mismatch immediately (it should at least).
With a fixed input data (combat participants, set of player moves), the output data is always the same. It's like a transcribed game of chess. So the device is not going to play through the battle several times and determine if it may be possible - it will just go through the fixed algorithm of one set combat and return the result which will always be the same. No animations, no waiting for input, no randomness, just calculating a function, which would not take much processing power per combat. Obviously it takes too much power to do it server side, or they would be doing it already, which is why distributing these small calculation to other random clients may be a viable solution.
It is not against any policies if the player accepts it, by using the app. No data would be transferred or handled unless you are actively playing the game. You help verify one combat just before you do your own combat - and you wouldn't even notice it happening.
You must have missed something in the description, but this is the precise problem that you solve by redistributing the check to other clients. You can't trust the local client to validate its own battles, it needs to send the battle data to the server, which then randomly distributes validation of the battle to other clients. Client-side detection is a non-starter unless the client is validating OTHER peoples' battles.
It's also likely not against any sort of app store policies: it's a game that already sucks up an incredible amount of battery with the 3d graphics alone, and you only need to be validating battles while the app itself is open/the user is engaged. These battles don't need to be validated quickly or at any particular time, once you catch a cheater 1 time and suspend the account it's months or years before they get back to doing it again on another account at that same level. Simulation calculations take up only a miniscule amount of battery compared to the 3d graphics and display related to the game.
Also, it's based on the assumption that running an active simulation is the way they detect cheating. I would think there are less computationally intense metrics they could compare the result to.
yes. all of this is done in the background, users totally unaware.
it is much less efficient in terms of net computing power, but it distributes the cost to the clients. currently, the server is little more than a database extension: it just records results from the client, it doesn't really do much in terms of heavy calculations. running simulations on millions of battles however takes extensive processing power, I have to assume the only reason they're not already doing this is because they've already run cost estimates and they're higher than they're willing to absorb. in an ideal world, they would do the sims on the server, but the only reason i can think of not to do that is computing power. cloud resources are expensive, setting up and maintaining in-house servers to do the work would be even more expensive than that. so, decentralize and send it out.
As far as being "really sketchy" goes- it's not even close to as sketchy as some of the current implementations- they literally send usage feedback for every button press, scroll action, character view, etc that you do in the cantina, or at least i'm pretty sure they used to do that. Battle data can relatively easily be sanitized before being pushed to the queue, so unless they're just not using standard security practices with their code there's little to no added risk to the user. I honestly wouldn't be surprised if they were able to determine with a high level of accuracy the risk of developing addiction to the game (or really addiction in general) that a given user has based on metrics on how they use the client.
In terms of detecting long shots- sure there are possibly some metrics they could use, but for everything else the math and mechanics are so complex that any minor modifications to battles that would be close but could be won are easy to slip through. in other words, catching blatant cheaters probably isn't that hard, but somebody looking for that little extra edge in a normally 30% arena match is a bit harder to prove to the point where you're comfortable banning an account.
traditional methods like hashing the client won't work because they don't address the core flaw: there's almost nothing that can be done to completely prevent client side memory tampering, especially since a huge part of the user base is on various builds of Android.
You want to know why they can't do a specific thing to detect cheaters and at the same time don't want to know anything about how the cheating happens? I'm not sure how to answer your inquiry 🤷♂️
Thats why its a question and not a statement
I get that it's a question. I just don't think it can be answered the way you want it to be (without at least some explanation of how the cheating happens).
It doesnt need to be answered really. I was hoping to spark new ideas by looking at it from a different angle.
Unfortunately it doesn't do what you hoped. There is no sure fire, or even high probability, way to detect this type of cheating on the client it occurs on. And to explain why someone would have to go into how the cheating happens which would go against both your wishes and forum rules.
Again, it also assumes running battle simulations is the best method for flagging potential cheaters, which I also think is impractical. It's going to be probability based, so investigation will be required regardless. Surely there are other metrics that can be used, whether they are more face value or technical.
That does not solve the problem I described. If the cheat sends results which are possible with the given setup, nothing will be detected. Very raw example: you have Jawa team and it can sometimes beat Geonosians. But depending on RNG it may not work all the time or you may loose quite few Jawas. You just need to see data of one battle where it work and cheat can send similar every time - cheaters Jawas will always win. We can go even further with mirror matches. To detect such anomaly, you need server side processing and some ML on top of historic data.
I don't how exactly Android / iOS development works. Is there a secure storage? Is there per app certificate store? Is it possible to play the game on rooted / jailbroken devices? Does rooting / jailbroking exposes secure storage or certificates? Because all this cheating is tampering with data send to server and that would not be possible if data were signed unless the cheat can access the signing key / certificate.
The problem you described isn't actually a problem. Gather the first random seed from the server, or base any random seeds off of the battle time and validate against it, sending duplicate battle results from old battles will no longer be an option if it even is right now. You don't need server side processing at all, you just need a way to validate a deterministic input in the data.
It's quite easy to "root" an Android operating system, some phones allow this out of the box others you can easily install a custom image over unlocked bootloaders. I think most of the tampering is probably done on emulators just because it's an easier environment to work with but I don't know that for sure. The actual cheating that has been unearthed recently and has gone undetected has to do with client-side memory tampering. There's not a great way to detect this because anyone able to do this kind of tampering is usually able to tamper with the checks as well, unless perhaps they came up with some clever trickery. Even detecting if an operating system is "rooted" is pretty much impossible for people who take the right steps. Even then, people with rooted systems are sometimes people who spend a lot of money on games and don't actually cheat.
As far as signed data goes, it's possible they could come up with a time-based hashing algorithm that's difficult to spoof, but if you can tamper with the application memory in the first place it's easy enough to just send the server the results it wants from any checksums or anything like that. It's also something you'd have to stay on top of every step of the way- literally any move can be tampered with at any time and one can revert right back to the stock data set.
if you can tamper with the program memory to do the client-side cheating we're seeing, then you can pretty easily spoof the validation data that the client would send to the server. that's why they're having trouble detecting some of these cheats. Using your car alarm analogy: remember that the car alarm is to stop the common street thief, it doesn't prevent your car from being stolen it just brings attention to tampering. If a smart enough thief comes by and has enough time, disabling or subverting that alarm isn't all that much trouble.
Could they make it so tampering with the memory forces the client to crash?
The amount of work scales with the total number of users, and a lot of users are active in the client when they're not really doing much. Most of the processing power needed for the game has to do with the graphics- battle simulations are fairly straightforward algebra that can easily be accomplished by any phone capable of running the game in down time.
Even if you run into a shortage of validators, you can always reduce the queue. not every battle needs to be validated necessarily either - roll it out on arena, tw, ga, expand to raids and special events. Worst-case scenario they could offer a small amount of crystals per 100 battles validated or something like that, this is what bitcoin and other cryptocurrencies do to award users for donating processing power to validate transactions. This decentralized, scalable, peer to peer form of validation isn't something new- the idea comes from tried and true technology. We're over a decade into the bitcoin thing at this point and despite massive incentive to hack it the protocol has held up incredibly well (though a lot of exchanges and other supporting systems haven't been so lucky)
Your right i have no idea about any of this but if one of my hair brained ideas makes someone say "that wont work but maybe this will" its 100% worth looking like an ****. A different perspective can shine a whole new light on things.
understood, just don't want you to rack your brain too hard going down the path of client-side detection as the solution when the cheating mostly stems from the client itself being untrusted.
Interesting. So what you are proposing actually means:
Are RNG implementations on all platforms same?
Isn't purpose of obfuscation to make it harder to to tamper with code and purpose of whitebox cryptography to make it harder tampering with client-side memory? I'm not saying they are solution, I'm just saying that tools to make this more robust even on rooted phone should be there. They cost quite lot of money and are very complex to use right but hey, game like this is making $100M+ so money to get the right tool and right devs are there. More money you put into obfuscation and whitebox crypto, more difficult it will be to get around them and you can even mess with that with every release of the game so hackers may find it not worth it if it will be changing every few weeks.
Saying that, there are enough money for them to run their own farm of phones doing just validation. Or simply make system like you are proposing server side - I find it more maintainable ,reliable and easier to make it robust and deliver.
For know it would be enough to start with battle log people are asking for. Not only to detect cheating but also to see what can beat your teams.
Which is exactly why you would do it server side. While cloud computing resources have a cost, "expensive" is a purely arbitrary term. Generally, compute and inbound data are relatively inexpensive compared to storage and outbound data. On-demand pricing for an hour of processing on a 96 vCPU EC2 instance is under $6.
The hourly cost of the resources needed to build your massively distributed validation platform is on the order of 20 times that, for a single hour - and building massively distributed platforms is non-trivial. It is certainly much more complicated than building a single, scalable event processor that could perform validation like that described above. If
compute and bandwidth are such a significant concern, you could always introduce some rule filtering to flag questionable matches and fine tune those rules as you learn more. However, I'm inclined to think that executing the rule set is probably more CPU intensive than just running the battle actions through the simulator/validator.
I'd even be willing to bet that if CG made the relevant portions of their battle algorithms available and a pipe to consume battle log messages, they'd have a couple of free solutions in place in short order.