dealing with unfair reviews - a rant

privateer
I was pleased to see that the unfair single star "review" Craig's new game received within an hour or so of publication has been removed. I suggest though that the likes of that reviewer have consigned other games to oblivion by similar meaningless arbitrary ratings.

That type of "reviewer" has a certain profile: many games in the sandpit, usually makes multiple "reviews" within a very short space of time, often tries to advertise their own games - often asking for good ratings or demanding they not receive bad ones, one star ratings of decent games are accompanied by short, meaningless comments which suggest (or even prove) they have not played the game, "reviews" and comments suggest illiterate or very young individuals - even taking into account the fact of non-native English users.

An attack of such "sandflies" can unfairly sink a game from public view. A good example is my father's game Curse! by Peter Edwards. A sandfly bit with a one star review saying he didn't know how to start the game. The game was then changed to make it simpler for those with less patience or imagination. However, another sandfly (who had been in conversation with the first on their own sandpit games) chimed in with a low rating of her own, admitting that she had not played the game but wanted to reiterate the point made by the first. She had not even engaged with the game to find out what it was, but was simply rating in support of her sandfly friend.

I'm sure there are other games that have also been blighted by immature abuse of the open ratings system. Although it's annoying, we had reconciled ourselves that perhaps it's just an unavoidable unpleasantness. But if blatantly thoughtless and unreasonable reviews of that sort can be identified and dealt with, perhaps it would be fair to have a procedure for appeal. (The trouble is, of course, that every sandfly would be complaining to the Admins about their own (quite deserved) low ratings!)

In the end, it's a shame that idiots can have such an effect on the visibility of games into which some thought and energy has been invested.

I wonder what the solution is? It would be nice if Curse! and other unfairly blighted games could get some justice too. Failing that, some sympathy and a cup of tea would do.

Rant over. Merry Christmas everyone (even sandflies)

davidw
While I agree 1 star reviews like that are certainly unfair, in a way they do a good job of balancing out the multiple 5 star reviews so many games on the site have. And let's be honest, some of the 5 star reviews are as bad as the 1 star reviews. From "Dream Pieces":

"First one I ever played! It's really fun!"

"Great Idea of a game"

"amazing game i love how its set up it all fits together perfeclty! these are the text adventures id like to see from others useing thier creativity and mind"

"Very creativy game"

privateer
What is being balanced out by that? *If* those high ratings are unfair as you intimate (and I don't suggest they are) then surely they accentuate the problem, not balance it out?

The trouble is, so few other people use the ratings system, these idiots get to have a huge influence on the position of any game that happens to catch their attention.

Democracy has its flaws. Ratings should have to be approved by an hereditary House of Lords...

privateer
From our most prolific games 'reviewer', who has rated no less than 50 games, sadly effecting games not only in the sand-pit:

Comment by Claire6129
14 Mar 2014

I rated this a 5 star just because i wanted to get you to like me, to rate my games more than 2 star.

Review for The survival Chapter 1
10 Mar 2014
I liked it. Since you aren't very considerate with what you rate my games, though, I'm giving it a 3.


Above are just 2 examples of her most recent contributions, all of which seem to be cynical attempts at horse-trading stars, blatant revenge scoring, or sycophantically echo-ing extant reviews without playing the games.

A network of children having fun with the ratings system in the sandpit is one thing, but when it leaves the sandpit it is a bit damaging if we want people to easily find the best games on Quest.

davidw
I'm curious as to whether anything will get done about this. There was a lengthy discussion about Quest games on the Int.fiction forum recently and one of the points commented upon is how so many Quest reviewers rate all games highly, making it difficult to tell which is a good game and which is a bad one.

Liam315
davidw wrote:... so many Quest reviewers rate all games highly, making it difficult to tell which is a good game and which is a bad one.


This is one of the reasons I stopped trying to find games to play that are published on the site, the amount of crap I had to try out before coming across something even half decent was more effort than it was worth. Now I just tend to stick with highly rated comp games, classic games that have a history of acclaim or high recommendations, or trying out games from people I recognize and respect from the forum.

Alex
Happy to consider any suggestions if anyone wants to have a go at solving this. Some way of marking reviews as helpful or unhelpful, perhaps? Some kind of reputation scoring system?

Pertex
Perhaps it's possible to combine the player rating with the number of plays (and the number of downloads). A good game is played more often than an bad one. And for Quest games it could be possible to analyse the game. A game with 10 objects is mostly not as good as a game with 50 objects. Hmmm, a game with 10 objects could be moved to the sandpit immediately :lol:

privateer
How about this (seeing as I made the complaint, I should probably try to suggest a solution):

We have a free democratic system of rating and reviewing, which is a Good Thing
We also have non-democratic institutions which are Good Things - the essential Sandpit, and the useful Editor's Pick

The root of the problem seems to be that non-Sandpit games vary SO widely in quality.

Since all games are checked by Mods, perhaps it would not be a significant increase in workload for them to categorise non-Sandpit games into 2 tiers, with Tier 1 games representing those which have (subjective issues of taste, etc not withstanding) obviously had lots of time and care invested in them, as opposed to those that have obviously not (but which were still not nonsensical enough to be condemned to the Sandpit.)

People could still review and rate freely. No doubt occasionally a Tier 2 game would get a glowing review saying it belongs in Tier 1. Well, that's good. It's a good review and the writer should be pleased, and the game would be more conspicuous as a result of the high rating. Likewise, a Tier 1 game might get 1 star and an unfavourable review. Everyone still gets their say, but in the meantime it's not impossible for people to find decent games.

A tier system might also work as an incentive for kids to put more effort into their games. The Sandpit is the Sandpit - necessary but not very encouraging, Tier 2 is a good way of encouraging kids to publish and have their games seen, Tier 1 becomes something to aspire to and be proud of.


I hope that suggestion is of some help - at least as a starting point for finding a solution.

george
I don't know what the solution is, but the IFDB doesn't seem to have this problem. Maybe it has a more diverse pool of raters and that balances things out?

Pertex
privateer wrote:
Since all games are checked by Mods, perhaps it would not be a significant increase in workload for them to categorise non-Sandpit games into 2 tiers, with Tier 1 games representing those which have (subjective issues of taste, etc not withstanding) obviously had lots of time and care invested in them, as opposed to those that have obviously not (but which were still not nonsensical enough to be condemned to the Sandpit.)


Alex is doing this most of the time but I am sure that he is not playing the games. I have checked some games , too, but I don't have time to check the complete game, just checking the first rooms/pages. It WOULD be a significant increase in workload to test the games more detailed

davidw
I don't know for certain, but I'm pretty sure if someone started posting nonsense reviews on IFDB - or posting positive reviews of games in order to encourage people to rate their games more highly - their reviews would be deleted and they'd be banned.

george
I'm not certain but I think iFDB has a feature where if enough people rate a review as unhelpful the review disappears (or goes to the bottom or something).

privateer
Pertex wrote:

"privateer"


Since all games are checked by Mods, perhaps it would not be a significant increase in workload for them to categorise non-Sandpit games into 2 tiers, with Tier 1 games representing those which have (subjective issues of taste, etc not withstanding) obviously had lots of time and care invested in them, as opposed to those that have obviously not (but which were still not nonsensical enough to be condemned to the Sandpit.)



Alex is doing this most of the time but I am sure that he is not playing the games. I have checked some games , too, but I don't have time to check the complete game, just checking the first rooms/pages. It WOULD be a significant increase in workload to test the games more detailed



I wasn't suggesting looking at the games in any more depth than they are already examined, just a refinement of the categories of Sandpit vs non-Sandpit.

privateer
george wrote:I'm not certain but I think iFDB has a feature where if enough people rate a review as unhelpful the review disappears (or goes to the bottom or something).


That's a good system, especially if non-registered people are allowed to participate in it.

(I suggest that because games with thousands of downloads and plays often have only 3 or 4 reviews - making me think the majority of those who play our games don't feel the need to register. And we need some sort of input from them to help counteract these 'sandflies'.)

jaynabonne
Not to be a fly in the soup, and perhaps it's my programmer sort of mind that likes well-defined things, but trying to crack down on "reviews we don't like" seems like a task that is either going to be tough to do or, worse, potentially stifling.

As an example, let's say we have a system where we can flag reviews that are "not helpful" (or perhaps more in Facebook fashion, we can upvote reviews - a sort of meta-review). Now let's say there is someone like the one who has been mentioned. I don't know what her reviews are like - can you tell that her reviews are not sincere by reading them? Short of some sort of "smoking gun" message like what you have received, how much is it possible to differentiate inaccurate reviews of games vs those which simply differ from our opinion? I have played (or tried to play) some games by a highly-ranked author on this site, and I eventually gave up. And looking through the walkthroughs, I could see why, as the commands needed to be input were often nonsense. (Don't get me started!) And yet there were clearly people who played and succeeded at the games. So I just took that as a hint that we were on different wavelengths and left it at that.

How do we draw the line? Let's say this claire person posts a review which, superficially, appears to be legitimate. Would it make sense to downvote such a person based on what you have seen so far instead of on the actual review itself? What is the criteria used to downvote someone?

I think something more along the lines of a reputation system would work better, where people earn rep through good reviews, perhaps voted by others with good rep. A bit of a bootstrap, but it would get rolling eventually.

I don't know if the above makes sense. I just would hate to turn things into a witch hunt, where we're looking to excise "bad" reviews (as defined by someone) from the site. Given the variability of human opinion - and the ability, it seems, for people to play and enjoy just about anything, or at least to want to offer something positive, I think it's a slippery slope to try to establish norms where none may reasonably exist. Not without removing the ratings from "the common folk" and having some sort of judging panel.

(A note: a lot of my experience with reviewing things comes from my time spent on writing review sites. It seems to be a pervasive problem that you end up with glowing, positive reviews for mediocre product. Whether that is people wanting to not hurt people's feelings, the desire to focus on the positive, etc. I'm not sure. But it's not necessarily people being deceitful or trying to curry favor. In the end, you end up making up your own mind anyway.)

Pertex
privateer wrote:
I wasn't suggesting looking at the games in any more depth than they are already examined, just a refinement of the categories of Sandpit vs non-Sandpit.


Even that is not so easy. If I wouldn't slow down, I would move 90% of all games into the sandpit :wink:

jaynabonne
A couple more suggestions:

1) We can't necessarily keep people from posting glowing reviews for what's (arguably) barely passible games. The counter for that is to post a review of your own, perhaps with more critical detail. Unfortunately, it could get swamped by a torrent of positive reviews. One solution to that would be an Amazon-type rating display - how many 5 stars, how many 4 star, etc. with links to the those. I know I tend to skip the 5-star reviews and go down to the 1-3's to see where the problems are with a product. And then judge from there based on how reliable those seem.

2) This is even more work, but have people not only rank a game but actually rank various technical elements of a game: writing (including grammar), ease of command input, etc. That doesn't eliminate the subjective element, but that would provide more axes, and those specific elements could then be averaged out to provide a more broad summary.

None of the above eliminate "bad" reviews or unbalanced ratings. I don't think there's a way to do that, and tryiing to attack thing from that point of view is doomed to failure (IMO). But making the process more visible by allowing people to see more than just a single star rating might allow people to more easily weed through the fluff to get a more truthful impression.

privateer
jaynabonne wrote:Not to be a fly in the soup, and perhaps it's my programmer sort of mind that likes well-defined things, but trying to crack down on "reviews we don't like" seems like a task that is either going to be tough to do or, worse, potentially stifling.

As an example, let's say we have a system where we can flag reviews that are "not helpful" (or perhaps more in Facebook fashion, we can upvote reviews - a sort of meta-review). Now let's say there is someone like the one who has been mentioned. I don't know what her reviews are like - can you tell that her reviews are not sincere by reading them? Short of some sort of "smoking gun" message like what you have received, how much is it possible to differentiate inaccurate reviews of games vs those which simply differ from our opinion? I have played (or tried to play) some games by a highly-ranked author on this site, and I eventually gave up. And looking through the walkthroughs, I could see why, as the commands needed to be input were often nonsense. (Don't get me started!) And yet there were clearly people who played and succeeded at the games. So I just took that as a hint that we were on different wavelengths and left it at that.

How do we draw the line? Let's say this claire person posts a review which, superficially, appears to be legitimate. Would it make sense to downvote such a person based on what you have seen so far instead of on the actual review itself? What is the criteria used to downvote someone?

I think something more along the lines of a reputation system would work better, where people earn rep through good reviews, perhaps voted by others with good rep. A bit of a bootstrap, but it would get rolling eventually.

I don't know if the above makes sense. I just would hate to turn things into a witch hunt, where we're looking to excise "bad" reviews (as defined by someone) from the site. Given the variability of human opinion - and the ability, it seems, for people to play and enjoy just about anything, or at least to want to offer something positive, I think it's a slippery slope to try to establish norms where none may reasonably exist. Not without removing the ratings from "the common folk" and having some sort of judging panel.

(A note: a lot of my experience with reviewing things comes from my time spent on writing review sites. It seems to be a pervasive problem that you end up with glowing, positive reviews for mediocre product. Whether that is people wanting to not hurt people's feelings, the desire to focus on the positive, etc. I'm not sure. But it's not necessarily people being deceitful or trying to curry favor. In the end, you end up making up your own mind anyway.)



I suppose there is no fixed criteria to down-vote someone, so that system becomes as vulnerable to abuse as the original rating system. I agree with you about witchhunts, and I suspect that those who do the witchhunting would be those prone to giving self-confessedly vindictive/ sycophantic reviews. That's why I prefer to keep the voting free, but with a little more staff categorisation to *indicate* quality. But who will bell the cat? I don't have time, which is why I labelled my OP as a "rant".

I also take Pertex's point about using 2 Tiers of non-sandpit game being time consuming. As he describes things, I now assume chucking games in the sandpit is such a rapid click-fest to keep up that it's not even possible to assess the quality of a game beyond "can i quickly and obviously see that there isn't really a game there at all?" [EDIT: though given games are selected for Editor's Picks, is this really the case?]

But if you think Claire, for example, is not abusing the system to curry favour, cast your eye over her reviews (and accompanying conversations in comment boxes.) She makes no secret of it. Honestly, this is not about "reviews we don't like". I'm not yet post-modern enough to think that there is no such thing as a pointless review that serves no purpose but to entertain a clique.

privateer
jaynabonne wrote:A couple more suggestions:

1) We can't necessarily keep people from posting glowing reviews for what's (arguably) barely passible games. The counter for that is to post a review of your own, perhaps with more critical detail. Unfortunately, it could get swamped by a torrent of positive reviews. One solution to that would be an Amazon-type rating display - how many 5 stars, how many 4 star, etc. with links to the those. I know I tend to skip the 5-star reviews and go down to the 1-3's to see where the problems are with a product. And then judge from there based on how reliable those seem.

2) This is even more work, but have people not only rank a game but actually rank various technical elements of a game: writing (including grammar), ease of command input, etc. That doesn't eliminate the subjective element, but that would provide more axes, and those specific elements could then be averaged out to provide a more broad summary.

None of the above eliminate "bad" reviews or unbalanced ratings. I don't think there's a way to do that, and tryiing to attack thing from that point of view is doomed to failure (IMO). But making the process more visible by allowing people to see more than just a single star rating might allow people to more easily weed through the fluff to get a more truthful impression.


Both your suggestions would make the rating procedure better...in the right hands.

Perhaps the only real solution is to somehow encourage a culture of reviewing so the significance of the "bad" reviews is minimalised. (And I don't think we need shy away from the fact that there IS such a thing as a bad review! - "Worst game I have played. I give it 5 stars lol", and so on...)

We either need to encourage more democracy, or mediate the democracy that exists with some top-down judgment on what deserves a higher profile. I can only think of doing that by having a category like "Tier 1/ recommended games" - a sort of expanded Editor's Pick where decent games are permanently housed.

(Speaking of Editor's Pick - surely the game checkers know what the promising games are? Else how to they be able to extract game for the Editor's Pick?)

davidw
I'm not sure the reputation system would work very well for pretty much the same reason as we have with the current system: namely, you'd just have people voting for each other to better their own reputation. ("Hey, mate! Vote me up and tell all your friends to vote me up and I'll vote you up in return.")

For myself, I tend to ignore the ratings system because everyone rates games differently. I've known people who will rate a game 5 out of 5 even if it was so bad they quit a minute into it. I've also known other people who will really enjoy a game, but give it a poor rating because there was the occasional typo. I usually look at the reviews themselves to see if a game is worth playing, and by reviews I mean actual proper in-depth reviews written by people who know what they're talking about. "Dis is great!" isn't a review, nor is "This sucks!" If the review isn't at least a couple of paragraphs in length, I skip over it.

privateer
davidw wrote:I'm not sure the reputation system would work very well for pretty much the same problem as we have with the current system: namely, you'd just have people voting for each other to better their own reputation. ("Hey, mate! Vote me up and tell all your friends to vote me up and I'll vote you up in return.")

For myself, I tend to ignore the ratings system because everyone rates games differently. I've known people who will rate a game 5 out of 5 even if it was so bad they quit a minute into it. I've also known other people who will really enjoy a game, but give it a poor rating because there was the occasional typo. I usually look at the reviews themselves to see if a game is worth playing, and by reviews I mean actual proper in-depth reviews written by people who know what they're talking about. "Dis is great!" isn't a review, nor is "This sucks!" If the review isn't at least a couple of paragraphs in length, I skip over it.


I would agree. It would just broaden the scope for the same abuse.

The real problem, as I see it, is how ratings affect the visibility of games. Good games get impossibly mixed up with the bad, and no-one knows where to look for them. If visitors are prepared to go through all the games and search for reviews that are both meaningful and positive then there's no problem.

Maybe an answer could be having various different ways of searching for games. We have genre categories, but perhaps a tag system like IFDB would be good. When you can only list games with reference to rating or newness, it's easy for a game to get lost from the public eye forever. Maybe a tag system would give all games a fair chance of being spotted in a search?

In other words, if we don't think it's feasible to successfully legislate for the silly-rating-culture, and it's too much work to have more top-down indications as to where the quality is (beyond Editor's Picks), maybe we could keep everything as it is, but introduce a system (eg. tags) that makes reviews less consequential to the exposure a game can expect.

privateer
http://textadventures.co.uk/games/view/ ... den-maze-3

This game hits number 5 in Quest's top games - due to six 5 star reviews (more than the number of downloads) within about 26 hours. The reviewers have a history of incestuously rating each others games, and at least some of them attend the same school.

Obviously this is not Russia invading the Ukraine so it's not something to lose real sleep over...but it does indicate a disfunction; I defy anyone to define those reviews as good by ANY criteria. And it's a game that a games tester might wonder how he let it slip through the sandpit filter.

Quest is really good and that's a TERRIBLE advert for it - misleading for players and to people wanting to find examples of Quest's potential.

jaynabonne
I see what you're talking about, but I don't see a good solution - at least not one that doesn't involve a lot of work for a number of people.

Looking at the reviews, it seems typical of what I see on other sites where "common man" reviews are given. I don't know what the motivation for these reviewers is. A cynical view could be that they are deliberately conspiring to intentionally give inflated reviews with the purpose of artificially pushing their friends' games up in the rankings. Another view (which I have seen more often) is that they are a circle of friends, and they all like each other, and they don't necessarily care about the larger scope or anyone outside their circle, and they give each other max reviews because they would never actually be harsh to friends (at least not in public), and they don't really play IF or have a gauge, and they're off in their own little world anyway.

In the end, I suppose it doesn't matter, as the result is the same - ratings are subjective and all over the map. It's frustrating, but short of eliminating people being able to review or having a "reviewer of reviews" with the power to delete (like Amazon, the bastards), there is not much you can do. To police it is to enforce a standard, and that has to come from somewhere. Which means someone has to decide.

One alternative is to have a panel of judges who alone review and rate. That keeps the reviews more consistent, but I think you'd need a broad selection of judges to keep it from being too one-sided - like you see on Rotten Tomatoes.

Another alternative is to have both, with people being able to review but not rate, and with Quest admins alone rating. Similar to the first option, but allows some "common" voice.

A third approach is to diminish the importance of ratings. Let people use them for their own purposes, but they don't factor into visibility or prominence on the site.

And the last I can think of at the moment is to get Quest games out there where the more valued reviewers can get hold of them. That's what was happening when Alex had Quest games automatically publish to IMDB. I think getting quality Quest games to IMDB is a good thing - but not automatically (just my personal preference as an author who wishes control over the crap he releases). Making it easier for authors to do that (some sort of link or button in the tool?) would help those who feel rightly proud of their games getting them out in the world. Would someone like the author of what you posted do that? I don't know. It depends on what their motivation is. If they're just posting games for their own circle of friends, then they won't. If there is text along with the button/link that basically says, "By doing this, you will be exposing your game to some of the toughest IF critics in the world who will rip your game to shreds. It's not pretty! Don't do it!" then one would hope they wouldn't.

In the meantime, the basic question is: can a high rating on the Quest actually mean anything? It seems not, just as it doesn't on any other site I have been on where all manners of people are able to freely express their point of view in an entirely subjective way, where you don't have large groups of reviews to statistically balance it out. Of course, if I went to that game and saw five-star reviews with comments like that, then I'd discount the reviews as well as the rating. It comes back to what you were saying about visibility, and perhaps that's the way to attack the problem (to the extent there is one? I'm still not sure, from a big picture point of view. If I saw that game, or others like it, on the textadventures home page to the detriment of quality games, then I'd worry.)

Sorry for making this so long. :)

jaynabonne
A thought and a cautionary tale: I was once part of a writing site called StoriesMania. I also have been part of a writing site called StoryWrite.

On StoriesMania, the web site owner wanted to allow only quality writing in the forums. So all pieces had to go through a review process where editors looked it over and sent it back until the quality standard was up to snuff.

On StoryWrite, anyone could post anything they wanted.

(You can probably see where this is going...)

StoriesMania became a ghost town and is in the process of shutting down since ghost towns get invaded by spammers and nobody cares enough to maintain something that nobody is actually using.

StoryWrite has booming traffic.

The stories on SM are still high quality (while it lasts), while the content and reviews on SW are what you'd expect - some real buried treasures but largely amateurish and overrated pieces (because, let's face it, not everyone has it in them to be truly critical, either due to lack of attention - how will someone critique someone else on punctuation when they don't bother themselves - or a more innate tendency of human beings to not enjoy tearing each other apart).

Take from that the moral you wish!

jaynabonne
Ok, one more thought. (Sorry. :) )

The reason there are so many "crap" Quest games is Quest's strength: it is accessible and usable by anyone. You don't have to be a writer-type (e.g. who can work out the English-like Inform script) or a programmer. It can be used by kids and adults alike, by amateurs and pros. I think that's a great thing, but then you end up with school kids alongside seasoned IF authors. There is no way to compare, and we really shouldn't. (One tack we took on SM was to split pieces up into Amateur and Advanced.)

What we really need is not so much to dis the honest efforts of people with less skill (to be honest, despite the poor spelling and grammar and over-usage of "colloquial" language, that game you highlighted had more logic in place than some Gamebook games I've seen on Quest. It looked like there was at least a decent amount of effort put in) as we need to promote what is good, to let people see what is possible with Quest in the right hands.

privateer
jaynabonne wrote:I see what you're talking about, but I don't see a good solution - at least not one that doesn't involve a lot of work for a number of people.

Looking at the reviews, it seems typical of what I see on other sites where "common man" reviews are given. I don't know what the motivation for these reviewers is. A cynical view could be that they are deliberately conspiring to intentionally give inflated reviews with the purpose of artificially pushing their friends' games up in the rankings. Another view (which I have seen more often) is that they are a circle of friends, and they all like each other, and they don't necessarily care about the larger scope or anyone outside their circle, and they give each other max reviews because they would never actually be harsh to friends (at least not in public), and they don't really play IF or have a gauge, and they're off in their own little world anyway.

In the end, I suppose it doesn't matter, as the result is the same - ratings are subjective and all over the map. It's frustrating, but short of eliminating people being able to review or having a "reviewer of reviews" with the power to delete (like Amazon, the bastards), there is not much you can do. To police it is to enforce a standard, and that has to come from somewhere. Which means someone has to decide.

One alternative is to have a panel of judges who alone review and rate. That keeps the reviews more consistent, but I think you'd need a broad selection of judges to keep it from being too one-sided - like you see on Rotten Tomatoes.

Another alternative is to have both, with people being able to review but not rate, and with Quest admins alone rating. Similar to the first option, but allows some "common" voice.

A third approach is to diminish the importance of ratings. Let people use them for their own purposes, but they don't factor into visibility or prominence on the site.

And the last I can think of at the moment is to get Quest games out there where the more valued reviewers can get hold of them. That's what was happening when Alex had Quest games automatically publish to IMDB. I think getting quality Quest games to IMDB is a good thing - but not automatically (just my personal preference as an author who wishes control over the crap he releases). Making it easier for authors to do that (some sort of link or button in the tool?) would help those who feel rightly proud of their games getting them out in the world. Would someone like the author of what you posted do that? I don't know. It depends on what their motivation is. If they're just posting games for their own circle of friends, then they won't. If there is text along with the button/link that basically says, "By doing this, you will be exposing your game to some of the toughest IF critics in the world who will rip your game to shreds. It's not pretty! Don't do it!" then one would hope they wouldn't.

In the meantime, the basic question is: can a high rating on the Quest actually mean anything? It seems not, just as it doesn't on any other site I have been on where all manners of people are able to freely express their point of view in an entirely subjective way, where you don't have large groups of reviews to statistically balance it out. Of course, if I went to that game and saw five-star reviews with comments like that, then I'd discount the reviews as well as the rating. It comes back to what you were saying about visibility, and perhaps that's the way to attack the problem (to the extent there is one? I'm still not sure, from a big picture point of view. If I saw that game, or others like it, on the textadventures home page to the detriment of quality games, then I'd worry.)

Sorry for making this so long. :)



Nothing wrong with this long post; all points well made.

Since pondering the problem and how others have responded to it, I have become more keen on your third option than my initial suggestions - making reviews less critical in deciding how visible a game is. Maybe the way forward now would be to think about how that's best done. I thought about search tags as a possibility, but I bet someone can come up with a better idea.

Maybe we could create an experience like browsing a bookshop or library? Games could be listed in genre categories (as now) but ordered by author (alphabetically). Games still have their ratings beside them....just a thought I had while writing this...

I also agree with your point about getting more exposure for Quest games by publishing on IFDB etc. Even if people who use that site don't publish their reviews here it doesn't matter too much. The important thing is that as many people as possible are able find and play the games they might enjoy. We can benefit from their ratings/ search facilities just as well as our own.

Cheers.

privateer
jaynabonne wrote:Ok, one more thought. (Sorry. :) )

The reason there are so many "crap" Quest games is Quest's strength: it is accessible and usable by anyone. You don't have to be a writer-type (e.g. who can work out the English-like Inform script) or a programmer. It can be used by kids and adults alike, by amateurs and pros. I think that's a great thing, but then you end up with school kids alongside seasoned IF authors. There is no way to compare, and we really shouldn't. (One tack we took on SM was to split pieces up into Amateur and Advanced.)

What we really need is not so much to dis the honest efforts of people with less skill (to be honest, despite the poor spelling and grammar and over-usage of "colloquial" language, that game you highlighted had more logic in place than some Gamebook games I've seen on Quest. It looked like there was at least a decent amount of effort put in) as we need to promote what is good, to let people see what is possible with Quest in the right hands.




I am honestly really not interested in dissing the games themselves, just drawing attention to potential (and occassionally realised) problems with how the ratings are being applied. I'm all for kids playing with Quest, and publishing their efforts, and seeing some of those make it into the main games listings. But I think you have agreed that it's not a perfect world when a game like the one I posted becomes an actual show-piece for Quest.

Incidentally, I think splitting into something like Amateur and Advanced is a great idea.

davidw
Thnking about it some more, the reputation system might work - but only if it was taken out of the hands of Quest users and placed solely with the moderators. You give it to the users, they'll just vote for each other to give themselves a great reputation, the same way they're currently doing with the games, but if you give it to the moderators, there's a far greater chance it'll work.

The way I'd do it would be:

> If a moderator feels a user's reviews are good or bad, they can either vote them up or down a sliding scale of reputation (a good review = goes into lots of details about why a great is game / it stinks; a bad review = very, very short / doesn't say anything more than "this is great" or "this sucks"). The higher up the scale, the more their rating of the game counts; the lower down, the less their rating counts.

> By default, reviewers with a bad reputation have their reviews hidden and their rating of the game either doesn't count or counts for far less than it should do (i.e. someone with a good reputation gives a game 5 out of 5, it counts as a proper 5 out of 5; someone with a bad reputation gives a game 5 out of 5, it only counts as 1 or 2 out of 5). Maybe allow it so that only registered users can see reviews written by people with a bad reputation; unregistered users only see reviews written by people with a good reputation.

> Allow users to nominate people for a better or worse reputation, but the final decision is in the hands of the moderators.

> Under this system, games rated highly by people with a bad reputation wouldn't show up on the front page of the main site - instead the games there would be the ones that are genuinely deemed to be good (whether they are or not is another matter entirely).

The system might take a bit of work to get off the ground, but it would essentially hide bad reviewers (and their reviews and votes) from casual visitors to the site and hopefully rid Quest of its current reputation. It would also allow people who are genuinely trying to write good games a chance to shine.

jaynabonne
The ironic thing is that if there were enough people writing reviews, then this would all be a non-issue - if you had twenty 4-5 star reviews, then a single 3 star from someone for invalid reasons wouldn't even make you bat an eyelid. Similarly, if someone has friends drop in eight 5-star reviews, they would be balanced out by the weight of 15 others giving a more accurate review. In the mean, these outliers would have no import. The problem is that so few people rate games that these aberrations take on greater weight.

Unfortunately, the real solution is to generate more traffic and more reviews, but I don't see that happening. It just feels like we're trying to solve the wrong problem... :) (but with great thoughts and ideas, I might add).

Liam315
But there you have a chicken and egg scenario. Which came first, not enough people rating games so the dud reviews carried more weight, or the dud reviews skewering the visible games so that serious reviewers couldn't be bothered sifting through the crap any more.

Also, while I appreciate the fact that younger people and amateurs can make use of quest to make games and have no wish to stop them, I don't have any interest in playing them. And because it takes much longer to create and publish a good game than it does a bad one, bad ones will always far outnumber good ones.

I think the solution to the problem isn't an overhaul of the review system, but a complete overhaul of the categorization system. Split "professional" and amateur, gamebook and text adventure, puzzle-driven and narrative-driven, difficulty etc. These are the things that determine the sort of game someone might want to play, for most it doesn't matter if it's sci-fi or historical if those other elements of gameplay and publishing standard are what they prefer. Such a system would work as follows:

There are a set of pre-defined tags that can be set to a certain value. (e.g. Tag: Game Type, Options: Gamebook, Text Adventure)
When the author publishes their game, they fill out a form about their game so the correct tags are applied.
Users can report or vote against a certain tag if they believe it has been wrongly classified by the author. If necessary moderators can change the value of the appropriate tag.

davidw
Jaynabonne:
I don't think the issue is that people aren't rating games, it's that the ratings they're giving in no way reflect the quality or otherwise of the game in question. Just looking at the latest activity page, I can see multiple 4 and 5 star ratings over the past few days often with meaningless reviews following:

"This writer is the next Shakepeare"

"I loved the poem(s)! :) "

"I loved this RPG! If you know anything about programming you should make a real video game of this. Very detailed and a great, classic storyline, wonderful."

"It's very interesting and it didn't frustrate me at all. I have no idea why t didn't"

"It's very interesting"

Those are the last five reviews following a rating, with not a single proper review among them. I guess the middle one might technically count as a review, but as it doesn't mention a single point about the game, and could instead have been written about pretty much any game in the world, I'm going to lump it in with the "not a review" pile.

Liam315:
I disagree. An overhaul of the caterigorization system might help, but it won't have much effect in the long run if people are still writing amazing reviews and giving great ratings to terrible games.

Right now, there's very little incentive to play a Quest game because I'd have to wade through a 100 terrible ones to find something worth playing, and there's even less incentive to review or rate them because my review / rating would quickly be swallowed by all the meaningless reviews / ratings. I don't see things improving until something is done to drastically cut down the number of meaningless reviews and ratings.

jaynabonne
I suppose I was taking a statistical point of view. But you're correct - if *nobody* ever gives a valid review, then you have nothing to balance out. And if that's the case, then ratings become meaningless, as they seem to be.

Liam315
davidw wrote:Liam315:
I disagree. An overhaul of the caterigorization system might help, but it won't have much effect in the long run if people are still writing amazing reviews and giving great ratings to terrible games.


That's a fair point.

To take a different tack then, on the site there is already a place that these sort of "reviews" should be- the comments tab. Maybe that should just start being enforced, anything posted as a review that shouldn't be classified as one should be moved over to the comment section. Any user that consistently misuses the review tab loses the ability to post reviews (while still being allowed to post comments). You can then give more weighting to ratings that are borne out by a proper review as opposed to a simple one-click star rating.

This obviously won't have any effect if the skewing is a result of a calculated effort among authors, but it will if it's just a matter of casual groups of users doing this kind of thing within their own cliques.

privateer
I think Liam and Jaynabonne have hit the nail(s) on the head there.

More voters can only be an improvement, while a categorisation system which divides games up into smaller (and more useful) chunks would mean the organistion of games is based on more than just their ratings and so the impact of the ratings is moderated.

Maybe a list of authors to click on would be useful. Let's say David is trying to find a good game; instead of wading through hundreds of games, he could look through the (much smaller) number of authors. Looking at one or two games per author would give him a fair idea of whether he's found a source of quality games, or whether he should move on to the next one. That's got to be more efficient that trying to find the needles in a haystack.

m4u
Good post, it would be interesting to see a solution implemented soon.

It would be helpful to explain somewhere how to write a good review. I didnt know till I read this post.

And, whats is wrong with a rate based on plays and downloads? It seems like a good metric to me. Or at least create a section with the most playable and downloaded. I have seen this system somewhere in pages about games.

Liam315
m4u wrote:And, whats is wrong with a rate based on plays and downloads? It seems like a good metric to me.


Because it creates a snowball effect regardless of game quality. A game played many times is ranked highly, so more people see it and play, thus increasing its play count and pushing the rating higher, repeat ad nauseum.

davidw
m4u wrote:And, whats is wrong with a rate based on plays and downloads? It seems like a good metric to me. Or at least create a section with the most playable and downloaded. I have seen this system somewhere in pages about games.


The ADRIFT site does that and while it's a nice idea, it doesn't work out that way. The top 10 most downloaded games list hasn't changed in ages because people new to the site simply see that list when they first arrive, download what's on there and ignore everything else. Unfortunately, most downloaded game doesn't necessarily equate to best game.

HegemonKhan
personally, after reading this thread's posts, and based upon my own experiences, I'd say:

have the reviews be moderated, either by:

*Also for both #1 and #2 below, anyone (any user and~or mod) can submit reviews too, if they're good reviews, then they'll be allowed~included with the other (good~professional~quality) reviews.

1. the site's mods will have to check the reviews to see if they're good (actual reviews: a detailed and accurate summary of the game in relation to the various categories for their rating system of that game; ie they need to show that they actually played the game enough to have the knowledge to truly review and rate it), and deleting~removing the bad~fake "reviews".

2. have a selected professional reviewers 'club' (of mods and~or users) who'll write the reviews for the games, as to how these reviewers are decided can be up to the site (its mods) on how they want to do this.

have a users comments section~feature:

this lets the users (anyone) post their comments about the game (good and~or bad in quality), so everyone can be included.

More categorization of the games (in accordance to as the # number of games grows of course) would be useful too:

to better help people 'search~find' the games that they're interested in. Also, maybe split the larger 'book genre' (and other such large) categories (like fantasy and sci-fi for the 'book genre' categories) into smaller sub-categories. Authorship (ie by author), is definately a needed category. Time to complete ~ Size of the game would possibly be good categories too. (I haven't looked at the play~download games categorization feature of quest in awhile, so my apologizes if a lot of this has already been done).

we should have a thread for brainstorming~deciding upon different categories, and the best ones get used for the quest site.

personally, I don't see any point in a "like~dislike" rating feature:

as it always just gets abused, due to humans' behaviors and~or motives, and thus it becomes nullified, whereas comments have their value, as people can choose what comments to read and care about in their decision towards trying a game or not.

lots of comments, and the (true) reviews, generate interest in the games, and does a good job in accurately rating games for people, as it provides 'food for thought' where they can then make a decision on whether that game is good or not, thus it's not abuse'able, whereas a simple 'like~dislike' rating button doesn't generate any interest, and it is abused to death galore of being of any true use~value to anyone.

--------

this allows for both professional~quality via reviews, but also for the free market of public comments too, so people can look at both, or one or the other as they prefer, to find the games (and good ones of them) that they're interested in playing.

----------

gamefaqs.com (just one site of many that) has a good system of reviews and user comments, for a *quick* example (using parasite eve game):

1. http://www.gamefaqs.com/ps/198265-parasite-eve/reviews (the formal reviews)
2. http://www.gamefaqs.com/boards/198265-p ... e/59649104 (thread's postings: informal comments)
3. http://www.gamefaqs.com/boards/198265-p ... e/60548636 (thread's postings: informal comments)

m4u
I see, you are right! A couple of ideas then:

1. What if we rank the reviewers and then we can filter the games for the best reviewers?

2. Or we can all agree to review the games, then we will have enough reviews. Or we can create a group of judges and recruit more judges from time to time.

Pertex
m4u wrote:
1. What if we rank the reviewers and then we can filter the games for the best reviewers?

But who watches the watchmen? You will have the same problem with the reviewer then so you will need chief reviewer...

davidw
I'd suggest that would be a job for the moderators.

m4u
HegemonKhan wrote:

gamefaqs.com (just one site of many that) has a good system of reviews and user comments, for a *quick* example (using parasite eve game):

1. http://www.gamefaqs.com/ps/198265-parasite-eve/reviews
2. http://www.gamefaqs.com/boards/198265-p ... e/59649104
3. http://www.gamefaqs.com/boards/198265-p ... e/60548636


It looks good. It seems a 10 points systems is better because I feel the 5 too close to the 1 or 3 :)

I'd suggest that would be a job for the moderators.



I agree, anybody else agree?

jaynabonne
HK, I think a lot of what you said sounds good as things to consider. There is one part I want to highlight, though, which I see as problematic and is my only cause for concern (and I'm not speaking to you as such, but to everyone in the discussion - you just expressed the clearest the part that concerns me :) ):

*Also for both #1 and #2 below, anyone (any user and~or mod) can submit reviews too, if they're good reviews, then they'll be allowed~included with the other (good~professional~quality) reviews.

1. the site's mods will have to check the reviews to see if they're good (actual reviews: a detailed and accurate summary of the game in relation to the various categories for their rating system of that game; ie they need to show that they actually played the game enough to have the knowledge to truly review and rate it), and deleting~removing the bad~fake "reviews".



The problem comes down to: what is "good"? What is "bad"? Where is the line drawn? HK gave some criteria there, and it sounds all right in theory, but we would need some fairly explicit and clear guidelines for people to follow in terms of establishing and enforcing "quality". What will moderators do? Delete reviews and/or nullify ratings if they don't feel they are up to snuff? Then we will have to have codified and published guidelines about what constitutes acceptable submissions so that when people come back and ask why their review/comment was deleted, there is somewhere clear to point them. It cannot be capricious on the whims of the "professional panel".

If someone who is not proficient with conveying their thoughts plays a game that so blows them away that they feel the need to try and put something into words, and the words they come up with are: "Wow! I absolutely loved the game. I can't wait to see what you come up with next. <smile>" - will that be considered a "bad" review, just because it doesn't delve into all the advanced analytical points that HK mentioned? I'm basing this, by the way, on my experiences with StoriesMania. Getting people to review at all can be a struggle at best; tell them that in order to review they have to jump through hoops to meet certain standard, and you might as well not even bother allowing them to post (for many), as this is all voluntary, and what is the incentive to make the effort?

Similarly, if someone rates a game with 4 or 5 stars, what is the criteria that will be used whereby someone can *objectively* state "That rating is not accurate, so we're going to delete it." Even assuming for the moment that people (in the ideal situation) vote honestly based on how a game affects them, there is no guarantee that they will rate based on whatever criteria is laid down.

I'm not trying to be a party pooper. I do see that there is a problem. But I see codifying and enforcing standards as being a very slippery slope, especially when in the realm of the subjective. And I would be quite nervous if it ended up being a *self-appointed* panel of "experts". (As in: who else cares enough to do it but us anyway?)

If it ends up being "we welcome all reviews as long as we agree with them" then we might as well not even bother with reviews from the public at all.

jaynabonne
To push things the other way, the only real solution I can see is to promote good games via other routes: editor's picks, a "hall of fame" that you get elected to by the more senior folks on the site, etc.

I have no problem with games being given laurels and raised up high by the elite. :) I just don't want to censor those who have differing opinions from being able to have their say in a way that doesn't necessarily keep people from finding the true gems here.

davidw
There's no ideal solution. The problem has arisen because certain individuals post glowing reviews and ratings irrespective of the quality of the game in question and it's unlikely they're going to change their mind any time soon. I don't see any of them posting in this thread so it's doubtful they're even aware this discussion is going on, or would care if they did know.

Maybe then instead of simply deleting reviews and ratings that aren't up to scratch, you make them far less prominent, at the same time as making the detailed reviews more prominent. Instead of a single place on the main site to write reviews, have two: one for amateur reviews that are just a few words in length, one for professional reviews that go into much more depth. Amateur reviews and ratings can be hidden away, perhaps only accessible to registered members, whereas professional reviews are there for all to see. This would allow the crowd who rate everything 5 out of 5 to carry on doing that while at the same time allowing the more in-depth reviews to stand out.

jaynabonne
To play devil's advocate: if someone writes a detailed review (more than a few words), emphasizes things that may or may not be "quality" (e.g. "I thought it was really funny when you had Jimmy jump off a cliff") and then gives the game 5 stars, is that amateur or professional? If it's just a question of the effort put into the review (based on word length, etc), then fair enough. But if it gets back to some judgment of "does this review accurately reflect what we consider quality", then your proposal is basically the same thing ("hidden" just a more minor form of "deleted"). Are we to say that some opinions are more valid than others?

I can understand the frustration about reviews that, for one reason or another, don't match what one or many might perceive as being an accurate reflection of the quality. But I have seen this on all the sites I go to because not everyone looks at reviews and ratings with the same criteria. Hiding away reviews we don't like just feels wrong to me.

If I review a cheap piece of junk on Amazon and I give it five stars because I happen to really like the way it matches the color of my wallpaper - despite the fact that it will fall apart if used for any length of time - to my knowledge, Amazon won't delete it. They're not in the business of assessing accuracy. They just want to be sure the review is honest and unbiased. So you need to look at a good assortment of them to try to divine some sort of reality.

I really think the problem is not so much the existence of the "bad" reviews and ratings as much as the weight given them in terms of visibility on the site. Instead of sweeping them under the rug as an embarrassment of sorts, we just need to focus on how we can get visitors to the Quest site in contact with good games - and the only way to do that is to have more experienced and analytical reviewers (and enough of them) whose opinions are given more weight than the random person off the street, especially if it's meant to try to meet standards of the wider IF community (which is really why I think getting them to weigh in on and put their seal of approval on the top games is the best way to go, if possible somehow).

HegemonKhan
"good quality" reviews merely follow a format (thus not being only a few lines ~ spam-like-posting as a so-called "review" = bad quality), with a paragraph of details explaining why they give the rating in a given category* to follow to ensure it being of good quality. Moderators should make sure the reviews follow their format, to have the reviews be more "professional and~or formal" vs a comment section, being more "informal and~or non-profession".

an example of a "good quality" review format: http://www.gamefaqs.com/ps/198265-paras ... iew-106956

-----

*Rating Categories:

Story~Plot, Media (music, art, vids, etc), UI Ergonomics ~ Play'ability (User Interface ~ "ease of game controls" ~ how frustrating the game is, in, in understanding how to play it), "Bug~Error~Glitch Count" (how error-free it is or not), Game Mechanics ~ Game Balance ~ Cool Features, Replayability ~ Customization, Overall, Detailed'ness ~ Complete'ness, etc

-----

as for more specific rules~criteria~quality'ness (ie: the oversight level of control that Jay is refering to and worried about) for their reviews, that's up to the site's moderators to decide upon, as they'd be setting it up, should they decide to have a more controlled~profession~quality of reviews feature here.

as for actual accuracy of the reviews... there's really only one way to do that: peer review (someone else have to have played the game, to know whether the review is accurate or a bunch of garbage, despite it looking "good", ie follows the format with paragraphs of details for their rating categories).

----------

if people want to comment, but can't or don't want to follow your official review format, they can post their own "review" in a comments (post) section on the game. This way, you got the controlled "quality~good~professional"-like REVIEWS, as well as a comments section, for anyone~everyone to say what they want to say, and~or to write their own reviews their own way or at a lower quality level than what is set for the "official" reviews for this site's games.

-------------

m4u wrote:It seems a 10 points systems is better because I feel the 5 too close to the 1 or 3


for me personally, 1-10 point system is too complex, lol. I mean trying decide whether the score is a 6 vs 7 for example... ugh! It's so arbitrary!

for myself, I have~use a simple "1-4" point system, hehe:

(1) epic ~ masterpiece ~ best of all time
(2) good
(3) average ~ decent
(4) bad

jaynabonne
All right, so that handles the review side of things - if it's not a "review" as such, then it gets demoted to a comment (if I read you right). That just leaves the policing of ratings, which is actually more (I think) where the problem lies, as reviews in and of themselves don't carry much weight - it's the rating numbers driving lower-quality games up on the site. (So while what you just proposed would work, it might not be necessary in terms of moderation - I'm just trying to keep the busy work down for those who would eventually have to do this!)

So what would be the plan for ratings? If it doesn't match the "peer rating", then it's eliminated? If so, then it comes back again to why bother with "public" ratings if you're only going to allow the ones that agree with what someone higher up has agreed with? You might as well just only allow the chosen ones (and I don't mean that in a negative way) to set ratings and not let the average joe rate at all since only the ones that have agreed upon get kept anyway.

That ends up being perhaps a simple solution: let people comment/review (and not even bother with a distinction since it doesn't matter anyway) but not let people rate, with that being left for the site's official raters. People have their say, but they aren't given total control to determine what is promoted on the site.

Or let people rate, but have another avenue for top game promotion, where you have "top rated by the masses" and then "site's official anointed ones". lol (Has Alex already done this with Editor's Picks?)

jaynabonne
As an example of what I mean, check this link out:

http://www.rottentomatoes.com/m/twilight/

Same film. Critics give it a 49%. Average audience goer gives it 73%. Why? I have no idea. But I can't help but think people are being honest on all sides. Which one carries more weight? It probably depends on what you're looking for.

So we have a distinction between "critically acclaimed" and "popular". There always has been, whether it be books, movies, music, or IF. They're quite often different - but both are valid, in different ways.

What we have now is a system where "popular" games are promoted, with little way to find "critically acclaimed" (that is, what someone on intfiction perhaps might consider good). And we need a way to make those more visible. I don't think we can overload a single rating system to support both. Either it's going to be popular or critic's choice. You can't munge the popular to make it be inline with critical just by eliminating what's not critical. You might as well not even allow the popular, then, since in the end it's irrelevant. ("You can choose any color as long as it's black.")

It seems that either we take over the ratings system where it's only determined by critics, or we find another way to promote technically excellent games.

xavea
If we do go the way of giving guidelines for reviews, why not implement a word minimum before one is allowed to rate a game? If someone tries to post a review that is too short, an error message could come up linking them to the guidelines and perhaps asking them if they've given them a look? I doubt that the people who are currently upvoting their friends' games are going to want to put in the extra effort, and if they do honestly believe the game is good or bad, they ought to be able to explain why in a minimum of one hundred words, for instance.

For reference, the above paragraph is 108 words according to MS Word counter, so reviews wouldn't even need to be exceptionally long for those that don't express themselves well, just a few more sentences. And prominently linking someone to a newly implemented reviews suggestion sheet or checklist could be a nice way of bringing the problem to these reviewers' attention.

Someone said a few of them go to the same school, so I'm guessing they are young (high school as opposed to post-secondary). I wasn't under the impression that in this community we eat our young and I'd like to keep it that way. Instead of harshly coming down out of the blue with strict guidelines or deleting posts for (what may appear to those that aren't in the forum to be) "no reason", maybe a helpful guide of suggestions on how to give good reviews could be looked at?

Also, I do prefer a ten point ratings system to a 5, because it is more nuanced, that may or may not work in conjunction with another plan, but won't help anything on alone.

m4u
I agree with Xavea.

(1) epic ~ masterpiece ~ best of all time
(2) good
(3) average ~ decent
(4) bad



Just mutiply your system by 2 :)

jaynabonne
Xavea, I like your idea about enforcing a word count at review entry time. Especially if the message that comes up (as you say) informs them that reviews have a minimum length and "if you'd like to leave the game author a short note, please go to the Comments section". If that's an easy change, it could be a simple start toward solving this problem, assuming that ratings only can be set for successful reviews.

Someone could pad their "review" out by just entering "la la la" 100 times, if they're so inclined, but those would be easy to spot and remove as clear attempts to circumvent. I agree with you that just having a word minimum count would be discouraging to those without a lot to say. :)

m4u
Now what? Do we need more people to agree or we send a pm to Alex to see if this can be implemented?

Pertex
So is there a solution? Sorry, but if I want to rate a game, I don't want to write a novel. Of how many characters are you thinking? 100,200, 500? Have a look at the top game The Mansion II ( http://textadventures.co.uk/games/view/ ... mansion-ii ). Which of the reviews there would be "valid"? 10%? In my opinion only 1!

Liam315
Pertex wrote:So is there a solution? Sorry, but if I want to rate a game, I don't want to write a novel. Of how many characters are you thinking? 100,200, 500? Have a look at the top game The Mansion II ( http://textadventures.co.uk/games/view/ ... mansion-ii ). Which of the reviews there would be "valid"? 10%? In my opinion only 1!


Have 2 sets of ratings, 1 for people who just want to rate it out of 5 stars and be done with it, but don't use this rating in determining the visibility of games (it can just be a separate poll). For visibility purposes, only use ratings in which a reviewer has taken the time to justify their opinion.

Enforce the difference between a review and a comment by moving the offending "reviews" to the comments tab.

If you don't want to write about what you've rated a game then that's fine, but without doing so your rating is largely meaningless to anyone browsing through the games since everyone uses different metrics to determine what a star is worth.

xavea
Pertex wrote:So is there a solution? Sorry, but if I want to rate a game, I don't want to write a novel. Of how many characters are you thinking? 100,200, 500? Have a look at the top game The Mansion II ( http://textadventures.co.uk/games/view/ ... mansion-ii ). Which of the reviews there would be "valid"? 10%? In my opinion only 1!


I used the example of 100 words in my post, because 100 words is really not a very challenging minimum to meet. However, if we knock that in half, more reviews are valid, but theoretically still explain at least a little bit why the reviewer rated it the way they did.

This may not end up being the best solution to the problem, I had hoped that people would add their thoughts to my half-idea to create something vaguely workable before trying to get it implemented. Isn't that sort of thoughtlessness what we are trying to discourage in the reviews? Let's not be complete hypocrites here.

For transparency sake, here's a link to the reviews that I have done in the past. Not all of them would reach my proposed 100 word minimum. I think they are all at least 50 words. Not all of them are good reviews, but even a small word minimum could bring some improvement over "This sucks!" or "awesome!". If all we want are short reviews to that effect, why don't we just implement "likes" and "dislikes" button? Cut out all those pesky words, because who reads these days anyway?

EDIT TO ADD: A word minimum that is too high will not only discourage people from writing reviews altogether, it will ensure that the reviews that are written are long-winded. I think the trick would be to find a balance between "wind-bag" and "keyboard equivalent of grunting".

jaynabonne
Regarding "thoughtlessness", I think we had had so many thoughts beforehand that there wasn't much left to add. :) Plus, what you had proposed could (possibly) be a simple thing to change, and it also happens to be independent of any other change, so it can be implemented on its own. It might not be the only thing changed, but on its own, it's a no-brainer (in my opinion).The appeal to me is that it covered what we had discussed before, namely that we wanted more in-depth reviews from people who rated - not just a jump in, several words, 5 stars and goodbye.

I agree the limit would have to be set with some thought. That paragraph above is 87 words - hardly a novel. Perhaps 50 would be a better amount, as was proposed. It does require some thought and effort to come up with words, but I personally don't have a problem with people "earning" their rating by putting a few minutes effort in.

Of course, I have little problem with words... :)

xavea
jaynabonne wrote:Regarding "thoughtlessness", I think we had had so many thoughts beforehand that there wasn't much left to add.


That's reasonable. I probably came off a bit harsher than I intended in my last post. It's a flaw of mine. For such a small forum, this topic is pretty well talked through as it stands; I guess I was just a little surprised that a new idea only brought agreement instead of invigorating the dialogue.

jaynabonne wrote:I agree the limit would have to be set with some thought. That paragraph above is 87 words - hardly a novel. Perhaps 50 would be a better amount, as was proposed. It does require some thought and effort to come up with words, but I personally don't have a problem with people "earning" their rating by putting a few minutes effort in.


This was more or less my point. I suspect that most of the people with nothing to say will stop saying it if they are heavily encouraged to articulate their thoughts better. And on the flip side, the ones who care enough to put some effort into what they want to communicate might find that their skills improve with some guidance.

Pertex
jaynabonne wrote:
I agree the limit would have to be set with some thought. That paragraph above is 87 words - hardly a novel. Perhaps 50 would be a better amount, as was proposed. It does require some thought and effort to come up with words, but I personally don't have a problem with people "earning" their rating by putting a few minutes effort in.

If you think this would help... Mansion II would lose 50% of it's rating then and I wouldn't say that the lost ratings are just rubbish. We need more ratings rather than less so that the law of large numbers applies. If all the people here in this thread would just rate the new games without writing lots of text we would have a much better rating. If I see a game rated by Jay I don't need a description of the reasons for this rating

george
I agree that we need to increase the number of (fair) ratings. Many times I'll rate a game on IFDB but I don't feel like writing a review.

TextStories
After reading all the issues and suggestions, while carefully weighing all the pros and cons, ultimately I have come to the conclusion to simply do away with the rating system altogether. If we do not have enough people to police the forums to see what is a good review or not or how heart felt it actually was, as in not a just a group of bad apples spoiling other people's games, then simply change the system.

Generally people will play the games with the higher ratings, regardless of how the games got their ratings. “Oh look! 5 star! Download!” We assume those ratings were there because of actual content, time played and not buddy comeuppances. Just as we assume bad ratings are gotten due to bad content, not mean spirited people. We also understand there will be a gem hiding some where with in the murky muck of low scores or no scores at all and if something happens to catch our eye, we may give it a try. But if people see that the only good games on Quest are crap games due to inflation, they may leave for good and no one wants that, not even these so called trouble makers I am sure. Perhaps if we have more people to vote as someone else suggested, this may not even be an issue, but as for now, it is what it is.

However, this still wont solve the problem of wading your way up to your neck through sewage trying to get to the promise land of good games out there. I have seen the attempt and if not already done so, we should have a sticky of player/editor recommended games, with thoughtful reasons why. Again you could have those who just want to talk about their buddies games, but it would be a lot easier for a mod to see it for what it is and simply take that game/comments off the list, opposed to hunting through many different threads for half hearty comments and trying to figure out if a games score was truly worthy or not, for the good or for the bad. The more people promote a game, the more likely it will be seen and the list can grow slowly, but surely. Maybe even make it a helpful first link to a new person either as a pop up or a message inside their profile page.

“Hey there! Glad you could join us. While you are settling in and getting familiar with the website, why not take a look at these recommended games!” or something to that effect. Some thing helpful to steer people in the right direction. Helps the community at large in my opinion, keeps the bad games at bay and hopefully keeps the new people coming back for more. Once they have played a few good games if they chose to follow our advice, it is up to them where they will go from there. But at least we gave it a good shot. The games should be recommended on it's merits, even if it was a rough game, but had great ideas or terrific story, etc.

Silver
I have mixed feelings on this now. Obviously it appears to be unfair if a game has five five star reviews and has suddenly rocketed onto the front page where as a much superior game that has had 10,000 plus plays is jettisoned off it.

The reason why the Jettisoned game has sunk in the ratings is because more people have played it and more realistic reviews have been left as a result. So it drops down in prominence as a result. Even though its a finer work, it stops the front page becoming stale and gives newer entries their day in the limelight.

The game that reached the front page from five five star reviews (perhaps even from friends of the author) will soon be slapped down if it isn't worthy of that position, one hopes. It isn't a reliable list of what is the 'top ten' as it were and perhaps a different code algorithm could be written to compile one of those.

Instead of editors picks maybe have the same system as is in place now (but removing games over time that seem to sit there) to keep things fresh but also have a top ten based on another calculation for people wanting to play the very best on here (amount of reviews is a massive signifier here, over and above who has the most five stars).

HegemonKhan
I've posted this somewhere before, but here it is again, my '2 cents' comments:

gamefaqs.com has done this very well, it's a good system

best would be a U.S.~Britain legislation system:

U.S. <--> Britain

Actual REVIEWS:

Professional Quality Panel of Reviewers (moderators and~or certain users) and their Ratings: Senate (100: 50 states x 2 per state) <--> House of Lords

and

Public Panel of Reviewers (the 'mob~people', aka the rest of the users): Congress (~ around 420) <--> House of Commons

---------

AND

a like~dislike rating system (it will be abused, but maybe we can hope the sheer quantity~statistics will balance out the abuse to it... lol)

AND

(as has already been expressed by us and being worked on by Alex and co., a better categorization system for finding specific types of games)

what MOST+FIRST gets my attention is simply the NAME of the game, and its genre~category and~or description:

is this even a type of game that interests me?

then, I try to figure out of it's a good quality game or not... lol

------------------------

though, this is a lot more work, as it's a more comprehensive system, which Alex may not have the manpower~mods or time for implementing and~or managing such a big system on his website.

TextStories
HegemonKhan wrote:I've posted this somewhere before, but here it is again, my '2 cents' comments:

gamefaqs.com has done this very well, it's a good system

best would be a U.S.~Britain legislation system:

U.S. <--> Britain

Actual REVIEWS:

Professional Quality Panel of Reviewers (moderators and~or certain users) and their Ratings: Senate (100: 50 states x 2 per state) <--> House of Lords

and

Public Panel of Reviewers (the 'mob~people', aka the rest of the users): Congress (~ around 420) <--> House of Commons

---------

AND

a like~dislike rating system (it will be abused, but maybe we can hope the sheer quantity~statistics will balance out the abuse to it... lol)

AND

(as has already been expressed by us and being worked on by Alex and co., a better categorization system for finding specific types of games)

what MOST+FIRST gets my attention is simply the NAME of the game, and its genre~category and~or description:

is this even a type of game that interests me?

then, I try to figure out of it's a good quality game or not... lol

------------------------

though, this is a lot more work, as it's a more comprehensive system, which Alex may not have the manpower~mods or time for implementing and~or managing such a big system on his website.


You lost me at the House of Commons... :shock: but had me at the Game FAQs way of doing things. :lol: I am familiar with them and actually like their reviews as well as their game walk thrus and codes, etc. Although I find it odd when the player scores are some times much higher than the mods and at times I tend to agree more with the player scores, although that may just be nostalgia for certain games...

And more what Silver was saying, maybe rotate every month for the best of games. No game will be highlighted twice in a row so newer games or old retirees get a chance in the lime light. And maybe even have a back log of all past picks so there would be a healthy dose of good games for people to know to try and be good.

Silver
Any progress on this? :D

I've just had a quick look at some stats.

The current number two game in the top games section is Star Wars: Escape the Flagship with an average user rating of 5.0

A recent number two game that has dropped a bit in the rankings is Any Which Way but Dead with an average user rating of 4.86

What's interesting is the former has six five star reviews whilst the latter also has six five star reviews but an additional four star review which gives it the lower average. If we were adding stars together rather than creating an average of them Any Which Way but Dead would have 34 stars against Starwars' 30.

A game that was the number one in top games when I first discovered this site little more than a year ago was First Times by Hero Robb. His game is now ranked as 22nd. Yet this game has an astonishing 42 five star reviews, 8 four star, 2 three star and 1 two star review. It gives him an average rating of 4.65 but if we were to add the stars together they would tally up to 250 stars, if I've done my maths correctly.

I'm not sure if adding stars rather than finding an average would work either as it would give a bias to games based on how long they've been published; unfairly keeping the same games eternally in the top spots (thus forever rendering everything else invisible - so less likely to elicit any attention to get better reviews to alter the situation. I feel Sir Loin and Mansion II sort of fall into that category). But as it is, mediocre games trump superior ones based on less reviews. IMO, natch.

I like the idea of giving people the option for thumbs up or thumbs down. I mainly play games online and once finished just close the window: perhaps closing the window could call up a small window using JS with the thumb up thumb down option? Intrusive, yes, but it might give a clearer view in the long run. The algorithm needs serious tweaking though. There's no way First Times should be 22nd whilst brief initial flirtations into CYOA are bouncing into the top six, IMHO.

Interestingly the number one game has an average user rating of 4.9. So there is an algorithm beyond just average rating although for the most part it doesn't appear so.

Edited typos and added bits.

jaynabonne
Here's a discussion about a weighting system based on both score and quantity. I have no idea how it works or if it's good, but it looks interesting.

http://math.stackexchange.com/questions ... view-score

Silver
jaynabonne wrote:Here's a discussion about a weighting system based on both score and quantity. I have no idea how it works or if it's good, but it looks interesting.

http://math.stackexchange.com/questions ... view-score


That does look interesting. I'd probably add time to it too. So games that have scored really highly will have their scores deteriorate over a period thus slowly pulling them down from the top spots to give rise to newer games. Not massively so, but it be taken into account. So you could have all time best games and current best games instead of editor picks I guess.

Alex
The ranking algorithm takes into account number of reviews as well as average. I quite like the idea of ratings that deteriorate over time though - there are certainly some games that have clung to the top for years.

Frostedge
While a deterio-rating system sounds like a good idea, (since it gives newer games with high scores more exposure and it prevents the elite games from becoming eliter while blocking other proper competitors) there's always one small problem. The ratings are still subject to inflation due to unfair 5-star reviews. We're still stuck with "sandflies" 5-starring poorly made games and 1-starring the good ones, causing the value of ratings to drop.

Games that have been crafted with a lot of blood, sweat and tears often get more critical acclaim from peers, giving it 4 stars with proper reviews and constructive feedback. (I don't believe in the highest scores, 5 stars, since I deem perfection impossible.)
Games that have been made without any serious effort or thought are produced in no time at all and receive truckloads of 5-star ratings because either their friends tuned in, they made alts, or they made review deals to give someone else 5-stars if they return the favour.

Inflation of the ranking system sets in either way. I'd say go with giving reviewers the option of leaving a minimal 100-word review alongside their rating, and then having those written reviews count stronger than just the rated ones. This, alongside the option to report a review if it has padding for the word count (such as "la la la") to weed out any dishonest folks trying to manipulate the system.
Or we could have a ranking system for reviewers themselves, but like someone else mentioned, that creates the "Who watches the watchmen?" issue.
I'm beginning to wonder whether or not there's an actual solution to the problem that doesn't negatively impact the very people/games we're trying to help by attempting to solve it.

Silver
The system is open to abuse by alts but who are we to say what rating other people give and whether the rating they have given is 'right' or not? It could be argued that Authors of primitive games receiving more favourable reviews than supposedly finer crafted works happens because they have made more of an effort to promote their games. There's really a lot of unknowns so a lot of this is speculation.

The main problem is not enough of a percentage of people who play games write a review. Have you reviewed every single game that you've played? A solution suggested earlier a quick thumbs up or thumbs down to get more people to add to a game's rating. Perhaps closing the game window could trigger some javascript that brings up a small ratings window with the thumbs idea... or maybe that's a bit intrusive.

Marzipan
The rating system here is so meaningless I'd honestly rather see it done away with altogether, and just let players rely on reviews. I'm sure we could throw together a pretty extensive 'recommended games' list here on the forum for new players. (The obstacle here is I think the forums link isn't nearly visible enough on the main site, but that's an issue for another thread...)

Instead of a 'top rated' section on the front page I'd love to see 'community picks' that rotate every week or so, or if that's not feasible just an expanded editor's picks section.

Alex
I like the idea of "community picks". If we can get some agreement about what they should be (maybe start a new forum thread) I'm happy to set them up on rotation where the editor's picks currently are.

Silver
Could be quite fun whilst encouraging more forum participation leading to community growth.

My suggestion is to list all the games that survived the sandpit that month and maybe do a poll? I suppose that ignores historical gems though and some months literally produces no games of note imv.

Silver
Another idea might be to collectively pick 3 games a month to play and review. Ones that aren't already front paged. Of course the reviews might be unfavourable but it would still give those 3 games a month in the sunshine. No idea how we would select them though.

Marzipan
I don't know if it even needs to be anything as formal as that. To begin with couldn't we try just making another one of those 'recommend a game' threads...if somebody thinks a game is really good they can post about it and say why, and if there's a consensus it can be added to a list in the OP.

I guess with some 'common sense' ground rules we could automatically rule out a good portion of the games here; let's say they had to have no major issues with spelling and grammar, not be half-completed demos, etc.

Of course it would be nice if we could be a bit more systematic about it, but this is a small community, I don't know how much of a formal 'process' is really practical, or necessary. Obviously we're not going to be able to sit there and go through literally every game to see if it meets minimum criteria but I'd hope the stand out gems would eventually get people's attention and rise to the top. (And anyway I don't really like the idea of 'meeting minimum criteria' guaranteeing inclusion, I'd think we'd want these all to be games that are genuinely well written and entertaining to put Quest's best face forward to newcomers. 'Not broken and awful' doesn't automatically equal 'really good', after all. :) )

Silver
But if there was some blurb on the front page saying something along the lines of 'what we're reviewing this month and perhaps you can too..' it isn't limiting the idea to forum users (who are relatively small in number) but to textadventures users too (who are considerably higher). People often have to be led by the nose to something. Well, if we don't build and encourage participation like that, nobody else will.

renagrade
Interesting stuff. I noticed these ratings really do change the outcome of a game. My game (first one ever) received 4 5/5 ratings and I was very very happy with this. I was fortunate enough to see my game rise to the #1 spot in RPGs. I was curious as to why.

I looked at N7 the (AMAZING) game that was just one place below it.

The only reason I passed him was because of ONE 3 star review. Even though this REALLY benefited my game (it's exposure has grown exponentially), I feel like N7 got cheated. One average review took him from the top spot. His game has 3000+ more plays than mine, but yet I'm above him.

All this to say that I think people who write 5/5 reviews are not the problem. It's the people who write the lower ones. They can really mess up a game's path. I am honestly SCARED to review people badly because I don't want my game to suffer the same fate because of a revenge review.

davidw
The problem is people writing dishonest reviews. If a game is great, say it's great. If it's bad, say it's bad. But don't give 5/5 reviews for barely playable messes.

Most of the reviews and ratings on this site are meaningless because there's no correlation between a 5/5 review and a great game.

renagrade
davidw wrote:The problem is people writing dishonest reviews. If a game is great, say it's great. If it's bad, say it's bad. But don't give 5/5 reviews for barely playable messes.

Most of the reviews and ratings on this site are meaningless because there's no correlation between a 5/5 review and a great game.

That about sums it up. There are some games that are just terrible, and I've given them bad reviews. That's how it should work. No revenge, just get better and don't make the same mistakes.

HegemonKhan
this is why you need professional reviewers (honest, impartial: no favoritism~chronyism nor hatred, people who write honest reviews, and who actually play the entire or most of the game, to write an accurate review of the game) along with public (amatuer) reviewers along with statistic counters ('likes~dislikes'), along with etc etc etc, as this thread is about the failure of statistical reviews~ratings... they can be easily skewed, dishonest reviews and~or 'likes~dislikes' (and~or reviews done by people who haven't played the game very far into it to write an accurate review).

you need a combination, many layers~types of ratings... but this is also more work for people to do...

Silver
The problem is that I might feel a game is good and worthy of four stars. But if I award it four stars I'm saying it's not as good as games that have been five starred to death but are inferior. So I'm forced to give five. The other option is to go to all the highly rated games I feel have been unfairly placed into the league of excellence and down vote them. But then this puts noses out of joint and upsets people. It's a complete mess and life's too short.

HegemonKhan
ratings is an unsolved problem for humans... laughs... every site or poll (such as political or whatever) or etc, has these same issues of

accuracy

(how do you turn subjectiveness into objectiveness? how do you elimate corruption, aka bias corruption: dishonest reviews likes~dislikes favoritism~chronyism hatred~revenge etc etc etc, which also destroys objectiveness?)

:D

Silver
I toyed with the idea of doing some in depth reviews either on the forum or on a blog. What's holding me back is that I have this fear that if I set myself up as a critic, it'll be the kiss of death of me being a designer.

Those who can't, teach...

davidw
I've found that people here don't tend to react positively to being told their terrible games are terrible.

Saying that, it would make a refreshing change to see a decent in-depth review of a Quest game as opposed to the hundreds of useless 1 line reviews on the site now.

george
Here's a workable solution -- redo the math that ranks games. If a lot of people are giving meaningless 5/5 scores to games, redo the math so the rankings are more accurate.

Silver
There's no accounting for taste though. :D

Odie_da_Bossé
Now, this is another really crap topic that I made, but I am a dude with many questions.

It sometimes really annoys me when someone who has made, maybe say, none or a sandit game, yet they rate about 5 other games with 1 star, that are probably a lot better than anything he/she made. Does this annoy anyone else, or do people think this is perfectly normal? Share why or how.

Also, do you guys like reviewing/playing other games, or do you like making your own games more than anything? Or are you a balanced person? Please share!

I may seem a bit rude with this topic (especially the first question), so sorry if I do. But anyways, please tell me!

Cryophile
The review comment has been discussed somewhat extensively in this thread: viewtopic.php?f=5&t=4061

As for your other question...

I haven't been particularly active in the community for... years. When I did, I was more interested in conversing, working out problems, and playing other games. I did have a handful of my own projects, but I never really finished any of them, and either lost what I had been working on or just gave up and moved on. I did eventually finish one of my titles ('Simple Game') but never bothered releasing it.

I would say I've stuck around mostly to lurk and see the community grow, with occasional spurts of activity and productivity, but I doubt I will announce or discuss any of my projects again - solely because I know myself well enough now to know I won't finish most of them! It is the journey, not the destination, for me, and I'd do something about my procrastination habits, but, well, whatever - maybe later.

Marzipan
Whether it annoyed me or not would depend on the specific situation. Yes, it's infuriating when someone just swoops in with a 'this sucks, one star', but let's face it, there are lots of games on this site that deserve that rating, and as long as the reviewer can give reasons as to why then more power to them. There's nothing wrong with someone who comes here just to play and review (IMO we could actually do with a few more of those people...) and their opinion is just as valid as anyone else's. I mean, a bad game is a bad game and is still bad whether or not the person saying so has written anything better themselves.

And if we're going to take this to its logical conclusion, we'd have to insist that players shouldn't be allowed to give out five star reviews either, because they can't possibly recognize a good game unless they've written a few of their own. :P Honestly, I know for a fact I've seen more worthless, unjustified or unexplained five stars than ones, so if anything is skewing ratings too much it's probably that.

And as for your question, I started off as a player/reader/reviewer, then one day the writing bug bit me. :wink: These days I prefer to work on my own projects in the limited time I have, but at the same time I feel obligated to review others at least occasionally. Do unto others and all that; whether it's a community for IF, CYOAs, or regular fiction, authors want feedback. There's nothing worse than releasing your 'baby' out into the world and having it ignored. And that's always a danger in a small community made up mostly of 'creators'. Everyone is too wrapped up working on their own stuff to pay attention to yours, so you lose the inspiration to write and participate and aren't around when they finally finish their masterpiece and want attention from you, then they lose their inspiration and so on and so on. It' s a vicious cycle. :D

Marzipan
Silver wrote:The problem is that I might feel a game is good and worthy of four stars. But if I award it four stars I'm saying it's not as good as games that have been five starred to death but are inferior. So I'm forced to give five. The other option is to go to all the highly rated games I feel have been unfairly placed into the league of excellence and down vote them. But then this puts noses out of joint and upsets people. It's a complete mess and life's too short.


This. This exactly. The real problem the site has is that these ratings mean something vastly different to everyone. Some see five stars as the default rating to give whatever you play and some see it as meaning what it says; that a game is fantastic. When a game is merely good, I want to give it four stars but unless I can come up with a specific criticism I'll usually bump it up to five just to not rock the boat.

In a perfect world a four star or even a three star 'okay' game could be one with no major bugs or problems beyond the writing being kind of uninspired or the game otherwise lacking that extra wow factor. But in reality three stars hurt our precious little feelings and I wind up giving it out as a 'at least you tried' consolation prize even to games I really wouldn't describe as anything near 'okay'.

But in the end, best way to easily tell if a rating is deserved? Just read the review it's attached to. The stars themselves are meaningless and always have been.

HegemonKhan
I think there's also the issue of:

amatuer game ratings vs professional game ratings

people understand that a lot of these games and their makers are noobs learning to make games, so a game can be considered 'good~decent~great' based upon this perspective, but that is vastly different from whether the game is truly good~decent~great (professional game rating perspective).

"hey, this game is really good for a person just learning to make games! but, as a game, it's really not that great... but I'm going to give him that 5 star rating, as this is really 'excellent' work for a person new to game making, it's a lot better than most of the games that I've played!"

---------

@marzipan:

a big problem with 'reading the reviews'.... (stars~likes~dislikes: aka counters, aren't the only things that can be abused~corrupted...)

some people write what looks like really detailed and well-done reviews... but they've never played the game or just played its very beginning... and until you play the game yourself... you would never realize how totally BOGUS that review that you've read, was... the review could have absolutely nothing to do with the game at all, and you'd never know it, unless you actually decided to try the game out yourself (which you wouldn't do if that BOGUS review was a negative review...)

the corruption of ratings is really ugly... there's really bad people out there... how do you prevent them and their bad behavior, from ruining a rating system (how do you design an accurate~objective ratings system? sadly, humans haven't discovered how to do this yet..) ???

davidw
HegemonKhan wrote:some people write what looks like really detailed and well-done reviews... but they've never played the game or just played its very beginning... and until you play the game yourself... you would never realize how totally BOGUS that review that you've read, was... the review could have absolutely nothing to do with the game at all, and you'd never know it, unless you actually decided to try the game out yourself (which you wouldn't do if that BOGUS review was a negative review...)


I find it pretty hard to believe people write really detailed and well-done reviews of games they haven't even played. While I'm sure that in the history of the world it's been done, I can't see it happening on a regular basis. If someone writes a long and detailed review of a game, it's a fair bet they've actually played it.

davidw
I never really understand why people have to written a great game themselves in order to ‘qualify’ to review other games. Most film critics have never directed or acted in a film, yet you don’t get people saying they're not qualified to review a film due to their lack of directing / acting experience.

The simple fact is that sometimes games just suck. You don’t need to be a writer yourself to realise that.

Silver
I sort of made the decision to do some IF Comp reviews. I gave up th ghost when I was still up at four in the morning writing one of them. And the critic-is-a-crap-designer fear.

Odie_da_Bossé
davidw wrote:I never really understand why people have to written a great game themselves in order to ‘qualify’ to review other games. Most film critics have never directed or acted in a film, yet you don’t get people saying they're not qualified to review a film due to their lack of directing / acting experience.

The simple fact is that sometimes games just suck. You don’t need to be a writer yourself to realise that.


You hold a very valid point, David. But if someone is constantly putting out those 'oh, I can't play this, one star' or 'you really suck, I'll give you a star for being a human' kind of reviews, about twice a day or something, it really frustrates me. Sure, once upon a time I hadn't given a single proper game out to the public yet, and I was giving two star reviews all over the shop. For example, if two people are doing apple bobbing, and then one has a go, but kinda fails miserably, the other person could say 'OMG your such a fail, actually try to get the apples out of the water you hollow head!' etc, and then they try it, and fail just as bad, it makes them look like a bit of a hypocrite.

Not that I'm saying you've got this all wrong; you've definitely persuaded me about this type of thing, but it's just sometimes it gets me so, so worked up! Just fairly recently, there was this guy (no naming) that kept giving horrible ratings to other people. And in return, people give him bad reviews for his games; 2 star average for normally, etc. So he let his fury out on the people who rated him bad, which led to arguments, swearing and eventually the moderators and staff had to get involved.

Again, true, sometimes it's just the games that suck, but if you give a one star review to a really good game, that didn't deserve a one star review, it would just end in everyone being annoyed and frustrated, and if it gets serious, fights and trouble. But that's something that hardly happens. So all in all, it's okay to give a one star review if you need to, but when it's unnecessary, it could come and bite you back.

Odie_da_Bossé
I didn't realise there was a topic like this, sorry guys! I created a topic that was nearly just for this kind of thing, but whatever.

As I said in the topic I created, it really annoys me when people just 1 star every single game they can find, or something like that. Also, the reverse way round is about as bad too. I spotted a group of people that would 1 star review most games, and then they would give five stars to a game; all of them would do the exact same games, nearly the exact same reviews. Maybe it's the same person, but whatever.

Silver
I've one starred quite a lot of games. The reason was because prior to the sandpit existing all games got published to the main site. Alex started a thread asking for people to rate unrated games, so I spent a few hours and most were terrible/short/unfinished. But nowadays they'd just go in the sandpit. In fact what tends to happen if you look at a lot of games in quick succession is that after twenty or so bad ones a semi okay-ish one will show up. Ordinarily it might be worth two or three stars but compared to everything else you've just seen you think it's a breath of fresh air!

jaynabonne
I think the basic problem is one of lack of honesty or integrity in those reviews more than whether they had made a game themselves or not. I personally think that authors and players wear two different kinds of hats, and they don't always commute. Even a single person is probably in a different mindset when creating a game than when playing, and it seems a lot of newbie game creators get more caught up in the authoring side than actually how it will affect players.

Which is why it's important for game authors to play games and why (sometimes) a game author can have a bit more compassion or have the ability to see at least what a game is trying to do that a player might miss, if they don't have much experience playing themselves. But there can be those who have played a wide range of games who can know good from not good, even without having put fingers to keyboard to create. (To echo David's example, there are many more readers of books than writers, and it's really how a book affects a reader that matters in the end, expert or not. The average reader's opinion is just as valid as an author's would be, though perhaps coming from a different place.)

The problem is that there is no absolute scale. I have seen people on Amazon who have one-starred a wonderful product simply because it wasn't delivered quickly enough. And, of course, Amazon has people looking through the reviews to see if they actually meet some standard, for at least (I assume) explaining why the rating was given. That might be another solution - let people rate, but then hold the *reviews* to some standards, deleting those that don't at least explain why they were given the ratings they were given. That might prevent the level of arbitrariness we're seeing.

Of course, what we really need are just a lot more quality people writing reviews. It might not completely offset a certain amount of trash, but it will certainly help to level things a bit. (But that's all been discussed a bit in the other thread. Let's keep it over there. :) )

In the end, though, as my wife would put it, "It's just a sodding website." Do what you can to make things better, but it's certainly not worth the anger or rage or any stress that might ensue from that. This is the internet after all. lol

Silver
Jay using Amazon as an example on another thread has got me thinking. If you search for a product you have choices such as relevence, most popular etc etc. Instead of just having highest rated - there could also be most played, most reviews, most comments etc selectable from a similar drop down bar. Maybe too much work for Alex, or maybe as simple as returning different values from a database.

I'm now envisaging the authors who get their mates to five star their games to also get them to leave loads of comments and repeat click on the 'play' button. :D

Cryophile
Perhaps a way to weigh user reviews based on user feedback? If your reviews are consistently upvoted by other top reviewers, your reviews count as slightly more than reviewers that have been downvoted. This, if not abused, might help the global ratings for games, and then poor reviews can be hidden with enough negative votes within the game page.

The community certainly does need more 'quality' reviewers, but I feel a peer review system is still necessary. There is really nothing currently stopping these people from just voting whatever they feel like for whatever reason.

Edit: At the cost of decreased visibility of a few posts, I merged the two relevant topics so future discussion stays in one thread.

Silver
I can't think of one system that can't be abused in some way. Even reviewing the reviewers might lead to a disgruntled orgy of down voting honest reviews. The only hope there is that players who don't write reviews are compelled to read them and vote on them also. Could be worthy of an experiment.

Cryophile
Perhaps those able to vote on reviews should be restricted to an approved list? At least there wouldn't be stagnation to the review process if peer review slacked off a bit through inactivity.

Silver
Could that lead to an elitist approach? Sorry, just playing devil's advocate. (And a bit tipsy, NYE early doors!) :oops:

Cryophile
It would, but if the system were entirely automated without moderation it would just fall apart. An approved peer review list would at least mean that upvotes/downvotes on reviews are warranted (presuming the approved ones aren't the problem). The actual reviews themselves would remain the same, but perhaps their weighting could be shifted to different ends of the spectrum with enough peer review.

Scenario:

10 people spam-vote 1 star, and they have a history of spamming arbitrary scores
2 person votes 4 stars with a history of decent reviewing criteria

What should the global rating be?


The ultimate solution would be to have reviewers that take it seriously, but the majority of people reviewing are probably underage or just don't care.

Silver
I think most people just don't review. Then you'll get a group of mates discover the site, they'll all have a crack at writing a game and vote each other up. First Times got a lot of reviews. It got mentioned in The Guardian (UK National Newspaper) which may be one reason. Not that The Guardian are keeping a beady eye on textadventures.co.uk, I think the fact it was released as an app for mobile devices brought it to wider attention.

Marzipan
If we end up going the 'rate the reviewers' route (though until Alex chimes in saying he's willing to make major changes to how the site handles ratings this is all theoretical anyway...) I think the best thing to get results while avoiding ruffling too many feathers is to make a 'thumbs up' available without resorting to a 'thumbs down'.

I sincerely doubt your average user cares enough about their respectability as a reviewer and how it reflects on their e-cred to make alt accounts or go around rounding up friends to rate up every single 'this game sucks' they type. While on the other hand people who routinely write detailed reviews will naturally accumulate thumbs up over time and the longer they keep at it the more weight their ratings will have.

On a different note, one simple thing that might protect games from the occasional undeserved rating is to make it so that a new member's first...let's say three ratings or so are ignored, or only count for half the usual weight. Now this is definitely elitist but I swear it would help! :P I've seen tons of users who have only played one game and given it one of those 'how u play there nothning to click???' reviews and then disappeared forever. And on the other end of the spectrum this would also protect us from 'great game made by best friend! 5 starz lol!!!'

By the time they play their third or fourth game it can probably be assumed they're interested in being a part of the community and are starting to learn how one game compares to another and so on.

Silver
They'd probably sit there and do the maths though.

Why has my game got seven five star reviews and still isn't on the front page...

Cryophile
They'd need to be aware of the system in order to circumvent the rules, and I would hope most people who spend the time to learn the system would also be dedicated or mature enough to not abuse it. The majority of poor reviews are either one-time reviewers, those new to IF in general, or authors upvoting themselves and downvoting competition.

Marzipan
Silver wrote:They'd probably sit there and do the maths though.

Why has my game got seven five star reviews and still isn't on the front page...


I meant the first ratings that new members give, not the ones they receive. So some random person dropping in, giving your game one star with no explanation and then vanishing forever wouldn't do any damage, while ratings from regular members would still have the same effect.

Silver
Sorry, I meant in the scenario which may happen or not where persons known to the Author are the ones giving the reviews/ratings.

davidw
Maybe length of reviews could count for something. Someone who writes a 1,000 word review has likely spent more time playing the game than someone whose review consists of "this game is great" or "this game sucks" or "the puzzles were ace". Not that every review has to be 1,000 word longs or anything like that, but the longer a review the better an idea of the overall game you get.

Cryophile
This is true, but are you going to write a full review if a game is absolutely terrible, or just a short comment and blast it with one star? If a game is mediocre, criticism and a proper review will go a long way, but some of these aren't even really games.

Granted, a lengthy or longer word count review usually indicates it is serious and the star rating is appropriate. I've also seen a few decent reviews end in "but you gave my game one star, so I am giving your game one star" so there's that :(

Marzipan
Cryophile wrote:I've also seen a few decent reviews end in "but you gave my game one star, so I am giving your game one star" so there's that :(


I'm not sure what the site's policy on revenge reviews is, I assume they would be deleted if pointed out? Really the way to nip the problem in the bud would be to have a clearly labeled zero tolerance, or at best 'one warning' policy. Revenge ratings (especially the kind that get openly gloated about) are right up there with blatant plagiarism in the Things You Don't Do list, and such a ridiculous, childish, spiteful thing anyway. I can't see anything good ever coming out of anyone who would even consider it, so kicking them to the curb isn't exactly a loss for the site.

Marzipan


Review by JCB_diggerbrown
07 Jan 2015
it is impossible to progress in this game in a group activity of 15 mins (year 7 class of 30)



One star review on The Lunastone.

This is such a weird feeling, I can't remember ever wanting to hit a child before. This is a good example of exactly the reason ratings from newbies shouldn't count, though. Why on earth was this posted as a review instead of just a comment?

Silver
Sounds like it came from a teacher and not a child. Or someone on a wind-up.

jaynabonne
More information than "impossible to progress" would be helpful as well. As it stands, there is no clue about what went wrong or what could be changed to make it better,

Marzipan
Silver wrote:Sounds like it came from a teacher and not a child. Or someone on a wind-up.


I would hope a teacher would know better. :( Seriously, what does their failure to teach their students have to do with the overall quality of the game?

I understand that terms like 'rating' and 'review' are probably pretty meaningless to most of the population, but I've been hanging out on various writing sites ever since I was in junior high and it has just been ingrained in me that there are some things you just don't do. I don't know, maybe part of it's just PMS or whatever (:P) but I see these horrible petty or ignorant 'reviews' and it's a like a part of my brain just shorts out. 'didn't really look at this but sabotaging the overall rating just because' is so unbelievably disrespectful to the author and straight up bannable in some places, or at the very least would get you set on ignore. Either way it makes me judge them so hard and question their overall worth as a human being, I just can't help it. It's the same gut reaction I get when people don't tip waitresses or delivery guys IRL. :lol:

And I'd say it's even less excusable here, with a perfectly convenient 'Comment' tab available so you can talk or ask questions about the game all you want without rating it.

Silver
Dunno, it just didn't strike me as something someone would say who was in a class; more something someone would say who was teaching a class. Unless they were making the point of "I'm not thick, loads of us were struggling!" Perhaps even encouraged by the teacher. I don't know what 'year 7' translates as (ten year olds?) but 15 mins isn't very long to be stuck at IF particularly.

Marzipan
I was assuming 7th grade, and wiki says it's an English thing with kids between eleven and thirteen, so close enough.

I don't like to make generalizations, but I have teachers in my family and my part time is at a day care and it's been my experience that a lot of kids that age have a reading comprehension level that would have been considered more appropriate for seven or eight year olds when I was growing up.

Still though I can't blame the kids...the teacher should have had the sense to check the game out beforehand to see if it was something their kids would be able to handle. What exactly did 'no progress' mean? Like, they literally couldn't figure out how to move, or did they get stuck on the puzzles? Was there any prior experience with IF on the part of the kids or the teacher before they were thrown into this? I'm guessing no just from the comment, but either way all of those details about introducing kids to IF would have actually been fascinating to read about.

Instead we have, what, a teacher showing kids a random file on the internet, on a site they're clearly unfamiliar with? I still can't believe they wouldn't have played it through themselves first, just to make sure everything past the first room wasn't racial slurs and tentacle porn.

Alex
There are definitely class groups that come along and use the main site, even though I set up activelit.com for that specific purpose - which gives the teacher a lot more control over what their students get up to. You can lead a horse to water...

I've deleted that review now. One day we'll have a way of ranking reviews as helpful or unhelpful, but I have quite a lot of other stuff I want to do first which I deem as more exciting.

Until then, you can always drop me an email and I'll remove any crap. It's very quick for me as I'm the only person who gets a Delete button!

Marzipan
Alex wrote:It's very quick for me as I'm the only person who gets a Delete button!


Ooh neat, where can I buy one of those?? :D

cdutton184
I've just had my game The Lunastone reviewed with 1 star again by user Jfriend. However, his comments were this:

ok
u
sux
The field Text must be a string with a minimum length of 10 and a maximum length of 10000.

Since Jfriend didn't specify where/when the error occurred, was it justified for him to give me a 1 star or not if the error was in my game?

At what situation does that 'field text' error occur (presuming it wasn't Jfriend not doing something stupid to force the error) so I can fix my game, anyone?

Alex
That "error" is what you see if you don't type a long enough review, so it's just somebody padding out text for their "review". Which is now deleted.

cdutton184
Thanks, Alex.

XanMag
Okay. My "rant" here... Evidently, I pissed Timmy off somewhere along the line. He "played" and reviewed both games within minutes of each and gave both a one star rating. If you read the comments he made during his review, it's pretty clear they are not warranted or accurate. One review was "absolut trash" and the other was "tu muk reeding". So... If you all happen to read this, gamers, disregard those reviews please and give Xanadu a try. It is, I believe, better than about 85% of the games on here. Also, please leave intelligent and thoughtful reviews! I will appreciate the praise or the criticism as long as it's intelligent! Thanks!

Marzipan
I don't see those comments now so I assume they were removed? But either way don't worry, I don't think many of us are going to see something like 'tu muk reeding' and go 'Wow, what an insightful review! This game must be terrible, I should avoid it!'

What little I've seen of your games so far has been very good by the way, I just have not had the time/cannot get myself into the proper mental state to play an IF game lately, let alone write out a detailed review for one, and I have a huge backlog to catch up on as it is.

InfectedBehaviour
Maybe if Quest had a feature of when someone has published their game. That the game it's self has a rating option once the game is finished by them. The player gets an option to rate the game or not. If rated, The rating/review gets sent online. That way non-registered and registered people can maybe rate the games properly. Plus also knowing that the person rating has actually played that game.

The Pixie
How about only allowed someone to rate a game after they have had at least 20 (or so?) turns playing it?

That would discourage people rating the game for revenge, as it would take quite a bit if effort to do, but would not discourage anyone who had actually played it. Unless your game is very short, in which case it should be in the sandbox!

davidw
What's to stop people typing LOOK 20 times in order to be able to give it a bad review? Not to mention that would only work for online plays.

XanMag
Been a while since I commented on this thread...

but, should I/we create a separate account for reviewing/commenting on games. Obviously in my comments I pissed someone off. lol.... sort of.

I went through and placed a few games fairly in proper categories. My comments are almost entirely positive or at least positive criticism (feel free to view them - tell me if they are not fair). Then... you get some offended troll that goes through and marks 1-star ratings for my games. I stopped reviewing games a couple months ago for this reason.

You may not like Furby McQuack or Xanadu Part I, but they are DEFINITELY NOT 1-star games. In fact, trying to put all biases aside, they are considerably better than any game I have "officially" reviewed.

I know Alex has Almighty Powers and can remove unwarranted reviews/ratings, but that is a hassle for both he and I. Bottom line... my attempt at constructive criticism and support for games will cease if it means I take unfair hits on quality games that I would like other people to play.

So... should I go to the trouble of creating a separate account to continue supporting/reviewing/helping game authors, or should I only write glowing review/comments and ignore the trash games?

Anyway, my rant is over. But, I would like some feedback on this and, if possible, a correction to those unfair reviews.

Thanks.

The Pixie
If I had responsibility for placing games in catagories, I would definitely do it with a different account.

OurJud
The Pixie wrote:If I had responsibility for placing games in catagories, I would definitely do it with a different account.

I'm a bit confused by this. Are you both referring to the moderator rights that Alex is currently granting, for anyone willing to help categorise the backlog of uncategorised games?

So the author can see who decided what category their games go in?

XanMag wrote:I went through and placed a few games fairly in proper categories. My comments are almost entirely positive or at least positive criticism (feel free to view them - tell me if they are not fair).

And Alex didn't say I had to add reviews for the games, too.

XanMag
No. You definitely do not have to comment or rate (I don't rate), but personally, I feel like if I play a game that someone took the time to make, I might as well add a positive (or at least constructive) comment behind. So... I guess, maybe I stop doing that and just throw games in categories as needed.

So much for trying to be helpful/thorough.

The Pixie
OurJud wrote:

"The Pixie"

If I had responsibility for placing games in catagories, I would definitely do it with a different account.


I'm a bit confused by this. Are you both referring to the moderator rights that Alex is currently granting, for anyone willing to help categorise the backlog of uncategorised games?

So the author can see who decided what category their games go in?

XanMag wrote:I went through and placed a few games fairly in proper categories. My comments are almost entirely positive or at least positive criticism (feel free to view them - tell me if they are not fair).

And Alex didn't say I had to add reviews for the games, too.


XanMag said "I went through and placed a few games fairly in proper categories" so I assume he has those moderator rights. I may be reading more into this than is actually there; I do not know how the web site works behind the scenes.

davidw
People don't like criticism and tend to react negatively to it. Even if the criticism is well intended, even if the game being criticised is downright awful, certain people will take that criticism to heart and lash out. Check out this thread:

viewtopic.php?f=10&t=3524

I make a comment about a game that has a crate which can't be examined and the next thing you know, I have someone going out of his way to criticise one of my games from years before and then even going on to admit that he hasn't even played it!

Alex
Authors can't see who moderated their game. If they respond to comments by trashing your games, let me know so I can decide on appropriate action to take.

You shouldn't need to create separate accounts for fear of reprisals - that is very much not the kind of community we have here. Please report all misbehaviour so I can deal with it.

OurJud
XanMag wrote:So much for trying to be helpful/thorough.

To be perfectly blunt, Xan, I'd say constructive criticism would be wasted on the majority of authors. I haven't even considered adding any comments because the games are so bad nothing good could be said, constructive or not.

If Quest does have a BR, it's now clear to me where it comes from. It's no exaggeration to say that 95% of the games I've sifted through so far (and it's a fair few) have had to go in the Sandpit category, and a good 80% of those were either nothing more than short stories written with Inklewriter, test games, unfinished, broken, bugged, or just kids messing around. The number of games with description such as 'jduyhhfdhfhfdheufhuihcbuieipf' and other gibberish is ridiculous.

I think a lot of authors uploading test games simply forget to switch to 'unlisted' before hitting publish, but even then you can tell just from their test game that the finished thing is unlikely to be any better.

This topic is now closed. Topics are closed after 60 days of inactivity.

Support

Forums