Collaborative rating system?

Tools    





I know we probably don't need one, but I did notice that a lot of movies aren't getting a lot of ratings here. We have our top 100 voter's polls, but that hasn't really provided the incentive needed to rate the movies. Lots of film websites have a collaborative rating system. Now this can either mean we don't really need one and we can stick to our various countdowns which are a lot of fun, or it means it could help make the site a little bigger. I have a unique system prepared as an example for one if necessary, but I want to see where this goes before anything.

Basically, the idea is to take all the ratings of movies, total scores by an algorithm and see which movies come on the top. I have an easy prototype system I can work on as an example, but I don't have access to a graph of the ratings like others do, so I might plan an example chart tomorrow on Letterboxd (and on the off-chance this becomes real, I'll keep the algorithm / system a secret just in case)


Btw, I don't think it's a huge deal, but it would be interesting to see where the talk goes. I just like lists.



I'm open to anything that boosts the amount of people that actually rate films on here (it's actually the simplest way to mark films off the various lists for those that like to do that).

My primary concern with any collaborative ratings-based system would be that it seems to almost inevitably attract the type of person that deliberately seeks to skew results to their own personal bias (i.e. downrating/uprating a film purely because of genre, when it was made, who it stars or who it was made by etc.). Personally I find individual lists and countdown lists (where most people that are open about their lists generally seem to play fair with their rankings) far more interesting.



Does rating a film when you write up a review for it mean that you have rated it on the site? (As long as you have marked the post as a review).

I will admit I don't totally understand how the lists, ratings, and reviews are all connected. Though that is largely because my main interest is discussion and not ratings or lists.



Does rating a film when you write up a review for it mean that you have rated it on the site? (As long as you have marked the post as a review).
Yes. If you submit a review and it's approved, the movie is marked as Seen and it saves the rating from your review for the overall average. You can override both on the movie's page, though.

Here's the Seen Log for those who haven't found it (it's still a newish feature so it's tucked away in the User CP).

I will admit I don't totally understand how the lists, ratings, and reviews are all connected. Though that is largely because my main interest is discussion and not ratings or lists.
They're mostly just connected as described above, and in that marking a movie as Seen in any capacity reflects in the other places, too (IE: checking off a list marks it as Seen on the movie's page, and vice-versa). Hopefully I've made it intuitive enough that how you'd guess/expect them to be connected is roughly the way they are.



Anyway, to the OP: We have a couple things we can utilize.

First, the ratings themselves. We have just under 28,000 of those, almost half of which are just pulled automatically from reviews. We could use those, and we could even give extra weight to the review-based ones.

Second, Favorites. We can rank/weight films based on how many users' Favorites lists they're on. We can exclude ones from users with no posts (or just a few posts), weight based on posts perhaps, or anything else like that. I actually have a prototype ranking system for this already made, I just haven't really decided how/when to use it.

Both are pretty easy to measure/calculate.



Yes. If you submit a review and it's approved, the movie is marked as Seen and it saves the rating from your review for the overall average. You can override both on the movie's page, though.

Here's the Seen Log for those who haven't found it (it's still a newish feature so it's tucked away in the User CP).


They're mostly just connected as described above, and in that marking a movie as Seen in any capacity reflects in the other places, too (IE: checking off a list marks it as Seen on the movie's page, and vice-versa). Hopefully I've made it intuitive enough that how you'd guess/expect them to be connected is roughly the way they are.
Cool. That's kind of what I figured. I already rate films on IMDb (for my own use, mainly), so I am not interested in rating on multiple sites. I think of IMDb as more of a data warehouse and this site as the place where I discuss and discover new films. Being able to review and rate in one go is very convenient!

I do know that some sites are able to import data from IMDb (or you can download your ratings from IMDb and then upload them elsewhere). At some point down the road that could be neat, because it would populate ratings from people like me who have films they have seen in the past but wouldn't necessarily rate/review them here. But I have no idea how such integration works or how burdensome it would be to add to the site.



Anyway, to the OP: We have a couple things we can utilize.

First, the ratings themselves. We have just under 28,000 of those, almost half of which are just pulled automatically from reviews. We could use those, and we could even give extra weight to the review-based ones.

Second, Favorites. We can rank/weight films based on how many users' Favorites lists they're on. We can exclude ones from users with no posts (or just a few posts), weight based on posts perhaps, or anything else like that. I actually have a prototype ranking system for this already made, I just haven't really decided how/when to use it.

Both are pretty easy to measure/calculate.

I really didn't see this one coming. Based on other smaller sites I expected a reply such as "just rating and reviewing is enough." Sometimes I'm just happy to be wrong. I really can't wait to see how this goes considering that other websites have a hard time getting it together sometimes.


My theory was a potential ranking based on a points system. This was influenced by the system of Besteveralbums in which lists are the dominant factor. The higher up on album is on the list, the more points it gets, especially if it's a large list. But this is easy to abuse and doesn't take ratings into account. Rateyourmusic has a user-weighting specific algorithm, but this can get in the way as an album with 500 ratings and a 3.9 average could potentially overtake an album with 700 ratings but a 4.0, which is the oppostie effect of what RYM wants done. I figured, why not combine the two?



In this regard, points would be given or taken depending on the rating. I was worried that it would be too simple, but now I'm thinking that both of your ideas would work with it by doubling the points for the reviewed ratings and adding points for the favorites, not that I expect my idea to be the system chosen, but I'm open to pretty much anything.


Now all that's left is a proper genre system. BEA didn't have one either.



Cool. That's kind of what I figured. I already rate films on IMDb (for my own use, mainly), so I am not interested in rating on multiple sites. I think of IMDb as more of a data warehouse and this site as the place where I discuss and discover new films. Being able to review and rate in one go is very convenient!
Yeah, there's a handful of sites that do something like this and we're late to the party so I can see why someone wouldn't wanna start over. However...

I do know that some sites are able to import data from IMDb (or you can download your ratings from IMDb and then upload them elsewhere). At some point down the road that could be neat, because it would populate ratings from people like me who have films they have seen in the past but wouldn't necessarily rate/review them here. But I have no idea how such integration works or how burdensome it would be to add to the site.
...I do have a rough version of this working, yes. It's a bit messy and I really need to finish it/clean it up. The main issue is that it's one of those things that a handful of people will definitely use and find very useful, but that number of people will still be quite small, particularly at first, so it keeps getting pushed behind other tasks. Also a lot of thorny technical stuff about how to handle situations where you've rated or marked a film as seen on both (which takes priority if there's a conflict?), and the edge cases where people have thousands upon thousands and the script needs to try to not choke on that.



I really didn't see this one coming. Based on other smaller sites I expected a reply such as "just rating and reviewing is enough." Sometimes I'm just happy to be wrong. I really can't wait to see how this goes considering that other websites have a hard time getting it together sometimes.
I love tracking stuff and playing with stats and I've spent a decent chunk of time the last few years trying to plug everything into one centralized place. This kinda stuff is definitely my jam.

My theory was a potential ranking based on a points system. This was influenced by the system of Besteveralbums in which lists are the dominant factor. The higher up on album is on the list, the more points it gets, especially if it's a large list. But this is easy to abuse and doesn't take ratings into account. Rateyourmusic has a user-weighting specific algorithm, but this can get in the way as an album with 500 ratings and a 3.9 average could potentially overtake an album with 700 ratings but a 4.0, which is the oppostie effect of what RYM wants done. I figured, why not combine the two?

In this regard, points would be given or taken depending on the rating. I was worried that it would be too simple, but now I'm thinking that both of your ideas would work with it by doubling the points for the reviewed ratings and adding points for the favorites, not that I expect my idea to be the system chosen, but I'm open to pretty much anything.

Now all that's left is a proper genre system. BEA didn't have one either.
Yeah, I think the main thing to keep in mind is that there's obviously no right answer: we see this with the curated lists, in that certain types of films seem to have an edge. Whatever weighting we choose (even if we choose none!) will have implications and it's always going to be arguable whether they're good or bad. Some will prioritize breadth of popularity, some intensity of popularity, and which matters more is just a fundamentally unanswerable question (nevermind how best to capture it, even if you have a clear opinion on that).

I'm up for some moderate experimenting on this, though. I've been meaning to "re-do" the Movies page (which is just a search box to finding movie pages for now) for awhile, maybe some kind of ranking would be part of that. I have a rough version of exactly that. Maybe I could just make intuitive choices myself, just to get something up and iterate-able?



Intuitive choices are a good way to create a prototype. It reminds me of the philosophy that in order to write a book, you should just keep writing and go edit it later. It doesn't matter if it's bad as long as you have something to go back to later.



Okay. God knows I don't need another to-do item this weekend, but gimme a week or two, tops, just to get something really rough up.

Only a couple weeks? Dude! After how sluggish BEA and RYM can be I was expecting a few months.



Well, just to make myself look cool, I'll point out I could have something up this afternoon (partially because I built something similar to this preemptively a year or two ago, and partially because it's pretty straightforward even from scratch), but I don't wanna rush it and there's a few fairly time-sensitive things I should probably do first, even though the shinier/newer/easier thing is always more tempting.



Well, just to make myself look cool, I'll point out I could have something up this afternoon (partially because I built something similar to this preemptively a year or two ago, and partially because it's pretty straightforward even from scratch), but I don't wanna rush it and there's a few fairly time-sensitive things I should probably do first, even though the shinier/newer/easier thing is always more tempting.

Two weeks is very quick for forums if you ask me. If you need to take your time, do so.



The main issue is that it's one of those things that a handful of people will definitely use and find very useful, but that number of people will still be quite small, particularly at first, so it keeps getting pushed behind other tasks.
Right. I would happily import ratings if you wanted me to (to have more data for the site to draw from), just like I started marking my reviews with the little checkbox. Seeing how movies are rated and ranked isn't the draw of this site for me---though I can't obviously speak for other users.

Or you could have one of those widgets that pops up random movies and you either rate them or mark them as not seen. I am not into looking up films just to rate them, but I would totally kill time clicking away at a widget if it was easy to access.



Ftr: personally don't like the idea of reviews being given more influence than just ratings only. Can understand the rationale behind it (to aid against people trying to game the system) but for those of us that don't particularly like or aren't good at writing reviews yet do take the time to give films honest ratings it would inherently be marking us down as inferior members.



Right. I would happily import ratings if you wanted me to (to have more data for the site to draw from), just like I started marking my reviews with the little checkbox. Seeing how movies are rated and ranked isn't the draw of this site for me---though I can't obviously speak for other users.
I think you're in the majority there. And to be honest, I'm not trying to attract people who are here for that kind of thing. I actually have specifically tussled with a few people over the years about just wanting to catalogue opinions but being uninterested in discussing them. A few things about the site and it's culture/aims have shifted over time (duh, it's been over 20 years!), but that might be one of the things that hasn't.

Of course, the fact that we didn't even have these options a few years ago means we've selected for this kind of interest, but it's an interest I want to select for anyway. I think it's a nice addition but if it ever becomes a bigger draw than the discussion/people, it means I've failed on some level.

Or you could have one of those widgets that pops up random movies and you either rate them or mark them as not seen. I am not into looking up films just to rate them, but I would totally kill time clicking away at a widget if it was easy to access.
This is actually a pretty fun idea (and 50% of "fun" for me is "relatively easy to implement"). I'll mark this down and give it a go. As far as time killers go I've seen a lot worse.



Ftr: personally don't like the idea of reviews being given more influence than just ratings only. Can understand the rationale behind it (to aid against people trying to game the system) but for those of us that don't particularly like or aren't good at writing reviews yet do take the time to give films honest ratings it would inherently be marking us down as inferior members.
Yeah, it's tricky. On one hand it obviously (indirectly) signals that, but on the other it'd just be an algorithmic thing done only with the hopes that it would produce a "better" list. Weighting based on post count would be kind of similar in this regard: sort of unfair to newer people, but probably producing better results, too.

Maybe to cover all (well, most) bases it could be either-or: just two levels of weighting, and either a few thousand posts or coming from a review both get the "higher" level. Dunno.

Anyway, need to fiddle with it, but FWIW I think the "worst" implementation of this from your perspective would still just be some boring behind-the-scenes thing done purely out of an eyeball-test for which list seems more emblematic. It would be much more about excluding the tons of Transformers and all-Harry-Potter favorites lists, or whatever, than about actually overrepresenting people who like to write reviews, if it happens at all. I will stay mindful of this in any event, though.



...if it happens at all. I will stay mindful of this in any event, though.
All I ask, thanks