by Doc
For almost the entire duration of the competitive amiibo scene, the Amiibo Dojo has been the hub of all competitive amiibo competition and discussion (these days, the scene has expanded out into many other servers). In fact, outside of Amiibo Doctor, they’re just about the only competitive amiibo content at all. It’s no surprise, then, that they have curated each iteration of the amiibo tier list over the years.
The recent tier list has drawn the ire of some segments of the community due to the method by which it was created: open discussion with many members of the community in real-time. The discussion was openly and intentionally declared to be focused on solid results instead of a combination of potential and proof, a first for amiibo tier lists. Previously, tier lists weren’t created in the open or with any explanation of the placements inside: rather, a very small group of trainers would argue over the tier list for a period of time, and then a new one would be created. This method drew complaints from the rest of the scene, who believed the tier list was full of in-group biases and wasn’t based solidly enough on the actual results.
Without addressing personal opinion on the matter, let’s discuss the question that this situation provokes: who and how should we determine tier lists, and what criteria should they be built on? I’ve asked the question of many members of the scene, and based these scenarios on their responses.
Who and how should we determine tier lists?
- Tier list placements should be openly determined by popular vote
- Tier list placements should be openly determined in real-time chat (the current list’s method)
- Tier list placements should be exclusively and quietly determined by trainers selected by the owners of the Amiibo Dojo (the previous list’s method)
Using color-coded bullet lists, let’s summarize the pros and cons of each one. Then we’ll come to a conclusion.
Popular vote
- Averaging vote totals leads to a better gauge of overall opinion
- All opinions being included leads to little room for complaints
- All known information is included somewhere in creating the tier list
- Mathematical justification of placements minimizes opposition
- Overall opinions may be arguably incorrect or unjustifiable
- Voters may not have all relevant information
- Single outlier votes could needlessly change the tier placement of amiibo
Real-time open determination
- Better gauge of opinion of many active users
- Explanations for each character placements are openly available in chat
- Discussion is held in real-time without restrictions on who can comment
- Inclusion of many voices in the scene lessens the complaints of exclusion, though not as effectively as popular vote
- Fun as hell to join in on, I tell you what
- Users not online have no say
- Real-time arguments are less subject to evidence-based longer-form discussion
- Users involved in the chat may not be as qualified as they ought to be
Very small, selective group with exclusive claim to tier list creation
- If chosen properly, group would consist of most qualified opinions
- Discussion can be much more thorough due to longer-term nature of the group
- Group insiders can be less influenced by outside opinions
- Group may or may not explain their placement decisions
- No one outside group has a say, which could lead to complaints
- Group likely to develop in-group biases or information blindnesses that spoil tier list
- Outsiders may have relevant information unknown to the group
- If not chosen properly, group system would function no better than other options
- Decision of who to include in the group is entirely subjective and handled by an even smaller group
Moving away from the how and towards the what, let’s talk about the criteria portion of tier lists.
What should tier lists be based on?
- Popular vote (visited above, no need to restate)
- Tournament results-based (current list)
- Potential-based
- Subjective mix of results and potential
It’s important to note that previously no official explanation has been given for tier list placements, so the previous methods cannot be properly analyzed.
Tournament results-based
- Almost completely reduces subjectivity of placements
- Inclusion/exclusion are irrelevant
- Mathematical justification for placements can be used
- More submissions of an amiibo can influence the number of wins
- Variety of tournament rulesets means some amiibo may undeservedly place differently
- Doesn’t reflect the potential of amiibo that may have been overlooked recently
Potential-based
- Can encourage trainers to experiment with amiibo and maximize previously hidden potential
- Is a “forward-thinking” tier list that theoretically is closer to the end of the metagame
- Potential is entirely subjective
- Completely detached from tournament results and any other hard data
- As training methods improve the understanding of amiibo potential changes as well, causing potential-based tier lists to quickly be outdated
Subjective mix of results-based and potential
- Can combine hard data with possibility of improvement
- More even-handed mix of input allows community to more easily explain placements
- Mix can be adjusted for availability of hard data
- Most benefits of results and potential carry over to the subjective mix
- Subjectivity still leaves room for complaints and representation issues
- Most issues with results and potential carry over to the subjective mix
- Determining how to use tournaments results vs. opinions on potential leaves a larger room for mistakes
Given all these pros and cons, one thing seems certain: tier lists are a pain in the ass.
Amiibo Doctor 