Retro Uprising - View Single Post

**d3scride** · 07-26-2014, 10:09 AM

Jace Hall gives some more insight into the new adjucation process (xposted from Twingalaxies.com forums):

After much consideration, we have concluded that this is a fair point. As such, we will attempt to provide some top level insight into the work that is being done. Discussing this matter has been avoided because we know that it will create more questions than answers and it will fuel speculation, which due to some of the unfortunate history that the community has been through, that may unnecessarily lean toward a semi-negative/cautionary interpretation of things.

For submission and adjudication, we are building an INTEGRATED system that is designed to address these items:

1.) Qualified community/peer review of submissions.
2.) A documented adjudication process per submission that is publicly accessible to all and permanently attached to the score/submission.
3.) The ability and process for the community to question and examine historical scores and permanently ratify, asterix, or expunge from the database.
4.) Adhere to Guinness-specific submission standards and adjudication criteria.
5.) A viewable historical public record of actions/successful adjudications for each participating community member which is used to calculate an overall "adjudication credibility" rating for the member. This rating will then factor into an "authoritative weighting system" that comes into play when voting on a submission.

As mentioned previously, this is an INTEGRATED system, which means that each part described above is tied to/and affects each other part in some way. A total effect is achieved and intended.

Below is all the detail we can provide at this time. Again, this posting is not attempting to answer all questions, and may in fact just create more questions than it solves, but our time is currently limited and we feel it is better spent actually doing the work required, rather than talking about it.

1.) Qualified community/peer review of submissions.
Submission videos will be uploaded through a TG interface, and once uploaded the video will not be alterable. It will allow easy selection of platform/game/variation. Once uploaded the videos will be made public immediately. The submitting user will have opportunity to completely describe everything and anything they want the adjudicators to know and attach that text description to the submission. While voting remains open, in progress vote results will not be visible. When all the voting criteria is met and voting completes, only PASS/FAIL final result is shown, not who voted on what. All of this takes place underneath paid TG Staff moderation and review - however, TG Staff can not vote, ever. Collusion, and popularity voting is not possible within the type of voting system we are implementing. MUCH thought and concern has been given to those issues. Rest assured.

2.) A documented adjudication process per submission that is publicly accessible to all and permanently attached to the score/submission.
A specific PUBLICLY VIEWABLE discussion thread will be created and directly attached to each submission. All participating adjudicators can place their comments and thoughts there. Areas of concern to look for, cheating speculation, etc. Every participating adjudicator will have access to all information as well as the submitting user and the submitting user can answer any questions or concerns raised during the voting period.

3.) The ability and process for the community to question and examine historical scores in the database from any time period and permanently ratify, asterix, or expunge from the database.
This is self-explanatory. It will follow a similar process to the normal score adjudication process, however there will be a few more logical hurdles to clear in order to ensure that the scores that are ultimately being looked at are truly of concern and not frivolous. Historical review is a big deal and is not going to be treated lightly. TG wants its database to become absolutely unquestionable over time and it is only through careful review and consideration that it will ever achieve this goal.

4.) Adhere to Guinness-specific submission standards and adjudication criteria.
The system being put into place has been taking into conscious concern items that have been discussed with Guinness over the last few months. We can not elaborate further. This is not a claim, announcement or public acknowledgement of anything other than the stated previous fact. Please bear with us.

5.) A viewable historical public record of successful adjudications for each participating community member which is used to calculate an overall "adjudication credibility" rating for the member. This rating will then factor into an "authoritative weighting system" that comes into play when voting on a submission.
Every adjudication has an outcome. Since there is a voting paradigm, there is a "winning" side of an adjudication. Where a participating adjudicating user "lands" relative to the final outcome will have an impact on that user's future voting credibility. The more often a user is "right" on a final adjudication result, the more credibility they will build. The more credibility a user has, the more impact they will have on the adjudication/voting process. This credibility has to be maintained and serviced, as it will always be in a very slow state of continual degradation. Also, as other users increase/decrease their relative credibility, the overall bell-curve of the system is affected, causing regular recalculation of the weighting system so that significant disparities can not occur. Under this mechanism, eventually, over time, some users will evolve to be almost referee-ish (in the old TG traditional sense) with the difference being that their authority and credibility was earned via the system and not bestowed by ownership. Additionally, every day and every adjudication becomes an effective checks-and-balances system for all adjudicating members. Regardless, no one single person will ever have the ability to adjudicate a score into the database.

What about private performances? What if I don't want people to see the technique I've developed to get my score? Uploading and making videos public really creates a problem for me!
We have not made a decision on this matter yet. While we completely acknowledge the desire to keep ones "secret sauce" secret, and also understand how it can theoretically increase the fun in competition by adding mystery into the equation, we also keep in mind that a player who wants to play in PRIVATE is asking to be acknowledged for recognition from the PUBLIC - and that can be perceived as a bit of a double standard by some people. We are evaluating this matter closely, and if we do decide to support it, it will require some special engineering and procedures to implement (which is possible to do.)

What if I don't have the ability to upload a video?
This is an exception to the rule, as most people these days do have a way to upload video (more than 48 hours of content is uploaded to Youtube per minute) - however, TG's goal is to try to have as many people participating as possible and is looking into potential 3rd parties that you could send your video tape to who could upload for you. This is an outstanding issue that is still on the table.

I want to record something now and I will submit it later! What are the rules?
Generally speaking, the long standing TG rules for recording a performance are remaining unchanged. However, please keep in mind that under this new system your performance is going to be scrutinized by many people. Please make sure that you include in your recorded performance whatever else you think may be necessary to give anyone watching the confidence they need to know that no cheating is taking place and that all the TG basic rules are followed. This system is hard on cheaters, but light on clearly good legit performances. Adjudicators will be focused on being correct in their voting so that they will build personal credibility, so no one is motivated to view your submission inaccurately on purpose. To do so would mean a drop in personal credibility because it is unlikely that everyone else voting would do the same. So the very nature of the system is just looking for truth. Be truthful in your recordings and you will have no problems!

We will repeat again that everything stated above is a greatly simplified, un-detailed, top level description of some of the work being done. It is not intended to be comprehensive and it does not discuss items such as:

New Track Variation Creation
Live On-Site Ajudication for Events
3rd party integrations that allow for rare earned one-of-a-kind non duplicateable custom certificates, badges, t-shirts, and other items made specifically for users that achieve different levels of status as an adjudicator or as a world record submitting player.
etc.

All of these things are being worked on one step-at-a-time so please continue to be patient with us.

Lastly, it is important to understand that everything that was said above, always will take place under internally paid TG STAFF moderation and review, and our system is not some kind of free-for-all. That being said, all TG STAFF will only ever be ADMINISTRATION, FACILITATION, RULES and STRUCTURE, never adjudication.

Since everything is connected, the engineering on all of this is extensive and complex. We are working on it. There are many questions we can not answer because we simply just don't know yet. Some of these systems have to be turned on (on our private duplicate TG SITE that we test on) and we need to see how they all play together in order to make adjustments and refinements. No system is perfect, and there will certainly be flaws in this new one, but we believe that it is a much better, credible, and transparent way for scores to be adjudicated and documented. It is a way to reward contributors, recognize achievement, involve Guinness (in theory **cough cough**) As you will see, our system is much more elaborate than other sites that may conceptually offer something similar.

Hopefully this post will be of some help - but of course all this is really going to do is raise more questions. Obviously there is a lot more to the system above than what was lightly touched upon in this post! Please bear with us and please do not consider what is said in this post as the final word in these matters. This is a lot of work and we are trying to do it right!

07-26-2014, 10:09 AM	#437
Donor Donor 95 posts View Stats	d3scride Re: Twingalaxies down Jace Hall gives some more insight into the new adjucation process (xposted from Twingalaxies.com forums): After much consideration, we have concluded that this is a fair point. As such, we will attempt to provide some top level insight into the work that is being done. Discussing this matter has been avoided because we know that it will create more questions than answers and it will fuel speculation, which due to some of the unfortunate history that the community has been through, that may unnecessarily lean toward a semi-negative/cautionary interpretation of things. For submission and adjudication, we are building an INTEGRATED system that is designed to address these items: 1.) Qualified community/peer review of submissions. 2.) A documented adjudication process per submission that is publicly accessible to all and permanently attached to the score/submission. 3.) The ability and process for the community to question and examine historical scores and permanently ratify, asterix, or expunge from the database. 4.) Adhere to Guinness-specific submission standards and adjudication criteria. 5.) A viewable historical public record of actions/successful adjudications for each participating community member which is used to calculate an overall "adjudication credibility" rating for the member. This rating will then factor into an "authoritative weighting system" that comes into play when voting on a submission. As mentioned previously, this is an INTEGRATED system, which means that each part described above is tied to/and affects each other part in some way. A total effect is achieved and intended. Below is all the detail we can provide at this time. Again, this posting is not attempting to answer all questions, and may in fact just create more questions than it solves, but our time is currently limited and we feel it is better spent actually doing the work required, rather than talking about it. 1.) Qualified community/peer review of submissions. Submission videos will be uploaded through a TG interface, and once uploaded the video will not be alterable. It will allow easy selection of platform/game/variation. Once uploaded the videos will be made public immediately. The submitting user will have opportunity to completely describe everything and anything they want the adjudicators to know and attach that text description to the submission. While voting remains open, in progress vote results will not be visible. When all the voting criteria is met and voting completes, only PASS/FAIL final result is shown, not who voted on what. All of this takes place underneath paid TG Staff moderation and review - however, TG Staff can not vote, ever. Collusion, and popularity voting is not possible within the type of voting system we are implementing. MUCH thought and concern has been given to those issues. Rest assured. 2.) A documented adjudication process per submission that is publicly accessible to all and permanently attached to the score/submission. A specific PUBLICLY VIEWABLE discussion thread will be created and directly attached to each submission. All participating adjudicators can place their comments and thoughts there. Areas of concern to look for, cheating speculation, etc. Every participating adjudicator will have access to all information as well as the submitting user and the submitting user can answer any questions or concerns raised during the voting period. 3.) The ability and process for the community to question and examine historical scores in the database from any time period and permanently ratify, asterix, or expunge from the database. This is self-explanatory. It will follow a similar process to the normal score adjudication process, however there will be a few more logical hurdles to clear in order to ensure that the scores that are ultimately being looked at are truly of concern and not frivolous. Historical review is a big deal and is not going to be treated lightly. TG wants its database to become absolutely unquestionable over time and it is only through careful review and consideration that it will ever achieve this goal. 4.) Adhere to Guinness-specific submission standards and adjudication criteria. The system being put into place has been taking into conscious concern items that have been discussed with Guinness over the last few months. We can not elaborate further. This is not a claim, announcement or public acknowledgement of anything other than the stated previous fact. Please bear with us. 5.) A viewable historical public record of successful adjudications for each participating community member which is used to calculate an overall "adjudication credibility" rating for the member. This rating will then factor into an "authoritative weighting system" that comes into play when voting on a submission. Every adjudication has an outcome. Since there is a voting paradigm, there is a "winning" side of an adjudication. Where a participating adjudicating user "lands" relative to the final outcome will have an impact on that user's future voting credibility. The more often a user is "right" on a final adjudication result, the more credibility they will build. The more credibility a user has, the more impact they will have on the adjudication/voting process. This credibility has to be maintained and serviced, as it will always be in a very slow state of continual degradation. Also, as other users increase/decrease their relative credibility, the overall bell-curve of the system is affected, causing regular recalculation of the weighting system so that significant disparities can not occur. Under this mechanism, eventually, over time, some users will evolve to be almost referee-ish (in the old TG traditional sense) with the difference being that their authority and credibility was earned via the system and not bestowed by ownership. Additionally, every day and every adjudication becomes an effective checks-and-balances system for all adjudicating members. Regardless, no one single person will ever have the ability to adjudicate a score into the database. What about private performances? What if I don't want people to see the technique I've developed to get my score? Uploading and making videos public really creates a problem for me! We have not made a decision on this matter yet. While we completely acknowledge the desire to keep ones "secret sauce" secret, and also understand how it can theoretically increase the fun in competition by adding mystery into the equation, we also keep in mind that a player who wants to play in PRIVATE is asking to be acknowledged for recognition from the PUBLIC - and that can be perceived as a bit of a double standard by some people. We are evaluating this matter closely, and if we do decide to support it, it will require some special engineering and procedures to implement (which is possible to do.) What if I don't have the ability to upload a video? This is an exception to the rule, as most people these days do have a way to upload video (more than 48 hours of content is uploaded to Youtube per minute) - however, TG's goal is to try to have as many people participating as possible and is looking into potential 3rd parties that you could send your video tape to who could upload for you. This is an outstanding issue that is still on the table. I want to record something now and I will submit it later! What are the rules? Generally speaking, the long standing TG rules for recording a performance are remaining unchanged. However, please keep in mind that under this new system your performance is going to be scrutinized by many people. Please make sure that you include in your recorded performance whatever else you think may be necessary to give anyone watching the confidence they need to know that no cheating is taking place and that all the TG basic rules are followed. This system is hard on cheaters, but light on clearly good legit performances. Adjudicators will be focused on being correct in their voting so that they will build personal credibility, so no one is motivated to view your submission inaccurately on purpose. To do so would mean a drop in personal credibility because it is unlikely that everyone else voting would do the same. So the very nature of the system is just looking for truth. Be truthful in your recordings and you will have no problems! We will repeat again that everything stated above is a greatly simplified, un-detailed, top level description of some of the work being done. It is not intended to be comprehensive and it does not discuss items such as: New Track Variation Creation Live On-Site Ajudication for Events 3rd party integrations that allow for rare earned one-of-a-kind non duplicateable custom certificates, badges, t-shirts, and other items made specifically for users that achieve different levels of status as an adjudicator or as a world record submitting player. etc. All of these things are being worked on one step-at-a-time so please continue to be patient with us. Lastly, it is important to understand that everything that was said above, always will take place under internally paid TG STAFF moderation and review, and our system is not some kind of free-for-all. That being said, all TG STAFF will only ever be ADMINISTRATION, FACILITATION, RULES and STRUCTURE, never adjudication. Since everything is connected, the engineering on all of this is extensive and complex. We are working on it. There are many questions we can not answer because we simply just don't know yet. Some of these systems have to be turned on (on our private duplicate TG SITE that we test on) and we need to see how they all play together in order to make adjustments and refinements. No system is perfect, and there will certainly be flaws in this new one, but we believe that it is a much better, credible, and transparent way for scores to be adjudicated and documented. It is a way to reward contributors, recognize achievement, involve Guinness (in theory cough cough) As you will see, our system is much more elaborate than other sites that may conceptually offer something similar. Hopefully this post will be of some help - but of course all this is really going to do is raise more questions. Obviously there is a lot more to the system above than what was lightly touched upon in this post! Please bear with us and please do not consider what is said in this post as the final word in these matters. This is a lot of work and we are trying to do it right!