Since the very first research about AI by the end of the 20th century, everybody has known how superior AI over humans is in many facets, especially about the ability to minimize errors in work. That’s why scientists have been having many difficulties in creating an algorithm to test its ability and limitations. However, there is a will, there is a way. After thousands of trials and experiments, one of the most famous “Big Four” Technology companies, Facebook, has released its unique test with a very familiar name recently:
#howmuchhaveyouchangedchallenge
Before discussing the “how much have you changed challenge”, let’s API introduce you to the grading method of the scientists who study AI diligently.
(Source: venturebeat.com)
BENCHMARK or DYNABENCH
Benchmark is the act of running one or more computer programs in order to assess the relative performance of an object. (Typically by running a number of standard tests and trials against it.)
Benchmarks once played a vital role in setting specific challenges, goals, and barem to help AI improve by overcoming its previous limitations. However, in some certain aspects that require thinking such as linguistics, the old barem is no longer suitable.
Douwe Kiela, a researcher at Facebook AI Research, once said: “Benchmarks can be very misleading. Focusing too much on benchmarks can mean losing sight of wider goals. The test can become the task.” (Heaven, 2020)
Instead of focusing on helping AI overcome the challenge set by a number of their standards, Facebook pursues a different pathway that is more suitable for their ecosystem - a social media with more than 2.4 billion users. A new barem, called “Dynabench”, was brought into life. This barem is actually a hybrid of “dynamic data collection” and “benchmarking” (the practice of comparing business processes and performance metrics.)
Dynabench works on the principle of comparing the results of an AI with the correct results conducted by humans. This process is called adversarial human annotators.
For instance, one scientist can leave a comment like “The restaurant was previously awful but now is getting better” as bait. The old AI can easily be tricked (as it interpreted the comment word-by-word) and rate the restaurant 1 star. However, after getting real feedback from our scientists (4/5 stars), AI will automatically learn its lesson and establish a new paradigm to train the future generation not to stumble over the same mistake. This automated loop that helps AI optimize its ability better [in the future] called Dynabench.
By now, you might have a question: How do all these things related to #howmuchhaveyouchangedchallenge?
Despite having no official statement from Mark Zuckerberg, we can implicitly predict some clever Facebook tactics to collect data that is suitable for Facebook AI Development. In one article published by MIT Technology Review, Kiela - Facebook AI Researcher - said that:
“.... You just need to find a way for people to upload their picture - or have them to draw - to test it: ‘The long-term vision for this is to open it up so that anyone can spin up their own model and start collecting their own data.” (Heaven, 2020)
Long story short, we have been empowering the researcher through our interactions on social media platforms and multimedia means, in general. About the challenge that we are facing, photos of our past and present can be utilized by Mark’s Researching Team as materials to test their child AI.
See the full post via this link.
---
Credits for API's Team:
Content: (in Vietnamese) Van Thanh.
Translation: (in English) Thanh Hai.
Cover photo: Tri Dung.
---
For any concern, please contact:
Email: [email protected]
Page: https://www.facebook.com/theapitales
Hotline: (+84)938327709 (Van Thanh Mr.)
(+84)702440636 (Giang Dang Ms.)
---
Reference:
Heaven, W. (2020, September 24). Facebook wants to make AI better by asking people to break it. Retrieved November 17, 2020, from https://www.technologyreview.com/2020/09/24/1008882/facebook-ai-test-benchmark-people-break-adversarial/
