The next generation of community machine learning

Original article was published on Artificial Intelligence on Medium


The next generation of community machine learning

How to supercharge crowdsourcing in AI

Are you amazed by how far machine learning has come in the last few years? Not only high performance infrastructure is available to run the algorithms, but more and more open source tools are becoming available for us to create awesome things. There are technological innovations every day, making the technology both more advanced and accessible for everyone. This is an amazing time to work in the field. If you feel motivated by this and want to help build the future with AI, we are alike.

We, the telesto.ai team, share this dream with you. Ever since we have started working in deep learning a few years ago, we wanted to bring machine learning powered solutions not only to the most advanced tech companies, but for everyone who can use it to improve their operations. To give you a concrete example, most of our team builds computer vision based algorithms for life science research in our day jobs. Working with top notch research groups, we see every day that even though the tools are there, it is not easy to find a proper team to develop, deploy and maintain a machine learning solution. Even if the actual algorithm is there, bringing it from proof of concept to production, preferably with an user friendly interface, is not an easy task and is often done poorly. We see this impeding progress too frequently.

On the other hand, we see millions of talented students, software developers, data scientists who are eager to hone their skills and constantly looking for a challenge. How can we find a way to channel all the talent where they are most needed? Our idea is a different kind of crowdsourcing. Crowdsourcing is an established method in the machine learning world, however, it is far from reaching its potential. We have an idea that will help it reach its apex.

The shortcomings of crowdsourcing

High level overview of the machine learning production life cycle

Machine learning is very complex. To deploy an algorithm in production, much more is required than training a model. Data collection, quality assurance, infrastructure engineering, API design are all part of the equation. And even when these are there, flaws often surface during deployment, so the entire process must be revisited and improved. This is the build-test-learn cycle. A stable and robust machine learning model arises after many iterations, almost never for the first time.

Current crowdsourcing solutions only contribute to a small portion of the development cycle and it is a one-time contribution, delivered in several months. An engineering team is still required from the consumer side to put the proof of concept into production.

Current crowdsourcing model

Even after a solution has been delivered, there are many banana peels to slip on.

  • Code is difficult to read.
  • Technical stack used by the solution is outside of the expertise of the engineering team.
  • The proof of concept model has enormous infrastructure requirements and thus not feasible. (Like many ensemble models.)
  • Private test data can be leaked and used by the competitors to overfit on the test set and produce useless solutions. (Like it happened to Kaggle recently.)
  • Unexpected new cases arise in production settings which are not covered properly by the model.

These are just the problems from the engineering team’s side. If you are a competitor, you might find the reward structure and participation terms unfair.

  • A one-time reward for the first few places, nothing for the rest of the hundreds of participants. Even if you were leading for weeks and only outcompeted in the very last hours.
  • Winners have to give up the training code and the entire model for good. The solution might be in use for years, only for a relatively small one-time fee.
  • Competition design mistakes, like a bad choice of evaluation metric, are set in stone. This leads to not only a useless solution, but to frustration of participants.

The issues are clear. How are we going to solve them?

A dynamic competitive crowdsourcing model

In essence, we think that the problem is the design of the crowdsourcing model in machine learning. The core problem solving process is isolated from the build-test-learn cycle of development, cutting off the feedback loop. This impacts both sides: the competitors can’t develop a production-ready model and the engineering team can’t properly rely on the winning solution. That is, if the customer has a software development team, which might not be the case. (It is certainly not in academia, where most of our team works.)

To solve these issues, we came up with the following idea. What if we make the competitions more dynamic and provide a fully containerized service directly to the customer each week? Instead of a one-time solution, we could constantly improve the performance and react to potential issues arising during continuous use.

Our idea of a dynamic crowdsourcing model

Of course, constantly improving solutions should be rewarded constantly. At the end of each period, the prediction code for the best models are uploaded and rewarded. We automatically package the model into an API in our side and deploy it for the customer. If the model is still the best for many weeks, the rewards keep coming. Period.

What is our progress?

Our platform is still under intense development, and we have a ton of features in mind to add. However, COVID-19 happened meanwhile, and we decided to use what we have built so far to help develop a new serology-based test, which will be free to use for everyone. So, we are launching an early version of our platform!

Each week’s winning solution will be deployed with a public API to help those who are in the front lines, fighting with the pandemic. If you are interested, you can find the details at our website telesto.ai!

We appreciate any feedback, may it be a bug report, a feature request, or a note about our ideas. If you would like to help, join our Slack community!

We are looking forward to build the future of AI crowdsourcing with you!