Matchmaking Problems and Solutions

🚨 New Hacking Academia video out on Matchmaking Problems and Solutions in research

It's *that* time of year again - submissions for the main international robotics conference ICRA2027 close in just over a month.

In today's video I cover:

💕 𝐲𝐨𝐮 𝐧𝐞𝐞𝐝 𝐚 𝐩𝐫𝐨𝐛𝐥𝐞𝐦 𝐭𝐨 𝐬𝐨𝐥𝐯𝐞. If the task you're developing solutions for is already near perfectly solved, it doesn't give you room for improvement (and improvement is usually hard to come by). You can typically find that room for improvement by considering the practical, real-world grounding for the task you're performing:

🔹 how can your robot perform that task in more challenging and varied environmental conditions?

🔹 how can your robot perform equally well with less training, or with less access to relevant training data

🔹 how can your robot become better at self-assessing when it is performing well or not

🔹 how can your robot achieve a level of accuracy, coverage, generality or robustness to adversarial interference that current methods cannot?

And so on: there is a reason robots are still not widely deployed, and you can find research inspiration in the gaps that remain to be bridged...

💕 𝐲𝐨𝐮𝐫 𝐬𝐨𝐥𝐮𝐭𝐢𝐨𝐧 𝐧𝐞𝐞𝐝𝐬 𝐭𝐨 𝐏𝐀𝐑𝐓𝐈𝐂𝐔𝐋𝐀𝐑𝐋𝐘 𝐟𝐢𝐭 𝐭𝐡𝐞 𝐩𝐫𝐨𝐛𝐥𝐞𝐦. There are many "generic" ways to improve performance: more training, more compute, using more data over time, using more sensors. Much of the key work has been done here already: there is a higher bar for demonstrating why your contribution is particularly noteworthy against all that prior work. 

A better situation is one where some key aspect of how your proposed solution *particularly* fits/solves the type of problem.

For example, in the early days of GANs - Generative Adversarial Networks - some networks demonstrated an impressive ability to re-generate real world photos under different conditions - for example turning a summer scene into a winter one - an application with great potential utility in making "all season" localization systems.

Another example is a localization system that uses multiple sensors where one sensor works particularly well in large open bland areas (e.g. GNSS in the middle of a desert) and one works well in cluttered environments with lots of landmarks (e.g. a camera in a city streets).

These are situations where the contribution moves beyond just "more/better is better" and has *specific* reasons why the change is particularly helpful.

This doesn't mean general improvements are not helpful, but the more intuition you have that your proposed approach is particularly suited to the problem, the more likely you will end up at a valuable, and publishable, contribution.

YouTube: https://youtu.be/VklLyfTGk5M

#research #academic #ICRA2027 #robotics #computervision #artificialintelligence #publishing #papers #writing #advice

Transcript

SPEAKER_00 0:00

Well, it's that time of year again. It's ICRA time. ICRA, of course, being the IEEE International Conference on Robotics and Automation, one of the largest premier robotics conferences in the year. We were there a few months ago. It had 8,700 attendees, and submissions for next year's conference, which will be in South Korea, close in just over a month. For today's Hacking Academia video, I wanted to cover a particular aspect of the publishing process, given that's what a lot of roboticists and related researchers are thinking about right now, which is the matchmaking process between problems that you're trying to solve and solutions that you are proposing or trying to develop. So the first key concept is simply that if you are trying to do research, you need to have a problem to address or solve where there is at least some room for improvement. So if the specific task you're addressing is one that is widely considered to already be pretty much nearly perfectly solved, that's obviously going to be an issue in a couple of different ways, at least. Firstly, there's not going to be much interest in further improvements a lot of the time. And the second is that even if you are interested in tackling the problem, it's going to be particularly hard to make meaningful improvements because the systems are already so good. A lot of very talented researchers have put a lot of research into refining and perfecting their approaches. Now, you have a bunch of possible solutions here. Probably the best way to think about it is to think about the real-world problem or application behind the research that you're doing. So let me just use an example from the field of autonomous vehicle camera-based positioning systems, which is one of the areas I work in. So you could argue that with sophisticated cameras under perfect daytime conditions in pre-mapped cities, this problem is more or less nearly perfectly solved. Now, I'm not going to go into the nuances of whether it is completely solved, but it's definitely a field that's received a lot of attention. So how do you find that room for improvement? Well, you think about the practical requirements or considerations of the problem. So, for example, you might think about the amount of training data that needs to go into these systems. And you may have a you may be interested in radically reducing the amount of training data, which would be very practically useful. It would be easier to set up an autonomous vehicle localization system in a new city. You wouldn't burn through as much power at training your systems. So that's clearly a practically useful thing to do. You might be interested in getting your system to work under adverse weather events. So, for example, not just during sunlight or during the daytime, but in the middle of the night when it's dark and the lighting is very different. Also, perhaps in the middle of a snowstorm or in the middle of a torrential rain downpour. So you're thinking about extensions of the practical task that you're trying to solve that introduce problems that are by no means perfectly solved by the field jet. Additional considerations would be an application like this is a critical life and death situation. You can't afford to get it wrong. One of the key things that might help you do that are processes like introspection. So helping the positioning systems or whatever the task is you're solving be self-aware of when they're performing well and when they're perhaps not so performing well, which would typically form a key component of a safety system. These are just a few examples of how you can go searching for meaningful, not yet fully solved problems to solve, but they all revolve around thinking about the actual practical implications of how would you roll out this system in practice, what are some of the constraints or things you'd like to make better, and you can almost always find areas where there is significant room for improvement and where you can make a contribution. The second key component, and this is probably the most important thing, is that ideally the solution or approach or technique that you are developing or proposing should be particularly suited to solving the problem that you are trying to address. So in robotics and lots of related fields, there are lots of generic ways to improve system performance. For example, you can just use a lot more data and train on a lot more data. You can use more than one sensor, you can use lots of sensors and fuse them together. If it's a moving platform, instead of making decisions based off just what you sense at that moment in time, you can integrate or filter information over a period of time. All of these approaches, sensibly implemented, should improve performance. The risk you run there, of course, is that there has been a lot of work with these general purpose improvement approaches, and you risk a reviewer or a reader of your paper basically saying, duh, you're using more, of course it's going to improve things. So one of the key ways you can get around this is to think about solutions or approaches where some particular property of the system you're developing particularly suits the nature of the problem. So to give you a couple of examples of this, in the early days of GANs, generative adversarial networks, people discovered that these systems are really good at taking a photo of the world and transforming that photo as if it would look like under very different conditions or from a different stylistic perspective. For autonomous vehicle or robot localization using cameras, this was really interesting because it raised the possibility that you might be able to simulate what a photo taken of a city street looked like during summer and simulate what that might look like during winter, perhaps even covered with snow or from daytime to nighttime. Now, regardless of how well you think that actually worked out in practice, this is a great example of a particular capability of the solution that seems, at least potentially, particularly suited to this type of problem that you are trying to solve. Another example, again, drawing from my field, which is localization and positioning for robots and autonomous systems, is you can always use more sensors together. More sensors generally make things better. Again, you risk the sort of so what? Of course, you're using more sensors, they cost more. I would expect them to perform better. A particularly potent combination of sensors might be something like combining a sensor that works well in one type of environment, for example, in an open featureless desert, so using a sensor like a GPS or GNSS sensor, and combining that with a camera-based sensor that doesn't work so well in that bland environment, but does work particularly well in clouded environments where there's lots of visual features and landmarks. So this sort of approach isn't just blindly throwing together more sensors and getting the expected performance gains, it's leveraging the specific properties of those different sensors in a really synergistic complementary way. That means that you have a strong intuition that putting them together will give you good performance across this diverse range of environments. So, again, it's a solution that's particularly suited to the nature of the problem, in this case, localizing both in open deserts and also in cluttered city streets. Now, does that mean general purpose improvements are not worthwhile of research or papers? Of course not. There's still a lot of great uh research to be done there. I guess the bar is a little bit higher because there has been so much general purpose research. And of course, if you're using more data, more training, more sensors, of course, you're gonna show an improvement. So the onus is to show why that's particularly interesting. In implementing these general purpose uh improvement methods, again, there are ways to still make a contribution. Usually the ways in which these methods are implemented have their own shortcomings, and so you can refine and improve the state of the art and how these conventional approaches are implemented. But again, a lot of people have already worked on this, so it can be a little harder to find your niche. The advantage of the intuitive approach where you're developing a solution that you believe has some key property that particularly suits the type of problem that you're addressing, is when things go a little bit differently, a few things go wrong in research, there's more chance that you're still gonna pop out the end with a really clever, elegant way to improve performance or solve a problem a little better than it was done before because you started with that intuitive concept that had a lot of promise and investigated it from there. If you are doing uh papers for Ica or for any other conference, this is just one particular approach. A lot of people don't need any help. They're great, they've been working on their research idea for a year or more. But if you do find yourself a little bit stuck, sometimes coming down to these simple concepts of finding a problem or a problem subset where there's room to improve, and then thinking about a solution or a technique where the properties of that technique particularly suit the particular problem you're trying to solve is a great way to substantially increase your chances of having some successful research with some great outcomes that are of utility to the community, and of course, hopefully, getting your paper published. Good luck.