Whats worth mentioning is that the reason for Andreas Madsens attempt to publish the summit paper is his experience of bumping into obstacles in the process of applying for doctoral admission.
A researcher who has a masters degree in machine learning, rich experience in machine learning development, and has published some important papers has been rejected by a number of professors only because he has not published top-level papers?
In his view, this phenomenon is more like a kind of new elitism in academia, which will eventually enlarge the inherent prejudice that has challenged the industry.
In this regard, he finally called for: there should be a clear academic, not only paper theory!
Here is his personal experience:
In April 2019, I decided to become an independent researcher in order to publish my thesis at an important academic conference. Finally, in December 2019, after a long seven or eight months of research without funding, I succeeded! My thesis has been accepted by ICLR. Whats more gratifying is that this thesis has been accepted as spotlight thesis!
After I published this paper on twitter platform, it spread rapidly, bringing hope to researchers like me. Therefore, I have received nearly 100 personal letters so far, and they are all asking for my opinions and want to know how to publish a paper as an independent researcher.
In this article, I will give you some suggestions in this regard. But more importantly, I want to talk about why I chose to be an independent researcher. You should know that working for seven months without any financial support is not a joke, which deserves careful consideration!
Fig1: Yes! My first top conference paper ICLR received as spotlight! We improved the performance of deepminds paper Nalu by 3-20 times. As an independent researcher, I spent 7-8 months to finish the job without any funding.
Paper address: HTTPS: / / openreview. Net / forum? Id = h1gnoehkps.
In March 2018, I published a paper visualizing memory in RNNs on the platform of until. Pub as the only author, which explained how to use an interactive saliency visualization technology for natural language processing tasks to prove that two models with almost the same accuracy rate have greatly different performance.
Paper address: https://distill.pub/2019/memory-in-rnns
Once upon a time, I thought that with the above mentioned thesis and my masters degree in machine learning, I would be able to apply for the position of doctoral candidate, or research software engineer, internship project or machine learning engineer. So I began to send doctors application letters to many professors, and at the same time I applied to Google, Microsoft, Rakuten, elementai, NVIDIA, hypefactors, Intel, JD, Amazon, Samsung, shifttechnology, Corti and other companies.
But in the end, I didnt even get an interview!
When I applied for research software engineer of Google, even a senior researcher of Google came to me. He encouraged me to apply for the position in their department and formally recommended me. To be on the safe side, I also asked another Google Senior Software Engineer to tweet me. He has worked with me on the open source node.js project.
However, I still didnt get an interview, no HR contact me, no news!
The reality is, one or two machine learning summit papers are very important.
During the application period, I received a reply from a professor:
Thank you very much for your email. The content is very interesting. Generally speaking, its very difficult to apply for a doctors degree without one or two machine learning / NLP papers in the top journal. The paper you published on distill and your experience in industry make up for that to some extent. Unfortunately, I dont have any vacancies to apply for right now. u2014u2014A professor from QS global top 100 university.
A friend of mine (who knows a former committee member of another university) received the following reply:
Of course, I also received some other emails (most of them didnt reply to me). The professors didnt say so directly, but they were not interested in my application at all.
I think they actually require applicants to have one or two top papers, but they dont want to admit it directly, because this poor standard is only good for those very lucky students. u2014u2014The university where I studied during my masters degree didnt encourage us to publish papers. But Denmark is also a good place to study for a masters degree.
So if you want to apply for a Ph.D. degree without internal promotion, you need to have half of the results you can get.
How to provide financial support?
How can I find my own research ideas? How can I fund myself? Thats the most questions Ive been asked so far. However, although these problems need to be solved, I dont think they are so important.
In fact, there are many ways to explore research ideas; if you dont have to bear the expenses of others, there are many ways to save money on your own. You can enter a state of living alone, but not giving up reason and hope, which will not make you feel very uncomfortable.
I was a freelancer from September 2017 to October 2018. When I first became an independent researcher, my friend Emily Bay recommended me to nearform, who wanted to do a new project called clinic. JS. This project not only requires engineers to have a detailed understanding of the internal implementation of node.js, but also requires statistical background and web visualization technology.
This project is just for me! Before that, I have done a lot of visualization work, participated in the work related to the internal implementation of node.js for up to 6 years, and just obtained the masters degree in machine learning. So, I got a lot of money in this project. If I can control my spending at a very low level, the reward is enough to support my research for three to four years.
Im very lucky in terms of money. However, Denmark is a country with a high level of consumption, and it may be easier to fund its own research in another country.
In 2019, they also invited me to develop tensorflow components for their IOT smart watch / BADGE launched on nodeconfeu2019. Although this income is not enough to support all my expenses throughout the year, it also helps me a lot.
Figure 3: Smartwatch device developed by the author
Where do good research ideas come from?
As I said, there are many ways to explore research ideas, so dont pay too much attention to my words, and give full play to your own creativity.
In February 2019, I went back to my alma mater to participate in the opening ceremony of an artificial intelligence student community, hoping to talk with my former tutor about applying for a doctor. Unfortunately, he could not offer me any chance to enter the school.
However, I met Alexander r Johansen, an assistant researcher looking for partners. In late March 2019, I wrote to him and he told me that several of his students tried to reproduce deepminds paper Nalu, but all failed. So, he asked me if I would like to study it, maybe we can write a paper on neurips.
My masters thesis and my thesis published on distill are critical thinking about other peoples exaggerated or misleading works, and then improving these works. And the difficulty of this research is optimization, which Im good at. So, I think I can do the job.
Almost all published papers exaggerate their performance, so improving others work is a feasible research strategy.
Thats what I suggest. This is not a very inspiring strategy to find research ideas. In this process, you will encounter some major challenges, but it is a feasible strategy.
Next, lets see whats most important!
No longer alone in the road of research, do not lose your sense and hope
As the first author of the thesis, its very difficult for you to write the thesis without the support of the people who feel the same with you. This is the first reason I suggest that you should not become independent researchers.
Everyone needs more or less encouragement. Dont think you can survive for seven months without any encouragement. I have been worried about the following situations: no solutions, unfair peer review, no useful experimental results, some important defects in the work, even if the paper is published, it will be too small and have no influence.
At the same time, there are also great risks in self financing research funds for seven months. If the final paper is not published, I will lose a lot. As an independent researcher, my chances of admission are below average because I get less feedback.
However, although you may not be supported by people you feel like, you can solve the problem in other ways. I do this:
1. I usually discuss research ideas with Alexander every week. Although Alexander didnt get his Ph.D., he had great critical thinking. I dont think its necessary to communicate with people who have many papers or many years of paper guidance experience. It is important that we communicate with those who can ask questions about our research work. Otherwise, we may become lazy and turn a blind eye to the mistakes in our papers. Please discuss your paper with others as much as possible, which will put a lot of pressure on you, so that you wont hope to be opportunistic.
2. In addition to the main research, I also carried out some additional projects. Its too risky to allocate all the time to the same job. Take the time to do small projects that you think are useful. Write an open source tool and reproduce a famous paper. Its also necessary to get out of the research and relax. In this way, even if the research project fails, at least some small projects have been completed. For me, its also a great encouragement to let these attached projects be recognized by famous researchers.
Figure 5: example of a side project completed in three days. The researchers who use this project will thank you very much, and you can feel the encouragement from it. (Twitter: Im glad to announce pipingstall ircurve here. It can be used to draw real-time learning curves, not as cumbersome as tensorboard. It contains a model independent interface to the keras help file.
How to write a good paper?
Only about 20% of all submitted papers will be accepted, and your peer reviewers will look for various reasons to reject your articles. You dont think its enough that the paper is good enough. You need to let your paper be praised as great!
However, you dont have any tutors to help you, and you havent published a paper before, so how can you write an amazing paper?
As far as Im concerned, my first paper was in the Journal of until. Pub. In retrospect, I was so lucky - distill was particularly concerned about the quality of the paper. Because, compared with peer reviewers (who may not be proficient in writing themselves), distill emphasizes that papers can explain clearly and educate others. For me, its much easier to write an article for explanation and education than to write an article that pleases peer review.
However, I was rejected by the editor when I first submitted my contribution to distill! They dont know what contribution my work has made. Is it a critique of nestedlstm, a new NLP task auto completion, or an interactive visualization technology?
There may be one or two important pieces of information in a paper. If they are misunderstood, the readers will be confused and rejected. So dont be afraid to repeat important messages to avoid these problems.
In my paper, the information mentioned above is: visualization can help people understand the model, but the accuracy is not.. However, you need to use a problem that anyone can have an intuitive feeling to reflect this. Obviously, the generation of Chinese poetry is not such a problem.
In my ICLR paper, the message is: the gating between heterogeneous units is much more difficult than it seems, but there are other problems to be solved before solving the gating problem, so we will solve the gating problem in future research..
Its not enough just to write like this. You need to express this idea repeatedly throughout the article so that even the laziest reviewers can see it.
I made a major revision of the paper, and submitted the revised version to distil again. This time, they are more open. Chris Olah and Ludwig Schubert on distill gave me very useful feedback before the paper went to peer review.
Without the experience from them, Im not sure if I can still be accepted by ICLR. You can read the skills of novelist McCarthy on how to write scientific articles, which covers most of the experience I have learned.
Finally, I would like to tell you that Alexander and I have spent a long time refining this paper, especially in the abstract and introduction, and his help is very valuable.
Rejected by neurips
We submitted our paper to neurips 2019, but it was rejected. At that time, I was suffering. After all, I spent so much time and got nothing. I feel that my dream is broken. I have no chance to pursue my dream any more. I cant engage in the machine learning research that I love.
Its a strange thing to leave your destiny in the hands of anonymous peer reviewers.
So why do we get rejected? The lessons I have learned are:
Some reviewers dont believe that we successfully reproduced the results of Nalu (the paper we improved). Many reviewers will ask, why is the result in the original paper much better than what you gave??
Some reviewers have asked us to do all the things that Nalus paper claims can be done in the submitted paper (even if we have provided clear evidence that Nalus model doesnt do these things to a satisfactory degree). If you read this paper, what we havent solved is division, and the gating between addition and multiplication. However, we have also improved other aspects.
In fact, the second point can also be attributed to the fact that some reviewers do not believe our experimental results and replication. This is the problem we may encounter in improving the exaggerated experimental results of others.
What I want to clarify is that the results given by deepmind at the end of the paper Nalu are not fake, they can be reproduced. But such a result is not the most appropriate way to build an extrapolation task. This task is the main goal. It makes the model look better than the first and second through reading (you need to read the experimental results more carefully). Moreover, Nalu only shows the results with a single seed, while our paper shows the results with 100 seeds. On one topic, we published a workshop paper measuring arithmeticextraction performance (address: https://arxiv.org/abs/1910.01888).
Whats different about the paper submitted to ICLR this time?
We have made some improvements to the paper before refuting the review opinions of neuroips2019. In the version submitted to ICLR, we added more evidence and experiments to support our view.
We did a very wise thing! We published the experimental setup and reproducing results of Nalu in the SEDL workshop of neuroips2019. We also tweeted the work and released the label of Nalu. He replied, great work! With a good benchmark, we can continue to improve the model. This is very helpful for us. We dont have to argue about the experimental setup and our new model any more. Instead, we can focus on our proposed model.
Figure 7: it is very difficult to implement mathematical extrapolation on neural networks. We propose a new benchmark and find that Nalu is very fragile. So how can we build a neural network that can learn arithmetic methods? Just using more data doesnt do that.
I really hope I can tell you that our improvement has played a great role, but the fact is, I think we are just lucky to meet a great reviewer. Because the comments are public, his comments on our work in openreview have caused more extensive discussion, deeper critical thinking and less abusive comments.
It is worth mentioning that we have received four comments on the review, which I think means that our domain presidents are very interested in it. One of the reviewers has also reviewed our paper on neurips (probably the reviewer Chen 3 at that time), and his review comments on neurips are also the most constructive.
After we made a series of changes, the reviewers opinion changed from weak rejection of neurips to weak reception of ICLR, and then changed to reception in ICLRs final review opinion. He even commented on the opinions of other reviewers, the contribution reflected in this article has made great progress..
The reviewers comments are as follows:
I understand the concerns of other reviewers. The model presented in this paper is obtained by incremental modification on other models. But I dont think the advantage of this paper lies in the model itself, but in the theoretical and experimental analysis embodied in the full text, so as to improve the performance of the model, and the code of this article is also open.
This article is very readable. It tries to do some work for the recently proposed model. It points out the defects of the model proposed by others, gives a detailed analysis process, and then improves the model and comparison benchmark. In this paper, we face up to the problem of recurrence of the original model and improve it greatly. Thats why I think this paper must be accepted.
Im not sure whether the model proposed here will cause great changes in this field, but this method may influence and inspire other researchers to make more thorough analysis.
Therefore, I will raise the score of this paper to receive.
I am very happy with this review opinion. At that time, our paper had two weak rejections, so there was still the possibility of rejection. But even if I was rejected, I would at least feel that I did not fail, but that the review process was flawed.
Is all this worth it?
Finally, our paper was received by ICLR. I hope I can say for sure that this article will help me find a research job, but to be honest, I dont know what the future will be like. I just saw an email saying that I need 2 top papers (preferably with famous researchers) to be admitted to the top doctoral program, but as an independent researcher, I cant do that. I hope its just the university that has this requirement.
Figure 9: the road of machine learning research is long. However, if you dare not take the first step, everything will look impossible. If you really want to do it, try to find other ways.
Q: How do you draw these beautiful pictures?
Q: Your work is useless. You are a joke.
A: Thank you. Have you ever thought of becoming a reviewer?
Q: How long did it take you?
A: I spend about 48 hours a week on this paper, sometimes up to 100 hours. But remember, Im also doing some side projects and free work.
A: Alexander can provide us with computing resources because he is an assistant researcher.
Q: My tutor is going to give me a chance to read a blog. Should I accept it?
Q: Im practicing now, but I feel like I cant learn anything because my tutor is not very responsible.
A: You should be responsible for your beliefs. Dont expect your tutor to come to you. You have to be glad you got the internship, I didnt get it before. You can start to arrange communication with your mentor, but also remember that in addition to mentors, you can ask others for advice.
A: If you can find a mentor who is open to this, you can try to publish a paper. At the same time, try to look for internships during your study. Most internships are open only to students. I used to get an internship but I couldnt because I had graduated.
Q: How can I improve my programming ability?
Q: What else do I need to post a spotlight?
Q: I see some people become Google researchers when they get their masters degree. How do they do it?
A: Indeed, the golden age of this phenomenon is from 2013 to 2015. If you had a masters degree and were lucky enough, you could have gone far.
Finally, I would like to reiterate the main recommendations for other independent researchers.
2. Plan for the worst and do your best. Only 20% of papers will be accepted. As an independent researcher, your probability of being accepted may be lower than the average. Dont be discouraged after being rejected once! You can also do side projects, so that even if you fail, you dont feel like youre wasting your time.
3. Avoid becoming an independent researcher. If you have a better choice, you shouldnt have any reason to be an independent researcher. Maybe your masters supervisor is not interested in you, but at least you can communicate with other doctoral students. If you become an independent researcher, you lose the very important network that supports you.
Figure 10: being an independent researcher is extremely difficult and lonely. However, if you are lucky enough, you may also find hope around the corner.
There should be a clear stream of learning, not only paper theory!
Finally, since we had a paper accepted by the relevant seminar, I did participate in neurips 2019. At the meeting, I had the honor to talk with several recruiters, professors and researchers. I am extremely shocked: there is a huge gap between the needs of recruiters, the willingness of researchers and the conditions that professors can provide.
What I want researchers to know is that in the past two years, the supply-demand relationship in this academic field has changed dramatically. Nowadays, it seems more difficult to get a chance to get a doctoral degree than to complete a doctoral degree.
If you have a masters degree by 2017 or earlier, you can still achieve the goal of being admitted to a good doctoral program.
Today, however, you need to publish 1-2 papers (preferably in collaboration with famous researchers) at top conferences such as (neuroips / ICLR / ICML) to meet the admission requirements.
I hope that researchers, professors, committee members and conference organizers can all help to curb this rapid spread of new elitism. It will only magnify the inherent biases that have challenged the industry.