• Lab Advice on Writing Research Papers

    Writing research papers is difficult but extremely valuable and rewarding. Writing is at the center of the research process -- not just a way to “write up” research that is completed. Writing will really help structure your ideas and identify gaps and new directions. Below is some advice on writing structure and Tips (below).


      Getting Started:

      Ideally 2 months before a conference deadline, start the ball rolling by proposing a draft title, abstract, and outline with a literature search of related work (start with Google Scholar and sort by most recent date). Discuss with Prof. Goldberg and potential co-authors, pointing out why the ideas are important and novel and build on the lab’s unique strengths.

      Paper Outline:


    • 1. Title and co-authors.
      The title encapsulates the new ideas. The lead author is responsible for coordinating and rallying the troops to meet the deadline! Be generous with co-authors. Usually the most senior co-author is listed last.
    • 2. Abstract.
      Summarize motivation, what's new, primary results/conclusions. Don’t over-claim, see Tips below. Quantify the results and provide intuition (in plain language avoiding jargon, use metaphors), Give the punchline: this is not a movie review ;) Again: Summarize the specific research contributions and do not overclaim.
    • 3. Introduction/Motivation.
      Take 3 steps back and explain to the reader why he or she should be interested in this paper. Know who your audience is! Your paper will be very different if you are writing for a general robotics conference vs an automation conference vs an algorithms conference. Give a concrete example where your problem arises and a few interesting applications. Describe what are the key prior methods that you will compare the new approach with and using which metrics. Summarize in more detail your contributions and why they are significant. Use bullets to summarize the very specific contributions in the Intro. Reviewers will greatly appreciate this.
    • 3a. Figure 1 (the “Money Figure”).
      This figure always goes at the top of the second column on page 1. It should conceptually summarize the project and contributions and have a self-contained caption (that is true for all figures but this one is especially important).
    • 4. Related Work.
      Demonstrate that you have a scholarly grasp of the published literature on your topic. Use Google Scholar (sort by date) to start and then follow the transitive closure of references (look up the references for each paper and recurse). Organize references by topic and then chronologically and summarize each prior work with a positive, appreciative tone emphasis on how your approach differs (** this is very important). One or two sentences for each ref. Typically 12-20 references for a conf paper, up to 40 for a journal paper.
    • 5. Problem Statement
      notation (refer to figures)
      assumptions: (collect and justify all assumptions here, this is critical, readers are open to almost any set of assumptions stated at the onset, but if you insert assumptions later it feels like you’re cheating and will aggravate readers and reviewers.)
      parameters: (list values/ranges you'll use and justify)
      input
      output
      metrics: (what you will compare and how you will quantify/evaluate the results, this should include sensitivity analysis)
    • 6. Experiments
      Details on all aspects of the experiments you performed and the prior methods you are comparing with. Enough info on hardware and software so others can repeat your experiments. Give specifics on the data sets, materials, what kind of computer you used, any human subjects info, Do tests to determine process noise and repeatability of the experiments.
    • 7. Results and Analysis.
      The most important section. Think carefully about how to present your results graphically. Report runtimes, complexity, metrics. In tables, highlight in bold the best performing method for each experiment. Be careful about using colors as they don't print in B&W: use dotted and dashed lines to distinguish curves.
      Always include Error Bars / Confidence Intervals for all data points (these were invented by Jerzy Neyman at Berkeley btw ;). If you forgot the formula here’s a good reference: https://en.wikipedia.org/wiki/Confidence_interval
      It is perfectly fine to have some negative results! In fact that usually strengthens the paper: a paper that only reports good news makes reviewers suspicious they aren’t getting the full story.
    • 8. Discussion and Future Work.
      Summarize again Acknowledge limitations of your paper, suggest open problems for future research. Don't be afraid to admit the limits: It is much better if you do this than others expose them later!
    • 9. Acknowledgements.
      Thank those who contributed ideas, references, proofreaders (see below), also thank sponsors with grant numbers. Ask me for this info.
    • 10. References.
      Be sure to double check how authors, title, source, and date of every publication cited is listed in the paper as LaTex / BibTex often makes errors. Each reference should clearly reference its source conference or journal, with the date, please always check this as BibteX can create errors.
    • 11. Submission
      Get all Paperplaza IDs for co-authors well before the deadline. Submit a “safety” version of the paper at least 12 hours before the deadline: conference submission sites (and ShareLatex) get very crowded in the last 3 hours.
    • ==========================

      General Tips:

      ==========================
    • Never Overstate.
      Always underpromise and overdeliver. Avoid unsupported or grandiose claims and always understate with phrases like “in the contexts we studied”, “we conjecture that”, “results suggest that”...For all claims think very hard about potential counter-examples, failure modes, etc. Even if we have collected the data showing statistically significant results, that is no guarantee that a method will work in all cases. The only guarantee is a proof, and a proof only holds when its assumptions are completely met.
    • Don't understate.
      Avoid words like "simple", "just", "easy", etc, it's nice to be modest but those words make your contributions sound weak and what is simple to you is not to your reader. better to describe as "standard, efficient, well-known".
    • State and justify all assumptions up front.
      e.g. We assume rigid polyhedral objects of known shape, etc. Expose all assumptions in the problem statement. Never “slip in” assumptions later in the text as that is very frustrating to reviewers, it’s always best to get these out of the way early.
    • Quantify and be specific whenever possible.
      Avoid vagueness: watch out for words like "usually, mostly, often, some, many...". Give crisp, refutable hypotheses.
    • Never Plagiarize.
      Describe prior work in your own words and never cut and paste text. Plagiarism can be easily detected and cause a paper to be instantly rejected.
    • Define before use.
      Always make sure you define terms/notation before you use them. Since you are so familiar with your topic and notation, be on guard for this and your ask your proofreaders to help with this.
    • Give Intuition before notation, proofs.
      Always provide the intuition / insight before the details. At the start of each section, step back and explain your ideas and insights in plain language before you give the technical details. use analogies to explain things.
      To test your approach, do experiments to compare it with the best existing method in terms of output, computation time, etc.. No method is perfect. Explain where yours is better and where it is not.
    • Illustrate whenever possible.
      Good figures really help clarify a paper. spend time on illustrations and ask friends for input. As an aside they will also make future presentations shine with less effort. Use SVG only.
    • Clarify Captions.
      I like long, self-contained captions. Each should have an explanatory title. Many reviewers and readers look over the figures and captions to get an initial impression of the paper and to decide whether or not to commit time to reading it. The caption for each figure should be self-contained and self-explanatory: describe what is being shown it in the first sentence and then give the intuition behind what it is illustrating. Describe all aspects including the variables for plots and any unusual words etc. Captions should be as long as needed.
    • Quantify.
      be specific whenever possible. Avoid vagueness!: watch out for words like "usually, mostly, often, some, many...". Give crisp, refutable hypotheses.
    • Experiments are used to evaluate, never to validate.
      The word “Validate” conveys experimenter bias. To evaluate your approach, be neutral and do experiments to compare it with the best existing method in terms of output, computation time, etc.. No method is perfect. Explain where yours is better and where it is not.
    • Try to construct cases where your method will fail.
      These counterexamples often lead to insights, or to a proof that your method will work in all cases.
    • Be Objective.
      Evaluate your method without bias. When reporting results, give enough data to evaluate the statistical significance of each experiment. Try to construct cases where your method will fail. These counterexamples often lead to insights, or to a proof that your method will work in all cases.
    • Acknowledge Limitations.
      In the discussion section, Clearly acknowledge limitations of your approach, and suggest future directions for improvement.
    • Be super meticulous with edit notes during the writing process!
      A great way to do this is to print out the edited version and mark every note with a red/blue pen when fixed, making sure that nothing gets skipped. When you can't fix an edit note, insert into the document a line like:

      ** Will revise tonight based on Prof. Goldberg’s edit suggestion **
      or
      ** I want to push back on Prof. Goldberg's edit suggestion to move this section because **
      (the “**” are important to to purge these before the deadline)

      Add also notes to comments you received via email.
    • Proofread.
      Always use spellcheckers and grammar checkers. Then ask your friends to proofread, ask 2 or 3 friends if English is not your native language.
    • Cultivate good proofreaders!
      Cultivate 3 or 4 students/friends you respect who will exchange papers with you for proofreading. It is helpful to have one or two outside the lab and even outside of your research topic. You'll learn about writing papers from reading their papers, and your papers will really benefit from their input. Make sure you thank them for criticizing your paper because they are saving you from being criticized by less friendly readers later!
    • Schedule the Home Stretch!
      The lead author on each paper should set up a Google doc ASAP to coordinate duties on experiments/writing and who has the edit token when to maximize parallel effort. For example list slots/duties in reverse order from the submission deadline:

      midnight Thurs: Submit
      8-11pm: First Author final polish with edits from X and Y...
      6-8pm: Proofreading by X and Y sending comments by 8pm

      Make sure you have the Paperplaza id for all co-authors and be sure all co-authors insert their availability. Note when you have classes, obligations, or sleep, and ask for help with writing and urge co-authors to take the edit token as often as possible. Everyone has to eat and sleep! But try to maximize available effort by maximal scheduling. Whenever you need help, ask Prof. Goldberg or others in the lab, many are avail to help proof and with writing.
    • ==========================

      Sanjay Krishnan added Tips ( from Mike Franklin and Joe Hellerstein):

      ==========================
    • Introductions.
      - In systems papers (i.e., all papers that are not studying a well-known open problem like TSP/MaxCut/etc. ), the introduction needs to motivate the problem statement and assumptions as well as the solution. Do not forget to motivate the assumptions in the intro as a common “lazy” review is one that just questions why the assumptions are sensible.
      - No typos or grammatical mistakes in the intro/abstract/background. Just looks sloppy and unprofessional, should be polished well before the mad rush of last minute experiments.
      - Don’t be afraid to say that a problem is well-studied, cite surveys, textbooks, and other summary references. Citing rare or unknown sources early (and making them look seminal) makes it look like you don’t have a grasp of what the field is doing.
      - Know the difference between a model (the way you chose to believe the world works), an algorithm (the way you chose to solve a problem given the model as true), and system/framework (a physical implementation of the model and algorithm).
      - Do not confuse the problem with the method -- the method may well have problems of its own!
    • Proofs and Formalism.
      - Know your audience. A statistical proof does not belong in the body of a database paper, nor does a logic-based argument belong in a statistical paper. Proofs should be moved to supplemental material if they are not the core competency of the reviewing community.
      - Don’t overdo formalism, some assumptions/conjectures are well-established and fine to make without statement, e.g., closed-world model, P != NP.
    • Results and Experiments.
      - Don’t underestimate the amount of time experiments require. Draw out all of the plots you want (sketch the curve you expect to see) on a whiteboard and a notebook BEFORE running any experiments.
      - If you “beat” a competitor’s algorithm and/or system, it is common courtesy to send them the results at least 1 week before submission.
      - NEVER CUT A Y-AXIS OF A PLOT--use a table if it is not informative
      - Scatter plots and CDFs should be used in the rarest circumstances since they are hard to interpret
      - Make the results section self-contained, some reviewers will read back-to-front
    • ==========================

      Jeff Tips:

      ==========================
    • Rough timeline that works for me:
      1. Read related work and write short summaries
      2. Plan experiments
      3. Build system to test
      4. Run at least some preliminary experiments to determine whether or not to proceed further
      5. Write paper outline (including where to place key figures) and decide on a story
      6. Get feedback
      7. Run at least one key experiment to support your intended claims
      8. Write detailed related work
      9. Revise outline as necessary from RW and experiments
      10. Draft figures (even if you don’t include them all it’s a great exercise in explaining as concisely as possible)
      11. Draft the whole paper (possibly minus final experiments and conclusion)
      12. Finish follow up experiments
      13. Iteratively polish experiments and paper with coauthors
    • How to decide what experiments to run:
      Based on your idea, come up with a hypothesis. Like for Dex-Net it was: “The number of iterations needed for robust grasp planning will decrease with increasing numbers of prior 3D models labeled with grasps and quality metrics.” This hypothesis should be directly linked to the story of your paper. Design and run an experiment to test this hypothesis. This should be the be primary experiment of your paper.
    • Key follow up experiments:
      • Sensitivity - test algorithm robustness to hyperparameters (e.g. number of iterations, noise in input, noise in output, any other parameters specific to the algorithm)
      • Runtime - characterize convergence, computational complexity if applicable
      • Application - show the result of applying your method to a particular application (e.g. implement on a physical robot for box packing)
    • ==========================

      Tips for after the paper is Accepted and preparing the final version!

      ==========================
    • Reviews are solid gold -- Take them seriously!
      Put the reviews into a shared Google Doc (with numbers for each point, Reviewer 1.a, 1.b, etc) and enter your initial responses, share with co-authors and think carefully about how to revise in response to every point raised.
    • Read the paper with a fresh eye and revise accordingly
    • Search for papers to cite that were published since you submitted it.
    • Consider adding new experiments.
    • Update figures and captions.
    • Check with Prof. Goldberg about who to cite in the Acknowledgements section.
    • Final submission is VERY Tricky!
    • The first step is to register for the conference and pay the overlength charges for each page over 6 (our papers are typically 8 so you have to specify 2 pages overlength when you register and the fees (typically $400) will be included in your registration fee and in your receipt you’ll get a CODE that is needed to upload your final paper.
    • We will reimburse you for the registration and travel after the conference, but alas, I can’t pay the overlength fee for you in advance (long story, ask me and I’ll explain)..
    • You also need to create a one-page powerpoint summary slide that goes into the conference booklet, and create a copyright transfer file...please budget time.
    • ==========================

      From ICRA 2017: Step-By-Step Guidelines to Submit Your Final Paper

      ==========================
    • Step 1: Prepare your final paper using the available LaTeX or Microsoft Word templates and following these paper preparation instructions. Make sure to have the latest templates and do not change the formatting in any way. If your paper is accompanied by a video, an explicit reference to the accompanying video should be present in the text of the paper. For RA-L papers with ICRA option: Please note that the final version for ICRA 2017 must be reformatted according to the aforementioned conference template.
    • Step 2: Check that the names of all authors and their order is identical in the PaperPlaza submission system, the paper, the summary slide, and the accompanying video (if applicable). Please write author names with only the first letter capitalized, that is, use “Firstname Lastname” in all instances and do not use “FIRSTNAME LASTNAME” anywhere.
    • Step 3: Create a PDF version of your paper and perform the IEEE PDF compliance test. If your paper does not pass the test, you cannot continue and may read the PDF Test Help to revise the format of your paper until it passes the test.
    • Step 4: Please prepare one PowerPoint slide (and only one) that summarizes your paper. You must only use the PowerPoint template available here. Please do not change the template and keep fonts and font sizes as prepared in the template.
    • Step 5: If your paper is accompanied by a short video (max. 10MB), please do not use special codecs (coders/decoders) in order to provide as much portability across platforms as possible. Using MPEG-1 or MPEG-4 video formats is strongly recommended. Accepted video contributions will be included in the conference proceedings and also in IEEE Xplore. There, the video will appear as an enhanced object next to the PDF of the summarizing text (a sort of short paper). If your paper features a video attachment, please prepare the following two plain text files:
      1. 1. Summary.txt file (max 1 KB). Describe in five sentences or less the contents or value of the video. This file helps IEEE Xplore users to make downloading decisions.
      2. ReadMe.txt file (max 10 KB). This file must include the following sections:
        1. Description: An overall description of the objects and what the audience can expect to gain by downloading them;
        2. Player Information: Provide the minimum version of the player software that is required to play the submitted files. Include the name of the software, the version number, and any special requirements for the player;
        3. Contact Information: The author should provide contact information in case users have questions regarding the extended material. IEEE will not provide any technical support.
      Note that both of the above files should be in plain text format and have a .txt extension. They are needed by IEEE in order to include the video in IEEE Xplore.
    • Step 6: Update the final submission information (title, author list, number of pages, abstract, copyright) in PaperPlaza. To process the copyright transfer for each contribution, follow the “Copyright Transfer” link, and you will be taken to the IEEE copyright transfer site (IEEE eCP) in a new window. Follow the steps as described on IEEE eCP pages and make sure that the paper title and author list are correct before your perform the copyright transfer.
    • Step 7: Register for the conference, indicating the paper number and any extra pages, through the author registration site. For RA-L papers with ICRA option: Please note that the extra page fee has to be paid only once, i.e. it is waived for the ICRA 2017 conference if the paper is accepted for publication in RA-L.
    • Step 8: Upload the following files in PaperPlaza
      1. A PDF file of your paper (max 6 MB) that has passed the IEEE Xplore compliance test,
      2. A single summary PowerPoint slide (max 1 MB) based on this template
      3. If applicable, a video (max 10 MB) plus a Summary.txt and ReadMe.txt file accompanying your paper (cf. Step 5).
    • Ken Goldberg, Oct 2004
      goldberg [at] berkeley [dot] edu