Observing the Ocean and Broadly
November 17, 2009 in BI Blog by Jim Yoder
My name is Jim Yoder and I oversee the academic programs at WHOI. I am currently a PI with COSEE-NOW and am also a researcher, although not as active as in the past. I worked at NASA Headquarters for a couple of years as a Program Manger in the 1980s, and again in the 1990s, and also at NSF as Director of the Division of Ocean Sciences from 2001-2004. At NSF, I was involved in the development of the OOI initiative and was also involved in many discussions on the broader impact criteria and how it was to be evaluated.
The purpose of this blog is simple. We would like to engage the ocean science community and other interested parties in a discussion as to how best to use ocean observatories and ocean observing initiatives to serve public outreach and education, as well as other “broader impacts”. Our hope is that this blog will be helpful to those of you planning to submit NSF and other proposals to federal agencies related to the use of ocean observing assets. Specifically, we hope that the discussion will help you develop better ideas for broader impact activities related to ocean observing and thus for more competitive NSF and other proposals. Although our focus will be on broader impacts in the context of ocean observing activities, we would like to start with a broader discussion on broader impacts.
First, some background. The National Science Board, the governing body of NSF, approved the use of two merit review criteria in March 1997, and each subsequent issuance of NSF’s Grant Proposal Guide (GPG) since 1997 has continued to strengthen the importance of broader impacts in the preparation and review of proposals submitted to NSF. Thus, the broader impact criteria is now more than 10 years old and well established in the NSF system. The NSF webpage lists a document that provides examples of broader impact activities (http://www.nsf.gov/pubs/gpg/broaderimpacts.pdf). A recent article in the ASLO newsletter (June, 2009, Vol 18, no. 2, openly available at www.aslo.org) co-authored by 3 NSF Program Managers from the Division of Ocean Sciences, Polar Programs and Environmental Biology provides some practical advice. In the concluding paragraph, this article states:
“The second major criterion is the project’s broader impacts. NSF is very serious about the broader impacts of a study ….. , but this criterion rarely supersedes intellectual merit. There are several types of broader impacts, but no expectation that a single proposal should cover all of them. Chose the one(s) that best fits your research and for which you can make a convincing case. …. Don’t rest on past accomplishments; we evaluate the broader impacts of the current proposal.”
This sounds like good advice, but what do you think? What are your experiences as well as your opinions on the broader impact criteria?
By the way, Janice McDonnell will share this part of the blogosphere with me, and we’ve also asked some former NSF Division of Ocean Science Program Managers to engage in our discussion.
Hi, Jim:
Glad to see someone interested in the broader impacts criterion (BIC)! Since it has been my major area of research for the past 5 years, I thought you and your group might be interested in some of my results: http://www.csid.unt.edu/topics/NSFbroaderimpactscriterion.html.
I’d love to hear feedback.
Britt
Hi Britt: Thank you for your comments and for your rearch. I am familar with your research and am intrigued by some of your findings. In one of your articles you quote a a recent article in American Physical Society
News that suggests that members of this scientific community are giving BIC
“mixed reviews,” noting that, while some support BIC, many view the criterion as “confusing, burdensome, inappropriate, or counterproductive,”. Our survey of ocean scientists so far (http://coseenow.net/2008/12/2008-scientist-survey/) does not paint the same picture. Although most surveyed, note the lack of time and resources to engage in BI work, they do not seem to hold resentment or question their involvement in education and public outreach. I hope we can discuss in greater detail your findings and compare notes to our ocean science community.
Look forward to more discussion,
Janice McDonnell
Hi, Janice:
Sorry about the delay in replying — I’m just figuring this out, and since I get informed via email about the blog posts, I didn’t realize I wouldn’t be informed of responses to comments. (Or maybe I was informed and just missed it. That’s very possible!)
In any case, thanks for turning me on to your survey! This looks really cool. Of course not all scientists are opposed to broader impacts activities, and it’s nice to have more evidence of that fact. What was so surprising to me about the APS News article was that folks were having such a negative reaction even ten years after the introduction of BIC. This suggests the possibility that there may be a gap in the education of STEM graduate students about BI activities, as well as a need for more interdisciplinary interactions between STEM folks and BI folks. Of course, the lack of time and resources even for those in favor of engaging in broader impacts activities might also support the idea of more interdisciplinary teamwork.
I’m sure I’ll have lots more questions once I take a thorough look at the report, but a few occur immediately:
Are you planning to conduct the survey again?
Do you know of other such surveys (e.g., in different disciplines, conducted by other groups)? One thing about which I’m quite curious is the extent to which differing attitudes toward BIC are discipline-dependent ….
As part of research my colleagues and I are conducting now, we are looking at both NSF and NOAA (among several other agencies) to find out about different groups’ attitudes toward the inclusion of societal impacts considerations in the peer review process. We’re also developing a survey of our own, so it will be very interesting to compare results!
Britt
HI Britt- I think there is a way to get a rss feed that will notify you when a comment has been posted. I will ask Sage on our team to post a comment on how one does that! In the meantime, thank you for your comment.
We are in fact planning to update the survey next month. We will be launching it to two different ocean professional organizations. We definitely want to compare results.
In addition, we are in the process of setting up case studies at each of our centers to supplement the survey data with more rich examples from different types of scientists (career stage, level of engagement in EPO, etc). This work will be starting in the next few weeks and you will be hearing about it on this blog site.
I completely agree that graduate students are perhaps the most important target audience for discussion and education about BIC. We also are involved in a project lead by Annette de Charon from University of Maine where we will be developing tools for getting graduate students engaged concept mapping and thinking about how to use concept mapping to improve the transfer of knowledge to public audiences. You can view concept mapping tools at http://cosee.umaine.edu/.
One of our biggest challenges in offering support to scientists around BIC writing is the lack of consistency in how BIC are reviewed and considered by NSF. It is almost completely up to the program manager whether they include BIC in the panel review process or not. We have heard stories that range from not considered, discussed at all in panel at all to bringing in someone specifically on the panel to serve as th BIC “expert”. Do you find the same lack of consistency in your work? We look forward to hearing more about your work. Please keep us informed!
Janice
Re: BIC
I thought you folks would be interested in our non-profit program, The Zephyr Education Foundation. We are located in Woods Hole. Our purpose is to enhance marine science aawareness and literacy by providing hands-on opportunities for students grades 5-16. Our program is designed to be a vehicle through which PIs at WHOI, MBL (and other institutions) can use for BIC and reach several hundred students per year. We go out on a collecting trip in Vineyard Sound (benthic collections and plankton tows) and also demonstrate oceanographic measurements. Other activities include a tour of MBL’s Marine Resources Center,a guided tour of a salt marsh and/or beach and occasionally the core lab or WHOI dock. We intend to add subjects such as OOI, climate change, ocean acidification and whatever other PI’s want to contribute. We hosted over 200 students in 2009. Our educator and I can meet with PI’s to design a short ‘unit’ on their project and present it as part of our program, with the intent that this will be viewed as an effective way to deliver BIC and public outreach and strengthen the grant proposal.
We have met with WHOI and MBL administrations and have their (non-monetary) support.
I’d love to hear your thoughts. Thanks
Rob Reynolds
Hi, Janice:
Great! I think we’ll have lots to discuss.
Yes — lack of consistency in how BIC is reviewed is a recurring issue, at least according to anecdotal evidence. We’ll have more hard data on this once we complete our current research. Interestingly, although we have often suggested the possibility of using BIC “experts” — the scare quotes you use here are essential, since the very idea of expertise in broader impacts is contested, even if it is clearly possible to identify experts in one or another area related to broader impacts — as reviewers, at least in an ad hoc fashion, we have consistently encountered resistance to the idea.
So, I’d love to hear from you and anyone else who knows of anyone who has been asked to serve as a reviewer for NSF specifically to address broader impacts.
In my experience, however, people (including program officers, proposers, reviewers, and just about anyone else you can think of) and agencies tend to think about peers along disciplinary, or even subdisciplinary, lines. (There are some interesting exceptions to this that we’ve found, however. More on that if anyone’s interested.)
My own opinion is that BIC considerations ought to be incorporated starting with the proposal writing stage, included during the review process, be integrated with the actual research, and continue well past the dissemination of the research findings (e.g., it would benefit everyone if we had long-term data on outcomes, as well as a set of direct research outputs). There are, of course, institutional barriers to such an approach; but there are also some institutional enablers, such as BIC itself. There are issues of what one might call “compliance” with BIC — including inconsistencies on the part of some POs and reviewers, as well as proposers — but there are also issues how the merit review process is designed (for instance, the issue of whether to weight broader impacts more, less than, or equally with intellectual merits, or not to specify a weighting) that affect “compliance.” So, it’s not just a matter of how the process is implemented; and it’s not only a matter of how the process is designed; it’s some combination of both of these, plus lots of other factors (such as the individuals involved, etc.). These are just the sorts of issues we are trying to draw out in our research.
Thanks, Janice and Jim for getting this started. I hope others out there are interested in carrying on what I think is both an interesting and a vital conversation!
Britt
Hi, Rob Reynolds:
I get the idea — and think it’s a good one — of partnering the Zephyr Education Foundation with scientists who are seeking NSF funding as a way to help those scientists enrich the BIC aspects of their proposals. A couple of thoughts occur immediately:
(1) Do you have a plan to make sure that the educational expertise you provide is incorporated into the research being proposed/conducted by the scientists, as opposed to being a mere add-on element?
(2) Do you mention BIC on your site at all? I glanced around quickly and didn’t see any explicit discussion of broader impacts.
(3) Do you have, for lack of a better term, a “marketing plan” to let scientists know about your organization?
Britt
Hi Britt – You raise the essential questions to an honest evaluation of the impact BIC can and could have in promoting effective communication of research science to public audiences. The point of this blog is to discuss and share information on these kinds of questions.
I will invite my colleagues who I know have stories about panelists being invited to serve based on their expertise in education and public outreach. The CAREER proposal solicitation jumps to mind as one where this has happened. JIm Yoder also has contacted some NSF program managers to weigh in on the discussion and offer their thoughts on the points you raise.
More soon
Rob – As a self proclaimed broker here at Rutgers, I seek out organizations such as yours as an outlet and effective partner to fulfil their broader impact requirements. The Ocean Institute (http://www.ocean-institute.org/about/index.html) is a wonderful facility and the only one that I know of that promotes their partnerships with scientists and their role as “translators”. I am sure there are others? I welcome others to post their thoughts on the questions Britt raises to Rob. Thanks for sharing this information Rob!
Janice
Janice and Britt-Thanks for your interst and comments, all good. Responding to Britt points:
1. I see it more that the research must be incorporated into our program, rather than the other way around (maybe I missed your point though). Our staff meets with each PI to design a ‘unit’ based on their research, that we tailor to each age group. The intent is to observe and measure, as much as feasible, as the PI does, while also introducing the relevance of the work to the students. In order to extend the effect of the program, each class can post their data on our website. This will allow comparison throughout the school year (and for subsequent classes) with data collected by other classes (different times of year, conditions, etc.) We also can provide study guides to take back to the classroom.
2. The website is developing, thanks for your comment. The original intent of the site was to be a source of information to attract classes. In order to attract PIs, I am contacting them rather piecemeal at WHOI and MBL (we have partnership agreements with them). I’ve assumed we can cover more in a face to face meeting than describe it on the site. We should expand the site to make it more obvious to PIs what we can provide to them.
3. At the moment, the marketing plan consists of internal communication from the administrations at WHOI and MBL to department heads, grant offices and individual PIs. I realize we have to be more aggressive on this, and would welcome your ideas. Additionally, there is no reason this can’t be done with PIs not located in Woods Hole, and I encourage this kind of outreach. Again, any suggestions are welcome.
Outside funding is still necessary, so maybe being a partner in a CAREER proposal would be a good idea, for example. I am looking for other sources of funding.
FYI, today we received strong interest for participation from four more schools.
Rob