bocklund.io

Why Jupyter Notebooks Won't Replace Academic Papers

2018-07-29T00:00:00-04:00

Recently there has been some buzz around Jupyter Notebooks in science, especially in light of the LIGO team sharing their detection and analysis of gravitational waves in Jupyter Notebooks. Others have claimed that Jupyter Notebooks will render traditional academic journal articles obsolete. The notebook or literate programming format improves on the reproducibility and disseminating of scientific work, however several key factors limit the notebook as a way to communicate science. In the article, I'll touch on these issues and explain why I believe that the current model of sharing computational science is here to stay.

Note: throughout this article I might refer to Jupyter Notebooks as notebooks, or the method as literate programming. I'm talking about the same thing, despite the second two terms being more generic.

Why Jupyter Notebooks Are So Promising

There are completely valid reasons to want to improve on the current publishing situation and Jupyter Notebooks are capable or have solved some of these problems.

Notebooks are open. Under the hood, notebooks are plain text formatted as JSON. They can technically be run or rendered by anyone who can reproduce an appropriate environment. There's no vendor lock-in. JOSS, the Journal of Open Source Software is a completely open publisher for scientific software that could support dissemination by Jupyter Notebook.
Arfon Smith, Editor-In-Chief of JOSS said on Talk Python to Me #157 shared a common sentiment that journal articles can sometimes be more like advertisements for the research. As a consumer of knowledge and software tools, literate programming allows me to not only run and reproduce the scientific result, but to use software as an application. Reproducibility is such a well known issue and for results generated by computer programs, there's no excuse to not share the exact recipe to reproduce a result.
As a producer of knowledge and tools, I can provide a resource so people can use (and cite!) my work. I can use feedback and create a community around tools and techniques to make them better.
Notebooks can provide more features to papers, such as interactivity with visualizations or datasets, sorting tables, or extracting data.

Why Notebooks Won't Take Over Yet

I want to avoid any discussion of compatibility, trendiness, environment or the technical choices of Python/Markdown, institutional publication models, and so on. These have been covered elsewhere. Some of these are valid arguments and others are part of the inertia that fights change. I think there are more fundamental problems with the literate programming as science model than these issues. If anything, the scientific community has proven its willingness to hang its hat on what some might consider to be legacy software: new FORTRAN code is still be written today.

Technical Issues

The main issue is that there's no balance to hide complexity or abstraction. Scientific methods can be complex and to "stand on the shoulders of giants" implies that we build knowledge and tools on top of each other into more and more advanced approaches. The notebook format is limiting for this. Scientific software should be packaged and not flattened into a notebook script. See most of the JOSS papers for examples of good software. Many of these projects contain notebooks to show how to run or use the software, but few of them have the software in the notebook, even though that's the key part we're supposed to be sharing.

There are many best practices lists for writing scientific software. The Scientific Cookiecutter project and the Data Science Cookiecutter project both give good ones. Some highlights are keeping I/O separate, writing for readability, functional programming approaches, and managing data separate from code. All of these conflict with the notebook model, which promises convenience of putting everything in one place. It's hard to write flexible, sustainable software in a notebook which encourages lumping everything together in what boils down to a script with inline outputs.

In addition, notebooks are challenging to maintain and iterate on. If I want to test a different idea, then I am copying cells, changing things and running them over and over. To share the notebook, I have to restart the kernel several times and make cleanup passes to make sure that everything is consistent. A scientist who comes along later might want to integrate their changes with my own, but they are suddenly stuck because everything has changed in my test and cleanup loop. This is a problem if we are expecting collaborators and derivative works to improve on the approach a publish the results, especially because usage and citations are key metrics for many academics.

A lot of software uses data. If not to gain insight, then at least as a comparison to validate a model or approach. How should this data be controlled? Where should the schema be described and validated? How can users supply their own data or extend the format schema?

Cultural Issues

Adam Rule discussed his work on seeing how Jupyter Notebooks are actually on Talk Python to Me #171. They analyzed a dataset of over 1 million notebooks and found several interesting conclusions that are discussed in their paper and on the podcast.

They found that people often used the notebooks for exploration and iteration, leaving behind messy notebooks without an overarching narrative. There's also no real guidelines or ideas of what the best practices are. What should a publication quality notebook look like? Once we have exposed all of the underlying steps in the notebook (distraction aside, see above), how do we strike a balance between letting data and methods speak for themselves and clarifying what part of a work is new knowledge? Academics with PhDs have arguably at least 10 years of training in writing more scientifically. As pointed out by Rule *et al.#, there's hardly formal training to use a Jupyter Notebook to the same standard of training to use a lab notebook.

There's also cultural inertia to what is considered scientific contribution. Rule mentioned in the podcast and the paper that PIs and group members alike see notebooks as half-baked, preferring slide decks and results over code. Much of scientific community largely seems to agree that the currency for sharing ideas in talks and at conferences is a polished slide deck. Even among the community at JOSS, I have seen that reviewers are much more critical of the software design and utility, even to go so far as requesting new features before recommending work for acceptance. Detailed vetting can lead to better reproducibility than the current status quo, but clearly a balance should be struck. Time will tell how efforts like JOSS or growing conferences like SciPy (see my favorite talks from 2017 and 2018) will shape the scientific community's view on software contributions, citations, and meaningful work.

In my experience, some people just don't like working in the open. Academia is competitive in jobs and funding. Not everyone wants to share. People seem to be getting better at publishing their software or data at the same time as their paper, but I have gotten pushback when suggesting that software and data be open from the start as we are doing with pycalphad and ESPEI.

Conclusion

Notebooks are a promising paradigm to share computational science, however there are some large and small roadblocks to overcome before they can be considered a replacement for current academic papers.

I want to mention that Jupyter Notebooks are a key part of my own daily workflow as a PhD student working in a computational science and as a maintainer of Python packages. Sketching ideas in a notebook that I can factor into a package is a huge productivity boost for me. I fully endorse the use of literate programming tools for doing science, but I don't think they are the right medium for sharing knowledge.

I am in complete support of the Jupyter team and highly suggest using Jupyter Lab, which has been ready for daily use since February 2018.

SciPy 2018 Talk Highlights

2018-07-29T00:00:00-04:00

Last year I gave a list of my favorite SciPy 2017 talks. The SciPy 2018 conference took place from July 9 to 15 and the talks and tutorials are now in a YouTube playlist created by Enthought. I have gone through all of this years talks and watched through any that seemed interesting. Read on for my suggestions!

Overall, it felt like there were a lot of machine learning and geoscience/geo-related talks and packages. A lot of the ML talks seemed to be missing practical insight into applying a specific method or seeing how someone solved a problem in a way that generalizes to other problems. The geoscience ones weren't very interesting to me, but might be worth checking out if you're interested.

Keynote: Democratizing Data - Tracy Teal

Tracy's keynote gave a great perspective on how we see and value data in science. It was fascinating to me that she showed results from a survey of Bioinformatics Resource Australia that training and community building were more requested than funding!

Recipe2Vec: How word2vec Helped us Discover Related Tasty Recipes - Meghan Heintz

Your user knows they want a healthyish but tasty pasta for dinner but aren't quite sure exactly which recipe to choose. How can you help narrow their search and show them closely related recipes to give them enough options without making their search exhausting? This talk will show you BuzzFeed/Tasty tech's solution to creating a consistent method for finding similar Tasty recipes using word2vec.

This talk gave an application of machine learning in industry and gave a great example of how ML was used to add value to a product.

Data Visualization for Scientific Discovery - Zan Armstrong

Choosing the visual form for a visualization is a decision about what aspects of the data matter most. Highlight or ignore outliers? Look at values, differences, or changes? Compare to 0, median, or mean?

In scientific analysis we risk missing discoveries by failing to notice important features of our data, yet we often use default parameters and charts without realizing what we might miss.

I will demonstrate how to translate questions about your data into chart parameters. Using Python examples, I'll illustrate powerful techniques like using color intentionally and creating 'small multiples' of charts that vary visual form or data.

This talk had a lot of really great practical tips for producing good visualizations for yourself and for others.

SciPy Tools Plenary Session 1, Session 2, and Session 3

Session 1

Session 2

Session 3

These were a series of 5 minute lightning talks from core packages in the SciPy ecosystem. They are definitely worth a watch to get up to speed in all that's happening in the ecosystem.

Filling ILLiad forms

2017-08-20T00:00:00-04:00

You may have seen my other post on quickly getting papers remotely through your own library with JavaScript. This method works well if your paper is easy to find and you just need to get around the paywall using the access your institution pays for. Sometimes, especially with older papers, you may be able to find a reference, but not a paper. Libraries can offer great utilities to supporting finding original research articles, but they can sometimes be clunky and slow to use. This post will give an example of using JavaScript to file requests on ILLiad, an article requesting service from OCLC used at Penn State, from only a bibtex reference to the article.

Too long, didn't read

See my JavaScript Gist with the code and/or bookmark this Fill ILLiad link. Run the script or bookmarklet on an ILLiad request page (possible PSU link) and fill the prompt with the BibTeX entry of the reference you'd like to fill.

Motivation

In thermodynamic modeling, it is desirable to have original references to experimental studies in order to properly interpret the results and prevent propagating mistakes. For many alloy systems, especially binaries, many of the original articles were written between (roughly) 1900 and 1950. Despite these being indispensable for modeling work, digital versions of the paper can be challenging to track down.

ILLiad is an InterLibrary Loan software service from OCLC offered by many institutions including Penn State, allows research articles (and book chapters) that are not accessible through the usual library resources. A typical ILLiad request page is below. The form serves its purpose well for requesting single papers, but is tedious for beginning literature searches for many articles that must be requested. Even for well formatted references, it is tedious to copy and paste seven or more fields in to request just one paper.

It would naturally be ideal to try to automate this. For serious automation of on the order of a hundred or more citations, it would be efficient to a script in Python to make submissions automatically. Since I'm not sure what type of systems other institutions use, I thought it might be helpful to write something more general in JavaScript that could be adapted to any kind of webform.

Getting our bearings

We need three pieces to be able to fill the above form (or any like it) automatically:

A standard citation format that will easily fill the forms fields
A way to convert the standard citation into a useful form in JavaScript
A place to fill our nicely formatted data

We'll stitch these together in a useful, portable bookmarklet, just like in my getting papers remotely post.

Reference format

More than any citation format, BibTeX files are the digital common denominator. Any reference management software worth using should be able to read and export to BibTeX. The fact that it can be stored and used as plain text makes it a natural choice to use for any digital reference management.

We will use BibTeX as input to our JavaScript bookmarklet. For Mendeley users, that will mean finding your reference, right-clicking and selecting BibTeX entry from the Copy As menu. See the reference software you use for instructions.

In our script, we'll just use a simple JavaScript prompt to prompt for user input.

Conversion

Another benefit of choosing an open format like BibTeX is that we should not have to worry about the syntax ourselves. I choose to use @mikolalysenko's parser and load it directly in the browser by using this accepted StackOverflow answer and rawgit. The loadScript function in the Gist is directly from that SO answer.

This parser will convert BibTeX entries into a JSON where each top-level key is the BibTeX citation key and the values are all the BibTeX fields, such as AUTHOR, TITLE, etc. We can assume that a user will only enter just one entry (and choose the last one if they enter more than one) just by using a simple loop through the keys and filling the form fields. See the replaceValues function to see the looping in action.

Though before we do this, we need to know where to fill these values.

Filling

Finally, we need targets to fill our data. All modern browsers have build in development tools. Here we've used Chrome to drill down through the elements and find the relevant textarea fields. You can see that the highlighted textarea has the id PhotoJournalTitle. From the inspection tree on the right, it appears that the other fields follow a similar format, e.g. PhotoJournalInclusivePages.

We can use JavaScript's getElementById function to grab each of these elements and let us change the values. For brevity and lack of creativity, I've hardcoded these in the Gist as the replacementDict. The replacementDict simply maps the name of the element id to the name of the key in the BibTeX JSON. If you don't use the same ILLiad system, you can inspect the page and you should only need to update the replacementDict.

Making it a bookmarklet

This example is only slightly more complicated than the getting papers remotely because it's not obvious how to squeeze down the content of the Gist to one line. Tools like Peter Coles' bookmarklet creator take a JavaScript script and condense it down to oneline and put it in a function to give things the correct scope. Though, notable this mangles the input with percent encodings.

See it in action

The input:

And the resulting filled form:

Note that the reference you entered is nicely put in the Notes section, in case your reference manager tries to pull the wool over your eyes when you import your ancient PDF.

Again, the link to the Gist and the Fill ILLiad bookmarklet.

Happy (semi-)automating!

Atomate - reproducible computational materials science workflows in Python

2017-08-05T00:00:00-04:00

If you've done any computational work on a high-performance computing system, odds are you are very familiar with shell scripting and running code in at least 3 different languages that was passed down through the years. You know the theory of how it all works, but can never find the time to weed through, organize and update someone else's FORTRAN code (that was out of date when it was written in 2002) to use the latest techniques. Atomate aims to solve these problems by leveraging the community developed materials science tools and analyses in pymatgen and the workflow management/database creation of FireWorks to document and allow for easy execution of community standardized and easily customizable workflows managed and written using Python.

Link to the paper: Mathew, K. et al. Atomate: A high-level interface to generate, execute, and analyze computational materials science workflows. Comput. Mater. Sci. 139, 140–152 (2017).

My other publications

SciPy 2017 Talk Highlights

2017-07-17T00:00:00-04:00

The full SciPy 2017 conference took place from July 10 to 16. The talks and tutorials are now live in a YouTube playlist created by Enthought. I have watched most of the available talks that seemed interesting from a materials science perspective, specifically talks geared towards building scientific packages and computational tools like pycalphad. Read on to see the four talks I found most interesting

Keynote - Coding for Science and Innovation

Computing has been driving forward a revolution in how science and technology can solve new problems. Python has grown to be a central player in this game, from computational physics to data science. I would like to explore some lessons learned doing science with Python as well as doing Python libraries for science. What are the ingredients that the scientists need? What technical and project-management choices drove the success of projects I've been involved with? How do these demands and offers shape our ecosystem?

Gael offered some interesting insights into how we should be developing packages and APIs for science that people actually want to use and are able to use. One takeaway for me was on slide 34, where Gael suggests to use example-driven development. Example-driven development is about writing codes and APIs that solve problems that people want to do. In my mind, this is a guiding principle behind pycalphad development. There are so many things to do in the CALPHAD space that the best path forward is to enable things that are hard or impossible with other software tools and iterate on those features to make them intuitive and easy to use.

Dask - Advanced Techniques

Dask enables parallel computing in Python. While commonly used for its parallel and NumPy, Pandas implementations, Dask is also capable of a variety of more advanced parallel computing workflows. This talk dives into these advanced features features and applications beyond the typical distributed dataframe to talk about asynchronicity, dynamic and self-building computations, multi-user workflows, and more.

This covered some of the advanced usage that we us in pycalphad and ESPEI with the distributed scheduler and Client. Note: you should be familiar with Dask or watch an introductory talk (from PyCon 2017) first.

Dash - A New Framework for Building User Interfaces for Technical Computing

If you are a data scientist today, it's actually pretty tough to build a data visualization web-application. If you're not a full-stack developer, you're practically out of luck.

But GUIs like sliders, dropdowns, and text inputs are extremely helpful to the data scientist or engineer. If you're an R programmer, you're in luck with Shiny. If you're a MATLAB programmer, you can use GUIDE (but good luck sharing it!). The dash project introduces a framework for building web-based technical computing apps (GUIs). It's like a Shiny for Python. dash is built off of plotly.js and react.js to provide rich interactive graphing and user interfaces and Python's flask to provide a simple but scalable web server.

This talk will introduce the scientific community to Dash. We'll go over motivations behind the project, the basic architecture of the framework, several interactive examples, and leave with a vision for the future of interactive and sharable technical computing.

Dash is a newly released package from the folks at Plot.ly. I like the idea of Dash because of its potential to generate interactive GUIs with visualizations in the browser. Easily, the best and most interesting part is that there is tooling to use JavaScript plugins so that all of the work building browser tools in that space can be reused.

Introducing JOSS The Journal of Open Source Software

JOSS is "A developer friendly journal for research software packages."

The Journal of Open Source Software (JOSS) is operated entirely in the open and offers an alternative for publishing software to the traditional 'write a paper with results' that also includes a description of the software. JOSS is mostly operated on GitHub and is transparent about the cost of publishing (free for authors). They have achieved several milestones since starting in May 2016 and Kyle Niemeyer discusses those milestones and the future of JOSS in this talk.

Bonus: I haven't watched any of the tutorials, but the Cython tutorial seemed interesting and new this year. If you're still looking for more talks, PyCon 2017 had a ton of interesting content. Specifically this 4d topology visualization by David Dumas and Jake VanderPlas's talk on the Python visualization landscape.

Getting Papers Remotely

2017-07-06T00:00:00-04:00

Whether you are at home or traveling to a conference, it can be helpful to have access to all of the literature online that you would have at your institution. This article covers some easy and practical ways to connect to your institution's library easily. Some of these techniques work on any website, and others are for Google Scholar, or with the Alfred app for the Mac. The examples here are specifically for connecting to the Penn State library, but the methods are general enough to work at many or all institutions.

This might seem trivial because it is. The time investment to figure out these things and write them up took much more time than going through the PSU library website for any article I was looking up at the time. Luckily there's an xkcd comic to justify my work (and your short investment to implement these!):

I'd say that it usually takes me a minute or so per paper to track down the link when I'm away from home. Usually I look things up in bursts, but maybe I could average it out to once per day. So over 5 years (roughly the length of a PhD) you give yourself an entire 24 hour day of reading papers instead of acquiring them.

Setting up the Get-It! button in Google Scholar

If you are on a university's campus network, you probably have seen the Get-It! @ University button around the web, including on Google Scholar. The Get-It! button will immediately take you to that university's page for the article. Clicking this button is often unnecessary on campus because you are often redirected automatically, but it is helpful to have this button while off campus. To set this up on Google Scholar, first visit the Google Scholar Settings page. Navigate to the Library Links setting and search for PSU and check the box for Pennsylvania State University. Now any Google scholar searches should show the Get-It! button, which will send you to the PSU authenticated journal article pages.

Bookmarklet for the library version of the article

When doing literature searches on any website you will likely end up on a web page for the paper you want, on ScienceDirect for example. Typically, if you are away from your institution, you can access the institution specific versions of any article through a library website. Penn State's library website is libraries.psu.edu. If you've already landed on the journal's website, it can be tedious to copy over the information to your library's website and click through to get to the article. If you did click through, you might notice that you get the same page from the library website as you had originally visited, but there the URL is slightly modified. At Penn State, the service changes the URL so that there is an .ezaccess.libraries.psu.edu after the .com, but before the rest of the URL to the specific page.

To get to the PSU libraries page in one click, we're going to use a JavaScript bookmarklet. Bookmarklets are supported in most browsers and are operate by executing a simple piece of JavaScript code. If you use Mendeley's web importer bookmarklet, it also uses JavaScript. Below are the steps to create the bookmarklet yourself, but you can also bookmark this link instead: PSUezaccess

This is only a very simple example of JavaScript in a bookmarklet, but the JavaScript can be complex. Some browsers make it more challenging to create bookmarklets by hand. If you have trouble, try creating the bookmarklet in another browser.

In your browser, create a new bookmark
In the URL for that bookmark, enter the following code:

javascript:window.location = window.location.href.replace(/\.(com|org)/, ".$1.ezaccess.libraries.psu.edu");

The first part, javascript: means that what follows will be JavaScript code, rather than a normal http URL. Then we are setting the new URL of the current page (window.location) to the old location (window.location.href), except that we are replacing either .com or .org with whichever one was matched plus ezaccess.libraries.psu.edu. So when this code is run, it would replace https://sciencedirect.com/stuff/to/page with https://sciencedirect.com.ezaccess.psu.edu/stuff/to/page. You can make the description/name of the bookmarklet whatever you want, I named mine PSUezaccess.

In the replace function, the first argument is a regular expression. You can test out different matching patterns several places online. I've found that most journal websites are either .com or .org websites, but if you find others, e.g. .net, you could simply add that to the regex in your bookmarklet and it will just work. The second part uses $1 to take the first match, so either .com or .org in this example. This allows us to easily use any arbitrary top level domain.

Easy searching with Alfred

Alfred is a productivity tool for Mac that is similar to the recent iterations of Spotlight in macOS, but much more powerful. There are many things that Alfred does and can be customized to do with Alfred Workflows, but most often I use it to open files, applications, and as a calculator. An interesting aspect of Alfred for getting papers is to set up DOI searches and Google Scholar searches.

Often I find that either from a paper or somewhere online, I have a plain text reference for a paper I want to look up or I have the DOI, but not the link. If you go to the Alfred preferences and the web searches page you can create a Custom Search in this dialog

Above is an example for creating a DOI search if you only have the DOI, e.g. 10.5334/jors.140 as it would appear in Mendeley. You simply set up the link as you would to visit the website, but instead replace the actual DOI with {query}. Then typing doi in Alfred, you can paste in the DOI (10.5334/jors.140), hit Return and go immediately to that article. You could then use the bookmarklet method above if you are working remotely.

Setting up a Google Scholar search works the same way, except you would use https://scholar.google.com/scholar?q={query} as the search URL. I set mine up to use the scholar keyword, so I can type scholar and paste in the text citation to search Google Scholar:

scholar Otis, R. & Liu, Z.-K. pycalphad: CALPHAD-based Computational Thermodynamics in Python. J. Open Res. Softw. 5, 1 (2017).

Using Google Scholar searching and following links would work well with either the bookmarklet or the Get-It! button methods above. All of these solutions are relatively straightforward to implement and use. The Penn State specific links for Google Scholar and the bookmarklet could easily be adapted to any institution that uses similar schemes for accessing paywalled journal articles.

Open-Source Learning

2017-06-06T00:00:00-04:00

New researchers, graduate and undergraduate students, spend much of their first 6 months to first year in their research group learning the techniques the lab uses to do their science. This is espeically true in computational fields, where there are numerous softwares (open- and closed-source) using in specific domains. Challenges associated with joining computation-focused research groups are exacerbated by the fact that many undergraduate curricula in applied research fields tend to lag behind in the use of software tools. In my experience, relatively few undergraduates in engineering consider themselves to be proficent software developers (CS, CE, etc. aside). Research groups need better ways to teach new scientists software fluency.

Most research groups have some kind of onboarding procedure for new members, which often involves an insurmountable stack of theory-filled textbooks and literature that are both too broad and too narrow to solve the introductory problems a new student should be exposed to. Eventually, these piles of literature are eschewed in favor of brute forcing the initial research tasks with heavy involvement and guidance from senior group members. This approach is wasteful for the student who may miss some key theoretical background or dwell too much on theory and be less productive, ultimately delaying the time to first publication.

The more senior researcher(s) tasked with guiding the new student as they orient themselves towards their research goals also suffer from the inefficieny of this process. Heavily guided introductions by senior group members is a cornerstone of academic culture and every early, mid, and late career scientist has been a part of this process at one time or another. Passing of knowledge down academic trees is not going away anytime soon, but it can be made more efficient.

Instead of teaching new students skills entirely case by case and in-person, each research group can develop a manual that couples theory and practice. Of course, this has been done many times over the years and is not at all a new idea and several models have been used for maintaining such documentation with varying degrees of success. In the past several years, better software development practices have become more accessible and more adopted by scientists. Scientists have readily adopted resources such as GitHub for hosting their code, but entire academic journals exist on GitHub as well as the DFT book by John Kitchin. In particular, the DFT book and the Hacking Materials group handbook by Anubhav Jain have served as inspiration to develop open guides to starting at the ground level with a new research group.

Thus, I am proposing that other research groups join this effort for opening up the onboarding process of new researchers and maintain open documentation for how to get started as a researcher and a member of the group/institution/company. I have initiated such a project for the Phases Research Lab at Penn State where we have an onboarding guide freely available on GitHub. The goals of the repository, called prl-onboard, is to guide new researchers (and possibly new materials scientists) through learning Python in context of the tools we are using in our group: pycalphad for computational thermodynamics and atomate for DFT calculations. Primarily this is accomplished by having a complete 'course' that students can progress through and alternate reading relevant material and actually using what they are reading about. Other goals are to introduce people to foundational knowledge required for success in our field and to getting settled at Penn State. Eventually, we hope that the prl-onboard repo will be useful to people outside of the Phases Research Lab.

Besides just being open for the sake of being accessible to people outside of our research group, the increased visibilty, ease of navigation and rich formatting offered by markup languages on services like GitHub can be coupled with a tight feedback loop to ensure that the project doesn't die after the person who initated the effort leaves or becomes to busy to maintain the resource.

This effort is still in its initial stages at the time of writing but I hope to address several long term goals and challenges in future posts. Some of these challenges are using closed-source software (we depend on VASP and Thermo-Calc) and textbooks for the reading material. We have access to these resources at the group scale, but scaling this beyond our group would make the time it takes to develop this material more worthwhile is currently a major challenge.