The University of Warwick’s FutureLearn MOOC Literature and Mental Health: Reading for Wellbeing, which began its first presentation February 2016, was identified as an opportunity to conduct some research into the course subject area, ‘reading for wellbeing’ or ‘bibliotherapy’. Since 2013, a substantial body of literature has emerged in the field of MOOC-related research, with the MOOC becoming both the subject of and vehicle for research. The research approach adopted in Literature and Mental Health was influenced by other, recent research studies conducted within MOOCs, and particularly by the first presentation of Monash University’s Mindfulness for Wellbeing and Peak Performance FutureLearn MOOC, which distributed a stress survey to its learners in the first and final weeks of the course, to assess the efficacy of the course’s mindfulness practices.
A number of reasons for trialling the use of this MOOC as a research tool were identified at the project’s outset. MOOCs give researchers access to large numbers of possible research participants, making MOOC research an attractive prospect, while the opportunity to gather valuable, potentially publishable data from free online courses may help to justify the time and resources expended during the production of new MOOCs. Several additional benefits of in-MOOC research were discovered during the process, including the potential for research activities to enrich the learner experience. However, a number of challenges and limitations were also encountered during the development of the study; the inevitable self-selection bias among MOOC learners, and the difficulty of establishing a control group within the MOOC activities, posed impediments to the gathering of useful, publishable data.
Although we were aware of other MOOCs which had been used as vehicles for research, the process of adapting Literature and Mental Health for this research study was nonetheless an illuminating and instructive experience. The purpose of this paper is to reflect on that experience, and to consider the lessons learned during the process which may be useful in informing future research studies conducted via Massive Open Online Courses.
The University of Warwick’s FutureLearn Massive Open Online Course (MOOC) Literature and Mental Health: Reading for Wellbeing began its first presentation in February 2016 (University of Warwick, 2016). In the early stages of its development, this MOOC had been identified as an opportunity to conduct some research into the subject area, using the MOOC participants to investigate the relationship between reading and wellbeing.
Literature and Mental Health: Reading for Wellbeing is a six-week MOOC, with each of its weeks organised around a different mental health condition; the first week concentrates on ‘Stress’, for instance, while the fifth is concerned with ‘Depression and Bipolar Disorder’. The course content is broadly divisible into two strands: the first is to explore how different mental health conditions have been written about in literature through the ages; the second is to consider the extent to which reading might have some therapeutic value in the treatment and prevention of certain mental health conditions. It was the latter of these strands which our research was designed to explore.
This paper will begin by putting the Literature and Mental Health research study into context, explaining the rationale behind the decision to conduct research via a MOOC, giving a brief overview of the history of MOOC research so far, and describing the research activities designed for Literature and Mental Health. The main body of the paper will be occupied with a discussion of the lessons learned about MOOC research during the first presentation of Literature and Mental Health, and will be organised under five headings: research participation rates; the problem of self-selection bias; balancing research and learning; impact findings; and the value of qualitative data.
The field of MOOC research is both relatively new and rapidly expanding. As such, it is important to reflect on the process of researching via a MOOC, as well as on the findings of the research conducted. We hope that our reflections on the Literature and Mental Health research project will encourage, influence and inform future research conducted via MOOCs.
The Literature and Mental Health MOOC was developed by the University of Warwick in conjunction with the organisation ReLit: The Bibliotherapy Foundation. A number of reasons for conducting a research study via this MOOC were identified at the project’s outset, some applicable to MOOCs in general, and others specific to the content of Literature and Mental Health. For any university, the prospect of generating publishable research via a MOOC may help to justify the time and resources expended during MOOC production; in the case of this particular MOOC, meanwhile, the opportunity to conduct large-scale research into the bibliotherapy practices advocated by ReLit was too valuable to be overlooked.
In 2012, Thomas L. Friedman heralded a higher education ‘revolution’ in his New York Times article about the newly developed Coursera MOOC platform (Friedman, 2012). The Massive Open Online Course, he suggested, was an innovation that would allow thousands of people to overcome the financial and geographical barriers that had previously prevented them from accessing university-level education (Friedman, 2012). These predictions of a bright and momentous future for MOOCs were inevitably followed by a backlash, with critics pointing to low completion rates and the limited demographic diversity of learners as evidence of the failure of MOOCs to revolutionise higher education (Zemsky, 2014). Yet although Friedman’s vision of an egalitarian online future for university learning has not been realised, nor can the MOOC be dismissed as an abject failure: MOOCs are, for the time being, here to stay (Friedman, 2013; Matkin, 2015).
Nonetheless, the novelty of the Massive Open Online Course for participating universities is diminishing, and the MOOC business model is under increasing scrutiny. A study in the Times Higher Education estimated that the average cost, for a university, of developing a MOOC was £29,356 in 2015, based on a survey of twenty courses from nine UK institutions (Parr, 2015). Although some universities and platform providers have found ways to profit from MOOCs (see, for example, Custer, 2014), many MOOCs yield negligible direct revenue, and so higher education institutions must look for other ways to make these online courses pay. One solution is to reuse MOOC teaching materials within paid courses. Another is to develop MOOCs tailored to attract students to particular degree courses offered by an institution. Some platforms and institutions have also started to offer MOOCs for credit, giving learners the opportunity to pay for and pass an additional assessment following the completion of a MOOC, conferring academic credit that can count towards a particular qualification or degree (FutureLearn, 2016). For universities, research output is another, potential form of revenue to be gained from MOOCs. This is an important reason for trialling MOOC research projects generally.
The reasons for conducting a research study via Literature and Mental Health, in particular, are related to the primary initiative of the organisation ReLit. The therapeutic reading advocated by ReLit is a modern and progressive approach in the field of mental health treatment, but the concept of bibliotherapy – or ‘reading for wellbeing’ – is rooted in ancient ideas. The entrance to the library of Pharaoh Ramses II reportedly bore an inscription declaring the building a ‘house of healing for the soul’ (Lutz, 1978, p. 36) and in the fifth century BC, the Greek tragedian Aeschylus wrote that ‘words are the physicians of the mind diseased’ (Kelly, 2014, p. 74). Despite boasting an anecdotal history spanning over 2000 years, however, the medical practice of bibliotherapy is supported by little substantive evidence. Some research studies investigating the therapeutic potential of literature in different clinical settings have been conducted in recent years (see, for example, Gregory et al., 2004; Moldovan et al., 2013), but as yet, none of these trials has yielded results significant enough to elevate bibliotherapy to the status of mindfulness, another form of non-pharmaceutical treatment proven to be ‘as good as drugs for preventing depression relapse’, and given its own page on the UK’s National Health Service website (NHS Choices, 2016). There exists, therefore, an incentive to gather data relevant to the potential link between reading and wellbeing at any available opportunity. The Literature and Mental Health: Reading for Wellbeing MOOC presented itself as one such opportunity.
Veletsianos and Shepherdson observe that a ‘deluge of empirical research’ concerning MOOCs has emerged since 2013 (Veletsianos & Shepherdson, 2016, p.198). In their literature review of the MOOC-related literature reviews published since this date, Veletsianos and Shepherdson identify four dominant themes of MOOC research publications: the distinction between cMOOCs and xMOOCs; the impact of MOOCs on education and pedagogical practices; the demographic of MOOC users; and the challenges encountered by MOOCs in the initiative’s infancy (Veletsianos & Shepherdson, 2016). Among the research proposals submitted to the MOOC Research Initiative, funded by the Gates Foundation and administered by Athabasca University in 2013, meanwhile, nine research themes were identified, including learner performance, MOOC platforms, MOOC-led improvements in educational technology, and the impact of MOOCs on higher education institutions (Gasevic et al., 2014).
As well as providing researchers with a new object of study, the MOOC has also encouraged the development of new research approaches: researchers are, increasingly, thinking through and with MOOCs, as well as looking at them. Admittedly, the distinction between research conducted on MOOCs and research conducted through MOOCs is not an easy one to draw, but for the purposes of this paper, research conducted through MOOCs will be defined as any research involving a MOOC, in which the findings have an application beyond that MOOC or MOOCs in general. Thus, papers which seek to identify and describe the differences between xMOOCs and cMOOCs (see, for example, Daniels, 2012; Siemens, 2012) are concerned with research on MOOCs, self-reflexive research that looks at MOOCs to determine something about MOOCs. The 2015 study by Southampton University, which analysed MOOC learner comments in order to evaluate different indicators of pedagogical activity, meanwhile, is an example of research conducted through a MOOC: although MOOC learner comments were the subject of the analysis, the aim of this research study was to compare different measures of pedagogical validity, and the findings could be applied to any online learning situation (O’Riordan et al., 2015).
The research project designed for and included in Literature and Mental Health seems to exist in a slightly different category again, in that the MOOC was used as a vehicle for research into a subject area entirely unrelated to MOOCs in general. As this paper will discuss, we have inevitably made some important findings about the impact and potential of the MOOC platform, but the aim of the research was to explore the relationship between reading and wellbeing, with the MOOC simply providing a convenient opportunity to do so. Other MOOCs have attempted similar research studies, though not necessarily on the same scale. The first presentation of the Mindfulness for Wellbeing and Peak Performance MOOC, produced by Monash University and hosted on the FutureLearn MOOC platform, gave learners the opportunity to take part in a ‘stress survey’ at the beginning and end of the course. This survey permitted learners to observe whether the mindfulness practices they had learned during the course were helping to reduce their stress levels, but learners were also given the opportunity to make their survey responses available to the lead educators, to be used as part of a research study (Monash University, 2015). The University of Warwick’s first MOOC, The Mind is Flat, meanwhile, invited learners to take part in a short psychological experiment related to the course content in each of its six weeks (University of Warwick, 2013). These experiments were included primarily to give learners an insight into the workings of the mind and into the psychological experimentation process, but the results could also be used as part of a research study.
It is notable, and perhaps not entirely coincidental, that the three in-MOOC research projects described here, which look beyond the MOOC to investigate a particular subject area, are all concerned with human psychology. Certain topics will naturally lend themselves more than others to in-MOOC research, and the remote access that MOOCs give researchers to human research participants seems suited to online experiments exploring aspects of psychology. The use of MOOC learners as research participants – and as research participants in a psychological experiment, in particular – raises a number of ethical questions, however, which are addressed below.
Although researchers are only beginning to use MOOCs as opportunities to involve their participants in these kinds of surveys and experiments, the use of internet-based platforms for large scale data-gathering is, in 2016, very well established. Survey design websites such as Qualtrics (which we used to build the surveys for Literature and Mental Health) and SurveyMonkey make it easy for anyone to distribute research surveys via email, or to embed them as links in webpages. Retailers regularly redirect online customers to questionnaires designed to gather information about their shopping habits and experiences. An initiative such as the National Student Survey utilises internet-based research to gather data on a national scale. To take another example, this time relevant to the topics explored in Literature and Mental Health, the University of Cambridge’s ‘Poetry and Memory’ project is using an online survey, accessed on the project’s website www.poetryandmemory.com, to ‘investigat[e] the value and experience of poetry in the memory, and examin[e] the relationship between memorisation and understanding’ (The Poetry and Memory Project, 2014). This last example, in particular, suggests that internet-based research methods are being used not only to gather large amounts of data, but also to investigate, in some depth, questions of a cognitive nature. The MOOC platform is primarily pedagogical, and MOOC research is therefore subject to more restrictions than other research conducted via the Internet; this is an issue discussed in more detail below. Nevertheless, there is certainly no lack of precedent or model for online data gathering, and it seems that MOOC developers could exploit the opportunity to conduct online research more readily than they currently do.
In his comprehensive discussion of MOOC ethics, Stephen Marshall alludes to a number of the challenges facing in-MOOC research projects, several of which we encountered during the development of Literature and Mental Health (Marshall, 2013). Issues of data protection are inevitably fraught when information is collected and stored online. We made the decision to design and conduct all of our research activities using the survey platform Qualtrics, which sets out its commitment to ‘the protection and reliability of customer data’ in its online security statement, as well as outlining the practices and regulations in place to ensure this data protection (Qualtrics, 2016).
In addition to the problem of data protection, internet-based research conducted within MOOCs poses further ethical challenges, in relation to learners’ awareness of and consent to their involvement in a research study. FutureLearn’s research ethics guidelines remind FutureLearn Partners that individuals taking part in FutureLearn courses are ‘engaging voluntarily with the courses with the intention of learning’ (FutureLearn, 2014). Learners have not signed up to be research participants, and while research on and through MOOCs is permitted, FutureLearn Partners have a responsibility to make learners aware that their participation in and contributions to a particular course may be monitored and analysed for research purposes.
Presumably for reasons of practicality, the FutureLearn research ethics guidelines do not require Partners to seek ‘opt-in consent’ from participants: provided learners are clearly notified of the research taking place, they ‘can be assumed to have given consent for participation in research’ (FutureLearn, 2014). Given the potentially sensitive nature of our research interests in Literature and Mental Health, we were concerned to ensure that learners knew precisely when they were engaging in a research activity, and that they were able to participate in the course without contributing to the research if they preferred. The research activities were therefore conducted on a separate platform to the remainder of the course. After being notified that their responses may be used for research, learners had to actively follow a link to launch the relevant survey or activity; alternatively, they were able to proceed to the next step without participating. Moreover, learners were given the option of supplying an email address as an identifier for their response only after they had completed a survey or activity, so that they were fully aware of any information divulged prior to supplying an identifier. This identifier would then be used to match all survey and activity responses created by the same learner, to build up a profile for that learner.
The FutureLearn research ethics guidelines stipulate that ‘[p]articipants should be given opportunities to access the outcomes of research in which they have participated’ (FutureLearn, 2014). In Literature and Mental Health, learners were given the opportunity to sign up for research updates at the end of the Start of Course survey. Emails will be circulated to those learners who asked to be kept updated, detailing the results of the course surveys and activities as well as outlining any future plans for research. The first of these emails, which shared with learners our findings about learner reading habits, was distributed on 23 June 2016, to 2665 recipients. Of these, 1372 or 51.5% opened the email, and 253 followed the link to download the full, in-depth report on reading habits.
Given the subject matter of Literature and Mental Health, it was deemed necessary to seek further ethical guidance on the specific content of the MOOC research activities, which falls outside the usual remit of Warwick Business School research. All surveys and research activities were therefore checked and approved by a clinical psychologist prior to their inclusion in the course.
Our MOOC research project comprised seven surveys and activities:
All of these surveys and activities were built using the survey platform Qualtrics. With the exception of the Follow-Up survey, all surveys and activities were accessed by learners via links from steps within the course. At the end of each survey and activity, learners were invited to supply an email address to be used as a unique identifier. This allows us to match survey and activity responses generated by the same individual, building up profiles for those learners who have chosen to supply an identifying email address.
In the Start of Course survey, learners were also given the opportunity to volunteer to participate in a post-course online study. This study is currently in the early stages of its development, and will serve as a more extensive and rigorous counterpart to the research activities conducted within the MOOC itself.
Perhaps the most obvious asset of the MOOC as a research tool is the access that it can afford researchers to thousands of potential research participants. The big numbers do tend to diminish rather quickly under scrutiny, of course. Of the 23,030 people who signed up to Literature and Mental Health, for instance, only 12,474 ever accessed the course. Nonetheless, with thousands of potential participants, these remain large numbers for a research study, as a comparison with some recent and influential mindfulness trials will demonstrate. Learner numbers typically reduce as a MOOC progresses, and participation in the Literature and Mental Health research activities inevitably dropped concurrently. Even ten weeks after the conclusion of the course, however, we found that over 1000 learners were willing to take part in a Follow-Up survey distributed by email; not only does this indicate that participant numbers remained high in comparison to non-MOOC research trials, but it also demonstrates that email distribution may be more successful than we had originally anticipated.
The participant numbers for recent clinical trials investigating the efficacy of mindfulness provide an apposite point of comparison, helping to put the participant numbers for the Literature and Mental Health research activities into some context. Like bibliotherapy, mindfulness-based cognitive therapy (MBCT) is a non-pharmacological approach to improving and maintaining mental wellbeing. Mindfulness has become increasingly popular in recent years: schools and universities offer mindfulness workshops to their students; businesses are beginning to introduce mindfulness to reduce workplace stress and increase productivity; in April 2015, The Mindfulness Colouring Book was an Amazon US top ten bestseller (Flood, 2015); and mindfulness is now prescribed on the NHS, with a section of the NHS website now dedicated to the practice (NHS Choices, 2016). Several influential publications in recent years have established the status of mindfulness as more than simply a popular trend, demonstrating that the practice is as effective at preventing relapse in recurrent depression as anti-depressant medication (ADM).
The clinical trial cited on the NHS website, which found MBCT to be an equally effective alternative to ADM for the prevention of relapse in recurrent depression, analysed 424 participants, randomly assigned to two equal groups (Kuyken et al., 2015). In 2010, researchers at the Oxford Mindfulness Centre published the design and protocol for a trial comparing MBCT with a control psychological treatment; allowing for an expected attrition rate of 20%, they intended to recruit 375 participants, in order to achieve a sample size of at least 300 (Williams et al., 2010). Another research project conducted by the Oxford Mindfulness Centre, which studied the effectiveness of mindfulness in treating health anxiety or hypochondriasis, recruited a total of 74 participants.
Evidently, a MOOC such as Literature and Mental Health: Reading for Wellbeing gives researchers access to a cohort of potential participants many times larger than the groups analysed in the most influential of mindfulness studies. During the Start of Course, learners were given the opportunity to volunteer to take part in a post-course online research study, which is currently being developed. Of the 9754 learners who responded to the survey, 4009 volunteered to participate in this post-course online study. This number is almost ten times larger than the number of participants in the influential 2015 mindfulness trial, although it is, of course, impossible to predict how many of these volunteers will ultimately take part in the online study.
Figure 1 shows the number of responses to each course activity, compared to the number of active learners participating in the same week of the course. An active learner is defined as a learner who marks at least one step of the course as ‘complete’. The graph demonstrates a dramatic but not atypical decline in the number of active learners participating in the course over the 6 weeks; the number of active learners in week 2 is 56.1% of the number in week 1, while the number in week 6 is 29.5% of the week 1 total. The corresponding decline in the number of learners participating in the course activities over the six weeks is therefore inevitable.
Notably, however, the percentage of active learners responding to the activities also diminishes across the six weeks, rather than remaining constant. As a percentage of the number of active learners in week 1, the number of responses to the first course activity (the Start of Course survey) is 97.1%. By contrast, the number of responses to the End of Course survey (the second activity in week 6) is just 62.9% of the number of active learners participating in that week. This difference may, in part, be accounted for by the fact that the Start of Course survey featured near the beginning of a week, in step 1.3. The second activity in week 1, the Poetry for Stress Relief activity (step 1.7), had a lower response rate of 84.9%, suggesting some attrition in participation rates within an individual week. We can identify this same trend in weeks 5 and 6, with fewer responses to the second activity in each week than the first. The End of Course survey features in step 6.15, near the conclusion of the week. The Missing Word activity, in week 5, is situated in a similar position, in step 5.14. The number of responses to this activity is 76.2% of the total number of active learners in week 5, lower than the percentage of active learners responding to the first activity in week 1, but not as low as the percentage of learners responding to the second activity in week 6 (62.9%). We see, therefore, that this attrition has taken place both within individual weeks and across the course as a whole.
Nonetheless, the numbers of research activity participants remain higher throughout the course than in any of the mindfulness trials discussed above. The number of responses generated by the follow-up survey surprised us, moreover. We had made the decision to include all surveys and activities within the course itself, as we believed this would result in higher levels of participation than if the surveys and activities were distributed via email links. We were pleased, therefore, that ten weeks after the conclusion of the course, 1339 learners (71.9% of the number that responded to the End of Course survey) responded to the Follow-Up survey distributed in an email. It should be noted that 87.9% of these respondents indicated that they had not viewed the course for at least a month.
Learners had been invited to supply an email address at the end of every survey or activity, which would be used as a unique identifier, allowing profiles of learner responses to be compiled. Figure 2 shows the total number of responses to each activity, alongside the number of responses with email addresses supplied. Figure 3 shows the percentage of responses to each activity in which an email address was supplied. In five of the seven surveys or activities, over 50% of responses included an email address to be used as an identifier. 972 email addresses could be matched between the Start of Course and End of Course surveys; this is 16.3% of the number of email addresses supplied in the Start of Course survey, and 79.8% of the number of email addresses supplied in the End of Course survey. Across all seven surveys and activities, meanwhile, 157 email addresses could be matched. The sample size therefore becomes much smaller if only those learners with a full profile of survey and activity responses are considered. Indeed, this number is smaller than the sample sizes of two of the three mindfulness studies referenced above.
We had anticipated large numbers of research activity participants, and the MOOC did not disappoint. Despite a drop-off as the course progressed, the majority of active learners participated in the research activities in every week of the course. Participant numbers remained far higher than those for comparable non-MOOC research studies, although as this paper discusses below, the Literature and Mental Health research exercises lacked the selection rigour of the mindfulness trials to which they have been compared. Upon reflection, then, participant numbers do indeed make MOOC research an attractive prospect; the challenge is maintaining and identifying a cohort of participants across multiple research activities.
In a research trial, the size of a sample group, while important, is by no means the only factor that needs to be taken into consideration. Despite the large numbers of responses generated by the Literature and Mental Health research activities, it must be acknowledged that there exists a self-selection bias among the MOOC learners. Every participant in the surveys and activities – indeed, every volunteer for the post-course online study – has signed up for an educational online course entitled Literature and Mental Health: Reading for Wellbeing. It seems probable, therefore, that these participants would be more likely to believe in or, at least, to be receptive to the concept of ‘reading for wellbeing’ than a random sample of the general population.
To demonstrate this, we asked learners in the Start of Course survey whether they thought that literature ‘can be used effectively to treat some mental health conditions’. 40.0% of respondents said ‘yes, probably’, and a further 40.4% said ‘yes, definitely’. 80.4% of Start of Course survey respondents, therefore, thought that literature at least ‘probably’ could be used in a therapeutic context. This percentage increases to 89.9% among those respondents whose email addresses can also be matched to the End of Course survey, and to 82.0% among those respondents who volunteered for the post-course online study. Any results from in-course research activities, or from a post-course online study, which support the hypothesis that literature can be therapeutically beneficial, are thus generated by a sample of which at least 80% of the participants already believe in the concept being tested.
The logic of the placebo effect would suggest that Literature and Mental Health learners, whether within the course or as part of a post-course study, are likely to experience an improvement in wellbeing when they read precisely because they expect to experience this improvement. Admittedly, the precise status of the placebo effect in non-pharmacological trials is the subject of some debate. The placebo effect refers to any change in a patient’s condition, whether positive or negative, which cannot be directly attributed to the treatment administered, and must therefore result from other factors: in particular, from the patient’s belief in the treatment, but also, perhaps, from the medical rituals of receiving treatment and consulting with a doctor. In pharmacological trials, the placebo effect is much easier to isolate. The real drug is administered to one group, and the placebo to another, and the results in both groups can be observed and compared, to identify which changes are caused by the drug itself. With non-pharmacological treatments, however, the surrounding factors that cause the placebo effect are often much harder to separate from the treatment itself. Trials have proven mindfulness to be more effective in reducing sensations of pain than both a placebo analgesic cream (actually petroleum jelly) and a sham mindfulness practice (Zeiden et al., 2015). However, there exists an extensive literature discussing the difficulty of distinguishing the placebo effect in psychotherapy (see, for example, Wampold et al., 2005; Brown, 2006; Justman, 2011; Wedge, 2012). Several researchers have argued that factors which would usually be regarded as contributing to the placebo effect, such as a patient’s confidence in and hope for the treatment, or the fact of regularly visiting a medical practitioner, are not extraneous in psychotherapy; rather, they are an integral part of the treatment itself (Justman, 2011; Wedge, 2012).
The status of the placebo effect for bibliotherapy-based treatments has not yet been elucidated by research; we do not yet know, in other words, whether a patient’s confidence in bibliotherapy is integral to the outcome of their treatment. The self-selection bias within the MOOC does not immediately or entirely negate the results of the course research activities, therefore, and nor does it render an extended online research trial involving MOOC participants a futile exercise. The fact that 6707 people believe in the therapeutic potential of literature should in itself be regarded as a positive finding, as should the fact that activity participants drawn from this cohort registered lower levels of stress after reading or listening to a poem. The bias must be acknowledged, however, and will inevitably influence the online trial design.
Self-selection bias will always be a relevant consideration for MOOC researchers. Whereas the participants in the mindfulness trials discussed above were selected for their suitability, as well as for their willingness to be involved in a study, participants in any MOOC research activity have all volunteered to learn about a particular subject, in a particular way. Whether research is being conducted into a subject related to the course content (as with Literature and Mental Health), or into the efficacy of an online teaching method, the potential influence of the participants’ self-selection bias on any findings from the research must be acknowledged. The Start of Course survey results indicate to us the extent of that bias.
Cited above, the FutureLearn research ethics guidelines describe learners as ‘[i]ndividuals… engaging voluntarily with the courses with the intention of learning’ (FutureLearn, 2014). FutureLearn Partners have an ethical responsibility to alert learners to any research being conducted on or via a MOOC, but they also have a responsibility, as educational providers, to ensure that any research taking place does not impinge on or impede the MOOC’s primary purpose, as a teaching and learning tool. Overall, the research activities developed for Literature and Mental Health were successfully integrated into the course learning experience. More than 80% of learners, according to the End of Course survey responses, enjoyed taking part in the interactive exercises developed for research purposes, and the discussion steps inspired by the surveys generated some of the highest numbers of comments of any steps throughout the course. We found that, designed and framed in the right way, research activities can enrich the learning experience. In Literature and Mental Health, as in some other MOOCs, the learners became part of the investigative process, exploring the same questions and concepts that we wanted to explore through our research. In order to prioritise the MOOC’s function as a teaching and learning tool, however, some important aspects of a standard research trial were sacrificed. Specifically, it was not possible to introduce a control into some of the activities, which therefore diminished the validity of the results.
When we began developing the research activities for Literature and Mental Health, we were conscious of the need to minimise the burden placed on learners by any surveys or exercises. In the absence of tests or quizzes, the three interactive poetry activities (the Poetry for Stress Relief activity, the Missing Word activity and the Poetry and Memory activity) were in fact designed, first and foremost, as opportunities for learners to explore and apply some of the ideas set out in the course videos. The Poetry for Stress Relief activity, for instance, gave learners the opportunity to find out whether reading or listening to a poem could make them feel ‘calm’, ‘relaxed’, ‘composed’ and ‘focused’, as lead educator Dr Paula Byrne suggests in step 1.5 (University of Warwick, 2016). Learners were exploring, at an individual level, the same question that we wanted to explore on a large scale: can reading a short poem help to alleviate stress? Other MOOCs have integrated potential experiments in a similar way. The University of Warwick’s FutureLearn MOOC The Mind is Flat: The Shocking Shallowness of Human Psychology invites learners to participate in a different psychological experiment in each of its six weeks. The learners effectively become research subjects, but the process is designed to supplement their understanding of the course content; according to step 1.13, introducing the Week 1 experiment, ‘each experiment illustrates something about how our minds work’ and ‘give[s] [learners] a sense of how psychologists use experiments to uncover… aspects of the mind’ (University of Warwick, 2013).
We see here how research and learning may productively coexist as functions of a MOOC, at least in theory. In Literature and Mental Health, in practice, learners felt that they gained less from the interactive research activities than from other aspects of the course. Figure 4 shows learner responses to a question in the End of Course survey asking which feature of the course they learned most from. ‘Interactive activities’ scored least well among the options given, although responses to the question ‘Which of the following features of the course did you learn least from?’ (Figure 5) indicate that ‘online discussions with other learners’ were least instructive for the highest number of learners. Just under a quarter of respondents felt that they learned least from the interactive research activities. However, in the same survey, a total of 84.9% of learners indicated that they either ‘somewhat’ or ‘definitely’ enjoyed taking part in these exercises. This would suggest that the research activities, although they were not as informative as the course educator videos or the interviews with writers and GPs, were neither burdensome to learners, nor detrimental to the overall course experience.
More difficult to integrate into the course learning experience were the three surveys. FutureLearn’s research ethics guidelines require Partners to be ‘sensitive to the problem of inundating learners with surveys.’ In addition to the concerns about learning safeguarding discussed above, it was our awareness of the need to minimise the number of surveys distributed to learners that caused us to leave out the PHQ-9, a validated wellbeing assessment used by medical practitioners in the diagnosis of certain mental health conditions (Kroenke et al., 2001; Martin et al., 2006). All research activities included within MOOC steps, moreover, were required to have a learning outcome. In the step immediately following each of the surveys, therefore, we invited learners to reflect on and, if they felt comfortable doing so, to discuss their survey responses. After the Start of Course survey, learners were encouraged to consider whether they had realised something new about their reading habits from taking part in the survey, and whether their responses would influence their reading habits. The Reading for Wellbeing survey was followed by a discussion of which books learners read for comfort, while the End of Course survey was used as the stimulus for learners to share their reflections on the course.
Other MOOCs have also encouraged the use of research surveys for learner self-reflection, though in some slightly different ways. The first presentation of Monash University’s FutureLearn MOOC Mindfulness for Wellbeing and Peak Performance, for example, gave learners the opportunity to fill in a ‘stress survey’ in its first and final weeks. Learners were invited to use the survey to ‘monitor [their] progress through the course and measure the impact of mindfulness on [their] life’, and were encouraged to print a copy of their responses in order to make it easier to compare their results before and after taking the course. The inclusion of these results in a research study was optional; learners were able to fill in the survey for their own records without making their responses available to the researchers (Monash University, 2015).
We see here that the relationship between research and learning need not be an antagonistic one; the two can exist, in practice, as mutually beneficial, rather than mutually exclusive processes. That the research surveys included in Literature and Mental Health were beneficial rather than detrimental to the learning experience is indicated by the popularity of the related discussion steps. The opportunities for reflection afforded by the three in-MOOC surveys generated some of the highest numbers of comments of any steps throughout the course. Indeed, the discussion step following the Start of Course survey, which invited learners to reflect on and discuss their reading habits in the light of their survey responses, generated 3717 comments, over 400 comments more than any other course step. This volume of responses may be attributed, in part, to the fact that this was the first discussion step of the course; likewise, the discussion step following the End of Course survey may have generated more comments (a total of 1071) than any other step in week 6 because it is the final discussion step of the course. By contrast, the discussion step after the Reading for Wellbeing survey in week 5 generated only the third highest number of comments for that week, with 958 comments.
Figure 6 compares the number of comments generated by each of the survey discussions to the number of comments generated by other discussion steps in the same week, as well as to the average number of comments across all steps in the same week. Although the Reading for Wellbeing survey inspired less discussion than the Start and End of Course surveys, it is evident that there was a high level of engagement in all three of the discussion steps following the course surveys. Each one received a higher than average number of comments not only for the week in which it featured, as Figure 6 demonstrates, but also for the course as a whole: the mean number of comments per step for the course as in its entirety is 936.4. The surveys had initially posed a greater challenge when it came to integrating research activities into the course. Whereas the interactive exercises were learning experiences in themselves, the surveys required additional steps in order to generate a learning outcome for participants. Ultimately, however, the surveys were integrated into the course learning experience as successfully as the interactive exercises, if not more so. The evident popularity of the discussion steps following the surveys suggests that the surveys gave learners food for thought: that learners had more to say after reflecting on and answering the survey questions. Thus, we can conclude that the surveys were beneficial to the course learning experience, encouraging learners to think and fostering conversation.
While we did not find research to be detrimental to learning in the MOOC, the necessary prioritising of the latter did, at times, inhibit the success of the former. In particular, we were not able to introduce a control into the course research activities. This had detrimental implications for the Poetry for Stress relief activity. Participants were asked to assess their current mood on six sliding scales, before and after either listening to or reading a poem. The six scales are set out below:
|Focused on one thing||Distracted by lots of things|
|Physically relaxed||Physically tense|
|Mentally calm||Mentally anxious|
|At ease, able to cope||Overwhelmed|
|Uplifted, upbeat||Unhappy, down|
In our analysis of participants’ responses to this activity, we found that there was an average reduction in stress levels – that is, a movement towards the left on the scales set out above – on every scale, for every version of the poem. On average, learners measured themselves to be more focused, more relaxed, calmer, more able to cope, more hopeful and more uplifted after reading or listening to W. H. Davies’ ‘Leisure’.
This has the potential to be an overwhelmingly positive result, indicating that reading and listening to poetry does help to reduce stress. Because our participants were learners in a literature MOOC rather than research subjects, however, we did not feel that it would be appropriate to create a control group within any of the course activities. Instead of reading or listening to a poem, a control group in the Poetry for Stress Relief activity would have spent an equivalent amount of time listening to a piece of music or looking at a photograph. Through comparison, it would have then been possible to begin to establish whether poetry was uniquely effective at reducing stress. Without a control group, however, we cannot be sure that the reductions in stress registered by learners in the Poetry for Stress relief activity were caused by the specific act of reading or listening to a poem. Any act of prolonged concentration on a single stimulus may have produced similar results.
It would, of course, have been both impractical and unfair to make some learners listen to a piece of music or look at a still image in order to form a control group in the Poetry for Stress Relief activity. This is one of the reasons why it is important for a post-course study to take place. In a post-course online study, we would not have the same responsibility to ensure that all participants receive an equivalent learning experience, and have the opportunity to take part in a literary exercise. It would therefore be possible to introduce control activities to provide a useful point of comparison.
When we began designing the research activities to be included in Literature and Mental Health, we were clear that our interest was in using the MOOC to find out about the relationship between reading and wellbeing. The course was the platform and the vehicle only: our research was not intended to find out about MOOCs. Reflecting on the outcomes of our research, however, it is apparent that we have made some notable findings about the impact of the MOOC. Indeed, our impact findings are perhaps the most substantial. For the reasons outlined above – namely, the problem of self-selection bias and the necessity of balancing research and learning – it has not been possible, through the MOOC alone, to definitively prove a link between reading and wellbeing. From the results of the follow-up survey, however, we can see that the MOOC itself has had an impact on learners’ reading and online learning habits.
Of the respondents who had not viewed the course for two months or more, 27.2% indicated that they were reading more regularly as a result of the course, and 41.2% said that they were reading different kinds of texts having taken part in the MOOC. Of that 41.2%, 87.7% had been encouraged to read more poetry; this is important because ReLit places particular emphasis on the reading of poetry, as the Stressed Unstressed poetry anthology demonstrates (Bate et al., 2016). Among respondents who had not viewed the course for at least one month, the figures were slightly higher, with 27.8% reading more regularly, and 45.0% reading different kinds of texts. This perhaps suggests that the impact of the course diminishes over time, as the percentage of respondents whose reading habits have changed is smaller among those who last viewed the course over two months ago, compared to those who viewed the course over one month ago. Further surveys would be needed to confirm this trend.
As well as asking about reading habits, the follow-up survey also asked respondents about their online learning habits. Of learners for whom Literature and Mental Health was their first experience of online learning, a substantial majority of 70.2% had already signed up for other MOOCs by the time the follow-up survey was distributed. This is a particularly encouraging figure, suggesting that Literature and Mental Health, as a free online course, gave learners a positive experience that they were keen to repeat. The follow-up survey also indicated slight changes in social learning habits. Of the learners who responded, 493 indicated that they ‘sometimes’ or ‘often’ made comments during online courses before taking part in Literature and Mental Health; this figure had risen to 525, based on the same group of respondents, after taking part in Literature and Mental Health.
Having matched responses from the Start of Course and End of Course surveys, we can also measure the impact of the course by looking at the changes in response to certain questions. Based on the same cohort of respondents, answers to the question ‘do you think literature can be used effectively to treat some mental health conditions?’ in the Start and End of Course surveys indicate that the course itself changed learners’ attitudes to ‘reading for wellbeing’. The number of learners responding ‘yes, definitely’ to the question rose from 434 at the start of the course, to 475 by the end. Moreover, the number of learners indicating that they were ‘undecided’ more than halved, dropping from 141 to 64. Although there was an increase, from 4 to 22, in learners responding ‘no, probably not’, therefore, these figures do indicate that the course was effective in encouraging learners to formulate definite opinions about the relationship between reading and wellbeing.
These figures correspond with the results from the follow-up survey, indicating that more learners were reading for comfort and reading for stress-relief as a result of taking part in the course. The number of learners reading for comfort ‘often’ increased by almost 100, from 313 to 410; the number reading for comfort ‘rarely’ or ‘never’, meanwhile, dropped from 212 to 128. Similarly, the number of learners reading for stress-relief ‘often’ increased from 273 to 399, while the number reading for stress-relief ‘rarely’ or ‘never’ dropped 273 to 151.
At most, of course, these figures tell us about the impact of the course ten weeks after its conclusion. 40.25% of respondents to the survey (506 people in total) had not viewed the MOOC for over two months, but the remainder of respondents were completing the survey within eight weeks of last visiting the course. We can conclude very little, therefore, about the long-term, lasting impact of Literature and Mental Health. The fact that changes in people’s reading habits have been sustained for two-month period does not mean that learners’ behaviour will still be influenced by the course six months or one year later. Nonetheless, the results of the follow-up survey do demonstrate that Literature and Mental Health has had an impact upon the habits and attitudes of its learners. We should be encouraged by these results, and consider seeking further follow-up opportunities once more time has elapsed.
It was the big numbers that made MOOC research seem an attractive prospect at the project’s outset: the opportunity to observe mass trends, and the potential for thousands of participants to respond positively to a bibliotherapy practice. We can learn much from the numerical data, as with the results of the follow-up survey discussed above. We had not realised, however, how much more there would be to learn from the qualitative data gathered during the course: from the stories of individual learners as opposed to mass statistical trends. Indeed, we had not anticipated how willing learners would be to supply this qualitative data, to respond to optional open-ended questions in the surveys, to make comments in the course discussions and to recount their personal experiences. Figure 7 shows the number of responses to each of the course research activities and, where available, the number of those responses which included at least one answer to an open-ended question. None of these open-ended questions was compulsory. From the graph, it is clear that the majority of responses to both the End of Course and the Follow-Up survey contained at least one answer to an open-ended question. 76.81% of End of Course survey responses, and 65.84% of Follow-Up survey responses contained at least one of these answers. The percentage for the Reading for Wellbeing survey, meanwhile, was 47.73%, with just under half of the total responses offering an answer to an open-ended question. It should be noted that the noticeably smaller percentage for the Start of Course survey may be due, in part, to the fact that the only open-ended questions were the ‘Other, please specify’ options in any drop-down lists, and a question asking anyone who has been employed in the medical or care professions to describe their role. Examples of the open-ended questions from the other three surveys, by contrast, include the following:
We had been conscious of burdening learners with open-ended questions such as these when we designed the surveys, including only a limited number and ensuring that all were optional. We were both surprised and pleased, therefore, to find that so many respondents were willing to engage with these questions. This was a MOOC in which many learners wanted to tell their stories, and to supply a descriptive or narrative response to a question rather than simply ticking a box. We had invited learners to become part of a conversation (University of Warwick, 2016), and this ethos of conversation was adopted by many learners in their survey responses.
The sheer volume of comments made during the course is further testament to the learners’ willingness to engage in a conversation, and to offer reflective, qualitative feedback in response to the course materials. Figure 8 compares the total number of comments made across all presentations of the University of Warwick’s five existing MOOCs. In its first presentation, Literature and Mental Health received almost twice as many comments as any other Warwick MOOC; it received more comments than other Warwick MOOC running at a similar time, and it received more comments than any other Warwick MOOC running for the first time. These higher comment numbers can, in part, be attributed to higher learner numbers. Literature and Mental Health is the only MOOC included in the graph to have over 10,000 learners, with a total of 12,485 learners taking part. It is no coincidence that the MOOC that generated the second highest number of comments – the second presentation of ‘Shakespeare and His World’ – also had the second highest number of learners, at 9,341.
Even allowing for the discrepancies in learner numbers, however, the comment total for Literature and Mental Health remains comparatively high. Figure 9 shows the average number of comments per learner for each presentation of the University of Warwick’s five MOOCs. We see here that the first presentation of Literature and Mental Health generated more than twice as many comments per learner as any presentation of The Mind is Flat, Big Data or Babies in Mind. It also generated more comments per learner than any run of Shakespeare and His World, by more than 1.2 comments.
Moreover, Literature and Mental Health had a higher percentage of social learners in its first presentation than any other presentation of a University of Warwick MOOC to date. 41.65% of learners made at least one comment. The first presentation of Babies in Mind came closest to this social learner figure, with 39.29% of learners becoming social learners. Social learner figures for presentations of Shakespeare and His World, meanwhile, ranged from 25.82% to 32.43%; fewer than 25% of learners made at least one comment in any presentation of The Mind is Flat for which the required records are available, and just under 19% of learners became social learners in both presentations of Big Data.
A greater proportion of the Literature and Mental Health learners chose to make at least one comment, to join the conversation. If we compare Literature and Mental Health with the most recent run of Shakespeare and His World (18th April 2016), we see that the average number of comments per social learner for Shakespeare and His World is actually higher than for Literature and Mental Health, at 19.45 compared to 15.87. However, the overall percentage of social learners is smaller by approximately one third: 26.04% of Shakespeare learners became social learners, compared to 41.65% of learners on Literature and Mental Health. In Shakespeare and His World, fewer learners were making more comments, whereas in Literature and Mental Health, a greater proportion of learners felt able to contribute to the discussions.
It is apparent that the learners who participated in the first presentation of ‘Literature and Mental Health’ were extremely and unusually willing to supply qualitative data, and that we consequently have a large amount of qualitative data to consider. While we have not begun any formal process of semantic analysis, we have already found learner comments and open-ended survey responses to be of considerable value, particularly in establishing the impact of the MOOC. These individual narratives are a useful and illuminating counterpart to the mass trends identifiable from the qualitative data.
Given the abundance of qualitative data available, this paper will limit itself to responses to three open-ended questions from the Follow-Up survey:
Several responses to these survey questions describe personal improvements in wellbeing brought about by the course. One participant wrote of how Literature and Mental Health had introduced them to poetry-reading as a form of stress relief, and described how they were now reading poetry aloud when they felt anxious, or when they were unable to sleep. A number of other respondents said that the course had helped them to better understand and manage a particular diagnosed condition, including major depressive disorder, post-traumatic stress disorder and mild cognitive impairment. For others, the course was a source of comfort during a difficult time, for example, following the death of a parent or during the illness of a loved one.
Such anecdotal evidence about the impact of Literature and Mental Health is powerful, telling us, in specific and human terms, how the course has changed the lives of some of the individuals who participated. Moreover, the responses to the Follow-Up survey’s open-ended questions also offer us some insight into the broader reach of the course. Several respondents described how they were now passing on the message of the course to others, both personally and professionally. Some stated that they were using what they had learned on the course to help loved ones, who were suffering either with stress or with the early stages of dementia. Others, meanwhile, described how they had introduced some elements of the course into a professional setting. One respondent had begun incorporating poetry into therapy sessions, one had distributed some poems in a local clinic waiting room, and another had created a mental health and wellbeing information board in the library of the school in which they work. In some instances, the actions described will have made ideas about reading and wellbeing available to hundreds of other people. Poems distributed in waiting rooms, and mental health and wellbeing boards established in school libraries, encourage people who may not have taken part in Literature and Mental Health to consider the therapeutic potential of reading. Public display boards raise awareness and start conversations about mental health and wellbeing.
The qualitative data gathered during Literature and Mental Health, then, offers us variety, subjectivity, the individual experience; but it can also hint, like the quantitative data, towards the big numbers. From the Follow-Up survey responses, we have discovered the course’s impact to be more far-reaching than we would otherwise have anticipated. Much MOOC-related research already incorporates semantic analyses of learner comments (see, for example, de Ward and Kukulska-Hulme, 2016; Hudson, 2016) and going forwards, we will consider a more rigorous and systematic approach to the analysis of open-ended responses from Literature and Mental Health. Given that learners did not seem burdened by open-ended questions, we feel that future MOOC research projects could benefit from providing more opportunities for this kind of data gathering.
Our experience of researching via Literature and Mental Health has been, overall, an instructive and encouraging one. We have found research to be compatible with teaching and learning within a MOOC, to the extent that research activities can actually enhance the learning experience. Given the resources expended during the production of a MOOC, therefore, it seems sensible for universities to harness Massive Open Online Courses as opportunities for research as well as for teaching wherever possible. Indeed, the ability of universities to capitalize on these research opportunities may be crucial to the future of the MOOC, as the MOOC business model comes under increasing scrutiny.
Researchers are already thinking creatively about MOOC research, and are already thinking beyond the MOOC itself, into subject areas (as with Mindfulness for Wellbeing and Peak Performance and Literature and Mental Health) and into other online teaching and learning platforms (see, for example, O’Riordan et al., 2015, on measures of online pedagogical activity). As more researchers begin to think through and with MOOCs, we would advise a greater emphasis on the gathering and analysis of qualitative data, which the Massive Open Online Course, in our experience, seems particularly suited to generating.
Our first attempt at researching through Literature and Mental Health has identified some notable weaknesses of the MOOC as a research tool. The self-selection bias that exists among a cohort of research participants drawn from a MOOC is both inevitable and ineradicable. The relative anonymity of learners also posed a problem for us, and may for other MOOC researchers, because of the difficulty of linking up responses made, by the same individual, to separate research activities. Nevertheless, we have found the MOOC platform to be surprisingly versatile: it can be adapted for research, thereby providing researchers with ready access to thousands of possible research participants. If more researchers begin to take advantage of this opportunity, then we may witness a new, research-driven era in the ongoing evolution of the MOOC.
Rachael Hodge is employed as eLearning Assistant and Administrator for ReLit: The Bibliotherapy Foundation.
Brown, W A (2006). Understanding and Using the Placebo Effect. Psychiatric Times, October 1 2006 http://www.psychiatrictimes.com/articles/understanding-and-using-placebo-effect
Custer, S (2014). How are universities evolving and earning revenue from MOOCs?. The Pie News, http://thepienews.com/analysis/how-universities-evolving-earning-revenue-moocs/
Daniel, J (2012). Making sense of MOOCs: Musings in a maze of myth, paradox and possibility. Journal of Interactive Media in Education, http://jime.open.ac.uk/articles/10.5334/2012-18/
de Ward, I and Kukulska-Hulme, A (2016). Self-directed informal learning by experienced online learners enrolled in FutureLearn MOOCs. FutureLearn Academic Network, June 7 2016 Conference presentation. http://cloudworks.ac.uk/cloud/view/9996.
Flood, A (2015). Colouring books for adults top Amazon bestseller list. The Guardian, April 5 2015 https://www.theguardian.com/books/2015/apr/05/colouring-books-for-adults-top-amazon-bestseller-list
Friedman, T L (2012). Come the Revolution. New York Times, May 15 2012 http://www.nytimes.com/2012/05/16/opinion/friedman-come-the-revolution.html
Friedman, T L (2013). The Professors’ Big Stage. New York Times, March 5 2013 http://www.nytimes.com/2013/03/06/opinion/friedman-the-professors-big-stage.html
FutureLearn (2014). Research ethics. February 24 2014 https://about.futurelearn.com/terms/research-ethics-for-futurelearn/
FutureLearn (2016). Major UK universities to award degree course credit through MOOCs for the first time with FutureLearn programs. May 26 2016 https://about.futurelearn.com/press-releases/universities-to-award-degree-credit-through-futurelearn/
Gašević, D, Kovanović, V, Joksimović, S and Siemens, G (2014). Where is Research on Massive Open Online Courses Headed? A Data Analysis of the MOOC Research Initiative. The International Review of Research in Open and Distance Learning, http://www.irrodl.org/index.php/irrodl/article/view/1954
Gregory, R J, Schwer Canning, S, Lee, T W and Wise, J C (2004). Cognitive Bibliotherapy for Depression: A Meta-Analysis. Professional Psychology: Research and Practice, http://psycnet.apa.org/index.cfm?fa=buy.optionToBuy&id=2004-95164-009
Hudson, L (2016). Smart City MOOC. FutureLearn Academic Network, June 7 2016 Conference presentation. http://cloudworks.ac.uk/cloud/view/10029.
Justman, S (2011). From medicine to psychotherapy: the placebo effect. History of the Human Sciences, DOI: https://doi.org/10.1177/0952695110386655 http://hhs.sagepub.com/content/early/2010/12/08/0952695110386655.abstract
Kroenke, K, Spitzer, R L and Williams, J B W (2001). The PHQ-9: Validity of a Brief Depression Severity Measure. Journal of General Internal Medicine, http://www.ncbi.nlm.nih.gov/pubmed/11556941
Kuyken, W, Hayes, R, Barrett, B, Byng, R, Dalgleish, T, Kessler, D, Lewis, G, Watkins, E, Brejcha, C, Cardy, J, Causley, A, Cowderoy, S, Evans, A, Gradinger, F, Kaur, S, Lanham, P, Morant, N, Richards, J, Shah, P, Sutton, H, Vicary, R, Weaver, A, Wilks, J, Williams, M, Taylor, R S and Byford, S (2015). Effectiveness and cost-effectiveness of mindfulness-based cognitive therapy compared with maintenance antidepressant treatment in the prevention of depressive relapse or recurrence (PREVENT): a randomised controlled trial. The Lancet, http://www.thelancet.com/journals/lancet/article/PIIS0140-6736(14)62222-4/abstract
Lutz, C E (1978). The Oldest Library Motto: ΨΥΧΗΣ ΙΑΤΡΕΙΟΝ. The Library Quarterly: Information, Community, Policy, https://www.jstor.org/stable/4306897?seq=1#page_scan_tab_contents
Marshall, S (2013). Exploring the ethical implications of MOOCs. Distance Education, http://www.tandfonline.com/doi/abs/10.1080/01587919.2014.917706
Martin, A, Rief, W, Klaiberg, A and Braehler, E (2006). Validity of the Brief Patient Health Questionnaire Mood Scale (PHQ-9) in the general population. General hospital psychiatry, http://www.ghpjournal.com/article/S0163-8343(05)00117-9/abstract
Matkin, G (2015). MOOC Research: What Can We Do with Big Data?. Open Education Global, Conference presentation. http://conference.oeconsortium.org/2015/presentation/mooc-research-what-can-we-do-with-big-data/.
Moldovan, R, Cobeanu, O and David, D (2013). Cognitive bibliotherapy for mild depressive symptomatology: randomized clinical trial of efficacy and mechanisms of change. Clinical Psychology & Psychotherapy, http://www.ncbi.nlm.nih.gov/pubmed/22941790
Monash University (2015). Mindfulness for Wellbeing and Peak Performance, https://www.futurelearn.com/courses/mindfulness-wellbeing-performance
NHS Choices (2016). Mindfulness. http://www.nhs.uk/conditions/stress-anxiety-depression/pages/mindfulness.aspx?tabname=Common%20problems
O’Riordan, T, Millard, D and Schulz, J (2015). Can you tell if they’re learning? Using a pedagogical framework to measure pedagogical activity. 2015 IEEE 15th International Conference on Advanced Learning Technologies. 2015, http://eprints.soton.ac.uk/376609/
Parr, C (2015). MOOCs: fluctuating rates in online investment. Times Higher Education, April 23 2015 https://www.timeshighereducation.com/news/moocs-fluctuating-rates-in-online-investment/2019816.article
Qualtrics (2016). Security Statement. https://www.qualtrics.com/security-statement/
Siemens, G (2012). MOOCs are really a platform. eLearnspace, http://www.elearnspace.org/blog/2012/07/25/moocs-are-really-a-platform/
The Poetry Memory Project (2014). http://www.poetryandmemory.com
University of Warwick (2013). The Mind is Flat: The Shocking Shallowness of Human Psychology, https://www.futurelearn.com/courses/the-mind-is-flat
University of Warwick (2016). Literature and Mental Health: Reading for Wellbeing, https://www.futurelearn.com/courses/literature
Veletsianos, G and Shepherdson, P (2016). A Systematic Analysis and Synthesis of the Empirical MOOC Literature Published in 2013–2015. International Review of Research in Open and Distributed Learning, http://www.irrodl.org/index.php/irrodl/article/view/2448/3629
Wampold, B E, Minami, T, Tierney, S C, Baskin, T W and Bhati, K S (2005). The placebo is powerful: Estimating placebo effects in medicine and psychotherapy from randomised clinical trials. Journal of Clinical Psychology, http://onlinelibrary.wiley.com/doi/10.1002/jclp.20129/abstract;jsessionid=81BDE64289A0747EA68F8ED0176374F9.f02t01
Wedge, M (2012). Is Psychotherapy a Placebo?. Psychology Today, January 10 2012 Blog. https://www.psychologytoday.com/blog/suffer-the-children/201201/is-psychotherapy-placebo.
Williams, J M G, Russell, I T, Crane, C, Russell, D, Whitaker, C J, Duggan, D S, Barnhofer, T, Fennell, M J V, Crane, R and Silverton, S (2010). The Staying Well After Depression Study: Trial Design and Protocol. BMC Psychiatry, http://www.oxfordmindfulness.org/wp-content/uploads/Williamsetal2010stayingwell.pdf
Zeiden, F, Emerson, N M, Farris, S R, Ray, J N, Jung, Y, McHaffie, J G and Coghill, R C (2015). Mindfulness Meditation-Based Pain Relief Employs Different Neural Mechanisms Than Placebo and Sham Mindfulness Meditation-Induced Analgesia. The Journal of Neuroscience, http://www.jneurosci.org/content/35/46/15307.full.pdf+html
Zemsky, R (2014). With a MOOC MOOC here and a MOOC MOOC there, here a MOOC, there a MOOC, everywhere a MOOC MOOC. The Journal of General Education, https://muse.jhu.edu/article/564411/summary