Toward Open Pragmatism Developing a Revised Framework for Openness Presenters Erik Christiansen Assistant Professor/Librarian Mount Royal University, Calgary, Canada Michael B. McNally Associate Professor University of Alberta, Edmonton, Canada 2 Background & previous work Copyright/Open Licensing Frameworks Accessibility/Usability Formatting Language 1. Choose Elements to Address 2. Effort and Willingness 3. Skill/Knowledge Required Support Costs Assessment Digital Distribution File Format Closed Mixed Most Open Cultural Considerations McNally, M.B., & Christiansen, E.G. (2019). Open enough? Eight factors to consider when transitioning from closed to open resources and courses: A conceptual framework. First Monday 24(6). https://doi.org/10.5210/fm.v24i6.9180 3 Closed Mixed Most Open Copyright/Open Licensing Frameworks Copyright/all rights reserved Less Open CC License Terms (NC/ND and arguably SA) CC-BY License/ Public Domain Accessibility/Usability Formatting Not formatted for accessibility Some accessibility formatting (e.g. closed captioning) Fully accessibility (e.g. compliance w/ US HHS 508 Compliant) Language Bi-lingual or includes Single Language (usually English) guides/steps for translation Multi-Lingual or includes guides/steps for translation and is bilingual Support Costs Paid resources Licensed library resources Openly Licensed Resources Assessment No assessment available Assessments made available Assessments tailored for self-assessment Digital Distribution Closed/available only to insiders (e.g. via LMS) Open and high discoverability Open but low discoverability (e.g. (e.g. YouTube or broadly available institutional repository) repository (e.g. Merlot, BCcampus) File Format PDF or other non-editable format Editable format but proprietary software (e.g. Word) Fully open format (e.g. html) Cultural Considerations No consideration for outside cultural users/includes culturally specific materials/content Some considerations for outside cultural users Generally devoid of culturally specific material 4 Literature review Conceptions of Openness › › › 50 shades of open (Pomerantz and Peek, 2016) 4 facet spectrum (social, technical, legal and financial) (Hodgkinson-Williams and Frameworks for Openness › Gary, 2009) › Expanding and contracting over time (Peter and Deimann, 2013) 11 approaches topology (Economides and › › Admission, free, OER, OEP (Cronin, 2018) › Perifanou, 2018) ALMS framework (Hilton et al., 2010) › Access to editing tools › Level of expertise › Meaningfully editable › Source-file access Gurell (2012) creates ALMS scoring framework D-Index (Abeywardena et al., 2012) › Desirability index that quantifies level of access 5 Why develop a framework? Conceptual framework is necessary for the following reasons 1. Better understand the state of OCW/OER a. What can we improve? b. What are we doing well? 2. Address the lingering concerns from educators a. Quality control b. Context and broader utility of these resources 3. Give educators a more robust ‘guide’ for developing new, or adapting existing, OCW/OER Photo by Startup Stock Photos from Pexels 6 Research Questions 1. Are these factors robust enough to analyze (or measure) the level of openness in OCW? 2. Are certain factors impractical for measurement and do some factors require modification and/or expansion? Photo by Suzy Hazelwood from Pexels 7 Pilot study design 1) Choose Repositories 2 chosen from 16 repositories examined Examined 5 of 38 MIT Courses (2016-) 2) Random selection 5 of 116 TU Delft Courses 97 TU Delft MOOCs removed CC0 image 2 evaluations x 3) Evaluation 8 factors x 10 courses Photo by Lukas from Pexels 8 Results ● Each of us independently evaluated the OCW sample ● We brought our results together, and did a final analysis to settle on the conclusions outlined by this study 9 Does the framework work? Yes, but there are caveats Some factors are too impractical or subjective › › Cultural considerations Usability Some factors needed rewording › › › Digital Distribution > Discoverability Accessibility/Usability > Accessibility Support Costs > Materials Photo by Startup Stock Photos from Pexels 10 Copyright & Open Licensing Frameworks › All courses fell under ‘mixed’ › Both institutions enforce somewhat restrictive CC-BY-NC-SA › Under different circumstances, categorizing the openness of OCW or OER could be more challenging. How would a course be classified if the documents within an open course each adhered to different licensing terms? 11 Accessibility › MIT ‘Most Open’: “...committed to accessibility for persons with disabilities and strives to meet W3C Web Content Accessibility Guidelines (WCAG) 2.0, Level AA, including validating HTML, captioning the video, and checking the accessibility of course content as part of the authoring process” (MIT, n.d.a) › TU Delft ‘Mixed’: No such formal commitment › Closed captions for video (though no transcript download option) › ‘Usability’ dropped as a consideration for this framework. Too subjective MIT Biological Chemistry II course 12 Language › Majority of classes fell under the ‘closed’ category › Notable exceptions include TU Delft’s Drinking Water Treatment 2 which provided final report example assignments in Dutch › Not surprising given the work required to translate OCW Supplementary lecture notes from TU Delft Public Hygiene and Epidemiology Course 13 Support costs Readings list from MIT Equity and Inclusion course › Majority of OCW analyzed categorized as ‘closed’ with one being ‘mixed’ and one being ‘most open’ › Closed OCW relied on paid textbooks. Many MIT courses featured Amazon links in their reading lists › TU Delft Public Hygiene and Epidemiology best example of an open course › Completely custom openly licensed course readings › Lack of openness 14 Assessment › All courses categorized as mixed, with some exceptions › Qualitative assessments were provided - sometimes with examples student work › Natural sciences courses had more quantitative assessments, often with assessments and solutions available › Factor that appears to be relatively easy to make open Assignment answer sheet from TU Delft Hydrology of Catchment, Rivers and Deltas course 15 Digital Distribution › All courses were categorized as most open › Each course was discoverable through a range of OER repositories such as OASIS, OER Commons, Merlot, etc. › Success story for OER/OCW, as ‘finability’, or a lack of federated searches, is a common instructor complaint CCCOER “Open CourseWare” website OASIS course search results for “Structured Electronic Design” 16 File format › Majority of courses analyzed were categorized as closed › Most assignments, assessments, and other course materials almost exclusively provided in non-editable PDF › Closed nature most surprising given how easy it is to upload multiple file formats to a given platform › Video: › Transcripts usually available as a separate file › What is an ‘open’ video format? › ‘Harvestability’ remains an unanswered question to be addressed in another study Screenshot of PDF lecture notes from MIT Public Transportation Systems course 17 Cultural considerations Discussion of Mumbai transit system in lecture from MIT Public Transportation Systems Course › Most difficult factor to address › More technical courses tended to be very open and had little jargon › Courses featuring culturally specific content, or depend on geographic examples were less open › Cultural Considerations factor ultimately decided to be too subjective to assess and was moved to ‘other considerations’ in the revised framework Reference to the Bible in “Classification of Life on Planet Earth” reading from TU Delft Public Hygiene and Epidemiology course 18 Revised framework › Digital Distribution changed to ‘Discoverability’ › Usability and Cultural Considerations moved to ‘Other Considerations’ section › Factors divided into two umbrella categories › Technical Factors › Pedagogical Factors 19 Lingering questions: Harvestability › ‘Harvestability’ an additional factor or consideration? › MIT allowed ‘full’ download of the course content - except video files. Those must be downloaded individually, as only transcripts & closed captions are provided in archive Screenshot of MIT’s “Download Course Materials” Function 20 Lingering questions: Openness of Video › Openness of video or audio formats still an open question › Should File Format address just the type of format provided (MP4, MP3, MVK, MOV, AAC, etc. › Or, should it also take into account editability - i.e. availability of unedited footage or audio 21 Lingering questions: Volume of Content › Framework focuses on openness, not quality › Volume of content also escapes evaluation › Course can be ‘open’ but have little content › E.g. MIT Introduction to Art History course had list of works for each week, but not lecture Screenshot of “List of Works” from MIT Introduction to Art History course 22 Conclusions: Where can we focus? Copyright › File format: Multiple editable formats as long as they’re commonplace › Eg. .docx., .xlsx, .pptx, .txt, etc. › Language: Can be addressed by little things › Eg. Glossary › Complete translation labour intensive and not always necessary › Materials: Use open academic articles instead of paid textbooks/closed articles where possible File Format Discoverability Language Assessment Accessibility Material Costs 23 Future research Comprehensive assessment of large OCW sample using revised framework Further exploration of OCW harvestability › Ability to download course once critical to geographic locations with limited bandwidth Better understand how instructors locate and adapt OCW to their own context Photo by Javier Allegue Barros on Unsplash 24 THANKS! Erik Christiansen ● ● Email: echristiansen@mtroyal.ca Website: erikchristiansen.net Michael McNally ● ● Email: mmcnally@ualberta.ca Website: https://apps.ualberta.ca/directory/person/mmcnally#Overview 25 Sources ● Abeywardena, I. S., Tham, C. Yoong, and Raviraja, S. (2012). Conceptual framework for parametrically measuring the desirability of open educational resources using D-index. International Review of Research in Open and Distance Learning, 13(2): 59-76. ● Cronin, Catherine. (2018). Openness and Praxis: A Situated Study of Academic Staff Meaning-Making and Decision-Making with Respect to Openness and Use of Open Educational Practices in Higher Education. Ph.D. Dissertation. https://aran.library.nuigalway.ie/handle/10379/7276 ● Economides, A. Anastasios, and Perifanou, Maria A. (2008). The many faces of openness in education. Proceedings of EDULEARN18 Conference. 2-4 Jul. 2018, Palma, Mallorca Spain, 3694-3703. http://smile.uom.gr/publications/conf/2018-EDULEARN%20-%20The%20many%20faces%20of%20Openness%20in%20Education.pdf ● Gurell, Seth Michael (2012). Measuring the technical difficulty in resusing open educational resources with the ALMS analysis framework. Ph.D. Dissertation, Brigham Young University. https://scholarsarchive.byu.edu/etd/3472/ ● Hilton III, John, Wiley, David, Stein, Jared, and Johnson, Aaron. (2010). The four ‘R’s of openness and the ALMS analysis: Frameworks for open educational resources. Open Learning: The Journal of Open, Distance and e-Learning, 25(1): 37-44. ● Hodgkinson-Williams, Cheryl & Gray, Eve. (2009). Degrees of openness: The emergence of open educational resources at the University of Cape Town. International Journal of Education and Development using Information and Communication Technology, 5(5): 101-116. ● McNally, M.B., & Christiansen, E.G. (2019). Open enough? Eight factors to consider when transitioning from closed to open resources and courses: A conceptual framework. First Monday 24(6). https://doi.org/10.5210/fm.v24i6.9180 ● Peter, Sandra and Deimann, Markus. (2013). On the role of openness in education: A historical reconstruction. Open Praxis, 5(1): 7-14. DOI: http://dx.doi.org/10.5944/openpraxis.5.1.23 ● Pomerantz, Jeffery, and Peek, Robin. (2016). Fifty shades of open. First Monday, 21(5). DOI: https://doi.org/10.5210/fm.v21i5.6360 26 Courses Examined MIT OCW ● Public Transportation Systems (Spring 2017) ● Equity and Inclusion: Local Policy Driven Strategies for Economic Development and the Just City (Spring 2019) ● Introduction to Art History (Fall 2018) ● Innovation Systems for Science, Technology, Energy, Manufacturing, and Health (Spring 2017) ● Biological Chemistry II (Spring 2016) TU Delft OCW ● Public Hygiene and Epidemiology ● Hydrology of Catchments, Rivers and Deltas ● System Validation ● Structured Electronic Design ● Drinking Water Treatment 2 27