FORGE Course

 

Full Height |  Two columns |  Parts | 

Introduction

Learning Analytics can be described as the “measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs”. The field of Learning Analytics is essentially a “bricolage field, incorporating methods and techniques from a broad range of feeder fields: social network analysis (SNA), machine learning, statistics, intelligent tutors, learning sciences, and others”.


Learning Analytics applies techniques from information science, sociology, psychology, statistics, machine learning, and data mining to analyze data collected during education administration and services, teaching, and learning. Learning Analytics creates applications that directly influence educational practice. For example, the OU Analyse project deploys machine-learning techniques for the early identification of students at risk of failing a course. Additionally, OU Analyse features a personalised Activity Recommender advising students how to improve their performance in the course.


Movie 1: Unpacking the core question for Learning Analytics. In this EdMedia 2014 keynote speech, Prof Simon Buckingham Shum discusses the main aspects of Learning Analytics.

Deployment in FORGE

With Learning Analytics it is possible to obtain valuable information about how learners interact with the FORGE courseware, in addition to their own judgments provided via questionnaires. In particular, we are collecting data generated from recording the interactions of learners with the FORGE widgets. We are tracking learner activities, which consist of interactions between a subject (learner), an object (FORGE widget) and are bounded with a verb (action performed). We are using the Tin Can API (also known as xAPI) to express and exchange statements about learner activities, as well as the open source Learning Locker LRS (Learning Record Store) to store and visualise the learner activities.


Figure 1 depicts the widget-based architecture adopted in FORGE. The FORGE widgets use LTI 2.0 for their integration within a Learning Management System (LMS). The FIRE Adapters function as a middleware between the FORGE widgets and the FIRE facilities (testbeds), while the FORGEBox layer offers a seamless experience while learners are performing a course, reading content and interacting with FIRE facilities. All the interactions performed by users on the course content and the widgets are recorded and stored in the Learning Locker LRS using the xAPI.

Figure 1: The widget-based FORGE architecture for Learning Analytics.

Learner activities on the FORGE widgets typically include the initialisation of an experiment, setting the parameters of the experiment and, finally, completing the experiment. Therefore, the learner activities captured by the FORGE widgets use the following types of xAPI verbs:

  • Initialized: Formally indicates the beginning of analytics tracking, triggered by a learner “viewing” a web page or widget. It contains the (anonymised) learner id and the exercise/widget that was initialized.

  • Interacted: Triggered when an experiment is started by the learner, containing the learner id, the exercise and possible parameters chosen by the learner. These parameters are stored in serialized JSON form using the result object, as defined by the xAPI.

  • Completed: The final verb, signalling completion of an exercise by the learner. We can also include the duration that a learner took to perform the experiment, formatted using the ISO 8601 duration syntax following the xAPI specifications.


More specialised learner activities are also recorded by the FORGE widgets depending on the functionalities offered by each widget. For example, the PT Anywhere widget that offers a network simulation environment records the following types of activities, reusing already defined vocabulary:

  • Device creation, update and removal: We use the verbs “create”, “delete” and “update" from "http://activitystrea.ms/schema/1.0/".

  • Link creation and removal (i.e., connecting and disconnecting two devices): The link creation and removal is expressed as a user creating a link that has its two endpoints defined as contextual information. Another alternative could have been to use non-existing connect/disconnect verbs to express that a user connects a device to another one (the latter should have been added as contextual information). However, we chose the first alternative because it reuses already existing verbs.


These statements are collected in the Learning Locker, which features a simple but effective dashboard, giving a quick overview of the activities over time, as well as the most active users and activities, as shown in Figure 2.


Figure 2: A screenshot of the FORGE LRS visualising learner activities.


FORGE provides learners with Learning Analytics dashboards in order to raise their awareness of their learning activities by providing an overview of their progress or social structures in the course context. Learners are offered with detailed records of their learning activities, thus being able to monitor their progress and compare it with the progress of their fellow learners. Additionally, the Learning Analytics dashboards targeted to educators provide an in-depth overview about the activities taking place within their courses, thus making the educators aware of how their courses and experimentation facilities are being used by their students.


In order to improve the ways we facilitate awareness and reflection for learners and educators, we are developing further ways of analysing and visualising the captured Learning Analytics data. Our goal is to help educators better understand the use of experimentation facilities by their students, as well as to allow learners to compare their use of the experimentation facilities with that of other learners. Towards this goal, we are developing graph models in order to visualise the different sequences of steps carried out by learners when conducting an experiment via the FORGE widgets. The following widget displays a model of the different sessions recorded by the PT Anywhere widget. This model is customised by the learner or the educator, who specifies the different levels to visualise, i.e. the number of steps or actions to be displayed. In this particular model, the different states for each level apply to a network device, which is part of a network simulation experiment, and refer to its creation (ADD), removal (DEL), update (UPD), connection (CONN) and disconnection (DISCONN). Additionally, a NOOP state is used to represent the lack of action in sessions with fewer actions recorded than levels shown.


Models such as the one featured in the widget below, allow educators to get a more detailed view of how learners conduct experiments using the FORGE widgets. Learners can also use these models to replay their sequence of interactions with the FORGE widgets, as well as view the sequences of interactions of other learners. On top of providing awareness, these models also enable learners to reflect on their learning process, for example by being able to compare the sequences of interactions of other learners with theirs, as well as by comparing their experimentation results with those of their peers. Additionally, educators can reflect on the design of the experimentation facilities and the associated learning materials by studying usage patterns that can reveal common difficulties that learners have in conducting experiments. Educators can also provide suggested sequences of interactions to their students as a means of scaffolding their experimentation tasks.

PT Anywhere user interactions

PT Anywhere user interactions 16/08/2016

Code of use

The following code of use has been adapted from The Open University’s Policy on Ethical use of Student Data for Learning Analytics.

Rationale

FORGE courses and tools collect and analyse student data as a means of providing information relating to student support and retention. 

In the context of the FORGE courses and tools, learning analytics is the use of raw and analysed student data proactively identify interventions which aim to support students in achieving their study goals. Such interventions may be designed to support individual students and/or the entire cohort.

Different organizations are contributing to FORGE with their own educational materials. Some of these organizations (like those participating in the Open Call) are external to the FORGE consortium. Other organizations might reuse existing materials with their own students. Therefore, there is a need  to establish common guiding principles which help provide a clear framework for the ethical application of learning analytics.

Problem statement

All data captured as a result of the interaction with the student has the potential to provide evidence for learning analytics. Data will, however, only be used for learning analytics where there is likely to be an expected benefit (which will be evaluated) to students’ learning.

The techniques used in learning analytics are based on standard statistical methods, but typically involve the development of complex models, the full working of which will only be apparent to those familiar with the data and with the statistical methods employed. It is likely, however, that users will want to understand how the models produce the outcomes which they then deploy. Students will want to understand why they have been selected for an intervention and, in some cases, may want to challenge the basis for their selection. A potential conflict exists therefore between creating models which provide the most reliable outcomes and those which work in ways that can be made transparent to users and subjects.

Learning analytics can be applied to individual students as well as to defined groups of students (as a result of identifying a student via combinations of characteristics and/or study behaviours), and to whole cohorts of students (as a result of amending the assessment regime on a module following observed behaviours and/or results, for example). The policy and principles created apply in all cases.

Any use made of data regarding individual students must be compliant with the Data Protection principles and policies of the institution or institutions running each course.

Definitions

The following definitions are intended to provide clarity about terms used throughout the Policy.

  1. Learning analytics has been defined as the collection and analysis of data generated during the learning process in order to improve the quality of learning and teaching.
  2. The term associate partner refers to educational institutions, research institutions, universities, individuals or small/medium/large enterprises adapting or using the FORGE tools for teaching purposes.
  3. The term student refers to individuals registered to study on a module or qualification offered by an associate partner. This also includes informal learners that study the open educational resources offered by the FORGE project.
  4. The term cohort may refer to, for example, all students linked to a qualification or students registered on a specific module-presentation.
  5. An intervention derived from a learning analytics approach may refer to information, advice and guidance directed from an associate partner to one or more students. Learning analytics at a module or qualification level may be used to inform changes to teaching and learning design.
  6. Data used for learning analytics typically falls into one of two categories: that captured at registration or at later points as a result of the student supplying information to the associate partner (typically labelled as Student characteristic data), and that derived from ways in which the student engages with FORGE tools as a result of their ongoing study (typically summarised as Study behaviour data). Access to this data is governed by existing policies such as the Data Protection Policy.
  7. Sensitive data: The Data Protection Policies define sensitive data and generally require an individual’s permission to collect and use sensitive data for a specified purpose. The items of sensitive data that FORGE does not collect are:
  • racial or ethnic origin
  • religious or similar beliefs
  • disability and other health matters
  • sexual life
  • political opinions
  • offences (including alleged offences)
  • criminal proceedings, outcomes and sentences
  • membership of trade unions
  1. In the context of this policy, informed consent refers to the process whereby the  student is made aware of the purposes to which some or all of their data may be used  for learning analytics and provides consent. Informed consent applies at the point of reservation or registration on to a module or qualification. Requests to students to participate in educational research will follow existing associate partner practices.

Scope

The FORGE associate partners use and apply information strategically (through specified indicators) to retain students and progress them to complete their study goals. This is done in two levels:

  • at a macro level to aggregate information about the student learning experience at an institutional level to inform strategic priorities that will improve student retention and progression.
  • at a micro level to use analytics to drive short, medium and long-term interventions.

In the future, the use of learning analytics may be extended to personalised learning paths, adaptive learning, personalised feedback, visualisations of study journey, intelligent e-tutoring, intelligent peer support, etc. Furthermore, new technological innovations might allow for more targeted, measured approaches.

In scope

Categories of data that might be captured by the associated partners as part of its interaction with students and potentially available as individual or combined data sets for use in learning analytics:

  • personal information provided by the student at registration
  • the student’s study record held by the associate partner
  • sensitive information that the associate partner has consent to use.
  • details of contacts between the enquirer or student and the associated partner
  • interactive content generated by enquirers or students; for example: completing diagnostic tests, student responses to surveys and research (subject to existing restrictions and approvals), etc.
  • system-generated data such as the date and frequency of accessing VLE pages; data derived by the associate partner from other data, for instance, whether a student falls into a widening participation category
  • data held or generated internally in combination with data provided by third parties may be used by the associate partner to tailor support, where there is agreement to do so from the third party concerned. For example, another associate partner which has used the same educational materials might agree to aggregate their records to compare student results and to improve these materials. Student data supplied to third parties is subject to existing policies such as the Data Protection Policy
  • anonymised data from external sites, e.g. social networking sites not owned by the associate partner, where this is used to generate information on the cohort rather than the individual student. For example, where this forms part of an activity within a module
  • miscellaneous sources of data, for example, forum posts could be anonymised and analysed to shape module design
  • data on student complaints

Out of scope

In its adoption of a learning analytics approach to provide student support, the associate partners do not intend to use the following types of data. This list is subject to review.

  • Data that identifies individuals created on external sites, e.g. social networking sites not owned by associate partners, third party sites where there is no permission to employ shared information, etc.
  • Sensitive information on religious belief and sexual life will not be used as part of the analytical models. Should any other sensitive data items be required for learning analytics, consent will have to be obtained by a suitable means, such as through changes to the Data Protection Policy. Any combinations of data or derived data that may contravene an individual’s right to respect for their private and family life will not be used.

Ethical issues relating to the use of student data for academic research

Applications to use student data for the purposes of research will need to be made in accordance with the standard processes in place currently in each associate partner (e.g., ethics committees). Bodies considering applications for research using learning analytics should assess if the projects comply with this policy. The bodies, within the remit of their own terms of reference, may approve research proposals that test the boundaries of this policy. If the outcomes of that research may then be applied to operationally targeting individuals or groups of students, further alignment with this policy will be required.

Policy statement

This policy aims to set out how associate partners should use student data in an ethical way in order to shape the student support provided. The document and accompanying guidelines are not regulatory in nature but are intended to inform and guide the ethical use of student data.

The policy is based around eight key principles discussed in more detail in the policy statement below.

  • Principle 1: Students should not be wholly defined by their visible data or our interpretation of that data.
  • Principle 2: The purpose and the boundaries regarding the use of learning analytics should be well defined and visible.
  • Principle 3: The associate partners are transparent regarding data collection, and will provide students with the opportunity to update their own data and consent agreements at regular intervals.
  • Principle 4: Students should be engaged as active agents in the implementation of learning analytics (e.g. informed consent, personalised learning paths, interventions).
  • Principle 5: Modelling and interventions based on analysis of data should be sound and free from bias.

Each of the above principles is linked to particular aspects of learning analytics.

Purposes and boundaries

Principles 3 and 4 make clearer why the associate partners adopt learning analytics as one of many means of providing effective and targeted student support whilst recognising that students, as real and diverse individuals, rather than data or information, drive appropriate student support.

Principle 1: Students should not be wholly defined by their visible data or our interpretation of that data.

  • Analysis based on the characteristics of individual students at the start of their study must not be used to limit the associate partners’ or the students’ expectations of what they can achieve.
  • Learning analytics will generate data and insights which enable us to provide targeted and specific support to student groups with shared characteristics and/or behaviours. For example, these may be students who fall within our widening access priority groups.
  • Predictive analytics reflect what has happened in the past, not the future. In their calculation of error rates, it is accepted that there will always be individuals whose behaviours do not follow the typical pattern.
  • We should guard against stereotyping. Students who do not fall within any priority group may encounter difficulties during their study which become apparent as a result of learning analytics data, and subsequently benefit from targeted interventions.
  • Caution needs to be exercised in the interpretation of data for a variety of reasons and guidance provided to staff will aim to support this. For example, individual members of staff may not have access to the full data set that is available to the associate partner and may have an incomplete view of the student and their experience.
  • The primary purpose of our approach to learning analytics is to support students in achieving their study goals: thus consideration can be given to forms of additional communication or support offered in line with our ‘open’ mission to support the diversity of student need. Services and support that are available potentially to all students will continue to be provided.

Principle 2: The purpose and the boundaries regarding the use of learning analytics should be well defined and visible.

  • The primary purpose of learning analytics activity must be to identify ways of effectively supporting students to achieve their declared study goals.
  • The purposes of using learning analytics within FORGE are to:
    1. identify aspects of a student’s record and engagement with learning activities which may enable us to match services more closely to need and to understand how we may do this effectively
    2. support the further development of our curriculum, policies and business processes including the delivery of Information, Advice and Guidance services to enquirers and students
    3. improve the likelihood that a student will achieve his or her stated learning outcomes.

Engaging students in the use of their data

Principles 3 and 4 reflect the shared responsibility of both the student and the associate partner for student learning.

Principle 3: The associate partners are transparent regarding data collection, and will provide students with the opportunity to update their own data and consent agreements at regular intervals.

  • Accurate and representative student data should not be the sole responsibility of the associate partner. Students also have a duty to accurately maintain their personal data and to inform the associate partner of any changes which might impact on their studies. The associate partner must provide students with the opportunity to periodically update their records.
  • This principle highlights the importance of a clear plan to communicate with students about our approach to learning analytics:
    1. in order that students understand our approach and feel reassured that data is used responsibly
    2. where we can share our interpretation of data with students we will do so unless there are good pedagogical reasons to do otherwise. It may, for instance, become good practice to highlight for students those avoidable factors such as late registration which analytics identify as increasing the likelihood that a student will fail to complete a module
    3. to support students in making informed decisions about their studies
    4. to reassure that this is about enhancing support services in order to improve qualification completion which will benefit all students.
  • Data may not be reliable, nor reflect a student’s current status, if it is not up to date. Students must have opportunities to check and update their personal data, including their consent to its use, at clearly defined intervals.

Principle 4: Students should be engaged as active agents in the implementation of learning analytics (e.g. informed consent, personalised learning paths, interventions).

  • Students should be actively involved in helping the associate partner to design and shape interventions that will support them.
  • It is essential to engage students with the development of our approach to learning analytics, because this is likely to enable us to:
    1. ensure that students understand their responsibility for keeping personal information up to date and can give informed consent
    2. achieve a more accurate interpretation of data relating to student behaviours
    3. improve our understanding of what forms of intervention and support are most appropriate
    4. know how to communicate with students in general and individually about our approach
    5. understand how to tailor a student’s learning journey to meet their needs, potentially as a personalised learning path
    6. produce outcomes that students will find useful and be able to respond positively to; this might include a decision to continue or discontinue with their studies.

Ensuring that data is used wisely

The final principle which support the policy relate to the need to ensure that any interpretation or manipulation of data to extract meaning is based on sound technique which is subject to expert peer review and, if necessary, through advice and mentoring by those more experienced in techniques of quantitative data analysis.

Principle 5: Modelling and interventions based on analysis of data should be sound and free from bias.

  • A set of best practice principles must be established for the development, maintenance, interpretation and review of the statistical models used for learning analytics to enable periodic internal audits.
  • All information collected by the associate partners is potentially available for the purposes of learning analytics, providing its use is consistent with these principles. Information selected for use in learning analytics should demonstrate its value in delivering the agreed outcomes. Analytical models should aim to be transparent, such that their method of working can be described to staff and students, they are based on standard statistical techniques, and they can be tested and audited to provide assurances that they use data which meet quality criteria, correctly apply that data and produce results that reach an agreed level of accuracy.