Crowdmark

Crowdmark

Instructors' Insights on Crowdmark

This page shares instructors' thoughts on using Crowdmark, as well as alternatives: Quizzes in Quercus and Quercus Assignments. While maintained by Jennifer Murdock, Professor, Teaching Stream, Department of Economics, it incorporates contributions of many within Economics and U of T more broadly. For recordings and slides of workshops on assessment design, see "Assessing Student Work and Providing Feedback in Quercus-Part I,"  "Assessing Student Work and Providing Feedback in Quercus-Part II," and "Open Book Assessment Options" on https://q.utoronto.ca/courses/46670/pages/quercus-videos-and-workshop-recordings. For Crowdmark support, please contact support@crowdmark.com or your divisional support (scroll down that page). I avoid overlap with the Crowdmark in A&S page. This page contains candid reports from the field: it is like an informal chat with real instructors about their experiences. Finally, this page evolves in real time as we continue to gain experience, correct inaccuracies, develop better best practices, and learn about new features from Crowdmark as they are released. The Road Map below explains how this site is organized.

Last updated: March 24, 2021

Road Map

For those new to Crowdmark, start with the section below What is Crowdmark?. (Each section ends with a link jump back to this road map.) Next, to put this site in context, all readers should see A short history of Crowdmark in Economics with samples. This is followed by Overall, what are the options for an online test/exam?, which is a pressing question for many of us planning for 2020/21. You may also see the session "Open Book Assessment Options" on the CTSI Online/Remote Teaching Webinar Series.

Important companion resources to this page further explore the major alternatives to Crowdmark, which include Quizzes in Quercus and Quercus Assignments. All online assessment alternatives have significant strengths and significant weaknesses. There is no magic solution. I use a mix.

Looking back and looking ahead to when we can have in-person, invigilated, paper-based tests and exams, see Why use Crowdmark over a 100% paper-based approach?. I find it useful to consider adopting technologies during emergency remote teaching that will continue to be useful in more normal times. I can say to most colleagues that if you are giving a paper-based test or final examination, you should be using Crowdmark. (The only exception to that strongly-worded advice is for courses where such assessments take the form of long essays.)

For those wishing to start using Crowdmark, see How do I access Crowdmark and link my students?. It includes a subsection on how to Sync your classlist in Crowdmark with Quercus.

The biggest section is Crowdmark: Assigned Assessments (remote). This goes over the most likely use case for 2020/21, which is remote assessments. It is detailed and technical. It includes multiple subsections. The first subsection addresses common questions around Late submissions and penalties Assigned Assessments (remote). The next is very important: Scanning for Assigned Assessments (remote). It is critical that you understand the scanning requirements and best practices to make sure that your assessment runs smoothly and does not cause undo stress for your students. This includes a link to the page Scanning for Crowdmark: Template, which helps you explain to your students what they will need to do. Ignore these at your own peril. The next subsection -- Choosing a completion window and more than one sitting for Assigned Assessments (remote) -- helps you navigate the choices about when students will need to complete your online assessment. Multiple Versions for Assigned Assessments (remote) explains how you can have more than one version of your questions in Crowdmark. (If you want lots of versions, you may consider Quizzes in Quercus.) The next subsections are special cases: Group submissions (remote) and Combining Quercus Quizzes and Crowdmark (remote). There is also a link to Exit survey: April 2020 online final assessment with Crowdmark, which shares some student feedback on remote Crowdmark assessments.

The section Crowdmark: Administered Assessments (in-person) goes over the logistics for our past and future uses of Crowdmark for in-person, invigilated assessments.

The section Grading Tips gives some advice on marking in Crowdmark. See also the section on Academic Integrity. The section on Accessibility and accommodations for Crowdmark covers both instances of Crowdmark (in person and remote).

Jump back up to the Road Map

What is Crowdmark?

Crowdmark Links to an external site. is an online marking platform integrated with Quercus. To quickly learn more about what it is, visit Crowdmark's website Links to an external site.. If you're new to it, start with their pages What can Crowdmark do? Links to an external site. and Introduction to Crowdmark Links to an external site.. For a short video introduction, see https://crowdmark.com/ Links to an external site.. I also recommend Crowdmark's help page https://crowdmark.com/help/ Links to an external site.. For how to use Crowdmark for a Quercus course, see How Do I Access Crowdmark and Link My Students? and the designated support page for Canvas users Links to an external site. (Quercus is U of T’s name for Canvas).

There are two types of Crowdmark assessments, which differ in who is responsible for administering the assessment. For both, the grading interface -- an important strength of Crowdmark -- is nearly identical.

  • Crowdmark, Assigned Assessments (remote)
    • Students are responsible for checking their U of T e-mail, viewing the questions in their web browser, writing their answers on plain paper, stopping writing early enough to leave time for scanning/uploading, scanning, and uploading to Crowdmark
  • Crowdmark, Administered Assessments (in-person)
    • The instructional team (professor, TAs, etc.) is responsible for specialized printing, handing each student the physical test papers in the exam hall, collecting these papers when time is up, scanning, and uploading

Obviously, making students responsible for so much of the administration of an assessment creates unique challenges that do not exist with Crowdmark, Administered Assessments. Hence, if you are new to Crowdmark in 2020/21, you are starting with the more challenging type of assessment to manage. The Crowdmark interface itself is easy to learn and use and TAs require minimal training in our experience in Economics. However, having students responsible for scanning and uploading adds complexity and issues. (These issues are addressed in Scanning for Assigned Assessments (remote).) However, for many courses in Economics, despite this downside, Crowdmark is the best option in certain circumstances.

For anyone teaching an A&S course in Fall 2020 or a Y course, we already know as of June 9, 2020 that you cannot require all students to write in-person tests or exams (see A&S FAQs for students), which means that for Crowdmark your focus will most likely be on Assigned Assessments for 2020/21.

Jump back up to the Road Map

A short history of Crowdmark in Economics with samples

Starting in 2014/15, a very small number of courses in Economics tried Crowdmark to manage term work electronically. We ran an A&S sponsored pilot in Summer 2015 that included extensive surveying of students, TAs, and instructors. Use expanded. In 2016/17, we began also using it for A&S final examinations: all constituents complained bitterly about going back to the old ways for the April 2015 and April 2016 final exams after using Crowdmark for term work. It has expanded organically every year as more faculty members, sessional instructors, and course instructors adopt it, including at UTM. Our students love it. Our TAs really like it. My favorite past quote from a colleague is “You can take Crowdmark from my cold, dead hands.” Most recently (June 2020), Nate Vellekoop -- who joined in 2019/20, teaches our huge ECO101H and ECO102H courses, and was a first-time Crowdmark user in Fall 2019 -- states "The learning curve at this point is not steep at all (I love it) and I really like how versatile it is (one can include pictures/graphs both inline and as attachments), and how intuitive the interface."

I have been using Crowdmark, Administered Assessments since Summer 2015: it is the single most important piece of educational technology I have adopted and it is continues to prompt evolution my assessment strategies. (However, remember that for 2020/21 many of us will not be able to use Crowdmark, Administered Assessments, which require in-person assessments.) For example, in 2019/20 with a large course -- three of five sections of ECO220Y1Y with about 500 students and this Download syllabus

-- I used Crowdmark, Administered Assessments for Test #4. Here is the Download question paper for Test #4 and the separate Download Supplement. After running it through Crowdmark, here is Download one unique paper and here are the Download unique papers for the class. (While each student gets a unique booklet by the QR codes in the top right, all questions are the same and you can pass out the papers in the exam room in any order: you don't have to give a specific booklet number to a specific student.)

A complete example would also show you what students see. I did not ask permission to share students' work for the March 2020 Test #4. However, I have permission to share samples from 2018/19. (Note that the links from Crowdmark are automatically anonymous: they exclude the first page containing students' identifying information.) Here is a sample of an A+ test paper, from Test #1, October 5, 2018, in ECO220Y: https://app.crowdmark.com/score/daff7a2d-47f3-49d7-93c5-1fd82b5f89f4 Links to an external site. for the same test, a B- and a C- paper: https://app.crowdmark.com/score/caab62e9-23d3-4004-b47f-1be2eff782ae Links to an external site. and https://app.crowdmark.com/score/55a6c5ce-c6e8-44fa-9d73-72bbe9ed95ae Links to an external site., respectively. Note that the preamble (top of these Crowdmark papers) provides access to that test, supplement, and solutions.

Up until March 2020, a majority of our experience in Economics with Crowdmark has been with Administered Assessments (in person), but some of us were also using Assigned Assessments (remote) for some term work. Before the crisis, I used Crowdmark, Assigned Assessments (for the first time) in 2019/20. After each of the four term tests (that were not impacted by the crisis), including the Test #4 example above, students submitted a test reflection assignment within a few days that allowed them to take another shot, reasonable collaboration permitted, at the most challenging question for them. It is along the lines of a two-stage test for those familiar with that term. For example, here is the Download Test #4 reflection assignment

(that students complete after the regular invigilated Download Test #4, which they hand in to us, but get to keep the Download Supplement, which they will need to complete the reflection assignment). The Download solutions are only posted after the due date. The Download syllabus defines reasonable collaboration.

Starting in mid-March 2020, in Economics we have gained more experience with Crowdmark, Administered Assessments, including in our very large introductory courses, upper-level electives, and graduate courses. This page shares what we have learned thus far.

In Fall 2020, for large courses with relatively tight completion windows, we did have 3 major instances in ECO (with the most recent being December 4, 2020), as well as smaller instances, of Crowdmark being unable to handle the volume of uploads and being down for periods ranging from minutes to hours. This caused a tremendous amount of extra work for instructors and TAs and a tremendous amount of stress on everyone, including students. However, starting in Winter 2021, we have not been experiencing performance issues.

Jump back up to the Road Map

Overall, what are the options for an online test/exam?

All online testing options have significant limitations and issues compared to regular invigilated tests and final examinations. All require considerable planning and effort. All are susceptible to impersonation (i.e. a student paying someone else do an assessment for them). This page discusses strategies to help. However, it would be wise to diversify your assessment approaches if your course had been structured around a big midterm test and a big final exam, unless you have developed strategies for online invigilation (a little more on that below). Your options:

  • Quercus Quizzes
  • Quercus Assignments
  • Crowdmark, Assigned Assessments (remote)

The CTSI page titled Online Tests and Exams compares and contrasts Quercus Quizzes and Quercus Assignments, but there is nothing about Crowdmark. A useful guide from Engineering at U of T -- Which Assessment Tool is Right for your Assessment? Links to an external site. -- includes all three (although not everything in the summary table as of June 15, 2020 is correct). To complement this page, I've also prepared Quizzes in Quercus and Quercus Assignments.

You may also find helpful the session "Open Book Assessment Options" on CTSI Online/Remote Teaching Webinar Series.

Online we’re concerned about cheating and about robustness (being sure things run smoothly). The three approaches -- Quercus Quizzes, Quercus Assignments, and Crowdmark -- have proven to be quite robust provided that you understand them well, keep your design as simple as possible (but not too simple), and can clearly communicate with students. Crowdmark requires the most work to make it robust (but confers other advantages). Further, with all online graded work, it helps tremendously to have students (and TAs and instructors) working within the same technologies on lower stakes assessments before getting to higher stakes assessments. In other words, you're strongly encouraged to consider things like weekly homework or frequent quizzes in addition to term tests and final assessments.

While all are susceptible to impersonation (i.e. a student paying someone else do an assessment for them), if your first concern is collaboration among students, there are ways to address that with multiple versions of questions, which are pretty-straightforward for two of three approaches: Quercus Quizzes and Crowdmark, but not Quercus Assignments.

Since March 2020, instructors and TAs have been entirely responsible for providing academic accommodations, which are normally provided by Accommodated Testing Services. This creates its own set of administrative and pedagogical challenges. However, accommodations can be made for all three options, including Crowdmark: see Accessibility and accommodations for Crowdmark.

While this page is focused on Crowdmark, it is hard to discuss it without considering what your alternatives are. My own experience has primarily been with Quercus Quizzes and Crowdmark (I use both) but another important option (which I also plan to use in 2020/21) is Quercus Assignments. Hence, I've created pages each for Quizzes in Quercus and Quercus Assignments.

Jump back up to the Road Map

Why use Crowdmark over a 100% paper-based approach?

There are many distinct advantages of Crowdmark that add up to a big advantage over the traditional 100% paper-based approach for in-person assessments. Of course COVID-19 changes our reference point BUT if you are considering adopting new technologies now, ideally they would be ones that stay relevant in the longer run.

  • Improved management of TA team during marking process
    • No passing of papers or need to mark in a secure physical location and no possibility of lost papers
    • Real time knowledge of everyone’s marking progress and exact marking time by TA and by question
    • Easy to monitor comments being used and marks being given: can fix any issues promptly
    • Improved communication: TA can ask about a paper via e-mail and instructor can see it
    • Marking is entirely anonymous: graders cannot access students’ identifying information
    • Reduces clerical errors in marking (no need for human addition, marks entry, copying to front, etc.)
  • Allow comments on students’ work to be efficiently reused and/or revised
    • This leads to students getting more and better comments on their work, even in large classes
  • Improved marking experience and marking quality even when instructor marks everything themselves
  • Instant return of students’ work when marking is finished
    • Do not need to delay until next class or tutorial
    • Do not need to waste class or tutorial time with administrative tasks
  • All students have ready, continuing, and secure access to their marked work organized by course
    • No uncollected papers and no misplaced work
    • No temptation to change answer and submit it for remark and no suspicion around remark requests
    • Students can write in pencil and use an eraser during invigilated work (no need to require using a pen
    • Improved confidence in accuracy of recorded marks and appreciation of increased transparency
    • Remark requests can be done electronically: no need for passing papers back-and-forth

As you gain more experience with Crowdmark and try more things, additional benefits can accrue:

  • In my 800+ student course (all sections, common exam), I’ve been able to abandon bubble forms, which had been ~30 percent of course work
    • New (and better) types of closed-form question now possible with this more flexible technology
    • Can use a higher fraction of open-ended questions
  • In my small 400-level course (30 – 35 students), Crowdmark greatly facilitated meaningful class participation every week that allowed even shy students to get feedback on their preparation (readings) and understanding of classmates’ presentations
  • Chunking invigilated writing to allow TAs to specialize and expertly and consistently mark students’ work
    • The Economics Department’s participation in the A&S WIT program (Writing-Integrated Teaching) predates Crowdmark but Crowdmark has allowed us more options (but Crowdmark is NOT integrated with Turnitin, but that's not needed for invigilated writing)
  • Opens door for promising pedagogical approaches: we keep thinking of new things
    • Two-staged tests
    • More flexible class participation and without cost to students (unlike iClicker or TopHat)
  • Managing a TA budget
    • Can divide marking duties far more than would make sense if TAs had to share physical papers and flip to a particular page
      • More specialization leads to better and more efficient marking
      • Can productively use TAs even if they have a very small amount of marking time
    • Can keep track of actual marking hours to ensure equity across TAs by end of course
  • Learn how prompts and questions translate into marking time
    • For example, asking students questions that require them to write a short paragraph take far less time to mark than I would have thought and can be quicker than more analytical questions
  • When writing letters of reference for former students, I can easily have a look at their actual work
  • Having ready and ongoing access to what your students have (and have not) learned, and your effectiveness in assessing it and giving feedback, creates a higher impact experience compared to reviewing course evaluations

Jump back up to the Road Map

How do I access Crowdmark and link my students?

Go to https://app.crowdmark.com/sign-in/utoronto Links to an external site.. Bookmark it. Once logged in (via your UTORid), you can import your course from Quercus by selecting "Quercus @ University of Toronto" from the drop down menu next to My Courses and then clicking the button "Import a course" (and then following the pop up screens). This will enrol your students.

Sync your classlist in Crowdmark with Quercus

As your class list changes when students add or drop, from the Crowdmark page for your course, you need to go to Students (left toolbar) and then click the button "Sync with Quercus" (and then follow the pop up screens). Make sure to sync before each assessment so that Crowdmark knows who are (and are not) your current students.

Note: If you have students not currently registered in your course who are writing a deferred assessment, they will not be synced with Crowdmark if they have the role of observer in your Quercus course. As a work-around, you can either enroll them as a student in Quercus (and then sync again) or see Multiple Versions for Assigned Assessments (remote).

Jump back up to the Road Map

Crowdmark: Assigned Assessments (remote)

This goes over the most likely use case for 2020/21, which is remote assessments. First, if you've not already read What is Crowdmark? and A short history of Crowdmark in Economics with samples, start there. Recall that Crowdmark, Assigned Assessments are a bit different from from Crowdmark, Administered Assessments (with the QR codes that many have used extensively for regular invigilated tests and final exams). Instead, students answer questions on ordinary plain pieces of paper and then take a picture/scan (often with their phones) and upload answers to each question. With Crowdmark, Assigned Assessments you can replicate a more traditional test or exam and the TA marking interface -- an outstanding feature of Crowdmark. Overall this is a good option is you want to ask questions that students would normally hand-write during a test/exam, which may include equations, graphs, sentences, etc. Note that this is also a possible tool for participation, homework, and other non-test/exam elements. Also, Accessibility and accommodations for Crowdmark explains how you can add accommodations (in a way that will be familiar to those using Quercus quizzes).

Very Important: Make sure to read Scanning for Assigned Assessments (remote) below. Further, add a clear page of instructions for your students to your Quercus site: see Scanning for Crowdmark: Template.

Important: A common question about Crowdmark, Assigned Assessments is whether students will get notified about your assessment before you are finished editing it, setting it up, and ready for them to start. Rest assured that students will not know anything about it until you take an active step to distribute it to them. I would recommend Distributing an assignment at a scheduled time Links to an external site.. Also, remember to Sync your classlist in Crowdmark with Quercus.

This section is detailed and technical. It includes multiple subsections. The first addresses common questions around Late submissions and penalties Assigned Assessments (remote). The next is very important: Scanning for Assigned Assessments (remote). It is critical that you understand the scanning requirements and best practices to make sure that your assessment runs smoothly and does not cause undo stress for your students. This includes a link to the page Scanning for Crowdmark: Template, which helps you explain to your students what they will need to do. Ignore these at your own peril. The next subsection -- Choosing a completion window and more than one sitting for Assigned Assessments (remote) -- helps you navigate the choices about when students will need to complete your online assessment. Multiple Versions for Assigned Assessments (remote) explains how you can have more than one version of your questions in Crowdmark. (If you want lots of versions, you may consider Quizzes in Quercus.) The next subsections are special cases: Group submissions (remote) and Combining Quercus Quizzes and Crowdmark (remote). There is also a link to Exit survey: April 2020 online final assessment with Crowdmark, which shares some student feedback on remote Crowdmark assessments.

Late submissions and penalties Assigned Assessments (remote)

A key question instructors have with any self-administered test is what happens with late submissions. Start with Crowdmark's helpful help pages: How are due dates and penalties enforced? Links to an external site. and Submitting on behalf of a student Links to an external site.. There is a key distinction between how Crowdmark handles late submissions versus Quercus Quizzes. Once the available window has expired for a Quercus Quiz students cannot submit. Instead, students can keep submitting to Crowdmark until marks are returned. That sounds crazy but it is actually a useful safeguard. First, students cannot overwrite a submission after the due time, they can only submit if they have never submitted before. Hence, if students run into technical trouble, they can still submit their work. Further, you have control over the lateness penalty at both the class level and the level of an individual student: for the later, see Accommodation options for assigned assessments Links to an external site. and scroll down to Customize lateness penalty. Both you and your students have access to their complete log file: see Verifying a student’s submission attempt Links to an external site..

See Accessibility and accommodations for Crowdmark, which allows you to give students requiring accommodations extra time so that they are NOT flagged as late.

WATCH OUT:  As of October 2020, more than one student and more than one instructor in Economics has been snagged by a late issue in Crowdmark. As stated above, students cannot RESUBMIT past their deadline. However, if they submitted once already and decide to use any remaining time to keep working (which many instructors allow) and if they click submit before their deadline but the submission is only processed after the deadline (could be a matter of seconds), they will be marked late. This sounds like only a theoretical possibility, but it has happened to at least 4 students in Economics in one week in October 2020. You can see these cases (you have access to their complete log file: see Verifying a student’s submission attempt Links to an external site.) and you CAN manually override the penalty (see Accommodation options for assigned assessments Links to an external site. and scroll down to Customize lateness penalty). These are cases where the log file will clearly show that the student has submitted everything well ahead of the deadline and was working on revisions, which should  be risk free from lateness penalties because they cannot resubmit late, but unfortunately is not risk free due to the fact that Crowdmark calculates the late penalty not based on when the student clicked submit (locking in their answers) but when that finished processing on Crowdmark's server.

For more advice on handling lateness and penalties, see Scanning for Crowdmark: Template.

Scanning for Assigned Assessments (remote)

Update for October 2020: Scanning and uploading have recently become much more efficient. I would recommend alerting your students to Scan a whiteboard, document, business card, or photo in OneDrive for iOS Links to an external site., which is a Microsoft support page that goes over how you can easily scan using the OneDrive app on your phone (available for iOS, iPadOS, and Android). Also, some folks in math have had a great experience with Office Lens Links to an external site., which is another Microsoft app. 

During 2019/20, before the COVID-19 crisis, I used Crowdmark, Assigned Assessments four times with about 500 students. (See A short history of Crowdmark in Economics with samples for details.) I did not have a single instance of a student reporting technical problems, but students were not operating under a timer: they could upload anytime within a multiple-day window. (Although, a number chose to submit very close to the deadline.) However, during timed tests and final assessments across ECO courses, in March/April 2020 and Summer 2020 there have been numerous cases of scanning/uploading issues by students. Some have been due to students working from home wireless networks all over the globe and not knowing how to scan efficiently (in some cases creating huge file sizes that surpassed their bandwidth) but a few instances have been due to Crowdmark. As of April 24, 2020 Crowdmark has resolved an issue where students would try to re-do an upload while their original upload was in progress that caused them to get stuck forever in uploading.

Here are ways to minimize uploading problems and to have robust plans for dealing with issues and minimizing stress for students and everyone:

  • Proactively communicate expectations early with your students: see Scanning for Crowdmark: Template.
  • Give students enough time to both complete the substance of your assessment and upload it. For example, add 15 minutes to the time you would normally give students in a regular invigilated test/exam.
    • Depending on your pedagogy and if it's open book, you may consider given even more time or keeping the time the same but asking fewer questions than you normally would. If you typically have students furiously writing during the entire 2 or 3 hours of your tests and exams and when time is up you sometimes have to fight the papers out of some students hands (or shout more than once pencils/pens down to stop people writing), you may consider that this will be hard to enforce in an online testing world. It is difficult to separate students that are not leaving enough time to scan and upload because they keep writing from those that are diligently following instructions but having trouble. (But, there is https://crowdmark.com/help/verifying-a-students-submission-attempt/ Links to an external site..)
  • Design your assessment to be judicious in the number of pages you ask students to scan and upload. In Summer 2020, some first time instructors made the mistake of asking students to upload 10 to 20 pages (!) in an effort to make marking easier for TAs: each subpart of a question could be marked separately. Do NOT do this. It is way too taxing on students to ask for this level of scanning and uploading within a short assessment window (e.g. 2 hours). This also may be a tempting pitfall for instructors experienced with Crowdmark, Administered Assessments. In those, it is very easy to mark each subpart (e.g. 1a), 1b), and 1c)) separately if you wish. However, with Crowdmark, Assigned Assessments, you will need to rethink your marking strategies and TA allocations given that you should minimize the number of pages to be uploaded and you will need to mark at the level of a page (and not at a finer level). With large classes (including ECO102H), uploading up to 6 pages has gone smoothly, however, I would try to keep it within 4 pages for your first time.
  • For big classes, you can consider having two parts: for example, one part in Crowdmark and one part in Quercus Quizzes.
  • If you're using Crowdmark for the first time, give students practice opportunities. One option is to have a practice test: for example, you could use some HW questions and/or old test/exam questions and let students practice submitting. In our experience in Economics, to get more than 50 percent of your class to take advantage of a practice test, you must require it (e.g. for participation points). Even better, you may consider using Crowdmark for lower-stakes assessments (e.g. weekly participation) such that your students are experts by the time they need to write higher-stakes assessments.
  • Make sure your classlist is up-to-date in Crowdmark before your assessment opens. Keep in mind that administrators sometimes put a student into your course after the add/drop period without telling you. (If you update after the assessment opens, Crowdmark will immediately send any initially missing students an e-mail invitation if you have scheduled automatic distribution but those students would have less time to finish before the deadline.) Also, for Academic Integrity, you do not want Crowdmark to e-mail copies of your assessment to students who are no longer enrolled in your course. See Sync your classlist in Crowdmark with Quercus.
  • You should clearly communicate two time limits to students: writing time and scanning and uploading time. For example, "You have 105 minutes to write Test #1 . After that, you have up to 15 minutes to scan and upload your answers to Crowdmark. If you encounter scanning or uploading problems during those 15 minutes, immediately e-mail [fill in] for help and to ensure you're not marked late. Please remember that Crowdmark keeps a log of your activity, which you can review (https://crowdmark.com/help/verifying-that-an-assignment-was-submitted/ Links to an external site.), and it should show that you're trying to promptly upload during those 15 minutes. This is like the time that we would be collecting everyone's paper during an in-person test/exam: you must STOP WRITING 15 minutes before the due time in Crowdmark."
  • Ideally the instructor or course TAs should be available while students are working and submitting. For example, you could have a drop-in room in Bb Collaborate or give students an e-mail address or call-in number. These supports much be clearly and repeatedly communicated to students so that any issues can be addressed.

Choosing a completion window and more than one sitting for Assigned Assessments (remote)

There are some important pedagogical, academic integrity, practical, and technological considerations when choosing how long of a completion window to set: ranging from a tightly-timed test to an extended take-home test. For a tightly-timed test, we know that remote learning means that our students can be anywhere in the world, which creates its own challenges for synchronous work (2pm in Toronto is 2am in China). Hence, you may consider having two sittings. As of July 27, 2020 Crowdmark introduced timers -- see Creating a Timed Assessment Links to an external site. -- which is a key feature that has always been available in Quercus Quizzes. Having the option of a timer is very helpful.

Here are some sample strategies:

  • Give a take-home test where you allow students to work on the questions for some number of hours (or days). (However, remember that Crowdmark is not compatible with Turnitin.)
  • Have a short window (e.g. 2 hours) scheduled during class meeting time. This may work well if you have a morning meeting time: morning in Toronto is reasonable for most other time zones.
  • Have a longer window (e.g. 24 hours) that overlaps with your class meeting time(s) and use a timer (e.g. 2 hours). Students can write whenever they want during that 24 hours, but once they start they have 2 hours to finish. (This is just like for Quercus Quizzes.)
  • [More complex] If you have more than one section (with different meeting times), have a short widow within each class meeting time. See Multiple Versions for Assigned Assessments (remote).
  • [More complex] You could have multiple versions and multiple sittings (e.g. two different versions from 9am-11am and two different versions from 3pm-5pm), which could help address both unwanted collaboration and time zone issues. See Multiple Versions for Assigned Assessments (remote).

Multiple Versions for Assigned Assessments (remote)

It is possible to do multiple versions of an Assigned Assessment in Crowdmark. There are two (possibly overlapping reasons) to wish to have multiple versions: (1) you have more than one sitting (e.g. 9am-11am and 3pm-5pm) and/or (2) you have more than one version of the questions (to limit unwanted collaboration). Multiple versions is a newer feature in Crowdmark and we have limited field experience as of mid-June 2020: proceed with caution. (Note that you do NOT need multiple versions to handle accommodations: instead, see Accessibility and accommodations for Crowdmark.)

To start, see Managing sections or multiple versions of an assigned assessment Links to an external site.. You have three options (where they all have issues):

  • [Use for small classes] Option 1: Manually assigned each student a version.
    • Advantage: This is conceptually straightforward and allows you to easily switch students from one version to another after setting up the assessment (should that be necessary) via the Manage students button under Students for that assessment version.
    • Disadvantage: It is tedious for large courses: you need to start typing the name of each student and select. This Download screen shot shows what that pop-up window looks like. Also, if you click away from that pop-up window before you hit the Save button, you will lose your data entry work. You also need to be careful with students that share the exact same name (you will have to double check ex post each such case). To give you an idea of the data entry time, a highly efficient person needs about 1 hour for a class of 300 students.
  • [Use for big classes] Option 2: Create a spreadsheet (.csv) that has a column to identify which version each student should get.
    • Advantage: There is no manual data entry, which is helpful for large courses.
    • Disadvantage: It is more conceptually complicated. Also, you cannot move students. Even if you change their version in the student list it will not update what version they will get for pre-existing assessments. Hence, you would have to discard any affected versions of your assessment and recreate them after updating the class list.
    • How to do it: See Creating sections in Crowdmark for assigned assessments with multiple versions.
  • [Another option for big classes] Option 3: Use UT Advanced Group tool to create sections
      • Advantage: This is done within Quercus (not within Crowdmark) and many instructors of large classes are already familiar with this. Within Crowdmark, under Course Settings, you just have to check the box "This course has sections" and, as always, remember to click the button Sync with Quercus, which is under Students for your Crowdmark course. When you create a Crowdmark assessment, it asks you about students and you can select the section of students you wish to assign that particular version of the assessment. 
      • Disadvantage: No matter how you do it, students will get at least some information on who else in the class is assigned the same version as them (because of how Quercus works). Also, the UT Advanced Group tool is confusing if you're not already familiar with it. This page is https://q.utoronto.ca/courses/46670/pages/groups helpful but you will need one-on-one IIT support to implement this solution and understand its pitfalls. (Note: As of October 6, 2020 the bug in Canvas (not just Quercus) with exporting sections to third-party platforms has been fixed, which is great.)

Regardless of which way to assign students to a section, you can use the duplicate assessment option in Crowdmark if you are using multiple versions.

Group submissions (remote)

While I have not tried this myself, students can form their own groups (can search the name and e-mail addresses of classmates) or the instructional team can manually form groups (using drag-and-drop). For more on this, see Creating a group assignment Links to an external site.. (Note: If students form their own groups, there is no way to limit the group size.)

Combining Quercus Quizzes and Crowdmark (remote)

For the April 2020 final assessment in ECO220Y1Y, we combined Quercus Quizzes and Crowdmark. Quercus Quizzes are great for multiple question versions but not great for open-ended answers. Crowdmark is great for open-ended answers but not so good for multiple question versions. Part of the online final assessment used Quercus Quizzes, where students answered a variety of machine-marked questions, including numeric answer and multiple choice. Another part had students access their open-ended questions via a Quercus Quiz (to allow randomization of the version that each student answers) but write up their answers on plain paper and upload to Crowdmark. (We are now aware of Multiple Versions for Assigned Assessments (remote), which gives another option for multiple versions, but it is still unclear what is the best approach.) Tethering Crowdmark to a Quercus Quiz does require considerable back-end coordination of the TA marking team and would likely be too complicated to be an attractive option for most courses.

However, the idea of simply having two parts: one Crowdmark part and one Quercus Quiz part is not as complicated and allows asking a greater variety of question types. In addition to doing this for the April 2020 online final assessment, we have also used a two-part approach to testing in Summer 2020. This allows students and the instructional team to take advantage of the best features of Quercus Quizzes and Crowdmark. Our summer mid-course survey reinforces what we have heard from students in April 2020: they prefer that the majority of the weight be on the Crowdmark portion because this allows showing work and partial credit and feels more like a real test/exam.

The page Exit survey: April 2020 online final assessment with Crowdmark shares students' perspectives when we combined Quercus Quizzes and Crowdmark in April 2020.

Some colleagues in Economics combined Quercus Quizzes and Crowdmark where they had students upload their scratch work to Crowdmark. This has the added benefit of deterring academic infractions.

Jump back up to the Road Map

Crowdmark: Administered Assessments (in-person)

When you again have the pleasure of giving an in-person assessment, use Administered Assessments. For these, the instructional team gives each student the paper with the questions and spaces to write their answers, collects those papers, scans those papers, marks, and returns students' work (electronically). You create a pdf file with your assessment that you upload to Crowdmark. It is ;so much easier than Assigned Assessments. Here are some tips.

  • Overall format: https://crowdmark.com/help/creating-an-assessment-template/ Links to an external site..
  • Give ample space right after each question (or part of a question) and tell students to write in PENCIL and use an ERASER.
  • Have an even total number of pages. For example, have six or eight pages (including the cover page), but not seven. Answers in the wrong spot complicate marking
  • If you have materials for students to review when answering, consider using supplement (not in Crowdmark). In other words, a separate attachment with any figures, tables, and/or excerpts that students consult when formulating their answers. This is a good idea in general as students can easily reference materials when writing their answers. Aside from making the test/exam more convenient for students, this lessens the number of Crowdmark pages (so that the pdf files to e-mail for printing are smaller and fewer pages need to be scanned after the test) and makes formatting the assessment substantially easier. As an example (from April 2019 in ECO220Y1Y), here is the Download starting version of a test and the Download Supplement.
  • Remember to Sync your classlist in Crowdmark with Quercus to make sure you have an accurate count of enrollment. (This is not as critical a step as it is for Assigned Assessments (remote).)
  • If you have a team of marking TAs, I would recommend having each specialize in a question or part of a question. Use the flexible grading features Links to an external site. introduced in December 2017. Think about how your TAs will mark when designing the assessment.
  • Crowdmark photocopy requests are submitted via usual protocols. It does require more explanation. Here is a sample photocopy request e-mail:
    • Sample e-mail with photocopy request: I am writing with a copy request for a Crowdmark test with a separate supplement for Test #5 in ECO220Y1Y, L0201, L0301, and L0401. The attached pdf file Download ECO220_Murdock_Test_5_447 booklets.pdf contains 447 unique tests where each test is eight pages long (for a total of 3,576 pages = 447*8). Double-sided, each test is four sheets. It should be stapled every 4 sheets (8 pages). However, I do not need copies of all 447 tests: only pages 1 to 3,456 of the attached file, which corresponds to test papers for 432 students (3,456 = 432*8). Also, I need 432 double-sided and stapled copies of the attached pdf file Download TT220_5_APR19_SUPPLEMENT.pdf, which is a standard (non-Crowdmark) request. I do NOT want the supplement stapled to each test paper.
  • With a traditional paper based test, TAs share test papers. With Crowdmark, Administered Assessments, TAs mark the questions (or parts of questions) they have been assigned online. This requires scanning. First, the staples need to be removed. This is efficiently done by cutting the edges off the pages. Scanning happens at the unit level. In Economics we have a TA Scanning Coordinator to facilitate and train the course TAs to use our RICOH machines efficiently and effectively.

Jump back up to the Road Map

Grading Tips

For a great summary of marking see: https://crowdmark.com/help/grading-tools/ Links to an external site.. Also, when marking in Crowdmark, notice "Shortcuts" in the top right, which is a useful quick reference. The short-cuts significantly improve marking efficiency. Also, make sure to use the comment library: https://crowdmark.com/help/using-the-comment-library/ Links to an external site..

In the top right of the "Grading Grid" in Crowdmark, you can see "Filter evaluations." Use the tags (shortcut t) to tag papers for review. This allows others to easily filter by these tags. (Note: This only works once the papers have some sort of mark on them. You cannot filter entirely unmarked papers.)

If you have a Head TA, make them a Facilitator In Crowdmark. Facilitators can do almost everything that instructors can. If you have a Head TA
(or a single TA), promoting them will enable your Head TA to help you more.

Check grading times. Crowdmark records time spent on active grading. In Crowdmark you can monitor (in real time) the time spent marking by TA: go to "Team" in Crowdmark to see marking time by TA or go to "Dashboard" to see marking time and progress by question. This enables both real time monitoring of progress, reality checks for your DDAH forms, and learning about how the question prompts you write translate into marking time.

Jump back up to the Road Map

Academic Integrity

You cannot use Turnitin with either Crowdmark or Quercus Quizzes. If your assessment is along the lines of a paper, essay, report, etc., you may seriously consider Quercus Assignments. Barring online invigilation, either by the instructional team (e.g. via Zoom) or through a paid services (e.g. ProctorU or Examity), all options, including Crowdmark and essays (via essay mills), are highly susceptible to impersonation (i.e. a student paying someone else do an assessment for them).

For discouraging mindless collaboration among people in you course (just copying each other), multiple versions works well. Multiple versions is straightforward for Quercus Quizzes, but requires more work (and is more limited) for Crowdmark: Multiple Versions for Assigned Assessments (remote). Of course it is time consuming to come up with multiple versions and check them over. Obviously your own original questions are best (not just for avoiding mass cheating enabled by insecure textbook question banks, but because they will be well aligned with your expectations and the content of your course). Of course, this is time consuming for the instructor (and experienced TAs). Also, remember to Sync your classlist in Crowdmark with Quercus shortly before your assessment starts to make sure that any students who have dropped your course do NOT have access to your assessment (and that any students who have added it do have access).

In April 2020, I used the A&S template as a starting point for an Download academic integrity document in ECO220Y

that students must complete. These generally should reference the University of Toronto Code of Behaviour on Academic Matters [July 1, 2019]. It is unclear how effective such statements are and some students both complete them and cheat.

Two strategies to deter academic infractions is to use timers and to distribute work over the term (multiple smaller assessments rather than one bigger one). For example, require weekly assessments (even if they are machine-marked) that have non-trivial weight and are actually marked (not just check/no check). This would make it tremendously inconvenient and expensive to hire a professional service to do all those hours of work for you. The higher the stakes, the more academic infractions there will be.

Also, see Quizzes in Quercus, which has a number of features to promote academic integrity.

Here are some links for more about Academic Integrity:

Jump back up to the Road Map

Accessibility and accommodations for Crowdmark

Remote tests/exams (Assigned Assessments)

Crowdmark released new features in August 2020 with documentation released in September 2020 that make accommodations substantially easier to manage: see Accommodation options for assigned assessments Links to an external site.. This means you do NOT have to create multiple versions and you can customize parameters at the level of the student.

In-person tests/exams (Administered Assessments)

For in-person tests/exams, you need to share unique booklets with Accommodated Testing Services (ATS). With Crowdmark, Administered Assessments unique copies of the test must be copied from the big pdf file containing all the unique tests for the entire class. This requires using Adobe Acrobat. This is a few minutes of extra work and double-checking for the instructor. Note well: Do not mess around with the big pdf file that Crowdmark produces. Keep that intact in your photocopy request: that is why the sample e-mail request above asked for only a subset of the large file to be printed. Crowdmark uses a special looping technology to keep the file sizes extra small and you do not want to mess that up. Instead, you can print (in Adobe) the last five tests (supposing that you have five students writing with ATS) and share those: the loss of looping does not matter because it is only a few tests anyway.

Jump back up to the Road Map