In lieu of an abstract, here is a brief excerpt of the content:

27 4¥¥ Identifying Indicators for Evaluation Data Collection FRANCESCA VENEZIA THE NEXT STEP IN evaluation planning is to decide how to collect information for the evaluation . More specifically, evaluators will need to carefully identify the relevant information needed to answer evaluation questions and to enable decision-makers to use the evaluation in the ways they desire. However, evaluators must still resist the urge to select the data-collection tools that seem appropriate for collecting relevant information (e.g., questionnaires , interviews, assessments, etc.). Rather, an important prior step is necessary to help ensure users get the specific types of information they need to answer their evaluation questions. This intervening step involves identifying evaluation “indicators,” or the different sources of evidence that will be measured, captured, or otherwise described in order to shed light on how the program is performing or functioning. In a simple sense, an indicator is a type of information or evidence needed to answer an evaluation question. For example, consider the following fictional evaluation question: “To what extent is the new,task-based teaching approach helpful for student learning?”To answer this question, one can imagine that there would be something happening in classrooms or elsewhere in the program—in terms of student behavior, performance, or some other phenomena—that would be suggestive of whether task-based teaching activities were helping students learn. That is, something in the program would“indicate” whether learning from task-based instruction was happening or not. Along these lines, a useful way to think about (and identify) indicators is to ask,“How will we know?”in conjunction with a particular evaluation question. For example, a way of conceptualizing an evaluation indicator for the evaluation question above would be to further ask,“How will we know the extent to which the new task-based teaching approach is helpful for student learning?” What would the answer to this question be? What would evidence of “helpfulness for student learning” look like? Some possible answers are the following:¥¥ Amount of interaction and negotiation happening during task-based classroom activities1¥¥ Level of student motivation and interest during task-based activities 28 Francesca Venezia¥¥ Student ability to successfully conduct important, task-based learning outcomes (e.g., writing a paragraph, asking for and understanding directions)¥¥ Ability to complete educational or professional tasks in real-life contexts Each of the above phenomena would arguably provide important insights into the extent to which task-based instruction was impacting student learning in helpful ways. For example, increased negotiation between students during classroom interactions is known to have important language-learning benefits, and if measured or assessed in some way, could well be regarded as indicating a helpful impact on learning from task-based language teaching.Likewise for motivation,which is known to increase when students engage in interesting tasks related to their language-learning needs. The same could be argued for successful completion of important task-based, course-learning outcomes or evidence from real-life contexts that students are better able to complete work- or education-related tasks—both, arguably, would be suggestive that a task-based approach had helped students ’ language development. Again, an indicator expresses what will be measured, captured, or otherwise described in order to answer an evaluation question. Identifying and listing indicators for each evaluation question is a specific strategy to help make evaluation more useful. The process of identifying indicators acts as a check against collecting the wrong type of evaluation information or selecting data-collection tools that capture information in nonuseful ways (points developed next).After determining evaluation questions and uses, evaluation planning should involve a collaborative process where stakeholders, users, and evaluators work together to identify the indicators for the evaluation prior to deciding how that information will be captured. Indicators versus Data-Collection Methods As hinted already, an indicator is not the same thing as a data-collection method. Recalling the evaluation question above, evaluators and users might be tempted to think that a questionnaire will indicate whether the task-based teaching approach is helping students learn. In one sense, this is true. However, it is important to understand that a questionnaire is not an indicator. It is, rather, a tool that reveals the indicator, or reveals some aspect of the task-based program suggestive of its helpfulness for learning. That is, a questionnaire is a data-collection strategy used to find out about something going on in the program that sheds light on whether task-based instruction is developing student learning (i.e., the...

pdf

Back To Top

This website uses cookies to ensure you get the best experience on our website. Without cookies your experience may not be seamless.