pdf Download PDF

Evaluation of Family Literacy Programs:
A Case Study of Better Beginnings, a Library-Initiated Family Literacy Bookgifting Program in Western Australia

This paper examines the nature of family literacy programs, with a particular focus on those that are based around the provision of free books to babies and young children, sometimes called “bookgifting” programs. First, the paper explores the rationale for family literacy programs in general and identifies some issues in their evaluation. It then focuses specifically on bookgifting programs. Using examples from several well-established programs, it reviews the research on which they are based, with particular reference to evaluation procedures. Next, the paper identifies some important issues that need to be addressed when planning and evaluating these programs, and notes some fundamental differences between particular programs that may have impacted on the results. It argues that this area of research needs stronger definition and a more inclusive approach to evaluation that includes both quantitative and qualitative methodologies. In order to illustrate the potential of a mixed-method approach, the paper examines the evaluation of the Better Beginnings bookgifting program for babies that has been initiated, developed, and extended by the State Library of Western Australia for over a decade. The paper concludes by suggesting that effective program evaluation is complex and multifaceted and must consider changes in behavior, confidence, and attitudes, as well as the ways in which such programs are experienced and integrated into family literacy practices. This approach provides insight into the contextual variables that determine the effectiveness of programs within and across families, and therefore inform further program development.

[End Page 19]


Family literacy programs are defined for the purposes of this paper as interventions, often with an emphasis on adults (usually parents) reading and talking about books with young children.1 The main focus is on those family literacy programs, sometimes called “bookgifting” programs, that involve “the distribution of free books to children and their families . . . particularly those that distribute books directly to babies and young children, alongside guidance for parents” (Sheffield Hallam University, 2014, p. 3). We begin by examining the research evidence that underpins the rationale for family literacy programs that focus on sharing books with young children.

Research suggests that the quality of experiences and interactions from birth have a major influence on the way the brain develops (McCain, Mustard, & Shanker, 2007). Recently, Hutton, Horowitz-Kraus, Mendelsohn, DeWitt, and Holland (2015), using functional magnetic resonance imaging (fMRI), found that for preschool children, greater exposure to listening to stories being read aloud was positively associated with the activation of brain areas that support mental imagery and narrative comprehension. Additionally, the vocabulary of books read aloud has been shown to be an important source of linguistic input for young children, particularly as picture books have been found to contain more unique word types than are found in parents’ regular child-directed conversations (Montag, Jones, & Smith, 2015).

Strong positive connections between shared book reading and early literacy learning have been demonstrated (Bus, van Ijzendoorn, & Pellegrini, 1995; De Temple & Snow, 2003; Karrass & Braungart-Rieker, 2005). Involvement in shared book reading has been found to increase young children’s motivation to participate in literacy activities (Baker & Scher, 2002; Sonnenschein & Munsterman, 2002) to support positive child–parent relationships (Hewer & Whyatt, 2006; Seden, 2008) and lead to positive educational and cognitive outcomes that last over time, “at least up to an age of 10 to 11” (Kalb & von Ours, 2013, p. 25).

There is further strong evidence from a large study by the Organisation for Economic Cooperation and Development (OECD) (2012) that the benefits of early book sharing are cumulative and long-lasting. From analyses of the PISA reading data of students from thirteen countries, it was shown that, regardless of family income, certain literacy practices in the home during children’s early years were related to student performance on reading tests when they were age 15. These home practices included using words within context, telling stories, singing rhymes and songs, and reading books to children by the time they were starting primary school. The most highly significant of these practices was reading books to children. At age 15, students whose parents had read to them “read at least as well as their peers one grade above them” (p. 18). Additionally, 15-year-old [End Page 20] students whose parents had read books and sung songs with them during their early years reported significantly higher levels of reading enjoyment. These findings suggest that promoting higher levels of parental involvement may increase students’ cognitive and affective outcomes, and may help reduce school-based performance differences across socioeconomic groups (Borgonovi & Montt, 2012).

The results of a series of meta-analyses on ninety-nine studies by Mol and Bus (2011) suggest that the effects of “exposure to reading” may be even longer lasting. These researchers examined “print exposure” across three age groups of students: preschoolers and kindergartners; students in grades 1–12; and college and university students. Moderate-to-strong correlations were found with print exposure on all measures of reading comprehension and technical reading and spelling. Exposure to print explained 12 percent of the variance in oral language in preschool and kindergarten, with the amount of explained variance rising for each educational level, to 19 percent in high school and 34 percent at the college/ university level. Mol and Bus suggest that the outcomes support an upward spiral of causality, in that shared book reading with young children may be “part of a continuum of out-of-school reading experiences that facilitate children’s language, reading, and spelling achievement throughout their development” (p. 267).

While such research has shown reading with young children to be associated with cognitive and affective outcomes over time, there is also some research to suggest that the type of interaction that takes place around the reading of books with young children is highly important in the shared book experience. The dialogic reading model has been shown to help develop children’s emergent literacy, language, and metalinguistic skills (Reese, Sparks, & Leyva, 2010). Dialogic reading is a method of reading picture books with young children in which parents/caregivers are shown how to encourage the child to actively participate in the reading of a book (Zevenbergen & Whitehurst, 2003), with the aim of the child eventually taking over the role of narrator. Adults support the child’s learning of story vocabulary and discussion of the plot through various questioning and explanatory techniques. Joint attention to text through shared, sustained thinking as parents/caregivers and the child jointly explore and extend concepts is a powerful means of supporting early language and literacy (Siraj-Blatchford, 2007). This model highlights the importance of dialogue during the reading, as parents scaffold, elicit, and respond to comments from the child about the pictures or text and hence adapt the discussion to the child’s level of understanding (Bus et al., 1995; Mol, Bus, & de Jong, 2009; Nyhout & O’Neill, 2013).

Although research has established the power of book sharing, it is also acknowledged that there are many different ways in which families from socially, linguistically, and culturally diverse communities engage children [End Page 21] in highly effective language and literacy learning (Anderson, Anderson, Friedrich, & Kim, 2010). However, research suggests that because of the emphasis on a book-based education system, some children may be disadvantaged at school due to a mismatch between home and school literacy practices (Sheffield Hallam University, 2014). Thus a number of family literacy programs emerged around the world as a means of introducing families to strategies that support young children’s literacy in ways that potentially help bridge the gap between home and school.

These include programs in the United Kingdom (Collins, Svensson, & Mahony, 2005; Hannon, Morgan, & Nutbrown, 2006; Moore & Wade, 2003), Europe (Carpentieri, Fairfax-Cholmeley, Litster, & Vorhaus, 2011), North America (Anderson et al., 2010), and Australia (Barratt-Pugh & Rohl, 2015; Elias, Hay, Homel, & Freiberg, 2006; Shoghi, Willersdorf, Braganza, & McDonald, 2013). They are informed by research suggesting that parents who are not used to sharing books may need ongoing support in selecting appropriate texts, and scaffolding interactions through meaningful talk (Bus, Leseman, & Keultjes, 2000; Neuman, 1996).

Evaluating Family Literacy Programs

Owen (2006) sees evaluation of an intervention as the process of making a judgment about the value of an object—that is, an intervention policy and/or program and its purpose as enhancing the quality of the intervention in order to “solve or ameliorate a problem within a social setting.” He also views the process as worthwhile if the knowledge produced is “reliable, responsive to the needs of policy and stakeholders, and can be applied by these stakeholders” (p. 1).

In recent years, there has been much interest in trying to quantify the magnitude of the impact of particular educational interventions through the use of meta-analysis, which provides an “effect size” for an intervention (Hattie, 2009, 2012). Hattie has explained the effect size as “a useful method for comparing results on different measures . . . or over time or between groups, on a scale that allows multiple comparisons . . . across content and over time” (2012, p. 3), and he identifies an effect size of 0.4 as the “hinge point” for identifying the effectiveness of an educational intervention. He does, however, point out that a smaller effect size for an intervention that requires few resources may be “more critical” than a larger effect size for a program that requires more resources. This may be particularly important for those family literacy programs that involve parents or other family members as the agents of intervention, interacting around books with their young children.

In his review of more than 900 meta-analyses of interventions that targeted student learning, Hattie (2012) included one meta-analysis classified as a “family literacy intervention” (van Steensel, McElvany, Kurvers, & Herppich, 2011), which, with an effect size of 0.18, is well below the 0.4 [End Page 22] “hinge point.” This review also included three meta-analyses of “exposure to reading” with effect sizes well above the 0.4 mark, ranging from 0.59 to 0.78.

Carpentieri et al. (2011), who conducted a review of six meta-analyses of family literacy interventions (including the van Steensel et al. [2011]) study, found this to be the only study with an effect size below 0.3, while four of the studies had effect sizes greater than 0.4, the highest being 0.68. They concluded from these findings that “family literacy interventions have a relatively large impact on child literacy acquisition” (p. 9). Nevertheless, Carpentieri et al. raise a number of issues concerning the reasons for such conflicting results of meta-analyses: the inclusion of different primary studies; the number of studies; the breadth of program type and measured outcomes; the intervention; and different interpretations of data. They discuss in detail the intervention characteristics that may impact on results, which include the duration and site of intervention; the nature, intensity, and quality of parent training; the quality of the implementation of the program; and participant characteristics in terms of family disadvantage and the child’s gender and age. Carpentieri et al. also point out that these factors can combine to limit confidence in the interpretation of meta-analyses that “often must synthesise the effects of interventions which are different enough to be considered apples and oranges” (p. 44). They emphasize the importance of complementing meta-analytic assessments with “high-quality quantitative primary research” and also “high-quality qualitative research” (p. 44).

Family Literacy Bookgifting Programs

Having examined the evaluation of family literacy programs in general, we now focus specifically on the nature and evaluation of bookgifting family literacy programs. Two of the most widely disseminated and researched programs that involve the provision of free books for babies and young children are the Bookstart and Reach Out and Read programs from, respectively, the United Kingdom and the United States, which have served as models for other programs. Reach Out and Read is a healthcare-based intervention run by a nonprofit organization that has been operating in the country since 1989 with the mission of giving “young children a foundation for success by incorporating books into pediatric care and encouraging families to read aloud together.” The program has “a special emphasis on serving those in low-income communities” and is presented at a number of regular health checks, when the child is between 6 months and 5 years of age (Reach Out and Read, 2014, n.p.). At each checkup a pediatrician or nurse explains to parents the importance of reading aloud from infancy onward, and the child is presented with a new book. In many program sites, books are provided in the waiting room and volunteers are available to model techniques for reading aloud to the child. The program [End Page 23] is implemented in 5,500 sites throughout the United States, and each year 6.5 million books are distributed to children (Reach Out and Read, 2014, n.p.). A total of ten pediatric checks may be conducted between the ages of 6 months and 5 years (Zuckerman, 2009.)

The UK’s Bookstart program was initiated in 1992 by the reading charity Book Trust, working with library services, health authorities, and the University of Birmingham as a pilot project involving three-hundred babies. It is now funded by the government and the book industry, and coordinated nationally by Book Trust and locally by the library service, in cooperation with early years settings and health professionals. Bookstart aims “to inspire a love of reading that will give children a flying start in life” (Bookstart, 2016, n.p.). It provides a “baby pack” to encourage parents to “engage in book-related activities with children from the earliest years by providing books and guidance on reading, for every child” (n.p.). This pack, distributed by health visitors in babies’ first year, consists of books and guidance for parents on sharing books. A “treasure pack” containing books, crayons, and paper is distributed to children ages 3–4 in “early years settings” (Sheffield Hallam University, 2014, p. 3). Parents and young children can also attend rhyme-time and story sessions that take place in libraries and various early years contexts. During 2013–2014 almost 1.5 million packs and 3 million books were provided to young children throughout England, Wales, and Northern Ireland, including almost 5,000 to children with additional needs.

Two of the most widely disseminated family literacy programs in Australia, Let’s Read and Better Beginnings, may be considered as broadly based on the Bookstart model. Let’s Read is a shared reading intervention for children from birth to age 4 and was developed by the Centre for Community Child Health at the Murdoch Children’s Research Institute (MCRI) and the Royal Children’s Hospital. The MCRI and the Smith Family Charity have partnered to implement Let’s Read within communities across Australia. This “multi-point intervention,” provided by a “trusted community professional,” has been extensively researched (Let’s Read, 2012, n.p.). It is designed to be delivered at four points during a child’s preschool years. Early literacy resources and materials, including tip sheets for parents, book suggestion lists, age-appropriate books, and key messages are delivered at each of the intervention points.

The Reach Out and Read and Let’s Read programs described above were created as preventative health initiatives; public libraries have been highly involved in the implementation of Bookstart since its inception. The OECD’s 2012 report Let’s Read Them a Story! The Parent Factor in Education shows the importance of parents reading to their children; it also points out the importance of parents taking their young children to the library and talking with them about the books they are reading. Over recent years, Australian libraries have been highly proactive in developing [End Page 24] and implementing family literacy programs. Further, libraries are increasingly partnering with family literacy programs and becoming part of the Let’s Read distribution and support network (Barratt-Pugh, Anderson, & North, 2013).

The Better Beginnings family literacy program that is explored further in this paper was launched in 2005 as an initiative of the State Library of Western Australia and designed “to provide positive language and literacy influences for young children through supporting parents as their children’s first teachers” (Barratt-Pugh & Rohl, 2012, p. 1). A reading pack is given to parents of young babies, usually by the community child-health nurse at the scheduled six-to-eight-week clinic health check. The pack contains a board book and other associated materials, including information on reading to babies and invitations to join the library and to library-based activities for babies and young children that feature rhyme-time and story sessions. At the health check, the contents of the pack and the importance of reading with babies and young children are discussed with parents/ caregivers. The program is funded by both state and local governments and by industry.

Evaluating Bookgifting Programs

It will be seen that there are a number of differences between the four programs described above. Recent reviews of the nature and outcomes of bookgifting programs have been undertaken as part of the evaluation strategies for Bookstart in the UK (Sheffield Hallam University, 2014) and Let’s Read in Australia (Shoghi et al., 2013). While recognizing the importance of sharing books with young children, these reviews have identified some methodological issues that are potentially problematic in researching these programs and drawing conclusions about their impact. The following classification of issues, which has many similarities to that of Carpentieri et al. (2011) for family literacy programs in general, is based on findings from both reviews about key elements that need to be considered in the design and interpretation of intervention studies:

  • The program: Content and intensity, including duration of the program, number of encounters with program staff, amount of support (such as feedback to participants), professional development of staff, and other literacy interventions that may be running alongside the target program.

  • The participants: Ages of the children, parents’ socioeconomic and linguistic backgrounds, and levels of attrition.

  • The study’s methodology: Type of research model, length of the study, sample size, types of data (quantitative and/or qualitative), and the timing of data collection.

Both reviews found the methodology of the study to be particularly important for planning and interpreting results. Short-term studies may [End Page 25] not be long enough to establish new practices, whereas in long-term studies it is possible that the impact of established programs may appear less, although this can be a direct result of a program’s success, in that ceiling levels leave little room for improvement (Sheffield Hallam University, 2014). Another issue is the size of the study: large-scale longitudinal studies, which often rely upon self-reporting questionnaires, may include variations in the implementation of the respective programs, such as the training of practitioners, timing of book distribution to participants, and level of intensity or guidance. On the other hand, smaller, randomized control trials (RCTs) in which “subjects are allocated to an experimental or control group according to a strictly random procedure” (Hattie, 2009 p. 4) require a high degree of standardization of implementation and assessment of children over time (Sheffield Hallam University, 2014), both of which are difficult to monitor in large-scale bookgifting programs. Nevertheless, Cara and Brookes (2012) have claimed that there is a lack of “systematic evaluations” in family literacy programs, including RCTs that are frequently used to investigate cause and effect in medical interventions and, as Hattie (2009) points out, are part of a move toward more evidence-based decision making in education.

Evaluating Bookgifting Programs through RCTs

There appear to be few published RCTs of bookgifting programs, and the results of those that have been reported are not consistent. The U.S. Reach Out and Read program has been well-researched using various methodologies, including RCTs. Mendelsohn et al. (2001) document the results of an RCT of this “clinic-based literacy intervention” in order to examine its effects on the language development of preschool children. The participants were 112 “impoverished,” inner-city Spanish- or English-speaking Latino and black families who attended one of two well-child clinics with their under age 6 child who did not attend kindergarten. The intervention group from one clinic had been exposed to a Reach Out and Read program for three years; the control group from the other clinic had been exposed to the program for only three months. All assessments were conducted in the child’s primary language. The results showed that the intervention families reported reading with their children significantly more often (approximately one day more per week), and that their children had significantly higher receptive-language scores. Further, the intensity of exposure to the intervention, as measured by the total number of contacts with the program, was associated with increased parent–child reading activities.

The UK Bookstart program has also been extensively researched through various methodologies. An RCT of the Bookstart treasure pack, which is given to all children between ages 3–4 (Demack & Stevens, 2013), was conducted in order to measure the impact of the pack by comparing [End Page 26] change over time (three months) among 138 parents in intervention and control groups on five key outcomes. These outcomes were

  • • the perceptions of parents about books, rhymes, and songs;

  • • the perceptions of parents about their child’s engagement with books, stories, rhymes, and songs;

  • • parental practices and the frequency of reading with their child;

  • • the use and membership of a public library; and

  • • the child’s book ownership.

The results showed a statistically significant difference in change over time on only one of these indicators, which was for fathers reading with their children. This difference was not noted for mothers, and the researchers identified a number of confounding factors that may have accounted for this result. They also identified some important factors that may well have accounted for the overall lack of positive results that include, but are not limited to, the low intensity of the program that was conducted over a short period of time and the presence of baseline ceiling effects. For example, nearly three-quarters of parents in both the intervention and control groups indicated in a preprogram questionnaire that someone read to their child every day, thus leaving limited room for improvement for many families. The addition of this questionnaire, which asked parents to report on their family literacy practices and perceptions of the program, allowed for data that complemented results of the RCT study and indicated that key stakeholders regarded the treasure pack as “a highly regarded and significant intervention” (Demack & Stevens, 2013, p. 6).

In Australia, Goldfeld et al. (2012) reported no measurable differences between intervention and control groups in their four-year, multipoint RCT study of the Let’s Read program. This was delivered by nurses at child health centers in five relatively disadvantaged areas at four points in time to mothers and their babies who attended the usual well-child care visits at 4–8 weeks, and 12, 18, and 42 months of age. At each visit, the nurses, who had received training by the research team, were asked to spend about five minutes delivering, modeling, and discussing the Let’s Read promotion messages with the mother. The nurses also gave the family a take-home pack of an age-appropriate picture book, book list, and guidance on shared, interactive reading activities and appropriate book selections. However, results indicated that there were no significant differences among the experimental and control groups on the outcome measure of a child’s emergent literacy skills at age 4. The researchers suggest that the study may have been limited by the facts that the parent participants living in the targeted disadvantaged area were among the more advantaged in their region; that there were no translating resources for families who spoke English as an additional language; and that for some families, the [End Page 27] intervention was delivered via telephone rather than face to face. They also suggest that a more intensive program might have produced more positive effects.

No RCTs have been reported for Better Beginnings. As can be seen, the results of RCTs for bookgifting programs presented above are inconsistent. There are many possible reasons for these results, including the intensity of the program. Families involved in Reach Out and Read could experience up to ten clinic-based intervention sessions, including advice and modeling provided by specifically trained pediatricians and nurses (Zuckerman & Khandekar, 2010). On the other hand, Let’s Read, with its four points of program delivery, and Bookstart, with two widely spaced points of intervention, are much less intensive.

Recently, Burden (2015) has presented a highly detailed critique of the use of RCTs, with a particular emphasis on their use in measuring the effectiveness of literacy interventions. He outlines a number of important issues that need to be considered in order to meet the minimal criteria of acceptability when using this methodology for evaluating an intervention, the first two of which are randomization and control. In terms of randomization, the issue is the creation of intervention and control groups that are truly representative of the target population as a whole, given the multiplicity of possible biases within samples. Control issues that may affect results include the integrity of the program, and the teaching provided to each group. Burden concludes that the process of learning is extremely complex in terms of context, teachers, and learners, whereas RCT methodology is underpinned by a simplistic view of the learning process. He proposes a more inclusive approach to program evaluation that features a cyclical review of context, plans, actions, and reflections that, in combination, can lead to a realistic evaluation of the effectiveness of a program and to decisions about its future directions.

In general, it seems that randomized control trials may not be able to capture the many complex factors and their interactions involved in most universal bookgifting programs. In particular, it seems that results depend on the ability to control for a range of variables, including randomization of the sample, the intensity of the program, and the social, cultural, and linguistic backgrounds of families. There have been some positive results for the high-intensity Reach Out and Read program when sociolinguistic variables are tightly controlled, showing that under these particular conditions strong evidence for the effectiveness of bookgifting programs can be obtained. Nevertheless, in view of the lower intensity and much more fluid nature of other universal bookgifting programs, which make randomization and control exceedingly difficult if not impossible, it seems that a more inclusive approach is needed. As discussed by Demack and Stevens (2013), in the RCT of Bookstart’s treasure pack, questionnaires that address family [End Page 28] perceptions and practices can provide highly valuable data that help explain results. This accords with the conclusions of Carpentieri et al. (2011) that high-quality quantitative and qualitative research methods are indicated when evaluating family literacy programs.

A Mixed-Methods Approach to the Evaluation of Bookgifting Programs

The need to take an inclusive mixed-methods approach that incorporates both quantitative and qualitative methodologies in the evaluation of book-gifting programs is indicated in order to capture the complexities of literacy learning. This approach is especially important in the current political climate that seeks to legitimize “hard evidence” as the most compelling means of demonstrating the effectiveness of interventions (Biesta, 2010). Many program providers are under pressure from funding bodies to demonstrate program effectiveness in relation to specific academic outcomes. In Australia there are mandated literacy and numeracy assessments that begin at school entry. Since they have to manage limited resources, policymakers and funding bodies are increasingly driven by “scientific” evidence and accountability based on short-term results (Lather, 2004). This approach, however, may limit our understanding of bookgifting programs, as it does not take into account program implementation and contextual variables within communities that help determine the significance and impact of programs for children and their families (Burden, 2015). Carpentieri et al. (2011) have argued that “insufficient evidence is available from primary studies to understand why, when, where and for whom they are effective” (p. 57), thus potentially misleading policymakers and funding bodies in their decisions about supporting particular programs.

An inclusive approach to the evaluation of bookgifting programs that involves the collection and use of both quantitative and qualitative data provides the opportunity to explore the complexity of program implementation and impact. Quantitative methods like surveys provide breadth of evidence, and qualitative methods like case studies, interviews, videos, and observations provide depth. Combining these methods has the potential to produce broad evidence from a wide range of participants and communities while also capturing the “lived experience” of participants within a particular time and place. Carpentieri et al. (2011) argue that impact should be identified through a wide range of measures, including recognition of the broader effect of programs, such as changes in attitudes, confidence, and self-efficacy. In addition, program implementation should take account of the context in which it is delivered because the ways in which programs are enacted are related to personal, social, economic, and cultural factors (Burden, 2015).

In order to illustrate the process and outcomes of an inclusive mixed-methods [End Page 29] approach to the evaluation of bookgifting programs, the following sections draw on the longitudinal evaluation of the Better Beginnings bookgifting program initiated by the State Library of Western Australia. This state covers an area of more than 2.5 million square kilometers (965,000 square miles—the combined size of Alaska and Texas) and has a population of 2.5 million people. Most of the local governments are in regional and remote Western Australia, the largest of which covers almost 4 million square kilometers and has a population of only 8,000. It can be seen that geographical constraints create enormous challenges to statewide accessibility to this program (Better Beginnings, 2015, n.p.).

The Better Beginnings Evaluation Process

The Pilot Study

In 2004 the Better Beginnings pilot was set up as an intervention program in six very diverse communities in Western Australia. Evaluation was built into this exploratory program from the start through the use of a formative experimental design that allows for the investigation of factors that contribute to or detract from the effectiveness of an intervention (Jay & Rohl, 2005; Reinking & Watkins, 2000). The evaluation took place in two of the six communities: a mining town with a significant Aboriginal and Torres Strait Islander population, 600 kilometers distant from the nearest major city, and an outer metropolitan suburb containing a range of residential settings and a small Aboriginal and Torres Strait Islander population. In carrying out this evaluation, we were mindful of Owen’s (2006) proposition that it should be a process that investigates the value of an intervention in order to “enhance its quality, and produce knowledge that is reliable, responsive to the needs of policy and stakeholders, and can be applied by these stakeholders” (p. 1).

The quantitative data for this pilot study consisted of 107 preprogram surveys completed by mothers (no respondent identified as “father or other primary carer”) before they received the reading pack, and a postprogram telephone survey completed by sixty-five of these mothers. Some qualitative data were also included in the surveys, as the mothers were encouraged to provide open-ended comments at the end of each section. Qualitative data were also provided by eleven mothers, who represented a cross-section of socioeconomic and educational backgrounds and were interviewed in their homes and observed reading with their children. In addition, five community health nurses who delivered the program, the central coordinators, and a librarian from each community were interviewed. In this way the voices of a diverse range of participants—mothers and professional library and health personnel—were heard. The results identified reported changes in the mothers’ perceptions and practices, some strengths of the program design and implementation, and issues that indicated key areas for modification and development. [End Page 30]

Extending the Program and Evaluation Cycle

Based on the results of the pilot study, some modifications were made to the program, and it has been gradually extended to include every family with a new baby in Western Australia and thus may be considered a “universal program.” Because the methodology used in the pilot evaluation had provided depth and breadth of information, it was decided to employ similar instruments in the following evaluations, but with some additions and minor modifications to reflect the development of the children from infancy to kindergarten. Two underlying questions have been addressed: How effective is the Better Beginnings program from the perspectives of the participants (parents/caregivers and professional library and health personnel)? How has the program, including its implementation, been developed and sustained?

In order to address these questions, annual evaluations were undertaken in four diverse communities (two metropolitan, one rural, and one remote). As in the pilot project, these evaluations included a survey of parents/caregivers (all of whom identified as “mothers”) from varying socioeconomic backgrounds and educational levels. The research instruments were somewhat modified each year to capture the ongoing impact and sustainability of Better Beginnings, but were similar enough to allow for comparisons over time. A longitudinal study of data from the first four evaluations was made in order to examine the implementation of the program from a longitudinal perspective (Barratt-Pugh & Rohl, 2012).

In addition to the surveys, each year we undertook interviews with forty case-study mothers drawn from the sample of parents, individual interviews with ten local librarians across the four communities, focus-group interviews with thirteen child-health workers who delivered the resource packs in the four communities, and interviews with the State Library coordinator. In the following year, based on recommendations of the four-year evaluation, additions were made to the study methodology; these included videos of the case-study mothers sharing books with their children, and interviews about books and reading with them.2

The following section summarizes major findings and how important issues identified by both the quantitative and qualitative data were addressed in order to modify and extend the program, in addition to some impacts on policy. These are discussed in relation to key elements identified earlier in this paper that need to be considered in the design and interpretation of intervention studies and include not only the program, methodology, and participants but also the role of the State Library and librarians.

The Program

Evidence from the evaluation demonstrated that the State Library, in collaboration with local librarians and child-health nurses, had the capacity [End Page 31] to deliver, sustain, and extend the Better Beginnings program. Feedback from the annual parent evaluations and the collection of data from child-health and library participants and stakeholders resulted in a number of strategies being implemented to strengthen the infrastructure of the program and increase its reach. Some of these strategies related to the central coordination of the program and the training and roles of library staff.

Central Coordination of the Program

Data from ongoing interviews with library staff informed the development of the role of the central coordinator as it evolved to encompass responsibility for maintaining partnerships with public libraries, organizing professional-staff development, delivering the program, reinforcing links with other agencies, and strengthening the networks within the program. Accordingly, the coordinator supported and nurtured the partnerships among the child-health nurses, librarians, and other professionals, which have impacted on policy in terms of a “memorandum of understanding” that has been developed between the State Library of Western Australia and various state health agencies to facilitate statewide provision of the program. Further, funding for Better Beginnings has become part of the State Library’s standing budget, thus ensuring ongoing support that allows for longer-term planning. Central coordination also facilitated a major media campaign to publicize Better Beginnings as a recognized brand in and across communities. This has reinforced and sustained the Better Beginnings message of sharing books with babies from birth and provided links across Western Australia to other early literacy family programs, such as Let’s Read. Central coordination has also allowed Better Beginnings activities to be included in the children’s section of the annual Perth Writers’ Festival.


Feedback from interviews with child-health nurses and librarians has informed the professional development of librarians based on identified needs. During the course of the evaluation, several librarians requested more information about practical aspects of the program and examples of practice; they also identified a need for information about networking with other librarians and ways of working with Aboriginal and Torres Strait Islander peoples and other culturally and linguistically diverse families. Accordingly, professional training was developed that addressed these needs. In view of diverse geographical challenges in Western Australia, delivery has moved from a print-based handbook to an integrated approach that includes, where possible, face-to-face, hands-on modeling and experiences and also web-based information and sharing of practice. Most recently, four online modules for librarians are being developed that have the potential to engage a wider audience. [End Page 32]

Role of the Librarian

Based on feedback about the responsibility for and management of the Better Beginnings program, several initiatives have been implemented in local libraries. First, where possible, a librarian has become the designated Better Beginnings coordinator who welcomes, informs, and encourages families to join the program and obtain library membership. Second, space permitting, family-friendly areas have been identified, with coded books for easy access and choice. Third, rhyme-time and story sessions have been timetabled on a regular basis and integrated into library services for families, thus providing librarians the opportunity to model book-sharing strategies for families and introduce them to library resources and other programs that target literacy for infants. An innovation to encourage and engage nontraditional library users has involved librarians taking Better Beginnings into the community, including mothers’ groups, play groups, and parks.

Involvement of the Families

The surveys and interviews with the mothers provided longitudinal evidence that, for the majority of them, the program helped to introduce or reinforce beliefs about the importance of sharing books. They reported an increase in the frequency of book sharing, the number of books in the home, self-confidence, and self-efficacy. There also appeared to be a “ripple effect,” in that other family members became involved in book sharing with the target child and other children in the family, including any additional newborns. Additionally, the mothers reported increased access to resources and book-sharing practices through library membership and attendance at library activities. In some remote Aboriginal and Torres Strait Islander communities, it appeared that the Better Beginnings gift book was the only source of children’s literature. Changes in child literacy practices were also reported, such as asking for books to be read, developing a love and understanding of books, and developing a positive self-concept as a reader. Other important outcomes reported by a small number of mothers included mother–child bonding, increased literacy skills for adults in the family who had low literacy levels, and a number of fathers or male partners becoming involved in sharing books and singing nursery rhymes.

In each year of the evaluation, the mothers were asked to identify the ways in which delivery, content, and activities could be improved or developed, and were also invited to add further comments about their experience with Better Beginnings. Accordingly, suggestions were made about ways of improving the program, and where possible these were incorporated into the development of the resources, delivery of the program, and library activities. From the open-ended survey questions and interviews with mothers, librarians, and child-health nurses, two key issues emerged. [End Page 33] The first was a strong interest (and some anxiety) by mothers about how to continue to facilitate their children’s literacy as they matured, and how to prepare them for formal school. The second was a concern by librarians and nurses about how to engage culturally and linguistically diverse families, including those living in remote Aboriginal and Torres Strait Islander communities, in the Better Beginnings program.

Extending the Program

Given the evolving evidence base for the effectiveness of Better Beginnings and in response to feedback about the program from the ongoing evaluation, it was decided, in consultation with the major funding bodies, to extend the program. As a result, a number of new programs have been developed, piloted, and evaluated—two of which, on the basis of the evaluative findings, have been continued.

Better Beginnings Four to Five

This program is delivered by local librarians through kindergartens in Western Australia. It consists of a reading pack given to the child and two sets of resources that may be borrowed; a discovery backpack containing a variety of picture books and electronic resources, puzzles, and reading tips; and a read-aloud book set for shared reading, along with suggestions (Better Beginnings, 2015, n.p.).

Read to Me—I Love It!

This program has been developed to support the specific early literacy requirements of children (ages 0–5) living in remote Aboriginal and Torres Strait Islander communities. The program is delivered to a hundred such communities in Western Australia twice a year by local distributors who are recognized as being part of the community. The reading packs include a range of specifically designed, culturally appropriate books and activities that have been developed to support the early literacy requirements of children living in remote communities; help library services become more relevant to the lives of Aboriginal and Torres Strait Islander Australians; and provide resources for communities who have limited access to books and library services (Better Beginnings, 2015, n.p.).

These programs show how evaluation identified the need for and informed the nature of new programs, and as a result has fine-tuned program content, delivery, and activities in a number of important ways. The content in the Better Beginnings Four to Five program is linked to Belonging, Being and Becoming: The Early Years Learning Framework for Australia (Department of Education, 2009) and the Australian Curriculum (Australian Curriculum, Assessment and Reporting Authority, 2014) and is aimed at facilitating links between families and schools. The resources in the Read to Me—I Love It! reading packs have been designed to reflect Aboriginal and Torres Strait Islander history, culture, and everyday community life. Where possible the reading packs are delivered by Aboriginal and/or Torres Strait Islander community members. [End Page 34]

The Study Methodology

The inclusion of qualitative and quantitative methodology in all the Better Beginnings evaluations posed some challenges in relation to both data collection and analysis, but is discussed here in relation to the Better Beginnings Birth to Three four-year longitudinal study.

The first issue relates to the decline in the number of mothers participating in the survey: there was a decrease in the number who responded to the surveys over the four years: Year 1, 300 mothers; Year 2, 177; Year 3, 102; Year 4, 84. Although perhaps not unexpected, given the nature of family demands and the transient nature of some communities in the study, this could also be a reflection of different levels of importance placed on literacy by mothers who remained in the study and those who withdrew. We were not able to recruit fathers and other male primary caregivers, who might have provided a different perspective on the Better Beginnings program.

The second issue concerns the validity and reliability of the self-report surveys and case-study interviews with a selection of mothers. The same research assistants contacted the mothers each year to conduct the telephone survey, and as they had knowledge of the mother and child, in some cases they had built up a strong rapport with the mothers. A desire to please and/ or give perceived “right” answers, as well as being prompted by a tele phone call about literacy practices, might have colored the mothers’ responses. On the other hand, this approach had the potential to build trust and to collect reliable data as researchers sought clarification, illustrations of practice, further information, and checked for misinterpretation. In the fifth year of the study, video observation of mother–child book sharing and child interviews were also undertaken, adding further layers to the picture emerging from the evaluation. In a subsequent evaluation of the Read to Me—I Love It! program in remote Aboriginal and Torres Strait Islander communities, where appropriate and possible, the interview data collection was modified to include conversations around the program. Rather than direct questioning, we chatted with participants within contexts that were part of their everyday experiences, such as around the shop, in the playground, and in the play group. Thus the data-collection methods were designed to help capture the ways in which Better Beginnings was experienced and enacted by families in different communities.

Our analysis of the data collection revealed some of the complexities, contradictions, and ambiguity of the effectiveness of Better Beginnings. The relationship between Better Beginnings and the myriad of family literacy practices embedded in cultural, social, and linguistic contexts was multifaceted. As well as the broad endorsement of the value and impact of Better Beginnings, a number of additional realities began to emerge from both the surveys and interviews with the mothers, which included the following: [End Page 35]

  • • Shared book reading as a means of forming relationships: mothers reported bonding with their baby

  • • Shared book reading as a means of supporting adult literacy: mothers reported fathers/partners “learning to read” with the new baby

  • • Shared book reading as creating conflict: in one family, being literate was viewed as “shameful” and thus the program posed a number of conflicts

  • • Shared book reading as a means of creating inclusivity: one mother talked about how she and her special-needs child felt included, for the first time, in a mainstream initiative

  • • Using the gift book as a means of recognizing and endorsing cultural diversity: several mothers reported the importance of the themes and illustrations in the resources that reflected their cultural experiences

  • • Using the gift book as a way of marking and celebrating a milestone in the baby’s life: some mothers reported this was their baby’s first book and thus worthy of celebrating and keeping

  • • Using the gift book as filling a void within a family: a small number of mothers reported that the gift book was their only book, and one talked about how she used it to fill a “silent house”

  • • Using the library as a safe place within the community: two mothers reported that the program had introduced them to the library, and this was the only place they felt safe with their children

This brief discussion of the study’s methodology and analysis has shown some limitations of the data collection. Nevertheless, the contextual adaptation of the data-collection methodology, which was informed by previous evaluations, and the qualitative data analysis have allowed some insights to emerge that illustrate the diversity of families and their contexts; it also allowed for additional interpretations of the “meaning” of the program to different families above and beyond its original aims. In terms of equity, it is important that universal family literacy programs are able to take into account such diversity in program design implementation, development, and analysis.


Our review of the evaluation of family literacy programs, with a focus on those that provide free books and resources to mothers, babies, and young children, illustrates a need to move beyond simple cause–effect relationships, and to identify the multiplicity of ways in which these programs are experienced within and across families. Specifically, our overview of the evaluation of the Better Beginnings program has attempted to show how the program is taken up differently by different families, suggesting that there is no absolute reality in terms of implementation, engagement, or outcomes. We have identified the complex interplay among the mothers and babies, the providers, and the librarians that is mediated through [End Page 36] the reading pack and the library activities. By offering a broad view of how bookgifting programs are experienced, and uncovering the assumptions that underlie the design, implementation, and evaluation of such programs, we hope to provide new insights about early literacy programs that honor diversity and ultimately make a difference to the future of our children. This approach demands a methodological framework that incorporates both quantitative and qualitative data: quantitative studies offer greater breadth of coverage, while qualitative studies have the potential to illuminate the complexity of literacy practices within the social, cultural, educational, and political contexts in which literacy programs are situated. In relation to the Better Beginnings program, our research suggests that libraries have an important role to play in continuing to support family literacy as key service providers of free resources and activities. The role of Better Beginnings is multifaceted, and its ongoing evaluation process has informed the ways in which it has grown to meet the needs identified by the participants, differentiating the program according to the community context, and making a difference to children and families in those communities.

Caroline Barratt-Pugh

Caroline Barratt-Pugh is a professor of early childhood and director of the Centre for Research in Early Childhood Education in the School of Education at Edith Cowan University, Western Australia (WA). She has extensive experience in teaching and research across early childhood settings and at tertiary institutions in the UK and Australia. Her research interests include language and literacy in families, communities, and formal educational settings, and the evaluation of early literacy programs. She has been a codirector and director of several state and national literacy research projects, and is currently director of the evaluation of the Better Beginnings family literacy program in WA (2005–2016). Numerous publications and engagement with professionals, parents, and other key stakeholders through her research has resulted in evidence-based recommendations for policy and practice across the early childhood sector.

Mary Rohl

Mary Rohl is a literacy specialist whose research has concentrated on literacy learning and teaching, with an emphasis on the early years, literacy learning difficulties, and prevention of literacy difficulties. Over a number of years she has been a member or codirector of various national and state children’s literacy-project research teams, which have focused on a wide range of literacy issues, including effective teaching practices in the early years of school and home and school literacy practices. She also has extensive teaching experience within a range of early childhood settings in the UK and Australia, and in teacher education in Western Australia.


1. The term parent is used to encompass mother, father, and primary caregiver.


Anderson, J., Anderson, A., Friedrich, N., & Kim, J. E. (2010). Taking stock of family literacy: Some contemporary perspectives. Journal of Early Childhood Research, 10(1), 33–54.
Australian Curriculum, Assessment and Reporting Authority. (2014). Australian curriculum. Retrieved from http://www.australiancurriculum.edu.au
Baker, L., & Scher, D. (2002). Beginning readers’ motivation for reading in relation to parental beliefs and home reading experiences. Reading Psychology, 23(4), 239–269.
Barratt-Pugh, C., & Allen, N. (2011). Making a difference: Findings from Better Beginnings a family literacy intervention programme. Australian Library Journal, 60(3), 195–204.
Barratt-Pugh, C., Anderson, C. K., & North, S. (2013). The changing perspectives of librarians in the Better Beginnings family literacy program. Australian Library Journal, 62(3), 183–195.
Barratt-Pugh, C., & Rohl, M. (2012). Making a difference: The report of the evaluation of Better Beginnings (0–3yrs) family literacy program 2007–2010. Retrieved from http://www.better-beginnings.com.au/research/research-about-better-beginnings/better-beginnings-making-difference
Barratt-Pugh, C., & Rohl, M. (2015). Better Beginnings has made me make reading a part of our everyday lives: Mothers’ perceptions of a family literacy program over four years. Australian Journal of Early Childhood, 40(4), 4–12.
Better Beginnings. (2015). Programs. Retrieved from http://www.better-beginnings.com.au/programs
Biesta, G. J. J. (2010). Why “what works” still won’t work: From evidence-based education to value-based education. Studies in Philosophy and Education, 29(5), 491–503.
Bookstart. (2016). About Bookstart; Research; Impact. Retrieved from http://www.bookstart.org.uk/
Borgonovi, F., & Montt, G. (2012). Parental involvement in selected PISA countries and economies (OECD education working papers, no. 73. OECD iLibrary). Retrieved from http://www.oecd-ilibrary.org/education/parental-involvement-in-selected-pisa-countries-and-economies_5k990rk0jsjj-en [End Page 37]
Burden, R. (2015). All that glitters is not gold: Why randomised controlled trials (RCTs) are of limited value in judging the effectiveness of literacy interventions. Educational and Child Psychology, 32(1), 11–20.
Bus, A. G., Leseman. P. P. M., & Keultjes, P. (2000). Joint book reading across cultures: A comparison of Surinamese-Dutch, Turkish-Dutch, and Dutch parent–child dyads. Journal of Literacy Research, 32, 53–76.
Bus, A. G., van Ijzendoorn, M. H., & Pellegrini, A. D. (1995). Joint book reading makes for success in learning to read: A meta-analysis on intergenerational transmission of literacy. Review of Educational Research, 65(1), 1–21.
Cara, O., & Brookes, G. (2012, November). Evidence of the wider benefits of family learning: A scoping review (BIS research paper number 93). London: Department for Business, Innovation and Skills.
Carpentieri, J., Fairfax-Cholmeley, K., Lister, J., & Vorhaus, J. (2011). Family literacy in Europe: Using parental support initiatives to enhance early literacy development (EAC/16/2009). London: NRDC, Institute of Education.
Collins, F. M., Svensson, C., & Mahony, P. (2005, January). Bookstart: Planting a seed for life [Final report]. London: Roehampton University. Retrieved from http://www.booktrust.org.uk/usr/library/documents/main/bookstart-planting-a-seed-for-life-roehampton-2005.pdf
Demack, S., & Stevens, A. (2013, March). Evaluation of Bookstart England: A randomised controlled trial evaluation of “Bookstart Treasure Pack.” Centre for Education and Inclusion Research, Sheffield Hallam University. Retrieved from http://fileserver.booktrust.org.uk/usr/resources/1016/bookstart-treasure-evaluation-2013.pdf
Department of Education, Employment and Workplace Relations for the Council of Australian Governments (DEEWR). (2009). Belonging, being and becoming: The early years learning framework for Australia. Canberra, ACT: Author.
De Temple, J., & Snow, C. E. (2003). Learning words from books. In A. van Kleeck, S. A. Stahl, & E. B. Bauer (Eds.), On reading books to children: Teachers and parents (pp. 16–36). Mahwah, NJ: Erlbaum.
Elias, G., Hay, I., Homel, R., & Freiberg, K. (2006). Enhancing parent–child book reading in a disadvantaged community. Australasian Journal of Early Childhood, 31(1), 20–25.
Goldfeld, S., Quach, J., Nicholls, R., Reilly, S., Ukoumunne, O. C., & Wake, M. (2012). Four-year-old outcomes of a universal infant-toddler shared reading intervention: The “Let’s Read” trial. Arch Pediatr Adolesc Med, 166(11), 1045–1052.
Hannon, P., Morgan, A., & Nutbrown, C. (2006). Parents’ experiences of a family literacy program. Journal of Early Childhood Research, 4(1), 19–44.
Hattie, J. A. C. (2009). Visible learning: A synthesis of over 800 meta-analyses relating to achievement. Abingdon, UK: Routledge.
Hattie, J. A. C. (2012). Visible learning for teachers. Abingdon, UK: Routledge.
Hewer, L., & Whyatt, D. (2006). Improving the implementation of an early literacy program by child health nurses through addressing local training and cultural needs. Contemporary Nurse, 23(1), 111–119.
Hutton, J. S., Horowitz-Kraus, T., Mendelsohn A. L., DeWitt, T., & Holland, S. K. (2015). Home reading environment and brain activation in preschool children listening to stories. Pediatrics, 136(3), 466–478.
Jay, J., & Rohl, M. (2005). Constructing a family literacy program: Challenges and successes. International Journal of Early Childhood, 37(1), 57–78.
Kalb, G., & van Ours, J. C. (2013). Reading to young children: A head-start in life? (Melbourne Institute working paper no. 17/13). Retrieved from https://melbourneinstitute.com/downloads/working_paper_series/wp2013n17.pdf
Karrass, J., & Braungart-Rieker, J. M. (2005). Effects of shared parent–infant book reading on early language acquisition. Journal of Applied Developmental Psychology, 26(2), 133–148.
Lather, P. (2004). Scientific research in education: A critical perspective. British Educational Research Journal, 30(6), 759–772.
Let’s Read. (2012). About. Retrieved from http://www.letsread.com.au/About/Research
McCain, M. N., Mustard, J. F., & Shanker, S. (2007). Early years study 2: Putting science into action. Toronto: Council for Early Childhood Development.
Mendelsohn, A. L., Mogilner, L. N., Dreyer, B. P., Forman, J. A., Weinstein, S. C., Broderick, M., [End Page 38] et al. (2001). The impact of a clinic-based literacy intervention on language development in inner-city preschool children. Pediatrics, 107(1), 130–134.
Mol, S. E., & Bus, A. G. (2011). To read or not to read: A meta-analysis of print exposure from infancy to early adulthood. Psychological Bulletin, 137(2), 267–296.
Mol, S. E., Bus, A. G., & de Jong, M. T. (2009). Interactive book reading in early education: A tool to stimulate print knowledge as well as oral language. Review of Educational Research, 79(2), 979–1007.
Montag, J. L., Jones, M. L., & Smith, L. B. (2015). The words children hear: Picture books and the statistics for language learning. Psychological Science, 26(9), 1489–1496.
Moore, M., & Wade, B. (2003). Bookstart: A qualitative evaluation. Educational Review, 55(1), 3–13.
Neuman, S. B. (1996). Children engaging in storybook reading: The influence of access to print resources, opportunity, and parental interaction. Early Childhood Research Quarterly, 11(4), 495–513.
Nyhout, A., & O’Neill, D. K. (2013). Mothers’ complex talk when sharing books with their toddlers: Book genre matters. First Language, 33(2), 115–131.
Organisation for Economic Cooperation and Development (OECD). (2012). Let’s read them a story! The parent factor in education. Retrieved from http://dx.doi.org/10.1787/9789264176232-en
Owen, J. M. (2006). Program evaluation: Forms and approaches (3rd ed.). London: Allen & Unwin.
Reach Out and Read. (2014). About us; Mission and model. Retrieved from http://www.reachoutandread.org/
Reese, E., Sparks, A., & Leyva, D. (2010). A review of parent interventions for preschool children’s language and emergent literacy. Journal of Early Childhood Research, 10(1), 97–118.
Reinking, D., & Watkins, J. (2000). A formative experiment investigating the use of multimedia book reviews to increase elementary students’ independent reading. Reading Research Quarterly, 35(3), 384–419.
Seden, J. (2008). Creative connections: Parenting capacity, reading with children and practitioner assessment and intervention. Child & Family Social Work, 13(2), 133–143.
Sheffield Hallam University, Centre for Education and Inclusion Research. (2014, January). The contribution of early years bookgifting programmes to literacy attainment: A literature review [Final report]. Retrieved from http://booktrustadmin.artlogic.net/usr/resources/1124/final-bookgifting-lit-review.pdf
Shoghi, A., Willersdorf, E., Braganza, L., & McDonald, M. (2013). Let’s Read literature review. Melbourne: Royal Children’s Hospital Centre for Community Child Health, Murdoch Children Research Institute. Retrieved from http://www.letsread.com.au/getmedia/5787744c-1674-4a3d-9298-72719a5cf944/2013-Let-s-Read-Literature-Review.pdf.aspx
Siraj-Blatchford, I. (2007). Creativity, communication and collaboration: The identification of pedagogic progression in sustained shared thinking. Asia-Pacific Journal of Research in Early Childhood Education, 1(2), 3–23.
Sonnenschein, S., & Munsterman, K. (2002). The influence of home-based reading interactions on 5-year-olds’ reading motivations and early literacy development. Early Childhood Research Quarterly, 17(3), 318–337.
Van Steensel, R., McElvany, N., Kurvers, J., & Herppich, S. (2011). How effective are family literacy programs? Results of a meta-analysis. Review of Educational Research, 81(1), 69–96.
Zevenbergen, A. A., & Whitehurst, G. J. (2003). Dialogic reading: A shared picture book reading intervention for preschoolers. In A. van Kleeck, S. A. Stahl, & E. B. Bauer (Eds.), On reading books to children: Parents and teachers (pp. 177–200). Mahwah, NJ: Erlbaum.
Zuckerman, B. (2009). Promoting early literacy in pediatric practice: Twenty years of Reach Out and Read. Pediatrics, 124(6), 1660–1665.
Zuckerman, B., & Khandekar A. (2010). Reach out and read: Evidence-based approach to promoting early child development. Current Opinion in Pediatrics, 22(4), 539–544. [End Page 39]