Development of Mass Media Text-Based Instructional Materials to Improve Critical Reading Skills of University Students

Critical thinking competency is one of the requirements for university graduates to succeed. Critical reading skills learning can accommodate the development of the critical thinking competency. Therefore, this study was designed to develop instructional materials for the students to learn the skills. The present study was carried out following some steps of research and development study that includes a preliminary study, the development of product prototype, tryout, and the effectiveness test. This study involved 82 pre-service Indonesian teachers studying at Universitas Negeri Makassar, Indonesia. Data was collected through a questionnaire, an experts’ validation sheet, and a test on critical reading skills. The results of the study indicated that instructional materials developed with a complete and integrated structure, involving the activity of conceptualizing, modeling, and practicing are effective in improving students’ critical reading skills. The characteristics of mass media texts are highly appropriate for habituating students to evaluate, respond to, and reflect on information critically. The instructional materials developed in this study can be used by the university students in learning and support the development of students’ 21st-century learning competency.


Introduction
Critical thinking skills have become crucial for every individual who lives in a competitive era such as of the 21st-century.As the main competency needed to succeed in this era (Trilling & Fadel, 2009), critical thinking skills are supposed to be a part of the curricula.Morocco, Aguilar, Bershad, Kotula, and Hindin (2008) identify four competencies that need to be improved by university graduates in facing challenges at the workplace.These competencies cover conceptual knowledge, critical thinking skills, creative thinking skills, and communication and collaboration skills.
The development of critical thinking skills should be a foundation of learning at a university.The students need the skills to help them achieve success and career in the future.Wagner (2008) categorizes critical thinking skills as primary skills required to survive and to solve problems in the new world of work.The rapid change of competency requirements in the workplace makes it difficult for the university graduates to follow the pace.One of the reasons is that because high schools or university they went to did not prepare them well to face the current situation (Wagner, 2008).
Critical reading belongs to a language skill which is possessed by the critical thinkers.The critical reading activity involves the activity to search for and analyze information, and develop a critical attitude towards a text.These activities actualize critical thinking skills which can be used to find a solution to a problem, make a decision, persuade, analyze, and evaluate an opinion (Johnson, 2002).Critical reading is a process in which university students are prepared to succeed not only in the classroom but also in the real world (Kadir, Subki, Jamal, & 1smail, 2014).
Previous related research has proven the importance of critical reading in improving critical thinking skills.Zing, Eng, and Rafik-Galea (2014) found that university students did not have an ability to identify writer's purpose and main idea of a text.In addition, Karabay, Kusdemir Kayıran, and Isik (2016) proved that university students' had a medium level of self-efficacy in reading critically.Their self-efficacy also depended on gender.Lun, Fischer, and Ward (2010) and Lee, Lee, Makara, Fishman, and Hong (2014) discovered that most of the Asian university students had lower critical thinking ability compared to the students who come from the West.One of the factors that might contribute to this issue was students' limited opportunity to perform their critical and creative thinking skills.Bharuthram (2012) revealed that many university students who are entering the higher level of education had insufficient ability to analyze, criticize, evaluate, and synthesize information from various sources.Even, findings by Bharuthram and Clarence (2015) also showed that around 80% of the students had a very limited view or did not understand the meaning of critical reading.These findings indicate that it is necessary to provide university students with the ability to read critically in order that they can be more confident in facing challenges in the 21st-century era.Critical reading activity is carried out by applying the critical thinking process in reading.Since it requires a higher order of cognitive thinking, readers need to do some activities which involve constructing knowledge, evaluating the quality and the validity of the information based on some criteria and evidence.Critical thinking skills which can be implemented in reading are interpretation skill, analysis skill, making an inference skill, evaluation skill, explanation skill, and self-regulation skill (Facione, 2013(Facione, & 2015)).The results of a study conducted by Thomas (2011) have revealed that university students' critical thinking competency can be improved through evaluating the quality of a reading material, analyzing an argument, and utilizing a rubric to do self-reflection.Research by Sultan, Rofiuddin, Nurhadi, and Priyatni (2017b) indicated that learning models which were able to encourage students to evaluate author's messages and purposes, reflect ideological assumptions, and respond to texts from many perspectives could help grow the students' critical awareness.Therefore, it can be hypothesized that structured and interactive learning activities can improve students' critical reading ability.
This study aimed to develop critical reading instructional materials for university students.Instructional materials are a crucial element of learning to achieve learning goals (Brown, 1995).Instructional materials offer students concepts and a set of structured activities which can enhance the students' critical reading skills.The results of a preliminary study indicated that it was necessary to develop instructional materials to facilitate critical reading learning at a university.An analysis of learning tools proved that instructional materials for critical reading learning are not yet available.Besides, it was found that university students were still unable to provide critical responses towards a text.In general, the students admitted that they accepted the information from the text as it is and did not attempt to understand something lies beyond the text.They also did not try to connect the information found in the text to another text and did not develop a new perspective which was supposed to be different from the author's (Sultan, Rofiuddin, Nurhadi & Priyatni, 2017a).
Instructional materials developed in this study were mass media text-based.It means that reading materials would cover texts from mass media, such as news item texts, editorial texts, advertorial texts, and opinion texts.These texts were used because the characteristics of the texts support critical reading activities.News item texts bring about a narrative genre which not only explains information but also persuades readers and expose an argumentation (Winterowd & Winterowd, 1992).Argumentative texts have a characteristic to convince readers to a certain point of view (the writer's point of view).Meanwhile, persuasive texts are to persuade readers to do a certain action.The characteristics of each text are relevant to the purpose of critical reading learning that is to train students' critical reading skills that include interpretation skill, analysis skill, making an inference skill, evaluation skill, explanation skill, and self-regulation skill.The results of a study conducted by Atay (1990) have shown that media mass texts are quite appealing to the university students so that they can be used as reading materials.However, research findings by Silva, Bargallo, and Prat (2017) suggested that the difficulties faced by pre-service teachers in reading critically might result from the students' inability to identify argumentation.The current study, thus, was oriented to develop critical reading instructional materials which contained activities that allowed students to find writer's arguments and respond to them from various perspectives.
Specifically, this study aimed to: 1. develop mass media-based critical reading instructional materials; 2. examine the validity of the product by experts' examination; 3. investigate the effectiveness of the instructional materials on university students' critical reading skills; and 4. measure the students' responses towards the instructional materials.

Methods
This research was designed as a Research and Development (R & D) study.Research and development is a research design used to develop a new product systematically through a field test until the product fulfills the predetermined criteria of qualities and effectiveness (Gall, Gall, & Borg, 2003).The R & D model applied here was adapted from the (R2D2) Recursive, Reflective Design, and Development model suggested by (Willis, 1995) and (RDR) Research-Development-Research model.The implementation of both models produced a design which consists of four main activities that are (1) conducting a preliminary study, (2) developing a product, (3) conducting a tryout, and (4) testing the effectiveness of the product.
This research combined the paradigm of qualitative and quantitative study.The qualitative approach was applied during the preliminary study, product prototype development, and tryout.The research began with a preliminary study conducted to collect data as the basis of the product development.A preliminary study was carried out by analyzing theories and doing field-observation to obtain data on critical reading learning.The data was collected through document analysis, a survey, and interviews.The development of product prototype was focused on: (1) developing critical reading competencies, (2) developing main concepts, (3) selecting texts, (4) examining the readability of the texts, (5) examining the attractiveness of the texts, (6) developing questions, and (7) packaging the product.
The product was validated by three experts and one practitioner of critical reading, instructional materials development, teaching reading, and learning assessment.The validators were required to check the accuracy and conformity of the learning materials and provide comments, critics, and recommendation for revision.The validation results were used as a guide to improve the product.
One group pretest-posttest experimental design was employed to measure the effectiveness of the product in improving the students' critical reading achievement.The design was selected according to the objective of the current study that was to observe changes in students' critical reading ability before and after the field test (Mertens, 2010).This design is normally used to test a new learning method or innovation (Cohen, Manion, & Morrison, 2007) or to evaluate the progress of one's knowledge or skills (Czaja & Sharit, 2013).There were 82 students from the Indonesian Language Department, Universitas Negeri Makassar, Indonesia, participating in this study.All participants had been registered to Reading classes.More than 80 percent of the participants were females.The process of examining the product effectiveness lasted in 12 meeting.Learning activities are conducted by utilizing text-based critical reading materials of mass media, such as news, editorials, advertorials, and opinions.
Data was collected through a questionnaire, an experts' validation sheet, and a test.The questionnaire was used to gather data related to the students' perception of the product with two alternative answers, yes or no.The expert validation sheet was developed in the form of Likert scales format with four alternatives (scored 1-4) that showed the quality (accuracy and appropriateness) of the product.The test was constructed based on the critical reading indicators.The validity and reliability of the test had been examined beforehand.The validity of the test was confirmed by the experts in critical reading and critical thinking.The results of the reliability test showed that every aspect of the test had met the criteria of Cronbach's alpha internal consistency coefficients (.70).
There were two categories of data collected in this study, qualitative data and quantitative data.The qualitative data was in the form of speech and behaviors.Speech data included responses, suggestions, and critiques from the experts, practitioners, and students.The qualitative data was used to revise the product.
The quantitative data was in the form of scores resulting from expert validation and pretest-posttest results.Data collected from the experts and practitioner was interpreted using the Aiken's V formula (Aiken, 1985).To define the results, the criteria determined were; High (>0.80),Average (0.40-0.80),Low (<0.40).Meanwhile, the pretest-posttest scores were analyzed using IBM SPSS Statistics 23 t-test.The significance level was determined at α = .05.Before analyzing the data, a normality test was conducted.The result of the normality test using One-Sample Kolmogorov-Smirnov Test showed a significance value of .052for the pretest and of .294for the posttest.This result indicated that both pretest and posttest data had a normal distribution; hence, t-test could be conducted afterward.

Instructional Materials Structure
Instructional materials developed were structured to train students' critical thinking skills in reading news item texts, editorial texts, advertisements, and opinion texts.Each text was presented in one learning unit which included the activity to guide students to (1) get to know the text, (2) identify the text's characteristics, (3) model the critical reading skills, (4) train their critical thinking skills, and (5) do self-reflection.All activities were oriented to provide students with conceptual knowledge, instances, and training of the skills.
In the introduction section, the students were required to relate their previous knowledge and experience to the text.The students could observe their surroundings and find recognizable phenomena which can be used as a proof that critical reading skills are useful.The students identified the text's characteristics to understand the nature, structure, and function of the information found in the text.
In modeling critical reading activity, the students were exposed to two main parts.First is "why" the texts (news item texts, editorial texts, advertisements, opinion texts) should be read critically and the second is "how" the texts should be read.The first part consists of the information about the potential of the text to contain elements of propaganda.The second part consists of the information on how to read the texts critically from the linguistic aspect, and how to pay careful attention to information, data, arguments, and facts presented in the texts.
Critical reading training is the main activity to develop students' critical reading skills.There are six critical reading skills that need to be developed; they are Activity 1: Interpreting, Activity 2: Analyzing, (3) Activity 3: Making an Inference, (4) Activity 4: Evaluating, (5) Activity 5: Explaining, and (6) Activity 6: Self-regulating.
The reflection activity aims to check students' understanding after doing several steps of critical reading.This activity belongs to self-evaluation activity to measure the students' achievement.

Instructional Materials Validation
Instructional materials being developed were validated by the experts and practitioners in reading, learning assessment, reading instructional materials, and teaching reading.The materials were evaluated from the theoretical and practical aspect.The results of the experts' validation are presented in Table 1.Table 1 suggested that the indicators of the five validated aspects had fulfilled the criteria of the product merit.Of all the evaluated items, only three indicators obtained average scores while the rest were categorized high.The results of the experts and practitioners evaluation indicated that the learning materials developed in this study were qualified and reliable so that it could be implemented.

The Effectiveness of Instructional Materials
The effectiveness test was used to measure the effect of instructional materials on students' critical reading skills.The results of the t-test show that t = 19.318and p = 0.000 (Table 1).It indicates that the effect was significant.Therefore, it can be concluded that the use of the instructional materials improved the students' critical reading skills in general.To measure the effectiveness of the instructional materials in a more specific way, a statistical analysis was conducted for each skill.Based on the information in Table 2, it can be concluded that the level of significance of all skills is .000(p = .000).It shows that the use of learning materials brought a significant effect on learning and was effective to improve the students' critical reading skills, particularly in interpreting, analyzing, making an inference, evaluating, and explaining the texts.Table 3 depicts the complete results of the statistical analysis.

Students' Responses to Instructional Materials
The students were required to respond to the instructional materials from the aspect of practicality, the functions, texts selected, and attitude towards learning.Based on the results of the analysis, it was found that the students' responses were positive.The complete results are recorded in Table 4.

Discussion
The validation of the product (instructional materials) has been justified by the experts and practitioners.The results of the t-test also show that the effect of the instructional materials on students' critical reading skills was significant.It indicates that the instructional materials have been developed according to the principles of the development of critical reading instructional materials.Also, the positive responses from the students have proven that the instructional materials are appropriate, effective, and able to suit the students' needs and characteristics.
The instructional materials integrate the activity of developing conceptual understanding and practicing.As a result, the combination of conceptualizing, modeling, and practicing critical reading skills is present in one set of instructional materials.Conceptual understanding is necessary since the students are not familiar with critical reading which requires a higher order of thinking.
An instructional material with theory-modeling-practice model provides an assist for the students to enhance their critical reading competencies.Learning materials which only present theory without practice or practice without theory will be burdensome for the students.Therefore, in line with Tomlinson's (1998), learning materials should help students learn easily through the presentation of concepts and modeling, for example.
Modeling is aimed to make the students able to relate their conceptual understanding to the critical reading activity they are doing.Blair, Rupley, and Nichols (2009) state that modeling is an important part of reading because it helps readers finish their training session.Modeling assists the students in constructing their conceptual understanding, applying it, and boosting their self-confidence.Good learning materials should be able to keep their self-confidence high (Tomlinson, 1998).
The exercises are provided for the students to practice their critical reading skills.In this part, they need to improve their interpretation skill, analysis skill, making an inference skill, evaluation skill, explanation skill, and self-regulation skill.Each skill is divided into some sub-skills which are then represented by questions.The students need to answer the questions to train their critical reading skills.This part becomes the main part of learning using the instructional materials being developed.Similarly, Blair et al. (2009) Tonegato andHendricks (2012) argue that students will be successful in reading if they are given an opportunity to apply their skills by doing some exercises.
Learning materials developed in this study contained mass media texts as a reference for reading.Texts variation is a key to the students' success in reading since there is a chance that the students can repeatedly apply their skills on reading different kinds of texts.Learning materials can be influential when they are presented in variation (Tomlinson, 1998).The mass media texts, for example, which provide updated information can help students promote knowledge of a wider context.This knowledge is needed to develop a critical attitude.
The selection of mass media texts to be used as reading materials implies the instructional materials' effectiveness.Mass media texts can be used to improve students' critical attitude through identifying values and perspectives found in the texts, interpreting the messages, evaluating the trustworthiness of the texts, and many others (Alvarado, 2012).Mass media texts are authentic.Therefore, they are appealing to students and motivating; students are interested in criticizing the texts.Authentic texts provide an opportunity for the students to explore authentic cultural information and real language use (Berrardo, 2006).
The results of the effectiveness test showed a significant difference in students between they learned using critical reading materials and before they used the materials.Texts in mass media which contain argumentative and persuasive information are relevant for the development of students' reading competences.The texts provide potentially biased events, data, and information.Editorial texts, for instance, serve a propaganda function.Advertorial texts contain information that can persuade readers to do as they are told.Opinion texts are persuasive.Argumentation, propaganda, and persuasion are texts elements that are effective in improving one's critical reading skills.Learning to criticize such texts from domination perspectives is therefore useful to promote students' critical reading skills (Sultan, Rofiuddin, Nurhadi, & Priyatni, 2017c).The use of domination elements found in mass media texts can enhance students' thinking and higher cognitive processes.
The results of the current study indicated that learning materials that were developed using analytical questions could empower students' critical ability.Analytical questions helped the students think deeply about the texts.They reflected on the information obtained from the texts.This finding was in line with the results of the research conducted by Wang and Seepho (2017) who found that analytical questions encouraged learners to critically think about writer's ideas or point of view to help them connect, interpret, and synthesize the text's purposes.In this study, students practiced their critical reading skills by answering some texts' comprehension questions in the form of multiple choice and essay.The multiple choice test was effective to train the students' interpretative skill in reading (Nuttal, 2005), while the essay test could be very useful to elicit the students' perspectives.
Critical reading skills trained through the instructional materials are adopted from Facione's six critical thinking skills (Facione, 2013(Facione, & 2015)).Those skills have a hierarchical structure determined by experts in critical thinking.The critical thinking skills have become important for the students to pursue their future career and life.Trilling and Fadel (2008) categorize three main skills needed in the 21st-century; they are learning and innovation skills, digital literacy skills, and career and life skills.Critical thinking skills and problem-solving skills are components of learning and innovation skills.Therefore, to prepare university graduates to face any kind of situation in their life, these skills need to be integrated into the school curriculum.
Language skills and critical thinking skills have a reciprocal relationship.Tung and Chang (2009) in their study report that learning to read can improve students' critical thinking skills and show the better disposition of critical thinking.Grosser and Nel (2013) have found out that there is a significant relationship between academic language proficiency and making inferences and critical thinking skills.On the other hand, Azin and Tabrizi (2016) have revealed that students who are competent in critical thinking are better at interpreting.Based on the results of the research, it can be said that the development of critical reading skills in the present study has a direct contribution to the improvement of students' critical thinking.
The instructional materials have been proven able to improve the students' critical reading skills.Therefore, it is recommended to use the instructional materials in the classroom.Tiruneh, Verburgh, and Elen (2014) assert that the effectiveness of critical thinking learning is influenced by the learning environment.Instructional materials, learning strategy, and instructional media are components that create a learning environment which supports the development of critical thinking skills.

Conclusion
Based on the results of the present study, it can be concluded that one of the most important aspects in developing a critical reading material is learning activities or learning processes that can encourage students as readers to think deeply.The six levels of critical reading skills developed in this research constitute a part of students' cognitive processes which help them develop their critical analytical thinking ability.Critical reading skills being developed have a unique characteristic to train students' critical reading skills.Instructional materials were developed with a complete and integrated structure, involving the activity of conceptualizing, modeling, and practicing.The structure of the instructional materials helps university students develop their critical reading competencies and motivate them and boost their self-confidence.
It can be concluded that the product (instructional materials) developed in this study is effective in improving students' critical reading skills.This finding also indicates the significance of the use of mass media texts in teaching critical reading.Mass media texts are highly relevant with the development of students' critical reading skills because they contain argumentation, propaganda, persuasion, and writer's purposes which habituate students to evaluate, respond to, and reflect on information critically.The use of mass media texts has provided an opportunity for the students to train their critical reading skills in various texts.

Table 2
Paired samples test on students critical reading skills in general

Table 3
Paired samples test on every aspect of critical reading skills

Table 4
Students' responses on instructional materials The Summary of Learning Activities in the Implementation of the Product Learning MaterialsIdentifying an advertorial text and building an understanding of the text Students identify the characteristics of the advertorial text from the structure, linguistic aspects, and information functions Students understand and take a close look at the examples of the use of persuasive messages, strategies to convince readers, and advertorial argumentation presented in the text Students discuss an example of a critic on an advertorial text.Lecturer guides the students to answer three questions: (1) Do you understand what the advertisers instill in readers?(2)How do the advertisers shape their product and service image?(3)What are the advertisers' strategies to persuade the reader?Identifying an opinion text and building an understanding of the text Students identify the characteristics of the opinion text from the structure, linguistic aspects, and information functions Students understand and take a close look at the examples of the use of language, opinions, ideas, data, and facts presented in the text Students discuss an example of an activity to read an opinion text critically.Lecturer guides the students to answer three main questions:(1) What ideas does the writer want to convey?(2) What does the writer intend to say behind the text?(3) What does the writer expect from the reader?